Text classification method and device and nonvolatile storage medium
Through the method of layer-by-layer feature extraction and similarity score update in the large model, the problem of unsatisfactory text classification efficiency is solved, and the model is early retreat and efficiency improvement is achieved.
Patent Information
- Application Number
- CN202411016001.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-12
AI Technical Summary
During the text classification process, large models have delayed inference and unsatisfactory efficiency due to complex network settings, and the existing technology has failed to effectively solve this problem.
By inputting multiple network layers in a predetermined order based on the text vector to be classified, the similarity score between the target layer and the pre-layer is determined, and the value is updated according to the score. If the threshold is exceeded, the target layer will be Classification results are used as the final result to achieve early withdrawal of the model to improve efficiency.
It improves the inference process efficiency of the large model, reduces the model processing time, and solves the problem of unsatisfactory text classification efficiency.
Smart Images

Figure CN118939800A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer science and artificial intelligence, and in particular to a text classification method, device and non-volatile storage medium. Background Art
[0002] With the rapid development of technology, large models are gradually becoming the core force driving the advancement of the field of artificial intelligence. However, large models contain hundreds of millions or even hundreds of billions of parameters, resulting in high computational costs in terms of memory consumption and latency, but low efficiency, which limits the deployment of models in real-world scenarios and their application in industrial scenarios. A neural network is a data model that simulates the neural thinking mode of the human brain. There are many types of neural networks, including BP neural network (BackPropagation), convolutional neural network, multilayer perceptron MLP (Multilayer Perception), Transformer (converter model), etc. In the text classification process, large models based on the Transformer framework in related technologies often have problems such as reasoning delay and overthinking due to the complex network settings of the model itself, resulting in unsatisfactory text classification efficiency.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present invention provide a text classification method, device and non-volatile storage medium to at least solve the technical problem of unsatisfactory text classification efficiency existing in the related art.
[0005] According to one aspect of an embodiment of the present invention, a text classification method is provided, comprising: based on a text vector to be classified, inputting a plurality of network layers in a predetermined order to perform layer-by-layer feature extraction; when a target layer is included in the plurality of network layers and there is a preceding layer before the target layer, determining a similarity score between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer, wherein the target layer is a network layer other than the last layer in the plurality of network layers; based on the similarity score, updating a predetermined time value to obtain an updated time value; when the updated time value is greater than a first predetermined threshold, using the layer classification result corresponding to the target layer as the target classification result of the text vector.
[0006] Optionally, the similarity score is negatively correlated with the confidence of the layer classification result of the corresponding network layer, and the predetermined number value is updated based on the similarity score to obtain an updated number value, including: when there are multiple preceding layers, based on the similarity scores between the multiple preceding layers and the target layer, determining a target number of preceding layers among the multiple preceding layers whose similarity scores are less than or equal to a second predetermined threshold value; when the target number is greater than a third predetermined threshold value, determining to perform increment processing on the number value to obtain the updated number value.
[0007] Optionally, the method of updating a predetermined number of times based on the similarity score to obtain an updated number of times includes: when there are multiple preceding layers and the multiple preceding layers are continuously adjacent to the target layer, if there is a preceding layer among the multiple preceding layers whose similarity score is greater than a second predetermined threshold, clearing the number of times to obtain an updated number of times; and if the similarity score corresponding to each preceding layer among the multiple preceding layers is less than or equal to the second predetermined threshold, incrementing the number of times to obtain the updated number of times.
[0008] Optionally, the method also includes: when the updated number value is less than the first predetermined threshold, based on the output vector of the target layer, inputting the next layer of the target layer for feature extraction until the next layer is the last layer, and determining the layer classification result corresponding to the last layer as the target classification result.
[0009] Optionally, the preceding layer is a layer above the target layer, and the predetermined number value is updated based on the similarity score to obtain an updated number value, including: when the similarity score between the preceding layer and the target layer is less than or equal to a second predetermined threshold, the number value is incremented to obtain the updated number value; when the similarity score between the preceding layer and the target layer is greater than the second predetermined threshold, the number value is cleared to obtain the updated number value.
[0010] Optionally, determining the similarity score between the layer classification result corresponding to the target layer and the layer classification result corresponding to the preceding layer includes: determining the distance between the layer classification result corresponding to the target layer and the layer classification result corresponding to the preceding layer; and determining the similarity score based on the distance.
[0011] Optionally, the method also includes: determining the text complexity of the text vector; a target update condition corresponding to the text complexity among multiple candidate update conditions, wherein the multiple candidate update conditions are respectively for different text complexities; based on the similarity score, updating the predetermined number value to obtain an updated number value, including: based on the similarity score, using the target update condition, performing update processing on the number value to obtain the updated number value.
[0012] Optionally, the method further includes: determining processing resources allowed to be used by the multiple network layers, and classification accuracy required for the text vector; and determining the first predetermined threshold based on the processing resources and the classification accuracy.
[0013] According to another aspect of an embodiment of the present invention, a text classification device is provided, comprising: an input module for inputting a plurality of network layers in a predetermined order to perform layer-by-layer feature extraction based on a text vector to be classified; a first classification result module for determining a similarity score between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer, when the target layer is included in the plurality of network layers and there is a preceding layer before the target layer, wherein the target layer is a network layer other than the last layer in the plurality of network layers; a time value calculation module for updating a predetermined time value based on the similarity score to obtain an updated time value; and a second classification result module for using the layer classification result corresponding to the target layer as a target classification result of the text vector when the updated time value is greater than a first predetermined threshold.
[0014] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, wherein the non-volatile storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any one of the text classification methods.
[0015] In an embodiment of the present invention, a text classification method is adopted, by inputting multiple network layers in a predetermined order based on the text vector to be classified to perform layer-by-layer feature extraction; when the target layer is included in the multiple network layers and there is a pre-layer before the target layer, the layer classification result corresponding to the target layer is determined, and the similarity score between the layer classification result corresponding to the pre-layer is determined, wherein the target layer is the network layer except the last layer in the multiple network layers; based on the similarity score, the predetermined time value is updated to obtain the updated time value; when the updated time value is greater than the first predetermined threshold, the layer classification result corresponding to the target layer is used as the target classification result of the text vector. The purpose of improving the use of early retirement of the model and reducing the model processing time is achieved, and the technical effect of improving the efficiency of the large model reasoning process is achieved, thereby solving the technical problem of unsatisfactory text classification efficiency existing in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0017] Figure 1 is a flowchart of an optional text classification method provided according to an embodiment of the present invention;
[0018] Figure 2 is a first schematic diagram of an optional text classification method provided according to an embodiment of the present invention;
[0019] Figure 3 is a second schematic diagram of an optional text classification method provided according to an embodiment of the present invention;
[0020] Figure 4 is a third schematic diagram of an optional text classification method provided according to an embodiment of the present invention;
[0021] Figure 5 is a schematic diagram of an optional text classification device provided according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0024] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:
[0025] Neural network, also known as artificial neural network (ANN), is a computational model inspired by biological neural networks. It consists of a large number of interconnected nodes or "neurons" organized into different levels, including input layer, hidden layer and output layer. Each neuron is connected to other neurons in its adjacent layer, processes input data through weighted summation or nonlinear activation function, and passes information to the next layer. This structure of neural network enables it to learn complex data representations and perform various cognitive tasks. There are many types of neural networks, including BP neural network, convolutional neural network, multilayer perceptron MLP, Transformer, etc.
[0026] An embodiment of the present invention provides an embodiment of a method for text classification. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0027] Figure 1 is a flowchart of an optional text classification method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:
[0028] Step S102, based on the text vector to be classified, multiple network layers in a predetermined order are input to perform layer-by-layer feature extraction;
[0029] It can be understood that the text is classified, the text data is converted into a numerical vector, and the preprocessed text vector sequence is used as the input of the input layer, and input into multiple network layers in a predetermined order to perform layer-by-layer feature extraction.
[0030] Optionally, data preprocessing is required before text classification. The data processing methods may include multiple methods, such as text cleaning (noise removal), word segmentation, stop word removal, stemming or word form restoration, etc., to prepare text data for subsequent processing.
[0031] Optionally, the feature extraction can be of multiple types, for example, deep learning-based word embedding technology (DeepLearning Feature Extraction), such as convolutional neural networks, recurrent neural networks, Transformer and other structures to automatically extract text features, use the hidden layer output of the model as the feature representation of the text, capture local features (the context of words or phrases) and global features (the overall semantics of the sentence) in the text, so as to effectively perform classification tasks.
[0032] Optionally, Figure 2 is a first schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 2 As shown in Figure 1, the backbone model is a superposition of multiple Transformer layers (i.e., multiple network layers), where L i is the i-th Transformer layer of the model (i can be 1 to n), parameter n is the number of layers in the Transformer network, C i is the classifier attached to the i-th Transformer layer, and the parameter s is the score of the similarity distance between different network layers. The smaller the value of parameter s is, the more similar the prediction results of adjacent layers are, and the higher the confidence of the model in the prediction results is. In addition, the parameter t is the predefined patience threshold, that is, the first predetermined threshold, and the parameter τ is the threshold of the similarity score, that is, the second predetermined threshold. That is, when s is less than the threshold for more than t times, the model stops reasoning and exits.
[0033] The input data is first embedded as a vector by the embedding layer:
[0034] h0=Embedding(x)
[0035] Among them, h0 is the text vector to be classified, and x is the input data.
[0036] Through Transformer modules L1 to L i Extract hidden features and calculate their hidden feature states h i , by each classifier C i , calculate the probability distribution p, that is, extract features through multiple network layers respectively, and use the classifier set by each network layer to obtain the layer classification result later:
[0037] h i =L i(h i-1 )
[0038] p i =C i (h i )
[0039] Among them, h i-1 is the feature output by the previous network layer, i is used to identify different network layers, and p i is the probability distribution of the i-th layer.
[0040] The similarity score s between the i and j layers included in the multiple network layers is calculated based on the predicted probabilities, and the model is evaluated whether it is confident in its prediction results. If the model is confident in the prediction results, the model stops reasoning and extracts the exit, while if the model is not confident in the prediction results, the results of the current layer will be input into the next layer to continue reasoning. If the model has not reached the exit criteria during this process, the prediction results of the last layer of the model are used as the final reasoning effect of the model.
[0041] Step S104, when the target layer is included in the multiple network layers and there is a preceding layer before the target layer, determining a similarity score between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer, wherein the target layer is a network layer other than the last layer in the multiple network layers;
[0042] It can be understood that there are multiple network layers, and the target layer is a network layer other than the last layer. In a neural network, the output of each layer will be used as the input of the next layer, so the previous layer of the target layer is the layer immediately above the target layer. The layer classification result refers to the output of each network layer, and the output represents the feature representation obtained by the model after the input data is processed by this layer. Calculate the similarity score between the layer classification result corresponding to the target layer and the layer classification result corresponding to the previous layer.
[0043] In an optional embodiment, determining a similarity score between a layer classification result corresponding to a target layer and a layer classification result corresponding to a preceding layer includes: determining a distance between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer; and determining a similarity score based on the distance.
[0044] It can be understood that the layer classification results of the target layer and the previous layer are obtained respectively, and the distance between the two results, that is, the distance between the two feature vectors, is calculated. Based on the calculated distance, the similarity score is further calculated. The similarity score calculated by this method helps to understand whether the feature extraction of the model at different levels is consistent and whether it can stably capture the important features of the input data.
[0045] Optionally, in the vector space, the distance method for calculating the feature vector can be multiple, for example, Euclidean distance, cosine similarity, Manhattan distance, etc. Among them, Euclidean distance is a method for measuring the distance between two points in a multidimensional space, which determines the coordinate values of the two points in their respective dimensions, adds the square values of the coordinate differences in all dimensions, takes the square root of the sum, and the result is the Euclidean distance between the two points.
[0046] Optionally, there are multiple methods for calculating the similarity score. For example, similarity scores are calculated by using the distillation object function of the probability distribution of the predictions of adjacent layers to evaluate the similarity of the prediction results of adjacent intermediate layers. The distillation object function represents the distance between the prediction results of adjacent layers. The smaller its value is, the smaller the gap between the prediction results of adjacent layers is, the closer the prediction results of adjacent layers are, and the prediction results are close to a stable state. When it is less than the threshold for a certain number of times, it means that the prediction results of the model are credible, and the reasoning can be stopped and exited. The larger the calculated distillation object function is, the greater the distance between the prediction results of adjacent layers is, indicating that the prediction results have not reached a stable state. The hidden state of the current layer needs to be used as input and passed to the next layer to determine whether the prediction results of the next layer are stable. This state continues until the standard is met. If the standard is not met, the prediction value of the last layer is output as the final prediction result.
[0047] Step S106, based on the similarity score, updating the predetermined number of times to obtain an updated number of times;
[0048] It can be understood that the similarity score is compared with a predetermined threshold value, and based on the comparison result, the number value is updated. This updating process can be an increase or a zeroing process to obtain an updated number value.
[0049] In an optional embodiment, the method also includes: determining the text complexity of the text vector; selecting a target update condition corresponding to the text complexity among multiple candidate update conditions, wherein the multiple candidate update conditions correspond to different text complexities; updating a predetermined number value based on a similarity score to obtain an updated number value, including: based on the similarity score, using the target update condition, performing update processing on the number value to obtain an updated number value.
[0050] It can be understood that based on the complexity of the text, an update condition and processing method corresponding to the complexity of the text are selected from multiple candidate update conditions. The complexity of the text can be divided into different predetermined levels, such as simple, general or difficult. For documents with relatively simple or general text complexity, a more concise candidate update condition can be selected for processing. For documents with relatively difficult text complexity, a complex candidate update condition can be selected for processing. Using a suitable candidate update condition as the target update condition, based on the similarity score, the update processing is performed on the number value to obtain the updated number value. Selecting different update conditions and processing methods according to the difficulty of the text can improve processing efficiency and prevent waste of resources.
[0051] In an optional embodiment, the preceding layer is the upper layer of the target layer, and based on the similarity score, the predetermined number value is updated to obtain an updated number value, including: when the similarity score between the preceding layer and the target layer is less than or equal to a second predetermined threshold, the number value is incremented to obtain an updated number value; when the similarity score between the preceding layer and the target layer is greater than the second predetermined threshold, the number value is cleared to obtain an updated number value.
[0052] It can be understood that the preceding layer is the previous layer of the target layer, and the similarity score of the target layer and its preceding layer is calculated each time, and the similarity score is compared with the second threshold value. If the similarity score is less than or equal to the second predetermined threshold value, the number value is added by 1, and if the similarity score is greater than the second predetermined threshold value, the number value is cleared, and at this time, the number value is 0. The number value update condition of this solution has a fast response speed, and the processing result can be obtained quickly.
[0053] Optionally, Figure 2 is a first schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 2 As shown, when the model reaches the current layer i, the similarity score s between the prediction results of the i-th layer and i-1 is calculated i,i-1 Among them, the similarity score s i,i-1 The smaller the value, the more likely the predicted probability distribution p is. i and p i-1 The more consistent they are, the higher the confidence of the model in the prediction result. In addition, parameter τ is set as the threshold of similarity score, and parameter cnt i It is used to store the number of times the similarity score is continuously less than the threshold τ, that is, the number of times, and the formula is as follows:
[0054]
[0055] That is, if the similarity score s between the prediction results of layer i and i-1 i,i-1When the similarity score is less than or equal to the predefined threshold τ, the number cnt i Increase by 1. If the similarity score s between the prediction results of the i-th layer and i-1 i,i-1 When the similarity score is greater than the predefined threshold τ, the number parameter cnt i Reset to 0. Repeat this process until cnt reaches the predefined patience value, that is, the first predetermined threshold, the model will stop the inference process and exit early. However, if this condition is not met, the model will output the result predicted by the classifier of the last layer as the final prediction result.
[0056] In an optional embodiment, based on the similarity score, a predetermined number value is updated to obtain an updated number value, including: when there are multiple preceding layers and the multiple preceding layers are continuously adjacent to the target layer, if there is a preceding layer among the multiple preceding layers whose similarity score is greater than a second predetermined threshold, the number value is cleared to obtain an updated number value; if the similarity score corresponding to each preceding layer among the multiple preceding layers is less than or equal to the second predetermined threshold, the number value is incremented to obtain an updated number value.
[0057] It can be understood that there are multiple preceding layers, and the similarity scores of the target layer and its preceding layers are calculated, and the similarity scores are compared with the second threshold. If the similarity scores corresponding to each of the multiple preceding layers are less than or equal to the second predetermined threshold, the number value is added by 1. If the similarity score is greater than the second predetermined threshold, the number value is cleared. At this time, the number value is divided into 0, and the output result can be of higher quality.
[0058] Optionally, the above number value update conditions can be multiple, Figure 3 is a second schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 3 As shown in the figure, when the model reaches the current layer i, the similarity score between layer i and its previous layers i–1 to i–m is calculated, where m is the identifier of the previous layers. If the similarity score s of the probability distribution predicted by the current layer and its previous layers is continuously less than the threshold τ of the similarity score, the parameter cnt storing the number of times the similarity score is continuously less than the threshold τ is stored. i Increase by 1, otherwise, the number cnt i Reset to 0. Repeat this process until the number cnt reaches the predefined patience value. The model will stop reasoning and exit early. Otherwise, if this condition is not met, the model will output the result predicted by the classifier of the last layer as the final prediction result.
[0059] In an optional embodiment, the similarity score is negatively correlated with the confidence of the layer classification result of the corresponding network layer, and based on the similarity score, a predetermined number value is updated to obtain an updated number value, including: in the case of multiple preceding layers, based on the similarity scores between the multiple preceding layers and the target layer, determining a target number of preceding layers among the multiple preceding layers whose similarity scores are less than or equal to a second predetermined threshold value; in the case of the target number being greater than a third predetermined threshold value, determining to perform increment processing on the number value to obtain an updated number value.
[0060] It can be understood that when there are multiple pre-layers, based on the similarity scores between the multiple pre-layers and the target layer, if the similarity score of at least one pre-layer is less than or equal to the target number of pre-layers of the second predetermined threshold, the number value is increased by 1; if there is no pre-layer with a similarity score less than or equal to the target number of pre-layers of the second predetermined threshold, the number value remains unchanged.
[0061] Optionally, the above-mentioned method for updating the number value condition at least includes: Figure 4 is a third schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 4 As shown, when the model reaches the current layer i, the similarity score between layer i and its previous layers i-1 to im is calculated. If there is a similarity score s between the current layer i and its previous layers that is less than the threshold τ of the similarity score, the parameter cnt that stores the number of times the similarity score is continuously less than the threshold τ will be stored. i Increase by 1, otherwise, the number cnt i unchanged. Assume that the similarity score s between the current layer and the previous layers is less than the threshold for a number of times that reaches the predefined patience τ. In this case, the model will stop reasoning and exit early. Otherwise, if this condition is not met, the model will output the result predicted by the classifier of the last layer as the final prediction result. The mathematical expression is as follows:
[0062]
[0063] Step S108, when the updated number value is greater than the first predetermined threshold, the layer classification result corresponding to the target layer is used as the target classification result of the text vector.
[0064] It can be understood that when the updated number value is greater than the first predetermined threshold, the model stops reasoning and uses the layer classification result corresponding to the target layer at this time as the target classification result of the text vector, that is, the final reasoning result of the model.
[0065] In an optional embodiment, the method further includes: determining processing resources allowed to be used by multiple network layers, and the classification accuracy required for the text vector; and determining a first predetermined threshold based on the processing resources and the classification accuracy.
[0066] It can be understood that the first predetermined threshold is a predefined patience value. The first predetermined threshold is determined by the processing resources of multiple network layers and the classification accuracy of the text vector. The first predetermined threshold should be less than or equal to the number of network layers. The value of the first threshold is determined in advance, which reduces the waste of resources caused by repeated iterations of the model.
[0067] In an optional embodiment, the method also includes: when the updated number value is less than a first predetermined threshold, based on the output vector of the target layer, inputting the next layer of the target layer for feature extraction until the next layer is the last layer, and determining the layer classification result corresponding to the last layer as the target classification result.
[0068] It can be understood that when the updated value is less than the first predetermined threshold and the layer below the target layer is the last layer of the model, the output result of this target layer is used as the input of the last layer, and the layer classification result of the last layer is calculated as the target classification result, that is, the final reasoning effect of the model.
[0069] Through the above step S102, based on the text vector to be classified, multiple network layers in a predetermined order are input to perform layer-by-layer feature extraction; step S104, when the target layer is included in the multiple network layers and there is a pre-layer before the target layer, the layer classification result corresponding to the target layer is determined, and the similarity score between the layer classification result corresponding to the pre-layer is determined, wherein the target layer is the network layer except the last layer in the multiple network layers; step S106, based on the similarity score, the predetermined time value is updated to obtain the updated time value; step S108, when the updated time value is greater than the first predetermined threshold, the layer classification result corresponding to the target layer is used as the target classification result of the text vector. It can improve the purpose of using the model to retire early and reduce the model processing time, and achieve the technical effect of improving the efficiency of the large model reasoning process, thereby solving the technical problem of unsatisfactory text classification efficiency existing in the related technology.
[0070] According to the above embodiments and optional embodiments, the present application also provides an optional implementation method. In step S1, the text passes through the Embedding layer (i.e., embedding layer) of the model. The Embedding layer converts the input text into a vector embedding model. Based on the text vector to be classified, multiple network layers in a predetermined order are input.
[0071] Step S2, the vector will pass through the Transformer module L1 to L i Extract hidden features and calculate their hidden feature states h i Then, each classifier C i Compute the probability distribution p.
[0072] The backbone model is a superposition of multiple layers of Transformer blocks, where L i is the Transformer layer of the model, parameter n is the number of layers in the Transformer network, C i is the classifier attached to the i-th Transformer layer, and the parameter s is the score of the similarity distance between different layers. The smaller the value of parameter s is, the more similar the prediction results of adjacent layers are, and the higher the confidence of the model in the prediction results is. In addition, the parameter t is the predefined patience threshold, that is, the first predetermined threshold, and the parameter τ is the threshold of the similarity score, that is, the second predetermined threshold. That is, when s is less than the threshold for more than t times, the model stops reasoning and exits.
[0073] The input data is first embedded as a vector by the embedding layer:
[0074] h0=Embedding(x)
[0075] Next, the data passes through the Transformer modules L1 to L i Extract hidden features and calculate their hidden feature states h i Then, each classifier C i Calculate the probability distribution p:
[0076] h i =L i (h i-1 )
[0077] p i =C i (h i )
[0078] After that, the similarity score s between layer i and layer j is calculated based on the predicted probability, and the model is evaluated whether it is confident in its prediction results. If the model is confident in the prediction results, the model stops reasoning and extracts the exit, while if the model is not confident in the prediction results, the results of the current layer will be input into the next layer to continue reasoning. If the model does not meet the exit criteria during this process, the prediction results of the last layer of the model are used as the final reasoning effect of the model.
[0079] Step S3, when a target layer is included in multiple network layers and there is a preceding layer before the target layer, determine the layer classification result corresponding to the target layer and the similarity score s between the layer classification result corresponding to the preceding layer, wherein the target layer is a network layer other than the last layer in the multiple network layers, and evaluate whether the model is confident in its prediction results.
[0080] Figure 2 is a first schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 2 , when the model reaches the current layer i, it calculates the similarity score s between the prediction results of the i-th layer and i-1 i,i-1 Among them, the similarity score s i,i-1 The smaller the value, the more likely the predicted probability distribution p is. i and p i-1 The more consistent they are, the higher the confidence of the model in the prediction result. In addition, parameter τ is set as the threshold of similarity score, and parameter cnt i It is used to store the number of times the similarity score is continuously less than the threshold τ, that is, the number of times, and the formula is as follows:
[0081]
[0082] That is, if the similarity score s between the prediction results of layer i and i-1 i,i-1 When the similarity score is less than or equal to the predefined threshold τ, the number cnt i Increase by 1. If the similarity score s between the prediction results of the i-th layer and i-1 i,i-1 When the similarity score is greater than the predefined threshold τ, the number parameter cnt i Reset to 0. Repeat this process until cnt reaches the predefined patience value, that is, the first predetermined threshold, the model will stop the inference process and exit early. However, if this condition is not met, the model will output the result predicted by the classifier of the last layer as the final prediction result.
[0083] Figure 3 is a second schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 3 , when the model reaches the current layer i, the similarity score between layer i and its previous layers i–1 to i–m is calculated. If the similarity score s between the current layer and the probability distribution predicted by its previous layers is continuously less than the threshold τ of the similarity score, the parameter cnt that stores the number of times the similarity score is continuously less than the threshold τ is stored. i Increase by 1, otherwise, the number cnt i Reset to 0. Repeat this process until the number cnt reaches the predefined patience value. The model will stop reasoning and exit early. Otherwise, if this condition is not met, the model will output the result predicted by the classifier of the last layer as the final prediction result.
[0084] Figure 4 is a third schematic diagram of an optional text classification method provided according to an embodiment of the present invention, such as Figure 4, when the model reaches the current layer i, the similarity score between layer i and its previous layers i-1 to im is calculated. If there is a similarity score s between the current layer i and its previous layers that is less than the threshold τ of the similarity score, the parameter cnt that stores the number of times the similarity score is continuously less than the threshold τ will be stored. i Increase by 1, otherwise, the number cnt i unchanged. Assume that the similarity score s between the current layer and the previous layers is less than the threshold for a number of times that reaches the predefined patience τ. In this case, the model will stop reasoning and exit early. Otherwise, if this condition is not met all the time, the model will output the result predicted by the classifier of the last layer as the final prediction result. Among them, the difference between this method and the inverse method 2 is that method 2 requires that the similarity score of the current layer and the previous layers is continuously less than the threshold, while this method does not require it, and only requires that the results of a certain number of layers are similar. Therefore, the calculation formula of the number value of this scheme is different from the above two methods, as shown below:
[0085]
[0086] Step S4, when the updated number value is greater than the first predetermined threshold, it is considered that the model is confident in the prediction result, and the layer classification result corresponding to the target layer is used as the target classification result of the text vector. The model stops reasoning and extracts exit.
[0087] In step S5, if the model has no confidence in the prediction result, the result of the current layer will be input into the next layer to continue reasoning.
[0088] Step S6. During this process, if the model has not reached the exit criteria and the updated number value is less than the first predetermined threshold, the output vector of the target layer is used to input the next layer of the target layer for feature extraction until the next layer is the last layer. The layer classification result corresponding to the last layer is determined, that is, the prediction result of the last layer of the model is used as the target classification result, that is, the final reasoning effect of the model.
[0089] The above optional implementation achieves at least the following effects: In the reasoning process, a new flexible early exit method and three different dynamic early exit judgment methods are proposed to improve the reasoning efficiency of the model. For confident prediction results, the model is allowed to exit early, reducing redundant calculations and speeding up the processing speed of the model. By adaptively processing texts of different complexities, the model can better adapt to various application scenarios, improving the generalization ability and robustness of the model.
[0090] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0091] In this embodiment, a text classification device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "device" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0092] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing a text classification method. Figure 5 is a schematic diagram of a text classification device according to an embodiment of the present invention. Figure 5 As shown, the above-mentioned text classification device includes an input module 502, a first classification result module 504, a frequency value calculation module 506, and a second classification result module 508. The device is described below.
[0093] An input module 502 is used to input a plurality of network layers in a predetermined order to perform layer-by-layer feature extraction based on the text vector to be classified;
[0094] A first classification result module 504 is connected to the input module 502 and is used to determine a layer classification result corresponding to the target layer and a similarity score between the layer classification result corresponding to the preceding layer when the target layer is included in the multiple network layers and there is a preceding layer before the target layer, wherein the target layer is a network layer other than the last layer in the multiple network layers;
[0095] The frequency value calculation module 506 is connected to the first classification result module 504 and is used to update the predetermined frequency value based on the similarity score to obtain an updated frequency value;
[0096] The second classification result module 508 and the frequency value calculation module 506 are used to use the layer classification result corresponding to the target layer as the target classification result of the text vector when the updated frequency value is greater than the first predetermined threshold.
[0097] In a text classification device provided by an embodiment of the present invention, an input module 502 is set to input multiple network layers in a predetermined order to extract features layer by layer based on a text vector to be classified; a first classification result module 504 is used to determine the layer classification result corresponding to the target layer and the similarity score between the layer classification result corresponding to the preceding layer when a target layer is included in the multiple network layers and a preceding layer exists before the target layer, wherein the target layer is a network layer other than the last layer in the multiple network layers; a frequency value calculation module 506 is used to update the predetermined frequency value based on the similarity score to obtain the updated frequency value; a second classification result module 508 is used to use the layer classification result corresponding to the target layer as the target classification result of the text vector when the updated frequency value is greater than the first predetermined threshold. The purpose of improving the use of model early retirement and reducing the model processing time is achieved, and the technical effect of improving the efficiency of the large model reasoning process is achieved, thereby solving the technical problem of unsatisfactory text classification efficiency existing in the related art.
[0098] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0099] It should be noted that the above-mentioned input module 502, the first classification result module 504, the number value calculation module 506, and the second classification result module 508 correspond to steps S102 to S108 in the embodiment, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules can be run in a computer terminal as part of the device.
[0100] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.
[0101] The above-mentioned text classification device may also include a processor and a memory, and the input module 502, the first classification result module 504, the frequency value calculation module 506, the second classification result module 508, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.
[0102] The processor includes a kernel, which retrieves the corresponding program unit from the memory. There can be one or more kernels. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0103] An embodiment of the present invention provides a non-volatile storage medium on which a program is stored. When the program is executed by a processor, a text classification method is implemented.
[0104] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: based on a text vector to be classified, multiple network layers in a predetermined order are input to perform layer-by-layer feature extraction; when a target layer is included in the multiple network layers and there is a pre-layer before the target layer, a layer classification result corresponding to the target layer is determined, and a similarity score between the layer classification result corresponding to the pre-layer is determined, wherein the target layer is a network layer other than the last layer in the multiple network layers; based on the similarity score, a predetermined number value is updated to obtain an updated number value; when the updated number value is greater than a first predetermined threshold, the layer classification result corresponding to the target layer is used as a target classification result of the text vector. The device in this article can be a server, a PC, etc.
[0105] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: when a target layer is included in multiple network layers and there is a preceding layer before the target layer, determining the layer classification result corresponding to the target layer and the similarity score between the layer classification result corresponding to the preceding layer, wherein the target layer is a network layer other than the last layer in the multiple network layers; based on the similarity score, updating a predetermined number value to obtain an updated number value; when the updated number value is greater than a first predetermined threshold, using the layer classification result corresponding to the target layer as the target classification result of the text vector.
[0106] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0111] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0112] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0113] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0114] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0115] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A text classification method, characterized in that: include: Based on the text vector to be classified, multiple network layers in a predetermined order are input to perform layer-by-layer feature extraction; In the case where a target layer is included in the multiple network layers and there is a preceding layer before the target layer, determining a similarity score between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer, wherein the target layer is a network layer other than the last layer in the multiple network layers; Based on the similarity score, updating the predetermined number of times to obtain an updated number of times; When the updated number value is greater than a first predetermined threshold, the layer classification result corresponding to the target layer is used as the target classification result of the text vector.
2. The method according to claim 1, characterized in that The similarity score is negatively correlated with the confidence of the layer classification result of the corresponding network layer, and the updating of the predetermined number value based on the similarity score to obtain the updated number value includes: In the case where there are multiple preceding layers, based on the similarity scores between the multiple preceding layers and the target layer, determining a target number of preceding layers whose similarity scores are less than or equal to a second predetermined threshold value among the multiple preceding layers; When the target number is greater than a third predetermined threshold, it is determined to perform an increment process on the number value to obtain the updated number value.
3. The method according to claim 1, characterized in that: The updating of the predetermined number value based on the similarity score to obtain the updated number value includes: In the case that there are multiple preceding layers, and the multiple preceding layers are continuously adjacent to the target layer, if there is a preceding layer among the multiple preceding layers whose similarity score is greater than a second predetermined threshold, the number value is cleared to obtain an updated number value; If the similarity score corresponding to each of the plurality of preceding layers is less than or equal to a second predetermined threshold, the number value is incremented to obtain the updated number value.
4. The method according to claim 1, characterized in that: The method further comprises: When the updated number value is less than the first predetermined threshold, based on the output vector of the target layer, the next layer of the target layer is input for feature extraction until the next layer is the last layer, and the layer classification result corresponding to the last layer is determined as the target classification result.
5. The method according to claim 1, characterized in that: The preceding layer is a layer above the target layer, and the updating of the predetermined number value based on the similarity score to obtain the updated number value includes: In the case where the similarity score between the previous layer and the target layer is less than or equal to a second predetermined threshold, incrementing the number value to obtain the updated number value; When the similarity score between the previous layer and the target layer is greater than a second predetermined threshold, the number value is cleared to obtain the updated number value.
6. The method according to claim 1, characterized in that The determining of a similarity score between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer includes: Determine a distance between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer; Based on the distance, the similarity score is determined.
7. The method according to any one of claims 1 to 6, characterized in that: The method further includes: determining the text complexity of the text vector; a target update condition corresponding to the text complexity among a plurality of candidate update conditions, wherein the plurality of candidate update conditions are respectively for different text complexities; The updating of the predetermined number value based on the similarity score to obtain the updated number value includes: based on the similarity score, using the target update condition, performing update processing on the number value to obtain the updated number value.
8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: determining processing resources allowed to be used by the plurality of network layers and a required classification accuracy of the text vector; The first predetermined threshold is determined based on the processing resources and the classification accuracy.
9. A text classification device, characterized in that: include: An input module, used for inputting a plurality of network layers in a predetermined order to perform layer-by-layer feature extraction based on the text vector to be classified; A first classification result module is used to determine a similarity score between a layer classification result corresponding to the target layer and a layer classification result corresponding to the preceding layer when the target layer is included in the multiple network layers and there is a preceding layer before the target layer, wherein the target layer is a network layer other than the last layer in the multiple network layers; A times value calculation module, used for updating a predetermined times value based on the similarity score to obtain an updated times value; The second classification result module is used to use the layer classification result corresponding to the target layer as the target classification result of the text vector when the updated number value is greater than a first predetermined threshold.
10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing a text classification method as described in any one of claims 1 to 8.
