Classification task processing method and device, terminal equipment and storage medium

Abstract

The invention discloses a classification task processing method and device, terminal equipment and a storage medium, input data of a classification task and a thinking token corresponding to a natural language processing model are obtained, the natural language processing model comprises a classifier layer and at least one converter layer, and each converter layer is connected with a probability prediction layer; adding the thinking token into the input data to obtain a target input sequence, inputting the target input sequence into a natural language processing model, processing the target input sequence through a converter layer to obtain a first output result output by the converter layer, the first output result comprising a token output result of the thinking token, and outputting the token output result to the converter layer; therefore, whether to execute the pre-exit operation or not can be determined according to the token output result, and the first output result is input to the classifier layer to obtain the classification result under the condition that the pre-exit operation is determined to be executed, so that subsequent operation does not need to be executed, the processing efficiency is improved, and the condition of computing resource waste is reduced.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a classification task processing method, apparatus, terminal device and storage medium. Background Art

[0002] With the rapid development of natural language processing (NLP) technology, natural language processing models have also been widely used in various language tasks. At present, traditional natural language processing models can be used to process various types of NLP tasks, such as text classification, named entity recognition, question-answering systems, etc.

[0003] However, traditional natural language processing models use a fixed computing path for processing. Even when processing data of different dimensions, the same computing path is used for processing. As a result, the same computing path will be used to process simpler data, and the same computing resources will be consumed, resulting in a waste of computing resources and low processing efficiency. Summary of the invention

[0004] The embodiments of the present application provide a classification task processing method, apparatus, terminal device and storage medium, aiming to solve the problems of waste of computing resources and low processing efficiency in existing natural language processing models.

[0005] In a first aspect, an embodiment of the present application provides a method for processing a classification task, the method comprising:

[0006] Obtain input data of the classification task and thought tokens corresponding to a natural language processing model, and add the thought tokens to the input data to obtain a target input sequence; wherein the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to a probability prediction layer, and the thought tokens are used to represent a global representation of the target input sequence;

[0007] Inputting the target input sequence into a natural language processing model, and processing the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result includes a token output result of the thought token;

[0008] When the token output result is input to the probability prediction layer and the early exit operation is determined based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain a classification result.

[0009] In a possible implementation manner of the first aspect, before processing the target input sequence through the converter layer to obtain the first output result of the converter layer, the method further includes:

[0010] Constructing an initial model; wherein the initial model is an untrained natural language processing model;

[0011] Acquire a sample input sequence, and train the initial model according to the sample input sequence to obtain a trained natural language processing model, and perform the step of processing the target input sequence through the converter layer based on the trained natural language processing model to obtain a first output result of the converter layer;

[0012] Wherein, the sample input sequence includes thought tokens.

[0013] In a possible implementation manner of the first aspect, the training the initial model according to the sample input sequence to obtain a trained natural language processing model includes:

[0014] Determining a classification label corresponding to the sample input sequence;

[0015] Inputting the sample input sequence into the initial model to obtain a sample output result corresponding to the sample input sequence;

[0016] Determining a model loss value of the initial model according to the classification label and the sample output result;

[0017] The model parameters of the initial model are optimized according to the model loss value to obtain a trained natural language processing model.

[0018] In a possible implementation of the first aspect, in the case where it is determined to perform an early exit operation according to the token output result, before inputting the first output result to the classifier layer to obtain a classification result, the method further includes:

[0019] For each converter layer, inputting the token output result of the converter layer into the probability prediction layer to obtain probability information of performing an early exit operation from the converter layer;

[0020] When the probability information is greater than or equal to a preset hyperparameter threshold and the converter layer is not the last layer, the early exit operation is performed.

[0021] In a possible implementation manner of the first aspect, inputting the token output result to the probability prediction layer to obtain probability information of performing an early exit operation from the converter layer includes:

[0022] For each converter layer, inputting the token output result of the converter layer into the probability prediction layer to obtain the first exit probability of the converter layer;

[0023] Obtain a second exit probability of a preceding converter layer, and generate cumulative probability information of performing an early exit operation from the converter layer based on the first exit probability and the second exit probability; wherein the preceding converter layer is a converter layer before the converter layer.

[0024] In a possible implementation manner of the first aspect above, the method further includes:

[0025] For each converter layer, when the probability information of the converter layer is less than the hyperparameter threshold and the converter layer is not the last layer, the first output result is input to the next converter layer, and the step of processing the target input sequence through the converter layer to obtain the first output result of the converter layer is repeated in the next converter layer;

[0026] Or, when the converter layer is the last layer, the classification identification data in the first output result is input into the classifier layer to obtain the second output result.

[0027] In a possible implementation manner of the first aspect above, the hyperparameter threshold is determined based on the model performance of the natural language processing model.

[0028] In a second aspect, an embodiment of the present application provides a device for processing a classification task, the device comprising:

[0029] An acquisition module, used for acquiring input data of the classification task and thought tokens corresponding to the natural language processing model, and adding the thought tokens to the input data to obtain a target input sequence; wherein the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to a probability prediction layer, and the thought tokens are used to represent a global representation of the target input sequence;

[0030] A processing module, configured to input the target input sequence into a natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result includes a token output result of the thought token;

[0031] A classification module is used to input the first output result into the classifier layer to obtain a classification result when the token output result is input into the probability prediction layer and an early exit operation is determined based on the output result of the probability prediction layer.

[0032] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processing method for the classification task provided in the first aspect above is implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processing method of the classification task provided in the first aspect above is implemented.

[0034] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0035] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0036] In an embodiment of the present application, by obtaining the input data of the classification task and the thinking tokens corresponding to the natural language processing model, the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to the probability prediction layer, the thinking token is added to the input data to obtain the target input sequence, and the target input sequence is input into the natural language processing model, the target input sequence is processed by the converter layer to obtain the first output result output by the converter layer, the first output result includes the token output result of the thinking token, the token output result is input into the probability prediction layer, and the thinking token can be used to represent the global representation of the target input sequence, then the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and when it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain the classification result, without the need to perform subsequent operations, thereby improving processing efficiency and reducing the waste of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of a method for processing a classification task provided by an embodiment of the present application;

[0038] Figure 2 is a flowchart of another method for processing a classification task provided by an embodiment of the present application;

[0039] Figure 3 It is a schematic diagram of the architecture of a natural language processing model provided by this application;

[0040] Figure 4 This is a flowchart of a method for processing a classification task provided by an embodiment of the present application;

[0041] Figure 5 It is a structural diagram of a classification task processing device provided by an embodiment of the present application;

[0042] Figure 6 It is a structural block diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] With the rapid development of natural language processing technology (NLP), natural language processing models have also been widely used in various language tasks, such as language representation models (Bidirectional Encoder Representation from Transformers, BERT), lightweight language representation models (A LiteBERT, ALBERT) and other natural language processing models. Generally, various natural language processing models will use the Transformer architecture, which is a deep learning model architecture based on the self-attention mechanism, allowing the model to be pre-trained on a large amount of data to capture language features, and then the natural language processing model can be used to process various types of NLP tasks, such as text classification, named entity recognition, question-answering systems and other tasks.

[0045] However, traditional natural language processing models use a fixed computing path for processing. Even when processing data of different dimensions, the same computing path is used for processing. As a result, the same computing path will be used to process simpler data, and the same computing resources will be consumed, resulting in a waste of computing resources and low processing efficiency.

[0046] Based on this, the present application provides a method for processing a classification task, by obtaining the input data of the classification task and the thinking tokens corresponding to the natural language processing model, the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to the probability prediction layer, the thinking token is added to the input data to obtain the target input sequence, and the target input sequence is input into the natural language processing model, the target input sequence is processed by the converter layer to obtain a first output result output by the converter layer, the first output result includes the token output result of the thinking token, the token output result is input into the probability prediction layer, and the thinking token can be used to represent the global representation of the target input sequence, then the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and when it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain a classification result, without the need to perform subsequent operations, thereby improving processing efficiency and reducing the waste of computing resources.

[0047] See also Figure 1 , Figure 1 A flowchart of a method for processing a classification task provided by an embodiment of the present application is shown, which may specifically include the following steps:

[0048] Step 101, obtain the input data of the classification task and the thinking tokens corresponding to the natural language processing model, and add the thinking tokens to the input data to obtain the target input sequence.

[0049] Among them, the classification task can be a task of classifying text data (such as sentences, paragraphs, documents and other text data) to identify the category to which the text data belongs. The text data can be composed of at least one natural language, such as Chinese and English natural languages. Exemplarily, the classification task can be a sentiment classification task, which can classify the text data into categories such as positive and negative sentiments; the classification task can also be a news classification task, which can classify the text data into different news categories, such as sports, entertainment, technology and other categories. The input data can be the text data that needs to perform the classification task. The natural language processing model can be a neural network model based on the self-attention mechanism, and the self-attention mechanism can be a mechanism for calculating the correlation between each element in the input sequence and all other elements. The natural language processing model may include an embedding layer, a classifier layer, a probability prediction layer, and at least one converter layer. The embedding layer can be used to preprocess the input sequence, including punctuation normalization, word lowercase and other processing. The converter layer can process the data based on the self-attention mechanism to obtain the output result of the input sequence. Specifically, the converter layer can be used to capture the correlation between each element in the input sequence. The converter layer can include a self-attention layer and a feedforward network layer. The self-attention layer can use the self-attention mechanism to process data to obtain the output result of the self-attention layer. The feedforward network layer can be used to perform nonlinear processing on the output result of the self-attention layer through an activation function to obtain the output result of the converter layer. The probability prediction layer can be a linear transformation layer that calculates the probability of performing an early exit operation based on the output result of the converter layer. The classifier layer can be used to convert the output result of the converter layer into a probability distribution of a category label to generate a category prediction result of the text data. The early exit operation can be an operation to exit the operation from the current converter layer. The thinking token can be used to capture and integrate the semantic information of all elements in the input sequence, that is, the thinking token can represent the global representation of the input sequence, and the natural language processing model can calculate the probability of performing an early exit operation based on the global representation of the thinking token, and specifically, the probability of performing an early exit operation can be calculated through the probability prediction layer.

[0050] When a classification task needs to be performed, a pre-trained natural language processing model can be loaded, and the thinking tokens in the natural language processing model can be obtained, as well as the text data required to perform the classification task, which is the input data for the classification task. Then, the thinking tokens can be added to the input data to obtain the target input sequence.

[0051] Specifically, the target input sequence may include text words representing the text and thought tokens. As an example, the target input sequence may also include classification tokens, which may be user-defined words. Classification tokens may be used to capture and integrate the semantic information of all elements in the input sequence, that is, classification tokens may represent the global representation of the input sequence. In the process of training the natural language processing model, thought tokens may be used to optimize the probability prediction layer, while classification tokens may be used to optimize the classifier layer.

[0052] In practical applications, in order to facilitate the thinking token to capture and integrate the global information of the target input sequence, the thinking token can be added to the front end of the input data to obtain the target input sequence, that is, the thinking token is the first element in the target input sequence.

[0053] It is important to understand that since the natural language processing model is a neural network model based on the self-attention mechanism, the self-attention mechanism can be a mechanism for calculating the correlation between each element in the input sequence and all other elements, and the thinking token is the first element in the target input sequence, the thinking token can be calculated first, so that the thinking token can capture the semantic information of all elements in the input sequence and calculate the correlation between each element in the input sequence and all other elements, thereby improving the accuracy of generating the global representation.

[0054] Step 102: input the target input sequence into the natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer.

[0055] Among them, the first output result can be a word vector used to represent the association between each word in the text data and other words. The first output result can include the main text output result and the token output result of the thinking token. The main text output result can be the word vector of the association between each main text word and other words in the target input sequence. The main text word can be the word in the target input sequence except the thinking token, and the other word can be the word in the target input sequence except the main text word currently being processed. The token output result can be a global representation of the target input sequence.

[0056] After obtaining the target input sequence, the target input sequence can be input into a natural language processing model, and then the target input sequence can be processed by a converter layer in the natural language processing model to obtain a first output result output by the converter layer.

[0057] In practical applications, each word in the target input sequence can be processed through the query matrix, key matrix and value matrix in the converter layer to obtain the query vector, key vector and value vector of each word. Then, based on the query vector and key vector of each word, the attention score between each word and other words can be calculated, and the attention score between each word and other words can be processed using a normalized exponential function to obtain the output result of each word in the converter layer. The first output result of the target input sequence in the converter layer can be generated according to the output result of each word.

[0058] As an example, the natural language processing model may also include a pre-trained vocabulary. Specifically, the vocabulary may be a list containing a large number of words, phrases and other language elements. The language elements included in the vocabulary cover the professional vocabulary of each business field, including not only basic nouns, verbs, adjectives and other vocabulary, but also phrases, idioms, place names, personal names, organization names and other vocabulary, and each language element in the vocabulary has a corresponding word vector, and the word vector corresponding to each language element may be determined by pre-training.

[0059] In practical applications, the target input sequence can be preprocessed through the embedding layer, and at least one word can be segmented from the preprocessed input sequence. Then, the language elements corresponding to each word in the target input sequence and their corresponding word vectors can be queried from the vocabulary, that is, the word vector corresponding to each word in the target input sequence is determined from the vocabulary, and the position information of each word in the target input sequence is determined, and the corresponding position vector is generated. Then, based on the word vector and position vector corresponding to each word, an embedding vector of each word can be generated, including the embedding vector of the thinking token. The embedding vector of each word can be used as the output of the embedding layer and input to the converter layer, so that the embedding vector can be processed by the converter layer in the above manner.

[0060] In an embodiment of the present application, before step 102, the following steps may also be included:

[0061] Construct an initial model, obtain a sample input sequence, and train the initial model according to the sample input sequence to obtain a trained natural language processing model, and then execute the step of processing the target input sequence through the converter layer based on the trained natural language processing model to obtain the first output result of the converter layer.

[0062] Among them, the initial model can be an untrained natural language processing model, the sample input sequence can be text data for training the initial model, and the sample input sequence can include thinking tokens, which can be user-defined words or words related to the business field.

[0063] In practical applications, a natural language processing model based on the self-attention mechanism can be pre-built, and the natural language processing model can include an embedding layer, at least one converter layer, a probability prediction layer, a classifier layer, and a feedforward network layer, which is the initial model.

[0064] After building the initial model, text data from different business fields can be obtained to generate sample input sequences for each business field. Each sample input sequence can then be input into the initial model. After being processed by the embedding layer, converter layer, probability prediction layer, and classifier layer in the initial model, the output results of each sample input sequence are obtained. Based on the output results of each sample input sequence, the model parameters of the initial model are updated.

[0065] Specifically, the model parameters of the initial model may include the weights of the embedding layer, the converter layer, the probability prediction layer, and the classifier layer during the forward propagation process, and the model parameters of the converter layer may include the weights of the self-attention layer and the feedforward network layer during the forward propagation process.

[0066] Exemplarily, in the process of building the initial model, user-defined word units can also be obtained as thinking tokens, and the thinking tokens can be added to the front end of the sample input sequence, so that the initial model can be trained using the sample input sequence after adding the thinking tokens, so that the trained thinking tokens can accurately capture the semantic information of each word in the input sequence.

[0067] Specifically, a word-element may be any meaningful basic unit appearing in a text, such as a word, punctuation mark, number, special symbol, etc.

[0068] In an embodiment of the present application, the initial model can also be trained in the following manner:

[0069] Determine the classification label corresponding to the sample input sequence, input the sample input sequence into the initial model, obtain the sample output result corresponding to the sample input sequence, determine the model loss value of the initial model according to the classification label and the sample output result, optimize the model parameters of the initial model according to the model loss value, and obtain the trained natural language processing model.

[0070] Among them, the classification label can be expressed as the category information of the sample input sequence, the model parameters can include the weights of the embedding layer, the converter layer, the probability prediction layer and the classifier layer in the forward propagation process, and the weights of the self-attention layer in the converter layer and the feedforward network layer in the forward propagation process, and the model loss value can be the loss value between the sample output result and the classification label.

[0071] In the process of training the initial model, the classification label corresponding to each sample input sequence can be determined, such as the technology category and entertainment category in the news classification task or the positive emotion category and negative emotion category in the emotion classification task, and each sample input sequence is input into the initial model in turn to obtain the sample output result corresponding to each sample input sequence. Then, the loss between the classification label and the sample output result corresponding to each sample input sequence can be determined, which is the model loss value of the initial model. The model parameters of the initial model can be optimized according to the model loss value to obtain the trained natural language processing model.

[0072] Specifically, the model loss L of the initial model can be calculated using the following formula:

[0073] L(x,y)=E i ~ p(i|x) [logp(y|x,i)-βKL(p(·|x)∥p(·|λ))]

[0074] Among them, x can be the sample input sequence, y can be the classification label corresponding to the sample input sequence, i can be the hidden variable, and E i～p(i|x) It can be the expectation of latent variable i, which is sampled from the conditional distribution p(i|x). The conditional distribution p(i|x) can be expressed as the probability distribution of latent variable i under the condition of a given input. logp(λ|x,i) can be expressed as the log-likelihood of target y under the condition of given input x and latent variable i, which can be expressed as KL divergence. KL(p(·|x)∥p(·|λ)) is used to measure the difference between two probability distributions, that is, to measure the difference between p(·|x) and p(·|λ). p(·|x) can be the probability distribution of a variable under the condition of given x, p(·|λ) can be the prior distribution, and β can be a hyperparameter used to balance the weight between the log-likelihood term and the KL divergence term.

[0075] Step 103, when the token output result is input to the probability prediction layer and the early exit operation is determined based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain the classification result.

[0076] The early exit operation may be an operation of exiting subsequent operations from the current converter layer, and the classification result may be a prediction result of the natural language processing model for classifying the target input sequence.

[0077] After obtaining the first output result, the token output result in the first output result can be input into the probability prediction layer to obtain the output result of the probability prediction layer, and whether to perform an early exit operation can be determined based on the output result of the probability prediction layer. If it is determined to perform an early exit operation, the first output result can be input into the classifier layer. The classifier layer can calculate the probability distribution of the target input sequence as each category in the classification task based on the first output result, so that the category of the target input sequence can be predicted based on the probability distribution of the target input sequence, which is the classification result of the target input sequence.

[0078] In an embodiment of the present application, by obtaining the input data of the classification task and the thinking tokens corresponding to the natural language processing model, the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to the probability prediction layer, the thinking token is added to the input data to obtain the target input sequence, and the target input sequence is input into the natural language processing model, the target input sequence is processed by the converter layer to obtain the first output result output by the converter layer, the first output result includes the token output result of the thinking token, the token output result is input into the probability prediction layer, and the thinking token can be used to represent the global representation of the target input sequence, then the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and when it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain the classification result, without the need to perform subsequent operations, thereby improving processing efficiency and reducing the waste of computing resources.

[0079] See also Figure 2 , Figure 2 A flowchart of another method for processing a classification task provided by an embodiment of the present application is shown, which may specifically include the following steps:

[0080] Step 201, obtain the input data of the classification task and the thinking tokens corresponding to the natural language processing model, and add the thinking tokens to the input data to obtain the target input sequence.

[0081] Step 202: input the target input sequence into the natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer.

[0082] For the related descriptions of step 201 to step 202, please refer to step 101 to step 102, and they will not be repeated here.

[0083] Step 203: for each converter layer, the token output result is input to the probability prediction layer to obtain the probability information of executing the early exit operation from the converter layer.

[0084] After obtaining the first output result, the token output result in the first output result may be input into the probability prediction layer to obtain probability information of performing an early exit operation from the converter layer.

[0085] It should be understood that since the natural language processing model is trained, and during the training process, the output results output by each layer of the converter layer and the final output results processed by all the converter layers can be determined, and then the loss between the output results of each layer of the converter layer and the final output results can be determined, as well as the token output results of the thinking tokens in the output results of each layer of the converter layer can be determined, and the token output results of each layer of the converter layer are input into the probability prediction layer for calculation to obtain the probability information of each layer of the autonomous attention layer, so that the weights of the probability prediction layer can be optimized based on the loss between the output results of each layer of the converter layer and the final output results and the probability information of each layer of the converter layer, so that the optimized probability prediction layer can accurately calculate the probability information of performing the extraction exit operation from the converter layer, that is, the optimized probability prediction layer can determine whether the first output result output by the current converter layer is equivalent to the final output result.

[0086] In practical applications, since the natural language processing model can include at least one converter layer, the target input sequence can be input into the first converter layer for calculation to obtain the first output result of the first converter layer, and the first output result of the first converter layer is used as the input of the second converter layer. Similarly, the target input sequence can be calculated through each converter layer in turn.

[0087] For each converter layer, after receiving the input of the previous converter layer or receiving the target input sequence, the first output result of the converter layer can be calculated, and the token output result in the first output result can be input into the probability prediction layer to obtain the probability information of performing an early exit operation from the converter layer. Then, it can be determined whether to perform an early exit operation based on the probability information of the converter layer, so that the early exit operation can be performed from any converter layer based on the probability information of each converter layer.

[0088] In an embodiment of the present application, step 203 may include steps 2031 to 2032:

[0089] Step 2031: for each converter layer, the token output result of the converter layer is input into the probability prediction layer to obtain the first exit probability of the converter layer.

[0090] The first exit probability may be based on probability information determined by a token output result of the current converter layer, ie, probability information obtained by inputting the token output result of the current converter layer into the probability prediction layer.

[0091] In practical applications, for each converter layer, the token output result of the converter layer can be input into the probability prediction layer to obtain the first exit probability of the converter layer.

[0092] Step 2032: Obtain a second exit probability of the preceding converter layer, and generate cumulative probability information of executing an early exit operation from the converter layer according to the first exit probability and the second exit probability.

[0093] Among them, the previous converter layer may be a converter layer before processing the current converter layer, the second exit probability may be probability information determined based on the token output result of each previous converter layer, and the cumulative probability information may be the cumulative value between the probability information of performing an extraction exit operation from the current converter layer and the probability information of performing an extraction exit operation from the previous converter layer.

[0094] Specifically, since each previous converter layer has input its own output results into the probability prediction layer before processing the current converter layer, the probability information of each previous converter layer is obtained, and then the probability information of each previous converter layer can be directly obtained, which is the second exit probability.

[0095] In practical applications, for each converter layer, the previous converter layer before processing the current converter layer can be determined and the second exit probability of each previous converter layer can be obtained, and then the probability information of performing an early exit operation from the converter layer can be generated based on the first exit probability and the second exit probability.

[0096] Specifically, the probability information p(i|x) can be determined by the following formula:

[0097]

[0098] h i =S(h i-1 )

[0099] Among them, Λ i It can be expressed as the exit probability of performing early exit operation from the i-th layer of the transformer, that is, the first exit probability, Λ j It can be expressed as the exit probability of performing early exit operation from the j-th layer of the transformer layer, that is, the second exit probability, It can be expressed as the token output result of the i-th layer transformer layer, h i It can be expressed as the first output result of the i-th transformer layer, h i-1 It can be expressed as the first output result of the i-1th converter layer, that is, the input of the i-th converter layer, and S can be the calculation function of the converter layer.

[0100] After obtaining the probability information of the current converter layer, the probability information of performing an early exit operation from the previous converter layer can be obtained, and the cumulative probability information of performing an early exit operation from the current converter layer can be calculated based on the probability information of the current converter layer and the probability information of the previous converter layer.

[0101] Exemplarily, the current converter layer may be a third converter layer, and the token output result of the third converter layer may be input into the probability prediction layer to obtain a first exit probability of the third converter layer. The previous converter layer before the third converter layer, that is, the first converter layer and the second converter layer are both previous converter layers, may be determined. Since the target input sequence has been processed by the second converter layer and the first converter layer before processing the third converter layer, the exit probability of the first converter layer and the exit probability of the second converter layer are obtained. In the process of processing the third converter layer, the second exit probabilities of the first converter layer and the second converter layer may be directly obtained, so that the probability information of performing an early exit operation from the third converter layer can be calculated based on the first exit probability and the second exit probability.

[0102] Similarly, since the target input sequence has been processed by the second converter layer and the first converter layer before processing the third converter layer, it is possible to directly obtain the probability information of performing an early exit operation from the second converter layer and the probability information of performing an early exit operation from the first converter layer, and accumulate the probability information of the first to third converter layers to obtain the cumulative probability information of performing an early exit operation from the third converter layer.

[0103] Step 204: When the probability information is greater than or equal to a preset hyperparameter threshold and the converter layer is not the last layer, an early exit operation is performed.

[0104] Among them, the hyperparameter threshold can be determined based on the model performance of the natural language processing model, and the hyperparameter threshold can be used to determine whether the first output result output by the current converter layer is equivalent to the final output result. The final output result can be the output result obtained after all converter layers operate on the target input sequence.

[0105] After obtaining the probability information, the probability information can be compared with the hyperparameter threshold, and then, when the probability information is greater than or equal to the hyperparameter threshold, it can be determined that the difference between the first output result output by the current converter layer and the final output result is small, and it can be considered that the first output result output by the current converter layer is equivalent to the final output result. Then, it can be determined to perform an extraction exit operation, that is, to perform an operation to exit the operation from the current converter layer, and use the first output result of the current converter layer as the output result of the target input sequence, without performing subsequent converter layer operations, thereby improving processing efficiency.

[0106] In an embodiment of the present application, the following method may also be included:

[0107] For each converter layer, when the probability information of the converter layer is less than the hyperparameter threshold and the converter layer is not the last layer, the first output result is input to the next converter layer, and the step of processing the target input sequence through the converter layer to obtain the first output result of the converter layer is repeated in the next converter layer, or, when the converter layer is the last layer, the classification identification data in the first output result is input to the classifier layer to obtain the second output result.

[0108] The classification identification data may be an output result obtained by processing the classification tokens through the converter layer, and the classification identification data may be a global representation of the target input sequence.

[0109] In practical applications, for each converter layer, it can be determined whether the converter layer is the last layer of all converter layers. When the converter layer is not the last layer, the probability information of the converter layer can be compared with the hyperparameter threshold. When the probability information of the converter layer is less than the hyperparameter threshold, it can be determined that the difference between the first output result output by the current converter layer and the final output result is large, and it cannot be considered that the first output result output by the current converter layer is equivalent to the final output result, and it is necessary to continue to propagate forward, that is, the first output result needs to be input to the next converter layer, and the above-mentioned steps of processing the target input sequence through the converter layer to obtain the first output result of the converter layer are repeated in the next converter layer, and the step of determining whether to perform an early exit operation based on the token output result in the first output result is executed.

[0110] When the converter layer is the last layer, it can be determined that the target input sequence is calculated through all converter layers to obtain the first output result of the last converter layer, which is the final output result. Then, the classification identification data in the first output result can be input into the classifier layer to obtain the second output result.

[0111] It should be understood that during the training of the initial model, user-defined classification tokens can also be added to the sample input sequence, and the parameters of the probability prediction layer in the initial model can be optimized through thinking tokens, and the classifier layer can be optimized through classification tokens, that is, thinking tokens are used to optimize the probability prediction layer, so that the optimized probability prediction layer can accurately calculate the probability information of performing early exit operations from each converter layer, and classification tokens are used to optimize the classifier layer, so that the optimized classifier layer can accurately calculate the probability distribution of the target input sequence as each category in the classification task.

[0112] See also Figure 3 , Figure 3 A schematic diagram of the architecture of a natural language processing model provided by an embodiment of the present application is shown. Figure 3 As shown, the natural language processing model 3 may include an embedding layer 31 , at least one converter layer 32 , a probability prediction layer 33 , and a classifier layer 34 .

[0113] The embedding layer 31 can receive input data and perform embedding processing to obtain a target input sequence. The embedding layer 31 can input the target input sequence to the first layer converter layer 32, and the first layer converter layer 32 processes the target input sequence after receiving it to obtain a first output result of the first layer converter layer 32. The first layer converter layer 32 can input the first output result to the probability prediction layer 33, and then the probability prediction layer 33 can calculate the probability information of performing an early exit operation from the first layer converter layer 32 according to the token output result in the first output result output by the first layer converter layer 32.

[0114] Exemplarily, after the target input sequence is input into the converter layer, the target input sequence can be processed by the self-attention layer in the converter layer to obtain the output result of the self-attention layer, and the output result of the self-attention layer can be input into the feedforward network layer to obtain the output result of the feedforward network layer. The output result of the feedforward network layer can then be used as the output result of the converter layer and input into the probability prediction layer to obtain the probability information of performing an early exit operation from the converter layer.

[0115] When the probability information is greater than or equal to the hyperparameter threshold, it is determined to perform an early exit operation, and the first output result output by the first converter layer 32 is input to the classifier layer 34 to obtain the classification result output by the classifier layer 34 based on the first output result output by the first converter layer 32.

[0116] When the probability information is less than the hyperparameter threshold, it is determined that the early exit operation will not be performed, and the first output result of the first converter layer 32 is input to the second converter layer 32 to obtain the first output result of the second converter layer 32. Then, based on the first output result of the second converter layer 32, the above step of determining the probability information of performing an early exit operation from the current converter layer can be repeated. By analogy, the second converter layer to the last converter layer can be processed in the above manner, that is, the first output result of each converter layer and the probability information of performing an early exit operation on each converter layer can be obtained.

[0117] When all converter layers 32 do not perform early exit operations, the classification identification data in the first output result of the last converter layer 32 can be input into the classifier layer 34, and then the classifier layer 34 can obtain the classification result based on the classification identification data.

[0118] Step 205, when the token output result is input to the probability prediction layer and the early exit operation is determined based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain the classification result.

[0119] For the relevant description of step 205, please refer to step 103 and will not be repeated here.

[0120] In an embodiment of the present application, by obtaining the input data of the classification task and the thinking tokens corresponding to the natural language processing model, and adding the thinking tokens to the input data, a target input sequence is obtained, the target input sequence is input into the natural language processing model, and the target input sequence is processed through the converter layer to obtain a first output result output by the converter layer. For each converter layer, the token output result in the first output result is input into the probability prediction layer. Since the thinking token can be used to represent the global representation of the target input sequence, the probability prediction layer can obtain the probability information of performing an early exit operation from the converter layer based on the token output result, and perform an early exit operation when the probability information is greater than or equal to a preset hyperparameter threshold and the converter layer is not the last layer. When the token output result is input into the probability prediction layer and it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain a classification result without performing subsequent operations, thereby improving processing efficiency and reducing waste of computing resources.

[0121] See, 4, Figure 4 A schematic flow chart of a method for processing a classification task provided by an embodiment of the present application is shown, which may specifically include the following steps:

[0122] Step 401, the process starts;

[0123] Step 402, obtaining input data, and performing preprocessing operations on the input data through the embedding layer to obtain a target input sequence;

[0124] Step 403, when the current converter layer is the first converter layer, input the target input sequence to the converter layer to obtain the first output result of the converter layer; when the current converter layer is not the first converter layer, input the first output result of the previous converter layer to the current converter layer to obtain the first output result of the current converter layer;

[0125] Step 404, inputting the token output result in the first output result to the probability prediction layer to obtain probability information of performing an early exit operation from the converter layer;

[0126] Step 405, determining whether to perform an early exit operation according to the probability information, if yes, executing step 406, if no, executing step 407;

[0127] Step 406, performing an early exit operation, and inputting the first output result of the converter layer into the classifier layer to obtain a classification result;

[0128] Step 407, determining whether the current converter layer is the last layer. If the current converter layer is not the last layer, repeating step 403. If the current converter layer is the last layer, inputting the classification identification data in the first output result of the current converter layer into the classifier layer to obtain a classification result.

[0129] Step 408, the process ends.

[0130] In an embodiment of the present application, by obtaining the input data of the classification task and the thinking tokens corresponding to the natural language processing model, the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to the probability prediction layer, the thinking token is added to the input data to obtain the target input sequence, and the target input sequence is input into the natural language processing model, the target input sequence is processed by the converter layer to obtain the first output result output by the converter layer, the first output result includes the token output result of the thinking token, the token output result is input into the probability prediction layer, and the thinking token can be used to represent the global representation of the target input sequence, then the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and when it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain the classification result, without the need to perform subsequent operations, thereby improving processing efficiency and reducing the waste of computing resources.

[0131] See also Figure 5 , Figure 5A schematic diagram of the structure of a classification task processing device provided in an embodiment of the present application is shown, and the device may specifically include the following modules:

[0132] An acquisition module 501 is used to acquire input data of the classification task and thought tokens corresponding to the natural language processing model, and add the thought tokens to the input data to obtain a target input sequence; wherein the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to the probability prediction layer, and the thought tokens are used to represent the global representation of the target input sequence;

[0133] A processing module 502 is used to input the target input sequence into the natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result includes a token output result of a thought token;

[0134] The classification module 503 is used to input the first output result into the classifier layer to obtain the classification result when the token output result is input into the probability prediction layer and the early exit operation is determined based on the output result of the probability prediction layer.

[0135] In one implementation, the device may further include the following modules:

[0136] A construction module is used to construct an initial model before processing the target input sequence through the converter layer to obtain the first output result of the converter layer; wherein the initial model is an untrained natural language processing model.

[0137] The training module is used to obtain a sample input sequence and train an initial model according to the sample input sequence to obtain a trained natural language processing model before processing the target input sequence through the converter layer to obtain a first output result of the converter layer, so as to execute the step of processing the target input sequence through the converter layer based on the trained natural language processing model to obtain the first output result of the converter layer.

[0138] Among them, the sample input sequence includes thought tokens.

[0139] In one implementation, the above training module may also be used for:

[0140] Determine the classification label corresponding to the sample input sequence;

[0141] Input the sample input sequence into the initial model to obtain the sample output result corresponding to the sample input sequence;

[0142] Determine the model loss value of the initial model based on the classification label and sample output results;

[0143] The model parameters of the initial model are optimized according to the model loss value to obtain the trained natural language processing model.

[0144] In one implementation, the device may further include the following modules:

[0145] The exit module is used to input the first output result to the classifier layer before obtaining the classification result, and input the token output result of the converter layer to the probability prediction layer for each converter layer to obtain the probability information of performing the early exit operation from the converter layer. When the probability information is greater than or equal to the preset hyperparameter threshold and the converter layer is not the last layer, the early exit operation is performed.

[0146] In one implementation, the above exit module may also be used to:

[0147] For each converter layer, the token output result of the converter layer is input into the probability prediction layer to obtain the first exit probability of the converter layer;

[0148] A second exit probability of a preceding converter layer is obtained, and cumulative probability information of performing an early exit operation from the converter layer is generated according to the first exit probability and the second exit probability; wherein the preceding converter layer is a converter layer before the converter layer.

[0149] In one implementation, the device may further include the following modules:

[0150] A processing module, for each converter layer, is used to input the first output result to the next converter layer when the probability information of the converter layer is less than the hyperparameter threshold and the converter layer is not the last layer, and repeatedly perform the step of processing the target input sequence through the converter layer to obtain the first output result of the converter layer in the next converter layer;

[0151] Or, when the converter layer is the last layer, the classification identification data in the first output result is input into the classifier layer to obtain the second output result.

[0152] In one implementation, the hyperparameter threshold is determined based on model performance of the natural language processing model.

[0153] In an embodiment of the present application, by obtaining the input data of the classification task and the thinking tokens corresponding to the natural language processing model, the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to the probability prediction layer, the thinking token is added to the input data to obtain the target input sequence, and the target input sequence is input into the natural language processing model, the target input sequence is processed by the converter layer to obtain the first output result output by the converter layer, the first output result includes the token output result of the thinking token, the token output result is input into the probability prediction layer, and the thinking token can be used to represent the global representation of the target input sequence, then the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and when it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain the classification result, without the need to perform subsequent operations, thereby improving processing efficiency and reducing the waste of computing resources.

[0154] It should be noted that the information interaction, execution process and other contents between the above-mentioned devices are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0155] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0156] See also Figure 6 , Figure 6 A structural block diagram of a terminal device provided by an embodiment of the present application is shown as follows: Figure 6 As shown, this embodiment provides a terminal device 61, which includes: at least one processor 611, a memory 612, and a computer program 6121 stored in the memory 612 and executable on at least one processor 611, and when the processor 611 executes the computer program 6121, the steps in any of the above-mentioned method embodiments are implemented.

[0157] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments can be implemented.

[0158] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the camera device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium.

[0160] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for processing a classification task, characterized in that: The method comprises: Obtain input data of the classification task and thought tokens corresponding to a natural language processing model, and add the thought tokens to the input data to obtain a target input sequence; wherein the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to a probability prediction layer, and the thought tokens are used to represent a global representation of the target input sequence; Inputting the target input sequence into a natural language processing model, and processing the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result includes a token output result of the thought token; When the token output result is input to the probability prediction layer and the early exit operation is determined based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain a classification result.

2. The method for processing a classification task according to claim 1, characterized in that: Before the target input sequence is processed by the converter layer to obtain a first output result of the converter layer, the method further includes: Constructing an initial model; wherein the initial model is an untrained natural language processing model; Acquire a sample input sequence, and train the initial model according to the sample input sequence to obtain a trained natural language processing model, and perform the step of processing the target input sequence through the converter layer based on the trained natural language processing model to obtain a first output result of the converter layer; Wherein, the sample input sequence includes thought tokens.

3. The method for processing a classification task as claimed in claim 2, characterized in that: The step of training the initial model according to the sample input sequence to obtain a trained natural language processing model includes: Determining a classification label corresponding to the sample input sequence; Inputting the sample input sequence into the initial model to obtain a sample output result corresponding to the sample input sequence; Determining a model loss value of the initial model according to the classification label and the sample output result; The model parameters of the initial model are optimized according to the model loss value to obtain a trained natural language processing model.

4. The method for processing a classification task according to any one of claims 1 to 3, characterized in that: In the case where it is determined to perform an early exit operation according to the token output result, before inputting the first output result to the classifier layer to obtain a classification result, the method further includes: For each converter layer, inputting the token output result of the converter layer into the probability prediction layer to obtain probability information of performing an early exit operation from the converter layer; When the probability information is greater than or equal to a preset hyperparameter threshold and the converter layer is not the last layer, the early exit operation is performed.

5. The method for processing a classification task as claimed in claim 4, characterized in that: The step of inputting the token output result to the probability prediction layer to obtain probability information of performing an early exit operation from the converter layer includes: For each converter layer, inputting the token output result of the converter layer into the probability prediction layer to obtain the first exit probability of the converter layer; Obtain a second exit probability of a preceding converter layer, and generate cumulative probability information of performing an early exit operation from the converter layer based on the first exit probability and the second exit probability; wherein the preceding converter layer is a converter layer before the converter layer.

6. The method for processing a classification task as claimed in claim 4, characterized in that: The method further comprises: For each converter layer, when the probability information of the converter layer is less than the hyperparameter threshold and the converter layer is not the last layer, the first output result is input to the next converter layer, and the step of processing the target input sequence through the converter layer to obtain the first output result of the converter layer is repeated in the next converter layer; Or, when the converter layer is the last layer, the classification identification data in the first output result is input into the classifier layer to obtain the second output result.

7. The method for processing a classification task as claimed in claim 4, characterized in that: The hyperparameter threshold is determined based on the model performance of the natural language processing model.

8. A processing device for a classification task, characterized in that: The device comprises: An acquisition module, used for acquiring input data of the classification task and thought tokens corresponding to the natural language processing model, and adding the thought tokens to the input data to obtain a target input sequence; wherein the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is respectively connected to a probability prediction layer, and the thought tokens are used to represent a global representation of the target input sequence; A processing module, configured to input the target input sequence into a natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result includes a token output result of the thought token; A classification module is used to input the first output result into the classifier layer to obtain a classification result when the token output result is input into the probability prediction layer and an early exit operation is determined based on the output result of the probability prediction layer.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images