A processing method and device of a classification task, a terminal device, and a storage medium

Abstract

The application discloses a processing method and device of a classification task, terminal equipment and a storage medium. The input data of the classification task and a thinking token corresponding to a natural language processing model are acquired, the natural language processing model comprises a classifier layer and at least one converter layer, each converter layer is connected with a probability prediction layer, the thinking token is added to the input data to obtain a target input sequence, the target input sequence is input to the natural language processing model, the target input sequence is processed through the converter layer to obtain a first output result output by the converter layer, the first output result comprises a token output result of the thinking token, whether to perform an early exit operation can be determined according to the token output result, and in the case that it is determined to perform the early exit operation, the first output result is input to the classifier layer to obtain a classification result, without performing subsequent operations, the processing efficiency is improved, and the waste of computing resources is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a processing method and device for a classification task, a terminal device and a storage medium. BACKGROUND

[0002] With the rapid development of natural language processing technology (NLP, Natural Language Processing), natural language processing models have been widely applied in various language tasks. At present, a traditional natural language processing model can be used to process various types of NLP tasks, such as text classification, named entity recognition, and question and answer systems.

[0003] However, the traditional natural language processing model uses a fixed calculation path for processing. Even when processing data of different dimensions, the same calculation path is used for processing. As a result, the same calculation path and the same calculation resources are used even when processing relatively simple data, which leads to waste of calculation resources and low processing efficiency. SUMMARY

[0004] Embodiments of the present application provide a processing method and device for a classification task, a terminal device and a storage medium, aiming to solve the problem of waste of calculation resources and low processing efficiency of existing natural language processing models.

[0005] In a first aspect, embodiments of the present application provide a processing method for a classification task, which comprises:

[0006] obtaining input data of a classification task and a thinking token corresponding to a natural language processing model, and adding the thinking token to the input data to obtain a target input sequence; wherein the natural language processing model comprises a classifier layer and at least one converter layer, and each converter layer is connected with a probability prediction layer, and the thinking token is used to represent a global representation of the target input sequence;

[0007] inputting the target input sequence into the natural language processing model, and processing the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result comprises a token output result of the thinking token;

[0008] In the case where the token output result is input into the probability prediction layer and it is determined to perform an early exit operation based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain a classification result.

[0009] In a possible implementation of the first aspect, before the processing of the target input sequence by the converter layer to obtain the first output result of the converter layer, the method further includes:

[0010] constructing an initial model, wherein the initial model is an untrained natural language processing model;

[0011] obtaining a sample input sequence, and training the initial model according to the sample input sequence to obtain a trained natural language processing model, so as to perform the processing of the target input sequence by the converter layer to obtain the first output result of the converter layer based on the trained natural language processing model;

[0012] wherein the sample input sequence comprises a thinking token.

[0013] In a possible implementation of the first aspect, the training of the initial model according to the sample input sequence to obtain the trained natural language processing model includes:

[0014] determining a classification label corresponding to the sample input sequence;

[0015] inputting the sample input sequence into the initial model to obtain a sample output result corresponding to the sample input sequence;

[0016] determining a model loss value of the initial model according to the classification label and the sample output result;

[0017] optimizing a model parameter of the initial model according to the model loss value to obtain the trained natural language processing model.

[0018] In a possible implementation of the first aspect, before the inputting of the first output result into the classifier layer to obtain a classification result in the case where it is determined according to the token output result that the early exit operation is performed, the method further includes:

[0019] for each converter layer, inputting a token output result of the converter layer into the probability prediction layer to obtain probability information of performing the early exit operation from the converter layer;

[0020] in the case where the probability information is greater than or equal to a preset hyperparameter threshold value and the converter layer is not the last layer, performing the early exit operation.

[0021] In a possible implementation of the first aspect, the inputting of the token output result into the probability prediction layer to obtain the probability information of performing the early exit operation from the converter layer includes:

[0022] For each of the transformer layers, input the token output result of the transformer layer to the probability prediction layer to obtain a first exit probability of the transformer layer;

[0023] obtain a second exit probability of a previous transformer layer, and generate cumulative probability information of performing an early exit operation from the transformer layer according to the first exit probability and the second exit probability; wherein the previous transformer layer is a transformer layer before the transformer layer.

[0024] In a possible implementation of the first aspect, the method further includes:

[0025] For each of the transformer layers, in a case where the probability information of the transformer layer is less than the hyperparameter threshold and the transformer layer is not the last layer, input the first output result to a next transformer layer, and repeat the step of processing the target input sequence through the transformer layer to obtain the first output result of the transformer layer in the next transformer layer.

[0026] Or, in a case where the transformer layer is the last layer, input classification identification data in the first output result to the classifier layer to obtain a second output result.

[0027] In a possible implementation of the first aspect, the hyperparameter threshold is determined based on model performance of the natural language processing model.

[0028] In a second aspect, an embodiment of the present application provides a processing apparatus for a classification task, and the apparatus includes:

[0029] an obtaining module, configured to obtain input data of a classification task and a thinking token corresponding to a natural language processing model, and add the thinking token to the input data to obtain a target input sequence; wherein the natural language processing model includes a classifier layer and at least one transformer layer, and each of the transformer layers is connected with a probability prediction layer, and the thinking token is used to represent a global representation of the target input sequence;

[0030] a processing module, configured to input the target input sequence to the natural language processing model, and process the target input sequence through the transformer layer to obtain a first output result output by the transformer layer; wherein the first output result includes a token output result of the thinking token;

[0031] a classification module, configured to, in a case where the token output result is input to the probability prediction layer and an early exit operation is determined to be performed based on an output result of the probability prediction layer, input the first output result to the classifier layer to obtain a classification result.

[0032] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the processing method of the classification task according to the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to implement the processing method of the classification task according to the first aspect.

[0034] It can be understood that the beneficial effects of the above-mentioned second aspect to the fourth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0035] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0036] In the embodiment of the present application, the input data of the classification task and the thinking token corresponding to the natural language processing model are obtained, the natural language processing model comprises a classifier layer and at least one converter layer, and each converter layer is connected with a probability prediction layer, the thinking token is added to the input data to obtain a target input sequence, the target input sequence is input to the natural language processing model, the target input sequence is processed by the converter layer to obtain a first output result output by the converter layer, the first output result comprises a token output result of the thinking token, the token output result is input to the probability prediction layer, the thinking token can be used to represent the global representation of the target input sequence, the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and in the case that it is determined to perform the early exit operation based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain a classification result, without performing subsequent operations, the processing efficiency is improved, and the waste of computing resources is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a step flowchart of a processing method of a classification task provided by an embodiment of the present application;

[0038] Figure 2 is a step flowchart of another processing method of a classification task provided by an embodiment of the present application;

[0039] Figure 3 is an architecture schematic diagram of a natural language processing model provided by the present application;

[0040] Figure 4 is a flow example diagram of a processing method of a classification task provided by an embodiment of the present application;

[0041] Figure 5 Fig. 1 is a structural schematic diagram of a processing device for a classification task according to an embodiment of the present application;

[0042] Figure 6 Fig. 2 is a structural block diagram of a terminal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0044] With the rapid development of natural language processing technology (NLP, Natural Language Processing), natural language processing models are also widely used in various language tasks, such as language representation models (Bidirectional Encoder Representation from Transformers, BERT), lightweight language representation models (A LiteBERT, ALBERT), etc. Generally, various natural language processing models will adopt the Transformer architecture, which is a deep learning model architecture based on self-attention mechanism, so that the model can be pre-trained on a large amount of data to capture language features, and then the natural language processing model can be used to process various types of NLP tasks, such as text classification, named entity recognition, question and answer system, etc.

[0045] However, the traditional natural language processing model adopts a fixed calculation path for processing, even if it is processing data of different dimensions, the same calculation path is used for processing, and then there will be a case that the same calculation path and the same calculation resources are spent even for processing relatively simple data, which exists the problems of waste of calculation resources and low processing efficiency.

[0046] Based on this, the application provides a processing method of a classification task. The method includes the following steps: obtaining input data of the classification task and a thinking token corresponding to a natural language processing model, the natural language processing model including a classifier layer and at least one transformer layer, and each transformer layer being connected with a probability prediction layer; adding the thinking token to the input data to obtain a target input sequence; inputting the target input sequence into the natural language processing model; processing the target input sequence by the transformer layer to obtain a first output result output by the transformer layer, the first output result including a token output result of the thinking token; inputting the token output result into the probability prediction layer; using the thinking token to represent a global representation of the target input sequence; determining, by the probability prediction layer, whether to perform an early exit operation based on the token output result; and inputting the first output result into the classifier layer to obtain a classification result when it is determined to perform the early exit operation based on an output result of the probability prediction layer, without performing subsequent operations, thereby improving processing efficiency and reducing waste of computing resources.

[0047] Referring to Figure 1 , Figure 1 A step flowchart of a processing method of a classification task provided by an embodiment of the application is shown, which can specifically include the following steps:

[0048] In step 101, input data of the classification task and a thinking token corresponding to a natural language processing model are obtained, and the thinking token is added to the input data to obtain a target input sequence.

[0049] The classification task can be a task of classifying text data (such as a sentence, a paragraph, a document, or the like) to identify a category to which the text data belongs. The text data can be composed of at least one natural language, such as Chinese, English, or the like. For example, the classification task can be a sentiment classification task, in which the text data is classified into categories such as positive sentiment, negative sentiment, or the like. Alternatively, the classification task can be a news classification task, in which the text data is classified into different news categories, such as sports, entertainment, technology, or the like. The input data can be text data on which the classification task is to be performed. The natural language processing model can be a neural network model based on a self-attention mechanism, which can be a mechanism for calculating the correlation between each element in the input sequence and all other elements. The natural language processing model can include an embedding layer, a classifier layer, a probability prediction layer, and at least one transformer layer. The embedding layer can be used to preprocess the input sequence, including punctuation normalization, word lowercasing, or the like. The transformer layer can process the data based on the self-attention mechanism to obtain an output result of the input sequence. Specifically, the transformer layer can be used to capture the correlation between each element in the input sequence. The transformer layer can include a self-attention layer and a feedforward network layer. The self-attention layer can process the data using the self-attention mechanism to obtain an output result of the self-attention layer. The feedforward network layer can be used to perform nonlinear processing on the output result of the self-attention layer using an activation function to obtain an output result of the transformer layer. The probability prediction layer can be a linear transformation layer for calculating the probability of performing an early exit operation based on the output result of the transformer layer. The classifier layer can be used to convert the output result of the transformer layer into a probability distribution of the category label, so as to generate a result of the category prediction of the text data. The early exit operation can be an operation for exiting the calculation of the current transformer layer. The thinking token can be used to capture and integrate the semantic information of all elements in the input sequence, i.e., the thinking token can represent the global representation of the input sequence, and the natural language processing model can calculate the probability of performing the early exit operation based on the global representation of the thinking token. Specifically, the probability of performing the early exit operation can be calculated by the probability prediction layer.

[0050] When the classification task needs to be performed, a pre-trained natural language processing model can be loaded, and a thinking token in the natural language processing model and text data on which the classification task is to be performed, i.e., input data of the classification task, can be obtained. Then, the thinking token can be added to the input data to obtain a target input sequence.

[0051] Specifically, the target input sequence can include text tokens representing the text and a thinking token. As an example, the target input sequence can also include a classification token, which can be a user-defined token. The classification token can be used to capture and integrate semantic information of all elements in the input sequence, i.e., the classification token can represent a global representation of the input sequence. During the training of the natural language processing model, the thinking token can be used to optimize the probability prediction layer, and the classification token can be used to optimize the classifier layer.

[0052] In practical applications, in order to facilitate the thinking token to capture and integrate the global information of the target input sequence, the thinking token can be added to the front end of the input data to obtain the target input sequence, i.e., the thinking token is the first element in the target input sequence.

[0053] It should be understood that since the natural language processing model is a neural network model based on a self-attention mechanism, the self-attention mechanism can be a mechanism for calculating the association between each element in the input sequence and all other elements. Since the thinking token is the first element in the target input sequence, the thinking token can be calculated first, so that the thinking token can capture the semantic information of all elements in the input sequence and calculate the association between each element in the input sequence and all other elements, thereby improving the accuracy of generating a global representation.

[0054] At step 102, the target input sequence is input into the natural language processing model, and the target input sequence is processed by the converter layer to obtain a first output result output by the converter layer.

[0055] The first output result can be a word vector representing the association between each token in the text data and other tokens. The first output result can include a text output result and a token output result of the thinking token. The text output result can be a word vector representing the association between each text token in the target input sequence and other tokens. The text token can be a token in the target input sequence except the thinking token. The other token can be a token in the target input sequence except the currently processed text token. The token output result can be a global representation of the target input sequence.

[0056] After obtaining the target input sequence, the target input sequence can be input into the natural language processing model, and then the target input sequence can be processed by the converter layer in the natural language processing model to obtain a first output result output by the converter layer.

[0057] In actual application, each word in the target input sequence can be processed through the query matrix, the key matrix and the value matrix in the converter layer to obtain the query vector, the key vector and the value vector of each word. Then, the attention score between each word and other words can be calculated based on the query vector and the key vector of each word, and the attention score between each word and other words can be processed using a normalization exponential function to obtain the output result of each word in the converter layer. The first output result of the target input sequence in the converter layer can be generated according to the output result of each word.

[0058] As an example, the natural language processing model can further include a pre-trained vocabulary. Specifically, the vocabulary can be a list containing a large number of language elements such as words and phrases. The language elements included in the vocabulary cover professional vocabulary in each business field, including not only basic nouns, verbs, adjectives and other vocabulary, but also phrases, idioms, place names, names, organization names and other vocabulary. Each language element in the vocabulary has a corresponding word vector, and the word vector corresponding to each language element can be determined by pre-training.

[0059] In actual application, the target input sequence can be preprocessed by the embedding layer, and at least one word can be segmented from the preprocessed input sequence. Then, the language element corresponding to each word in the target input sequence and the corresponding word vector can be queried from the vocabulary, i.e., the word vector corresponding to each word in the target input sequence and the position information of each word in the target input sequence are determined from the vocabulary, and the corresponding position vector is generated. Then, the embedding vector of each word, including the embedding vector of the thinking token, can be generated based on the word vector and the position vector corresponding to each word. The embedding vector of each word can be taken as the output of the embedding layer and input to the converter layer for processing the embedding vector in the converter layer in the above manner.

[0060] In an embodiment of the present application, before step 102, the following steps can also be included:

[0061] An initial model is constructed, a sample input sequence is obtained, and the initial model is trained according to the sample input sequence to obtain a trained natural language processing model. The step of processing the target input sequence through the converter layer to obtain the first output result of the converter layer is performed based on the trained natural language processing model.

[0062] The initial model can be an untrained natural language processing model, and the sample input sequence can be text data for training the initial model. The sample input sequence can include a thinking token, which can be a word unit customized by a user or a word unit related to a business field.

[0063] In practical applications, a natural language processing model based on a self-attention mechanism can be constructed in advance, and the natural language processing model can include an embedding layer, at least one transformer layer, a probability prediction layer, a classifier layer, and a feedforward network layer, i.e., an initial model.

[0064] After the initial model is constructed, text data in different business domains can be obtained, sample input sequences for each business domain can be generated, and then each sample input sequence can be input into the initial model. After processing by the embedding layer, the transformer layer, the probability prediction layer, and the classifier layer in the initial model, an output result of each sample input sequence is obtained, and based on the output result of each sample input sequence, the model parameters of the initial model are updated.

[0065] Specifically, the model parameters of the initial model can include the weights of the embedding layer, the transformer layer, the probability prediction layer, and the classifier layer in the forward propagation process. The model parameters of the transformer layer can include the weights of the self-attention layer and the feedforward network layer in the forward propagation process.

[0066] For example, in the process of constructing the initial model, a user-defined token can also be obtained as a thinking token, and the thinking token can be added to the front end of the sample input sequence. The initial model is trained using the sample input sequence with the added thinking token, so that the trained thinking token can accurately capture the semantic information of each word in the input sequence.

[0067] Specifically, the token can be a word, a punctuation mark, a number, a special symbol, or any meaningful basic unit that appears in text.

[0068] In an embodiment of the present application, the initial model can also be trained in the following manner:

[0069] The classification label corresponding to the sample input sequence is determined, the sample input sequence is input into the initial model to obtain a sample output result corresponding to the sample input sequence, the model loss value of the initial model is determined according to the classification label and the sample output result, the model parameters of the initial model are optimized according to the model loss value, and a trained natural language processing model is obtained.

[0070] The classification label can represent the category information of the sample input sequence, the model parameters can include the weights of the embedding layer, the transformer layer, the probability prediction layer, and the classifier layer in the forward propagation process, and the weights of the self-attention layer and the feedforward network layer in the transformer layer in the forward propagation process, and the model loss value can be the loss value between the sample output result and the classification label.

[0071] In the process of training the initial model, a classification label corresponding to each sample input sequence can be determined, such as a science category in a news classification task, an entertainment category, or a positive sentiment category and a negative sentiment category in a sentiment classification task, and each sample input sequence is sequentially input into the initial model to obtain a sample output result corresponding to each sample input sequence, and then a loss between the classification label corresponding to each sample input sequence and the sample output result can be determined, that is, a model loss value of the initial model, and the model loss value can be used to optimize the model parameters of the initial model to obtain the trained natural language processing model.

[0072] Specifically, the model loss L of the initial model can be calculated by the following formula:

[0073] L(x,y)=E i ~ p(i|x) [logp(y|x,i)-βKL(p(·|x)∥p(·|λ))]

[0074] where x can be a sample input sequence, y can be a classification label corresponding to the sample input sequence, i can be a hidden variable, E i～p(i|x) may be an expectation of the hidden variable i, the hidden variable i is sampled from the conditional distribution p(i|x), the conditional distribution p(i|x) can represent a probability distribution of the hidden variable i under the condition of a given input, logp(λ|x,i) can represent a log-likelihood of the target y under the condition of a given input x and the hidden variable i, can represent a KL divergence, KL(p(·|x)∥p(·|λ)) is used to measure the difference between two probability distributions, that is, the difference between p(·|x) and p(·|λ), p(·|x) can be a probability distribution of a certain variable under the condition of a given x, p(·|λ) can be a prior distribution, and β can be a hyperparameter used to balance the weight between the log-likelihood term and the KL divergence term.

[0075] In step 103, when the token output result is input into the probability prediction layer and the early exit operation is determined to be performed based on the output result of the probability prediction layer, the first output result is input into the classifier layer to obtain a classification result.

[0076] The early exit operation can be an operation of exiting subsequent operations from the current converter layer, and the classification result can be a prediction result of the natural language processing model for classifying the target input sequence.

[0077] After obtaining the first output result, the token output result in the first output result can be input to the probability prediction layer to obtain an output result of the probability prediction layer, and it is determined whether to perform the early exit operation according to the output result of the probability prediction layer. Further, in a case where it is determined to perform the early exit operation, the first output result can be input to the classifier layer. The classifier layer can calculate a probability distribution of each class in the classification task based on the first output result, so as to predict the class of the target input sequence based on the probability distribution of the target input sequence, that is, the classification result of the target input sequence.

[0078] In the embodiment of the present application, the input data of the classification task and the thinking token corresponding to the natural language processing model are obtained. The natural language processing model includes a classifier layer and at least one transformer layer, and each transformer layer is connected with a probability prediction layer. The thinking token is added to the input data to obtain a target input sequence, and the target input sequence is input to the natural language processing model. The target input sequence is processed by the transformer layer to obtain a first output result output by the transformer layer. The first output result includes a token output result of the thinking token. The token output result is input to the probability prediction layer. The thinking token can be used to represent the global representation of the target input sequence. The probability prediction layer can determine whether to perform the early exit operation based on the token output result. In a case where it is determined to perform the early exit operation based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain the classification result, without performing subsequent operations, thereby improving the processing efficiency and reducing the waste of computing resources.

[0079] Referring to Figure 2 , Figure 2 A step flowchart of another classification task processing method provided by an embodiment of the present application is shown, which can specifically include the following steps:

[0080] In step 201, input data of a classification task and a thinking token corresponding to a natural language processing model are obtained, and the thinking token is added to the input data to obtain a target input sequence.

[0081] In step 202, the target input sequence is input to the natural language processing model, and the target input sequence is processed by a transformer layer to obtain a first output result output by the transformer layer.

[0082] The related descriptions of steps 201 to 202 can refer to steps 101 to 102, which are not repeated here.

[0083] In step 203, for each transformer layer, the token output result is input to the probability prediction layer to obtain probability information of performing the early exit operation from the transformer layer.

[0084] After obtaining the first output result, the token output result in the first output result can be input to the probability prediction layer to obtain the probability information of performing the early exit operation from the transformer layer.

[0085] It needs to be understood that, since the natural language processing model is trained, and in the training process, the output result output by each transformer layer and the final output result processed by all transformer layers can be determined, and then the loss between the output result of each transformer layer and the final output result can be determined, and the token output result of the thinking token in the output result of each transformer layer is determined and input to the probability prediction layer for calculation to obtain the probability information of each autonomous attention layer, so that the weight of the probability prediction layer can be optimized based on the loss between the output result of each transformer layer and the final output result and the probability information of each transformer layer, so that the optimized probability prediction layer can accurately calculate the probability information of performing the extraction exit operation from the transformer layer, that is, the optimized probability prediction layer can determine whether the first output result output by the current transformer layer is equivalent to the final output result.

[0086] In actual application, since the natural language processing model can include at least one transformer layer, the target input sequence can be input to the first transformer layer for calculation to obtain the first output result of the first transformer layer, and the first output result of the first transformer layer is taken as the input of the second transformer layer, and so on, and the target input sequence can be sequentially calculated through each transformer layer.

[0087] And for each transformer layer, after receiving the input of the previous transformer layer or receiving the target input sequence, the first output result of the transformer layer can be calculated, and the token output result in the first output result is input to the probability prediction layer to obtain the probability information of performing the early exit operation from the transformer layer, and then whether to perform the early exit operation can be determined according to the probability information of the transformer layer, so that the early exit operation can be performed from any transformer layer based on the probability information of each transformer layer.

[0088] In an embodiment of the present application, step 203 can include steps 2031 to 2032:

[0089] Step 2031, for each transformer layer, input the token output result of the transformer layer to the probability prediction layer to obtain the first exit probability of the transformer layer.

[0090] The first exit probability can be determined based on the probability information of the token output result of the current transformer layer, that is, the probability information obtained by inputting the token output result of the current transformer layer to the probability prediction layer.

[0091] In practical applications, for each layer of the converter, the token output result of the layer of the converter can be input to the probability prediction layer to obtain the first exit probability of the converter layer.

[0092] In step 2032, the second exit probability of the previous converter layer is obtained, and the cumulative probability information of performing the early exit operation from the converter layer is generated according to the first exit probability and the second exit probability.

[0093] The previous converter layer can be a converter layer before processing the current converter layer, the second exit probability can be probability information determined based on the token output result of each previous converter layer, and the cumulative probability information can be a cumulative value between the probability information of performing the early exit operation from the current converter layer and the probability information of performing the early exit operation from the previous converter layer.

[0094] Specifically, since each previous converter layer has input the token output result to the probability prediction layer to obtain the probability information of each previous converter layer before processing the current converter layer, the probability information of each previous converter layer can be directly obtained, that is, the second exit probability.

[0095] In practical applications, for each layer of the converter, the previous converter layer before processing the current converter layer can be determined, and the second exit probability of each previous converter layer can be obtained, and then the probability information of performing the early exit operation from the converter layer can be generated according to the first exit probability and the second exit probability.

[0096] Specifically, the probability information p(i|x) can be determined by the following formula:

[0097]

[0098] h i =S(h i-1 )

[0099] Wherein, Λ i may represent the exit probability of performing the early exit operation from the i-th layer of the converter, that is, the first exit probability, Λ j may represent the exit probability of performing the early exit operation from the j-th layer of the converter, that is, the second exit probability, may represent the token output result of the i-th layer of the converter, h i may represent the first output result of the i-th layer of the converter, h i-1 may represent the first output result of the i-1-th layer of the converter, that is, the input of the i-th layer of the converter, and S can be a calculation function of the converter layer.

[0100] After obtaining the probability information of the current transformer layer, the probability information of performing the early exit operation from the previous transformer layer can be obtained, and the cumulative probability information of performing the early exit operation from the current transformer layer can be calculated according to the probability information of the current transformer layer and the probability information of the previous transformer layer.

[0101] For example, the current transformer layer can be the third transformer layer, and the token output result of the third transformer layer can be input into the probability prediction layer to obtain the first exit probability of the third transformer layer. The previous transformer layers before the third transformer layer can be determined as the first transformer layer and the second transformer layer. Since the target input sequence has been processed by the second transformer layer and the first transformer layer before processing the third transformer layer, the exit probability of the first transformer layer and the exit probability of the second transformer layer can be obtained, and the second exit probability of the first transformer layer and the second transformer layer can be directly obtained during processing the third transformer layer. Therefore, the probability information of performing the early exit operation from the third transformer layer can be calculated based on the first exit probability and the second exit probability.

[0102] Similarly, since the target input sequence has been processed by the second transformer layer and the first transformer layer before processing the third transformer layer, the probability information of performing the early exit operation from the second transformer layer and the probability information of performing the early exit operation from the first transformer layer can be directly obtained, and the probability information of the first to third transformer layers can be accumulated to obtain the cumulative probability information of performing the early exit operation from the third transformer layer.

[0103] In step 204, when the probability information is greater than or equal to the preset hyperparameter threshold value and the transformer layer is not the last layer, the early exit operation is performed.

[0104] The hyperparameter threshold value can be determined based on the model performance of the natural language processing model. The hyperparameter threshold value can be used to determine whether the first output result output by the current transformer layer is equivalent to the final output result. The final output result can be the output result obtained after all transformer layers process the target input sequence.

[0105] After obtaining the probability information, the probability information can be compared with the hyperparameter threshold. If the probability information is greater than or equal to the hyperparameter threshold, it can be determined that the difference between the first output result output by the current transformer layer and the final output result is small, and it can be considered that the first output result output by the current transformer layer is equivalent to the final output result. Therefore, it can be determined to perform the extraction exit operation, that is, to perform the operation of exiting the calculation of the current transformer layer, and to take the first output result of the current transformer layer as the output result of the target input sequence, without performing the calculation of the subsequent transformer layer, thereby improving the processing efficiency.

[0106] In an embodiment of the present application, the method can further include the following steps:

[0107] For each transformer layer, if the probability information of the transformer layer is less than the hyperparameter threshold and the transformer layer is not the last layer, the first output result is input to the next transformer layer, and the steps of processing the target input sequence by the transformer layer to obtain the first output result of the transformer layer are repeated in the next transformer layer, or if the transformer layer is the last layer, the classification identification data in the first output result is input to the classifier layer to obtain the second output result.

[0108] The classification identification data can be the output result obtained by processing the classification token by the transformer layer, and the classification identification data can be the global representation of the target input sequence.

[0109] In actual application, for each transformer layer, it can be determined whether the transformer layer is the last layer in all transformer layers. If the transformer layer is not the last layer, the probability information of the transformer layer can be compared with the hyperparameter threshold. If the probability information of the transformer layer is less than the hyperparameter threshold, it can be determined that the difference between the first output result output by the current transformer layer and the final output result is large, and it cannot be considered that the first output result output by the current transformer layer is equivalent to the final output result. Therefore, it is necessary to continue to forward propagate, that is, to input the first output result to the next transformer layer, and to repeat the above steps of processing the target input sequence by the transformer layer to obtain the first output result of the transformer layer in the next transformer layer, and performing the step of determining whether to perform the early exit operation according to the token output result in the first output result.

[0110] If the transformer layer is the last layer, it can be determined that the first output result of the last transformer layer obtained by the calculation of the target input sequence by all transformer layers is the final output result. Therefore, the classification identification data in the first output result can be input to the classifier layer to obtain the second output result.

[0111] It needs to be understood that in the process of training the initial model, the user-defined classification token can also be added to the sample input sequence, and the parameters of the probability prediction layer in the initial model are optimized by thinking token, and the classifier layer is optimized by classification token, that is, the thinking token is used to optimize the probability prediction layer, so that the optimized probability prediction layer can accurately calculate the probability information of executing the early exit operation from each transformer layer, and the classification token is used to optimize the classifier layer, so that the optimized classifier layer can accurately calculate the probability distribution of the target input sequence for each class in the classification task.

[0112] Referring to Figure 3 , Figure 3 An architecture diagram of a natural language processing model provided by an embodiment of the present application is shown. As shown in the figure, the natural language processing model 3 can include an embedding layer 31, at least one transformer layer 32, a probability prediction layer 33, and a classifier layer 34. Figure 3

[0113] The embedding layer 31 can receive input data and perform embedding processing to obtain a target input sequence. The embedding layer 31 can input the target input sequence to the first transformer layer 32, and the first transformer layer 32 processes the target input sequence to obtain a first output result of the first transformer layer 32 after receiving the target input sequence. The first transformer layer 32 can input the first output result to the probability prediction layer 33, and the probability prediction layer 33 can calculate the probability information of executing the early exit operation from the first transformer layer 32 according to the token output result in the first output result output by the first transformer layer 32.

[0114] For example, after inputting the target input sequence to the transformer layer, the target input sequence can be processed by the self-attention layer in the transformer layer to obtain an output result of the self-attention layer, and the output result of the self-attention layer can also be input to the feedforward network layer to obtain an output result of the feedforward network layer, and then the output result of the feedforward network layer can be taken as the output result of the transformer layer and input to the probability prediction layer to obtain the probability information of executing the early exit operation from the transformer layer.

[0115] In the case where the probability information is greater than or equal to the hyperparameter threshold, it is determined to execute the early exit operation, and the first output result output by the first transformer layer 32 is input to the classifier layer 34 to obtain a classification result output by the classifier layer 34 based on the first output result output by the first transformer layer 32.

[0116] ​In a case where the probability information is less than the hyperparameter threshold, it is determined that the early exit operation is not performed, and the first output result output by the first transformer layer 32 is input to the second transformer layer 32 to obtain the first output result of the second transformer layer 32. Then, the step of determining the probability information of performing the early exit operation from the current transformer layer based on the first output result of the second transformer layer 32 can be repeatedly performed, and the second transformer layer to the last transformer layer can be processed in the above manner, that is, the first output result of each transformer layer and the probability information of performing the early exit operation of each transformer layer can be obtained.

[0117] In a case where all the transformer layers 32 do not perform the early exit operation, the classification identification data in the first output result of the last transformer layer 32 can be input to the classifier layer 34, and then the classifier layer 34 can obtain the classification result based on the classification identification data.

[0118] In a case where the token output result is input to the probability prediction layer and it is determined that the early exit operation is performed based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain the classification result.

[0119] For details of step 205, refer to step 103, which will not be repeated here.

[0120] In the embodiments of the present application, the input data of the classification task and the thinking token corresponding to the natural language processing model are obtained, and the thinking token is added to the input data to obtain a target input sequence. The target input sequence is input to the natural language processing model, and the target input sequence is processed by the transformer layer to obtain the first output result output by the transformer layer. For each transformer layer, the token output result in the first output result is input to the probability prediction layer. Since the thinking token can be used to represent the global representation of the target input sequence, the probability prediction layer can obtain the probability information of performing the early exit operation from the transformer layer based on the token output result. In a case where the probability information is greater than or equal to a preset hyperparameter threshold and the transformer layer is not the last layer, the early exit operation is performed. In a case where the token output result is input to the probability prediction layer and it is determined that the early exit operation is performed based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain the classification result, without the need to perform subsequent operations, thereby improving the processing efficiency and reducing the waste of computing resources.

[0121] Referring to FIG. 4, Figure 4 FIG. 4 shows a flowchart of a method for processing a classification task according to an embodiment of the present application, which can include the following steps:

[0122] Step 401: The flow starts.

[0123] Step 402, obtaining input data, and pre-processing the input data through an embedding layer to obtain a target input sequence;

[0124] Step 403, in the case that the current converter layer is a first layer converter layer, inputting the target input sequence to the converter layer to obtain a first output result of the converter layer, and in the case that the current converter layer is a non-first layer converter layer, inputting the first output result of the last layer converter layer to the current converter layer to obtain a first output result of the current converter layer;

[0125] Step 404, inputting the token output result in the first output result to a probability prediction layer to obtain probability information of performing an early exit operation from the converter layer;

[0126] Step 405, determining whether to perform the early exit operation according to the probability information, if yes, performing step 406, and if no, performing step 407;

[0127] Step 406, performing the early exit operation, and inputting the first output result of the converter layer to a classifier layer to obtain a classification result;

[0128] Step 407, determining whether the current converter layer is a last layer, in the case that the current converter layer is a non-last layer, repeatedly performing step 403, and in the case that the current converter layer is the last layer, inputting classification identifier data in the first output result of the current converter layer to the classifier layer to obtain a classification result;

[0129] Step 408, ending the flow.

[0130] In the embodiment of the present application, by obtaining input data of a classification task and a thinking token corresponding to a natural language processing model, the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is connected with a probability prediction layer, the thinking token is added to the input data to obtain a target input sequence, and the target input sequence is input to the natural language processing model, the target input sequence is processed through the converter layer to obtain a first output result output by the converter layer, the first output result includes token output result of the thinking token, the token output result is input to the probability prediction layer, the thinking token can be used to represent global representation of the target input sequence, the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and in the case that it is determined to perform the early exit operation based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain a classification result, without performing subsequent operations, the processing efficiency is improved, and the waste of computing resources is reduced.

[0131] Referring to Figure 5 , Figure 5A structural diagram of a processing apparatus for a classification task is shown, and the apparatus can specifically include the following modules.

[0132] The acquisition module 501 is configured to acquire input data of the classification task and a thinking token corresponding to a natural language processing model, and add the thinking token to the input data to obtain a target input sequence; the natural language processing model includes a classifier layer and at least one converter layer, and each converter layer is connected with a probability prediction layer, and the thinking token is used to represent a global representation of the target input sequence.

[0133] The processing module 502 is configured to input the target input sequence to the natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer; the first output result includes a token output result of the thinking token.

[0134] The classification module 503 is configured to, in a case where the token output result is input to the probability prediction layer and it is determined to perform an early exit operation based on an output result of the probability prediction layer, input the first output result to the classifier layer to obtain a classification result.

[0135] In an implementation manner, the apparatus can further include the following modules.

[0136] The construction module is configured to, before processing the target input sequence through the converter layer to obtain the first output result of the converter layer, construct an initial model; the initial model is an untrained natural language processing model.

[0137] The training module is configured to, before processing the target input sequence through the converter layer to obtain the first output result of the converter layer, acquire a sample input sequence, and train the initial model according to the sample input sequence to obtain a trained natural language processing model, so as to perform the step of processing the target input sequence through the converter layer to obtain the first output result of the converter layer based on the trained natural language processing model.

[0138] The sample input sequence includes the thinking token.

[0139] In an implementation manner, the training module can be further configured to:

[0140] determine a classification label corresponding to the sample input sequence;

[0141] input the sample input sequence to the initial model to obtain a sample output result corresponding to the sample input sequence;

[0142] determine a model loss value of the initial model according to the classification label and the sample output result;

[0143] The model parameters of the initial model are optimized according to the model loss value, and a trained natural language processing model is obtained.

[0144] In an implementation manner, the apparatus can further include the following modules.

[0145] The exiting module is configured to, in a case where it is determined according to the token output result that the early exiting operation is performed, input the first output result to the classifier layer to obtain the classification result, before the input, for each transformer layer, input the token output result of the transformer layer to the probability prediction layer to obtain probability information of performing the early exiting operation from the transformer layer, and in a case where the probability information is greater than or equal to a preset hyperparameter threshold and the transformer layer is not the last layer, perform the early exiting operation.

[0146] In an implementation manner, the exiting module can be further configured to:

[0147] for each transformer layer, input the token output result of the transformer layer to the probability prediction layer to obtain a first exiting probability of the transformer layer;

[0148] obtain a second exiting probability of a previous transformer layer, and generate cumulative probability information of performing the early exiting operation from the transformer layer according to the first exiting probability and the second exiting probability, wherein the previous transformer layer is a transformer layer before the transformer layer.

[0149] In an implementation manner, the apparatus can further include the following modules.

[0150] The processing module is configured to, for each transformer layer, in a case where the probability information of the transformer layer is less than the hyperparameter threshold and the transformer layer is not the last layer, input the first output result to a next transformer layer, and repeat the steps of processing the target input sequence through the transformer layer to obtain the first output result of the transformer layer in the next transformer layer.

[0151] Or, in a case where the transformer layer is the last layer, input classification identifier data in the first output result to the classifier layer to obtain a second output result.

[0152] In an implementation manner, the hyperparameter threshold is determined based on model performance of the natural language processing model.

[0153] In the embodiment of the present application, the input data of the classification task and the thinking token corresponding to the natural language processing model are obtained, the natural language processing model includes a classifier layer and at least one transformer layer, each transformer layer is connected with a probability prediction layer, the thinking token is added to the input data to obtain a target input sequence, the target input sequence is input to the natural language processing model, the transformer layer processes the target input sequence to obtain a first output result output by the transformer layer, the first output result includes a token output result of the thinking token, the token output result is input to the probability prediction layer, the thinking token can be used to represent the global representation of the target input sequence, the probability prediction layer can determine whether to perform an early exit operation based on the token output result, and in the case that it is determined to perform the early exit operation based on the output result of the probability prediction layer, the first output result is input to the classifier layer to obtain a classification result, without performing subsequent operations, the processing efficiency is improved, and the waste of computing resources is reduced.

[0154] It should be noted that the information interaction and execution process between the above devices are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought about can be referred to the method embodiments part, which will not be repeated here.

[0155] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0156] Reference is made to Figure 6 , Figure 6 A structure block diagram of a terminal device provided by an embodiment of the present application is shown, as Figure 6 shown, the embodiment provides a terminal device 61, which includes at least one processor 611, a memory 612, and a computer program 6121 stored in the memory 612 and executable on the at least one processor 611, and the processor 611 implements the steps in any of the above method embodiments when executing the computer program 6121.

[0157] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in any one of the method embodiments.

[0158] The embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal is enabled to implement the steps in each of the method embodiments.

[0159] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the embodiment of the present application can implement all or part of the processes in the above method embodiments by means of a computer program instructing relevant hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each of the method embodiments can be implemented. The computer program includes computer program code. The computer program code can be in the form of source code, object code, executable files or some intermediate forms of the code. The computer readable medium at least includes any entity or device capable of carrying the computer program code to a photographing device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium.

[0160] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of processing a classification task, characterized in that, The method comprises: obtaining input data of a classification task and a thinking token corresponding to a natural language processing model, and adding the thinking token to the input data to obtain a target input sequence; wherein the natural language processing model comprises a classifier layer and at least one transformer layer, and each transformer layer is connected with a probability prediction layer, and the thinking token is used to represent the global representation of the target input sequence; inputting the target input sequence into the natural language processing model, and processing the target input sequence through the transformer layer to obtain a first output result output by the transformer layer; wherein the first output result comprises a token output result of the thinking token; in the case of inputting the token output result into the probability prediction layer and determining to perform an early exit operation based on the output result of the probability prediction layer, inputting the first output result into the classifier layer to obtain a classification result.

2. The processing method of a classification task according to claim 1, wherein, Before the step of processing the target input sequence through the transformer layer to obtain the first output result of the transformer layer, the method further comprises: building an initial model; wherein the initial model is an untrained natural language processing model; obtaining a sample input sequence, and training the initial model according to the sample input sequence to obtain a trained natural language processing model, so as to perform the step of processing the target input sequence through the transformer layer to obtain the first output result of the transformer layer based on the trained natural language processing model; wherein the sample input sequence comprises a thinking token.

3. The processing method of a classification task according to claim 2, wherein, The step of training the initial model according to the sample input sequence to obtain a trained natural language processing model comprises: determining a classification label corresponding to the sample input sequence; inputting the sample input sequence into the initial model to obtain a sample output result corresponding to the sample input sequence; determining a model loss value of the initial model according to the classification label and the sample output result; optimizing the model parameters of the initial model according to the model loss value to obtain a trained natural language processing model.

4. The processing method of a classification task according to any one of claims 1 to 3, wherein, Before the step of inputting the first output result into the classifier layer to obtain a classification result in the case of determining to perform an early exit operation according to the token output result, the method further comprises: for each transformer layer, inputting the token output result of the transformer layer into the probability prediction layer to obtain probability information of performing an early exit operation from the transformer layer; in the case that the probability information is greater than or equal to a preset hyperparameter threshold value and the transformer layer is not the last layer, performing the early exit operation.

5. The processing method of a classification task according to claim 4, wherein, The step of inputting the token output result into the probability prediction layer to obtain probability information of performing an early exit operation from the transformer layer comprises: for each transformer layer, inputting the token output result of the transformer layer into the probability prediction layer to obtain a first exit probability of the transformer layer; obtain a second exit probability of a previous converter layer, and generate cumulative probability information of performing an early exit operation from the converter layer according to the first exit probability and the second exit probability; wherein the previous converter layer is a converter layer before the converter layer.

6. The processing method of a classification task according to claim 4, wherein, The method further comprises: for each of the converter layers, in a case that the probability information of the converter layer is less than the hyperparameter threshold and the converter layer is not the last layer, inputting the first output result into a next converter layer, and repeating the processing of the target input sequence through the converter layer to obtain the first output result of the converter layer in the next converter layer; or, in a case that the converter layer is the last layer, inputting classification identification data in the first output result into the classifier layer to obtain a second output result.

7. The processing method of a classification task according to claim 4, wherein, The hyperparameter threshold is determined based on model performance of the natural language processing model.

8. A processing device for a classification task, characterized by The apparatus comprises: an obtaining module configured to obtain input data of a classification task and a thinking token corresponding to a natural language processing model, and add the thinking token to the input data to obtain a target input sequence; wherein the natural language processing model comprises a classifier layer and at least one converter layer, and each of the converter layers is connected with a probability prediction layer, and the thinking token is used to represent global representation of the target input sequence; a processing module configured to input the target input sequence into the natural language processing model, and process the target input sequence through the converter layer to obtain a first output result output by the converter layer; wherein the first output result comprises a token output result of the thinking token; a classification module configured to, in a case that the token output result is input into the probability prediction layer and it is determined to perform an early exit operation based on an output result of the probability prediction layer, input the first output result into the classifier layer to obtain a classification result.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program implements the method of any one of claims 1 to 7 when executed by the processor.

Images