Natural Language Processing Method, Device, Storage Medium, and Program Product

Abstract

The embodiment of the present application provides a natural language processing method, device, storage medium and program product, the method comprising obtaining a target text, inputting the target text into a natural language processing (NLP) model, obtaining a target processing result, the target processing result comprising each label and a corresponding probability value, inputting the target text into a confidence model, obtaining a confidence corresponding to the target processing result, the confidence being related to the distribution of the probability values of the labels, and processing the target processing result according to the confidence. The embodiment of the present application can determine the confidence of the target processing result obtained by the NLP model by obtaining a confidence model obtained through special training without relying on manual labor, and the range of the confidence can be a value between 0 and 1, thereby improving the accuracy of the confidence.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning technology, and in particular to a natural language processing method, device, storage medium, and program product. Background Art

[0002] With the increasing popularity of intelligent devices, research on human-computer voice interaction systems has expanded from scientific research to various terminal manufacturing fields such as smart TVs, in-car navigation, and smart speakers. In human-computer interaction systems, the ability to effectively understand user expressions is crucial to the success or failure of the entire interaction process. Therefore, natural language processing (NLP) is a crucial research area for intelligent interaction systems. The NLP field encompasses multiple tasks, such as classification and sequence labeling. For example, when an NLP model inputs a target text, it outputs various category labels and corresponding probability values between 0 and 1. The probability value for a category simply represents the degree of separation between that category and other categories, and does not represent the confidence score that the target text belongs to that category. This is not conducive to effectively utilizing the categorization results of the target text.

[0003] In the prior art, the confidence of the final category result of the target text can be evaluated by setting a threshold.

[0004] However, in the process of implementing this application, the inventors found that there are at least the following problems in the prior art: in the above-mentioned method of using thresholds, the confidence level only includes two values 0 and 1, which has low accuracy, and the setting of the threshold level relies entirely on manual labor, which further affects the accuracy of the confidence score. Even with the support of experts, high accuracy cannot be guaranteed. Summary of the Invention

[0005] Embodiments of the present application provide a natural language processing method, device, storage medium, and program product to improve the accuracy of confidence scores.

[0006] In a first aspect, an embodiment of the present application provides a natural language processing method, comprising:

[0007] Get the target text;

[0008] Input the target text into a natural language processing (NLP) model to obtain a target processing result; the target processing result includes each label and its corresponding probability value;

[0009] Inputting the target text into a confidence model to obtain a confidence level corresponding to the target processing result; the confidence level is related to the distribution of probability values of the labels;

[0010] The target processing result is processed according to the confidence level.

[0011] In one possible design, the target processing result is a classification result or a sequence labeling result.

[0012] In one possible design, processing the target processing result according to the confidence level includes:

[0013] If the confidence level is greater than a preset threshold, the final result is output according to the target processing result.

[0014] In one possible design, before inputting the target text into a natural language processing (NLP) model, the method further includes:

[0015] Training the first to-be-trained model using the first sample set to obtain the NLP model;

[0016] Inputting each second sample in the second sample set and the annotation results corresponding to each second sample into the NLP model to obtain a predicted value of the processing result;

[0017] The second model to be trained is trained according to the predicted value of the processing result and the second sample set to obtain the confidence model.

[0018] In one possible design, training the first to-be-trained model using the first sample set to obtain the NLP model includes:

[0019] Inputting each first sample in the first sample set and the labeling results corresponding to each first sample into the first to-be-trained model;

[0020] The parameters of the first model to be trained are updated by a gradient descent algorithm to obtain the NLP model.

[0021] In one possible design, training the second to-be-trained model according to the processing result prediction value and the second sample set to obtain the confidence model includes:

[0022] Inputting each second sample in the second sample set and the annotation results corresponding to each second sample into the second to-be-trained model to obtain a confidence prediction value corresponding to the processing result prediction value;

[0023] According to the loss function, the parameters of the second model to be trained are updated by a gradient descent algorithm to obtain the confidence model; the loss function is constructed based on the processing result prediction value and the confidence prediction value.

[0024] In one possible design, the loss function includes a probability constraint term and a probability distribution constraint term, wherein the probability constraint term is related to the difference between the probability prediction value of the true label and the corresponding confidence prediction value of each second sample in the processing result prediction value; the probability distribution constraint term is related to the difference between the labeling result of each label of each second sample in the processing result prediction value and the probability prediction value.

[0025] In one possible design, the NLP model includes an embedding layer, a representation layer, and a first output layer connected in sequence; the second model to be trained includes the embedding layer, the representation layer, and a second output layer connected to the representation layer of the NLP model.

[0026] In a possible design, the representation layer is a transformer model Transformer or a bidirectional long short-term memory model Bi-LSTM.

[0027] In a second aspect, an embodiment of the present application provides a natural language processing device, including:

[0028] Acquisition module, used to obtain target text;

[0029] A first input module is used to input the target text into a natural language processing (NLP) model to obtain a target processing result; the target processing result includes each label and its corresponding probability value;

[0030] A second input module is used to input the target text into a confidence model to obtain a confidence level corresponding to the target processing result; the confidence level is related to the distribution of probability values of the labels;

[0031] A processing module is used to process the target processing result according to the confidence level.

[0032] In a third aspect, an embodiment of the present application provides a natural language processing device, comprising: at least one processor and a memory;

[0033] The memory stores computer-executable instructions;

[0034] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect is implemented.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect and various possible designs of the first aspect.

[0037] The natural language processing method, device, storage medium and program product provided in this embodiment obtain a target text, input the target text into a natural language processing (NLP) model, and obtain a target processing result. The target processing result includes various labels and their corresponding probability values. The target text is input into a confidence model to obtain a confidence corresponding to the target processing result. The confidence is related to the distribution of the probability values of the various labels. The target processing result is processed according to the confidence. The confidence of the target processing result obtained by the NLP model can be determined without manual labor by obtaining a confidence model through special training. The confidence range can be a value between 0 and 1, thereby improving the accuracy of the confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A schematic diagram of the principles of the text classification model provided in the embodiments of the present application;

[0040] Figure 2 A schematic diagram of the principle of the sequence labeling model provided in the embodiment of the present application;

[0041] Figure 3 A flowchart of a natural language processing method provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the training process of the NLP model and confidence model provided in the embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of a natural language processing device provided in an embodiment of the present application;

[0044] Figure 6 A block diagram of a natural language processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] With the increasing popularity of intelligent devices, research on human-computer voice interaction systems has expanded from scientific research to various terminal manufacturing fields such as smart TVs, in-car navigation, and smart speakers. In human-computer interaction systems, the ability to fully understand user expressions is crucial to the success or failure of the entire interaction process. Therefore, natural language processing (NLP) is a crucial research area for intelligent interaction systems.

[0047] NLP mainly includes intent recognition and slot extraction. Intent recognition is to understand the user's intention domain, while slot extraction is to focus on the key parameter information under the execution of the intention. We can model the former as a text classification problem, and the latter as a sequence labeling problem. To deal with text classification problems, you can use Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM, Transformer model Transformer, Bidirectional Encoder Representations from Transformer, BERT) and other models to extract text features, and then pass through the sigmoid or softmax function to output the probability value of 0 to 1 corresponding to the decibel of each category as the classification processing result. To deal with sequence labeling problems, you can use LSTM+Conditional Random Fields (CRF), Bi-directional Long Short-Term Memory (BERT) and other models to extract text features. Models such as NLP (Non-Recorded Memory, Bi-LSTM) + CRF perform text representation and label state transfer. Finally, a maximum a posteriori probability function is used to map the scores of each label to the range of 0-1, which is used as the sequence labeling result. Regardless of the target text, these models output a probability value for any label. This probability value only represents the degree of separation between that category and other categories. For example, if a user inputs "I want to see Andy Lau's movie," the text classification model in the NLP model will obtain a classification score of 0.6 for the film and television category and 0.4 for the music category. These two scores only indicate that the text is more likely to belong to the film and television category and do not indicate the confidence score for the film and television category. When the NLP model processes samples that it has difficulty recognizing or misidentifies, the user is unaware of the model's errors or confusion, and may misuse the model's results, causing failure in the entire comprehension task. This is not conducive to effectively utilizing the categorization results of the target text.

[0048] In existing technology, a threshold can be set to assess the confidence of the final classification result of the target text. This threshold strategy is used to mitigate the problems caused by the above model. However, this strategy is too simplistic and crude, relying solely on prior knowledge, and cannot effectively handle unknown errors caused by unknown samples. Moreover, this approach cannot handle cases where the model has a clear classification bias but this bias is incorrect, resulting in low accuracy.

[0049] To solve the above problems, the inventors have discovered that the confidence level of the target processing result corresponding to the target text can be determined by specially training a confidence model. This confidence level is related to the distribution of the probability values of each label and can be a value between 0 and 1, not just 0 or 1. Compared with the confidence level obtained by using a threshold strategy, it has higher accuracy. Based on this, the embodiment of the present application provides a natural language processing method that improves the accuracy of the confidence level. This method is highly versatile and can be applied to a variety of natural language processing tasks, such as sequence labeling tasks and text classification tasks.

[0050] Figure 1 This is a schematic diagram of the principle of the text classification model provided in the embodiment of this application. Figure 1 As shown in the figure, the NLP model includes an input layer, an embedding layer, a representation layer, and an output layer. The input layer is used to input text, labels, and knowledge. Text refers to the target text to be classified, labels refer to classification category labels, and knowledge refers to external knowledge used to assist in identifying the target text. The embedding layer is used to embed characters, words, and labels, that is, to convert them into vectors. The representation layer can use the Transformer model to extract features from text, labels, and related knowledge through the powerful text representation capabilities of the Transformer. The output layer can include a fully connected network, such as a Softmax fully connected network or other fully connected networks to output the text classification category results.

[0051] Figure 2 This is a schematic diagram of the principle of the sequence annotation model provided in the embodiment of this application. Figure 2 As shown in the figure, the NLP model consists of an input layer, an embedding layer, a representation layer, and an output layer. The input layer receives text input, which refers to the target text to be sequence labeled. The embedding layer embeds characters, converting them into vectors. The representation layer can use a Bi-LSTM model. Leveraging the Bi-LSTM's ability to represent dependencies within the target text, it extracts features from each position in the target text. This is then processed using a CRF to characterize the label transition probability. Finally, the output layer outputs the label for each position. The output layer can include a CRF layer and a Softmax layer.

[0052] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0053] Figure 3 This is a flow chart of the natural language processing method provided in the embodiment of the present application. Figure 3 As shown, the method includes:

[0054] 301. Get the target text.

[0055] 302. Input the target text into a natural language processing (NLP) model to obtain a target processing result; the target processing result includes each label and its corresponding probability value.

[0056] 303. Input the target text into a confidence model to obtain a confidence level corresponding to the target processing result; the confidence level is related to the distribution of probability values of the labels.

[0057] 304. Process the target processing result according to the confidence level.

[0058] The execution subject of this embodiment can be a computer, tablet, mobile phone terminal, display device, smart speaker and other data processing equipment.

[0059] In this embodiment, the target text may be text content converted from the voice information input by the user.

[0060] In this embodiment, NLP involves many tasks, such as text classification and sequence labeling. Accordingly, the NLP model can output a variety of target processing results, such as classification results for text classification tasks and sequence labeling results for sequence labeling tasks. In this embodiment, there is no limitation on the specific tasks that the NLP model performs.

[0061] In this embodiment, the confidence output by the confidence model is related to the distribution of the probability values of each label in the target processing result. For example, for a two-category task processing, the target processing result output by the NLP model includes two categories and their corresponding probability values. For example, the processing result obtained by inputting the first target text into the NLP model includes a probability value of 0.6 for the first category (film and television field) and a probability value of 0.4 for the second category (music field). The processing result obtained by inputting the second target text into the NLP model includes a probability value of 0.9 for the first category (film and television field) and a probability value of 0.1 for the second category (music field). If the probability distribution corresponding to the first target text is compared with the probability distribution corresponding to the second target text, the processing result of the second target text has a higher credibility, that is, a higher confidence. If the probability values of the two categories are 0.5 and 0.5, then the credibility will be lower than 0.6 and 0.4, that is, the corresponding confidence will be lower. The confidence of the target processing result output by the confidence model provided in this embodiment can quantify the credibility of the target processing result based on the distribution of the probability values of each label, and accurately characterize the credibility of the target processing result output by the NLP model.

[0062] In some embodiments, processing the target processing result according to the confidence level may include: if the confidence level is greater than a preset threshold, outputting a final result according to the target processing result. Figure 1 As shown, the confidence obtained by the confidence model is 0.73. Assuming that the preset threshold is set to 0.5, the confidence of 0.73 corresponding to the current target text is greater than the preset threshold 0.5. Then, the final result can be output based on the target processing result corresponding to the target text. For example, each label in the target processing result and the probability value corresponding to each label can be directly output as the final result. Alternatively, only some labels and corresponding probability values can be output. The specific adjustment can be made according to actual needs, and this embodiment does not limit this.

[0063] In some embodiments, processing the target processing result according to the confidence level may further include: returning null if the confidence level is less than or equal to a preset threshold. Figure 2 As shown, the confidence score obtained by the confidence model is 0.73. Assuming that the preset threshold is set to 0.8, the confidence score of 0.73 corresponding to the current target text is less than the preset threshold 0.8, and the current target processing result is determined to be unreliable and not adopted, and the output is empty. Of course, other results can be returned according to actual needs, such as unrecognizable, unreliable recognition results, etc., which can be set as needed and are not limited in this embodiment.

[0064] It should be noted that, in this embodiment, the execution order of step 302 and step 303 is not particular, and they can also be executed in parallel, which is not limited in this embodiment.

[0065] The natural language processing method provided in this embodiment obtains a target text, inputs the target text into a natural language processing (NLP) model, and obtains a target processing result. The target processing result includes various labels and their corresponding probability values. The target text is input into a confidence model to obtain a confidence corresponding to the target processing result. The confidence is related to the distribution of the probability values of the various labels. The target processing result is processed according to the confidence. The confidence of the target processing result obtained by the NLP model can be determined without manual labor by obtaining a confidence model through special training. The confidence range can be a value between 0 and 1, thereby improving the accuracy of the confidence.

[0066] Figure 4 Schematic diagram of the training process of the NLP model and confidence model provided in the embodiment of this application. Figure 4 As shown, the method includes:

[0067] 401. Train a first model to be trained using a first sample set to obtain the NLP model.

[0068] 402. Input each second sample in the second sample set and the annotation results corresponding to each second sample into the NLP model to obtain a predicted value of the processing result.

[0069] 403. Train a second to-be-trained model according to the predicted value of the processing result and the second sample set to obtain the confidence model.

[0070] In this embodiment, the training of the first model to be trained using the first sample set to obtain the NLP model may include: inputting each first sample in the first sample set and the annotation results corresponding to each first sample into the first model to be trained; and updating the parameters of the first model to be trained using a gradient descent algorithm to obtain the NLP model.

[0071] In this embodiment, the training of the second model to be trained based on the processing result prediction value and the second sample set to obtain the confidence model may include: inputting each second sample in the second sample set and the annotation results corresponding to each second sample into the second model to be trained to obtain the confidence prediction value corresponding to the processing result prediction value; updating the parameters of the second model to be trained by a gradient descent algorithm according to the loss function to obtain the confidence model; the loss function is constructed based on the processing result prediction value and the confidence prediction value.

[0072] In some embodiments, for training convenience, the first sample set and the second sample set may be the same sample set.

[0073] Exemplarily, the loss function may include a probability constraint term and a probability distribution constraint term, wherein the probability constraint term is related to the difference between the probability prediction value of the true label and the corresponding confidence prediction value of each second sample in the processing result prediction value; the probability distribution constraint term is related to the difference between the labeling result of each label of each second sample in the processing result prediction value and the probability prediction value.

[0074] In this embodiment, in the process of training the second to-be-trained model using the second sample set to obtain the confidence model, the samples in the second sample set can be input into the second to-be-trained model in batches. The loss function of the second to-be-trained model used to train the confidence model during training can include a probability constraint term, and the probability constraint term can be in the form of a mean square error, specifically as shown in Expression (1):

[0075]

[0076] in, The trained NLP model infers the probability value of sample i in the current batch on the true label, c i is the output of the second network to be trained. n is the number of samples in the current batch.

[0077] Taking into account the loss caused by the distribution of the NLP model inference results, the loss function of the second model to be trained can also include a probability distribution constraint term, which can be expressed by the variance of the distribution variance. Taking the text classification task as an example, the probability distribution constraint term is as shown in expressions (2) and (3):

[0078]

[0079]

[0080] Among them, p ij is the probability value of the NLP model for sample i in category j in the current batch, The probability value of the true label of sample i in the current batch for the NLP model inference is in category j, and k is the total number of categories in the classification task. Therefore, the final loss function can be defined as expression (4):

[0081]

[0082] Among them, L true is the probability constraint term, L dis is the probability distribution constraint.

[0083] To improve model training efficiency, the parameters of a trained NLP model can be used as initialization parameters for a second model to be trained to obtain a confidence model. The NLP model includes an embedding layer, a representation layer, and a first output layer connected in sequence; the second model to be trained includes the embedding layer and representation layer of the NLP model, and a second output layer connected to the representation layer.

[0084] In some embodiments, the representation layer in an NLP model can use different models to complete different NLP tasks. For example, when processing text classification tasks, the representation layer can be a Transformer model, while when processing sequence labeling tasks, the representation layer can be a Bidirectional Long Short-Term Memory (Bi-LSTM) model.

[0085] The following uses text classification and sequence labeling as examples to describe the training and reasoning application processes of the NLP model and confidence model used for text classification, and the training and reasoning application processes of the NLP model and confidence model used for sequence labeling.

[0086] For example, the training of the NLP model and the confidence model corresponding to the text classification task may include the following steps:

[0087] Step 11: Input the training data (for example, the first sample in the first sample set) into the first to-be-trained model in batches, and use the gradient descent method, such as the stochastic gradient descent method, to update the parameters of the first to-be-trained model for the results of each batch of training data until the indicators no longer improve, thereby obtaining a trained NLP model.

[0088] Specifically, the first batch of first samples is input into the first model to be trained to obtain the first batch of corresponding processing results. Based on the processing results, the parameters of the first model to be trained are updated by the stochastic gradient descent method to obtain the first model to be trained after the updated parameters. The second batch of first samples is input into the first model to be trained after the updated parameters are obtained to obtain the second batch of corresponding processing results. Based on the processing results, the parameters of the first model to be trained are updated by the stochastic gradient descent method to obtain the first model to be trained after the updated parameters are obtained again. The third batch of first samples is input into the first model to be trained after the updated parameters are obtained again, and so on, until the indicator no longer improves, that is, the input of training data is stopped, and the first model to be trained obtained by the current training is determined as the trained NLP model. The indicator here can be the value of the loss function or objective function of the first model to be trained.

[0089] Step 12: Input the first batch of samples in the training data (e.g., the second sample in the second sample set) into the NLP model and output the prediction results p in each category. ij .

[0090] Step 13: Input the first batch of samples into the second trained model to obtain the confidence model, and output the prediction result c I .

[0091] In this embodiment, in order to improve the model training efficiency, the parameters in the embedding layer and the representation layer in the NLP model can be used as the initial values of the embedding layer and the representation layer in the second model data to be trained.

[0092] Step 14: Calculate the loss L of the second model to be trained according to the loss function (Expression (4)) conf , using a gradient descent method, such as a stochastic gradient descent method, to update the parameters of the second model to be trained to obtain the second model to be trained after the parameters are updated.

[0093] Step 15: Input the next batch of samples from the training data into the NLP model and output the prediction results p in each category ij .

[0094] Step 16: Input the next batch of samples into the second training model with updated parameters and output the prediction result c i .

[0095] Step 17: Calculate the loss L of the second model to be trained according to the loss function (Expression (4)) conf , using a gradient descent method, such as a stochastic gradient descent method, to update the parameters of the second model to be trained after the parameters are updated, to obtain a second model to be trained after the parameters are updated again.

[0096] Similarly, steps 15 to 17 are repeated until the value of the loss function does not increase, then the input of training data is stopped, and the second to-be-trained model obtained by the current training is determined as the trained confidence model.

[0097] It should be noted that, in this embodiment, the execution order of step 12 and step 13 is not specific, and they can also be executed in parallel. The execution order of step 15 and step 16 is not specific, and they can also be executed in parallel.

[0098] Exemplarily, the reasoning application of the NLP model and the confidence model corresponding to the text classification task may include the following steps:

[0099] Step 21: Input the target text into the NLP model and output the prediction results p in each category j .

[0100] Step 22: Input the target text into the confidence model and output the prediction result c.

[0101] Step 23: If the confidence score c is greater than the preset threshold, the classification prediction result is output, otherwise it returns empty.

[0102] It should be noted that, in this embodiment, the execution order of step 21 and step 22 is not particular, and they can also be executed in parallel.

[0103] For example, the training of the NLP model and the confidence model corresponding to the sequence labeling task may include the following steps:

[0104] Step 31: Input the training data (for example, the first sample in the first sample set) into the first model to be trained in batches, and use the gradient descent method, such as the stochastic gradient descent method, to update the parameters of the first model to be trained for the results of each batch of training data until the indicators no longer improve, thereby obtaining a trained NLP model.

[0105] For details, please refer to step 1 of the NLP model training in the above text classification task. I will not go into details here.

[0106] Step 32: Input the first batch of samples in the training data (e.g., the second sample in the second sample set) into the NLP model and output the label prediction result p for each position. imj, subscript m is the position index of the text sequence.

[0107] Step 33: Input the first batch of samples into the second to-be-trained model for training the confidence model, and output the prediction result c im ;

[0108] Step 34: Calculate the loss L of the second model to be trained according to the loss function (Expression (5)) conf , using a gradient descent method, such as a stochastic gradient descent method, to update the parameters of the second model to be trained to obtain the second model to be trained after the parameters are updated.

[0109] It should be noted that since sequence annotation predicts the label probability of each position, the confidence model can calculate the confidence of the output of each position. Compared with the embedding of the text classification task, the loss calculation adds a position dimension m. The specific form is shown in expression (5):

[0110]

[0111]

[0112] Among them, L true is the probability constraint term, L dis is the probability distribution constraint, m is the sequence position index, s is the sequence length, n is the number of samples in the current batch, The trained NLP model infers the probability value of sample i at position m in the current batch on the true label, c im is the output result of the second network to be trained, p imj is the probability value of the NLP model for sample i at position m in category j in the current batch, Infer the probability value of the true label of sample i at position m in category j in the current batch for the NLP model.

[0113] Step 35: Input the next batch of samples in the training data (e.g., the second sample in the second sample set) into the NLP model and output the label prediction results p for each position. imj , subscript m is the position index of the text sequence.

[0114] Step 36: Input the next batch of samples into the second training model with updated parameters and output the prediction result c im .

[0115] Step 37: Calculate the loss L of the second model to be trained after parameter update according to the loss function (Expression (5)) conf , using a gradient descent method, such as a stochastic gradient descent method, to update the parameters of the second model to be trained after the parameters are updated, to obtain a second model to be trained after the parameters are updated again.

[0116] Similarly, steps 35 to 37 are repeated until the value of the loss function does not increase, then the input of training data is stopped, and the second to-be-trained model obtained by the current training is determined as the trained confidence model.

[0117] It should be noted that, in this embodiment, the execution order of step 32 and step 33 is not specific, and they can also be executed in parallel. The execution order of step 35 and step 36 is not specific, and they can also be executed in parallel.

[0118] For example, the reasoning application of the NLP model corresponding to the sequence labeling task and the reasoning application of the confidence model may include the following steps:

[0119] Step 41: Input the target text into the NLP model and output the prediction results p in each category mj .

[0120] Step 42: Input the target text into the confidence model and output the prediction result c m .

[0121] Step 43: If the confidence score c m If the value is greater than the preset threshold, the classification prediction result is output, otherwise it returns empty.

[0122] It should be noted that, in this embodiment, the execution order of step 41 and step 42 is not particular, and they can also be executed in parallel.

[0123] The natural language processing method provided in this embodiment obtains a confidence model by training based on a loss function including probability constraints and probability distribution constraints. The confidence output by the confidence model can accurately reflect the probability distribution of the NLP model reasoning results, and obtain accurate confidence based on the probability distribution. Under the constraints of the probability constraints, it is ensured that the confidence deviates less from the probability value of the sample on the true label, which is more credible and more accurate.

[0124] Figure 5 This is a schematic diagram of the structure of the natural language processing device provided in the embodiment of the present application. Figure 5 As shown, the natural language processing device 50 includes: an acquisition module 501, a first input module 502, a second input module 503 and a processing module 504.

[0125] The acquisition module 501 is used to acquire the target text.

[0126] The first input module 502 is used to input the target text into a natural language processing (NLP) model to obtain a target processing result; the target processing result includes each label and its corresponding probability value.

[0127] The second input module 503 is used to input the target text into the confidence model to obtain the confidence corresponding to the target processing result; the confidence is related to the distribution of the probability values of the various tags.

[0128] The processing module 504 is configured to process the target processing result according to the confidence level.

[0129] The natural language processing device provided in the embodiment of the present application obtains the target text through the acquisition module 501, the first input module 502 inputs the target text into the natural language processing NLP model to obtain the target processing result, and the target processing result includes each label and the corresponding probability value. The second input module 503 inputs the target text into the confidence model to obtain the confidence corresponding to the target processing result, and the confidence is related to the distribution of the probability values of the each label. The processing module 504 processes the target processing result according to the confidence, and can determine the confidence of the target processing result obtained by the NLP model without relying on manual labor, by obtaining a confidence model through special training. The range of the confidence can be a value between 0 and 1, which improves the accuracy of the confidence.

[0130] The natural language processing device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0131] Figure 6 This is a block diagram of a natural language processing device provided in an embodiment of the present application. The device can be a computer, a message sending and receiving device, a tablet device, a medical device, etc. The device can be set on an unmanned vehicle.

[0132] The device 60 may include one or more of the following components: a processing component 601 , a memory 602 , a power component 603 , a multimedia component 604 , an audio component 605 , an input / output (I / O) interface 606 , a sensor component 607 , and a communication component 608 .

[0133] The processing component 601 generally controls the overall operation of the device 60, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 601 may include one or more processors 609 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 601 may include one or more modules to facilitate interaction between the processing component 601 and other components. For example, the processing component 601 may include a multimedia module to facilitate interaction between the multimedia component 604 and the processing component 601.

[0134] The memory 602 is configured to store various types of data to support operations on the device 60. Examples of such data include instructions for any application or method operating on the device 60, contact data, phone book data, messages, pictures, videos, etc. The memory 602 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0135] The power supply component 603 provides power to the various components of the device 60. The power supply component 603 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 60.

[0136] The multimedia component 604 includes a screen that provides an output interface between the device 60 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 604 includes a front camera and / or a rear camera. When the device 60 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0137] The audio component 605 is configured to output and / or input audio signals. For example, the audio component 605 includes a microphone (MIC) that is configured to receive external audio signals when the device 60 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 602 or transmitted via the communication component 608. In some embodiments, the audio component 605 further includes a speaker for outputting audio signals.

[0138] The I / O interface 606 provides an interface between the processing component 601 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0139] The sensor assembly 607 includes one or more sensors for providing various aspects of the status assessment of the device 60. For example, the sensor assembly 607 can detect the open / closed state of the device 60, the relative positioning of components, such as the display and keypad of the device 60. The sensor assembly 607 can also detect changes in the position of the device 60 or a component of the device 60, the presence or absence of user contact with the device 60, the orientation or acceleration / deceleration of the device 60, and temperature changes of the device 60. The sensor assembly 607 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 607 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 607 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0140] The communication component 608 is configured to facilitate wired or wireless communication between the device 60 and other devices. The device 60 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 608 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 608 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0141] In an exemplary embodiment, the device 60 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned methods.

[0142] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the instructions can be executed by the processor 609 of the device 60 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0143] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0144] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0145] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0146] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the natural language processing method performed by the natural language processing device as described above.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A natural language processing method, characterized in that: include: Get the target text; Training the first to-be-trained model using the first sample set to obtain an NLP model; Inputting each second sample in the second sample set and the annotation results corresponding to each second sample into the NLP model to obtain a predicted value of the processing result; Training a second to-be-trained model according to the predicted value of the processing result and the second sample set to obtain a confidence model; Inputting the target text into a natural language processing (NLP) model to obtain a target processing result; The target processing result includes each label and its corresponding probability value; Inputting the target text into a confidence model to obtain a confidence level corresponding to the target processing result; the confidence level is related to the distribution of probability values of the labels; The target processing result is processed according to the confidence level.

2. The method according to claim 1, characterized in that The target processing result is a classification result or a sequence labeling result.

3. The method according to claim 1, characterized in that The processing of the target processing result according to the confidence level includes: If the confidence level is greater than a preset threshold, the final result is output according to the target processing result.

4. The method according to claim 1, wherein The step of training the first to-be-trained model using the first sample set to obtain the NLP model includes: Inputting each first sample in the first sample set and the labeling results corresponding to each first sample into the first to-be-trained model; The parameters of the first model to be trained are updated by a gradient descent algorithm to obtain the NLP model.

5. The method according to claim 1, wherein The step of training the second to-be-trained model according to the predicted value of the processing result and the second sample set to obtain the confidence model includes: Inputting each second sample in the second sample set and the annotation results corresponding to each second sample into the second to-be-trained model to obtain a confidence prediction value corresponding to the processing result prediction value; According to the loss function, the parameters of the second model to be trained are updated by a gradient descent algorithm to obtain the confidence model; the loss function is constructed based on the processing result prediction value and the confidence prediction value.

6. The method according to claim 5, characterized in that The loss function includes a probability constraint term and a probability distribution constraint term, wherein the probability constraint term is related to the difference between the probability prediction value of the true label and the corresponding confidence prediction value of each second sample in the processing result prediction value; The probability distribution constraint item is related to the difference between the labeling result of each label of each second sample in the processing result prediction value and the probability prediction value.

7. The method according to claim 1, characterized in that The NLP model includes an embedding layer, a representation layer and a first output layer connected in sequence; the second model to be trained includes the embedding layer, the representation layer of the NLP model, and a second output layer connected to the representation layer.

8. The method according to claim 7, characterized in that The representation layer is a transformer model Transformer or a bidirectional long short-term memory model Bi-LSTM.

9. A natural language processing device, characterized in that: include: Acquisition module, used to obtain target text; A first input module is used to input the target text into a natural language processing (NLP) model to obtain a target processing result; The target processing result includes each label and its corresponding probability value; A second input module is used to input the target text into a confidence model to obtain a confidence level corresponding to the target processing result; the confidence level is related to the distribution of probability values of the labels; a processing module, configured to process the target processing result according to the confidence level; The device further includes: training a first to-be-trained model using a first sample set to obtain the NLP model; Inputting each second sample in the second sample set and the annotation results corresponding to each second sample into the NLP model to obtain a predicted value of the processing result; The second model to be trained is trained according to the predicted value of the processing result and the second sample set to obtain the confidence model.

10. A natural language processing device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the natural language processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the natural language processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the natural language processing method according to any one of claims 1 to 8 is implemented.

Images