Natural language processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a natural language processing method and device, electronic equipment and a storage medium, belonging to the field of data processing. The method comprises the following steps: inputting to-be-processed data into a preset language processing model, the language processing model is obtained by updating a trained model by using an attention matrix corresponding to new training samples, the trained model is obtained by updating an initial model by using original training samples, and the attention matrix is obtained by performing attention processing on the original training samples and the new training samples. Therefore, the language processing model can well adapt to the new training samples, the to-be-processed data can be recognized and processed by using the language processing model, a preliminary processing result with higher accuracy can be obtained, the accuracy of the processing result selected from all prediction data of the preliminary processing result is higher, and the accuracy of the natural language processing is greatly improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a natural language processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. NLP is primarily applied to machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, sentiment analysis, and Chinese OCR.

[0003] Natural Language Processing (NLP) typically employs machine learning models. Currently, NLP often utilizes incremental learning-based continuous learning methods, employing online learning algorithms to update the model with new data without system downtime. However, models using incremental learning algorithms exhibit poor learning performance and stability, thus impacting the accuracy of NLP. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a natural language processing method, apparatus, electronic device and storage medium that can improve the problem of poor learning effect and stability of incremental learning-based models affecting the accuracy of natural language processing.

[0005] To achieve the above objectives, the technical solution adopted in the embodiments of this application is as follows:

[0006] In a first aspect, embodiments of this application provide a natural language processing method, the method comprising:

[0007] Acquire the data to be processed and input the data to be processed into a preset language processing model; wherein, the language processing model is obtained by updating the trained model with the attention matrix corresponding to the newly added training sample, the trained model is obtained by training the initial model with the original training sample, and the attention matrix is ​​obtained by performing attention processing on the original training sample and the newly added training sample;

[0008] The language processing model is used to identify and process the data to be processed to obtain preliminary processing results; wherein, the preliminary processing results include at least one predicted data and the confidence level of each predicted data.

[0009] Based on the confidence level of each predicted data, at least one predicted data is selected from all predicted data as the processing result of the data to be processed.

[0010] Optionally, the method further includes the step of updating the trained model using the attention matrix of the newly added training samples to obtain the language processing model, including:

[0011] Extract the features of each of the newly added training samples, convert each feature into a feature vector, and form a new sub-matrix corresponding to the newly added training sample;

[0012] Based on the newly added submatrix, update the weights of multiple original submatrixes of the original training samples, and obtain the weights of the newly added submatrix.

[0013] By combining the weights of the newly added submatrix and the multiple original submatrixes, the representations of each feature vector of the newly added submatrix and the original submatrix are updated to obtain the attention matrix;

[0014] The attention matrix is ​​used to update the trained model to obtain the language processing model.

[0015] Optionally, the step of updating the representations of each feature vector of the newly added sub-matrix and the original sub-matrix by combining the weights of the newly added sub-matrix and the plurality of original sub-matrixes to obtain the attention matrix includes:

[0016] By combining the weights of the newly added submatrix and the weights of the plurality of original submatrixes, the importance of each submatrix is ​​calculated to obtain an importance score for each submatrix; wherein, the submatrix is ​​any one of the newly added submatrix and the plurality of original submatrixes;

[0017] For each submatrix, the representation of each eigenvector of the submatrix is ​​updated according to the weight of the submatrix to obtain a new submatrix.

[0018] The attention matrix is ​​calculated based on the importance scores of the submatrices corresponding to all the new submatrices.

[0019] Optionally, the step of combining the weights of the newly added submatrix and the weights of the multiple original submatrixes to calculate the importance of each submatrix and obtain the importance score of each submatrix includes:

[0020] Calculate the sum of the weights of the newly added submatrix and the plurality of original submatrixes;

[0021] For each submatrix, the ratio of the weight of the submatrix to the sum of the weights is used as the importance score of the submatrix.

[0022] Optionally, the step of updating the representation of each eigenvector of the submatrix according to the weights of the submatrix to obtain a new submatrix includes:

[0023] For each eigenvector of the submatrix, the product of the eigenvectors is used as a new representation of the eigenvector;

[0024] The matrix formed by the new representations of each of the eigenvectors is taken as the new submatrix of the submatrix.

[0025] Optionally, the step of calculating the attention matrix based on the importance scores of the submatrices corresponding to all the new submatrices includes:

[0026] The attention matrix is ​​calculated by using the attention matrix formula and combining the importance scores of all the new submatrices and the submatrices corresponding to the new submatrices.

[0027] The attention matrix formula includes:

[0028]

[0029] Where A represents the attention matrix, S i Let M represent the importance score of the submatrix corresponding to the i-th new submatrix, k represent the total number of new submatrixes, and M represent the importance score of the submatrix corresponding to the ith new submatrix. i Let i represent the i-th new submatrix.

[0030] Optionally, the step of updating the weights of multiple original sub-matrices of the original training samples based on the newly added sub-matrix, and obtaining the weights of the newly added sub-matrix, includes:

[0031] Calculate the sum of weights of all eigenvectors in the plurality of original submatrices and the newly added submatrices to obtain the total weight sum;

[0032] For any submatrix, calculate the sum of weights of all eigenvectors of the submatrix to obtain the submatrix weight sum, and use the ratio of the submatrix weight to the total weight sum as the weight of the submatrix; wherein, the submatrix is ​​any one of the newly added submatrix and the plurality of original submatrixes.

[0033] Secondly, embodiments of this application provide a natural language processing device, including a data input module, a data processing module, and a result determination module;

[0034] The data input module is used to acquire data to be processed and input the data to be processed into a preset language processing model; wherein, the language processing model is obtained by updating an already trained model using an attention matrix corresponding to a newly added training sample, the already trained model is obtained by training an initial model using the original training samples, and the attention matrix is ​​obtained by performing attention processing on the original training samples and the newly added training samples;

[0035] The data processing module is used to identify and process the data to be processed through the language processing model to obtain preliminary processing results; wherein, the preliminary processing results include at least one predicted data and the confidence level of each predicted data;

[0036] The result determination module is used to select at least one predicted data from all predicted data as the processing result of the data to be processed, based on the confidence level of each predicted data.

[0037] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the natural language processing method as described in any possible embodiment of the first aspect.

[0038] Fourthly, embodiments of this application provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the natural language processing method as described in any possible embodiment of the first aspect.

[0039] The natural language processing method, apparatus, electronic device, and storage medium provided in this application input the acquired data to be processed into a preset language processing model. This language processing model is obtained by updating an already trained model using an attention matrix corresponding to newly added training samples, and the already trained model is obtained by updating an initial model using the original training samples. The attention matrix is ​​obtained by performing attention processing on the original training samples and the newly added training samples, thereby enabling the language processing model to adapt well to the newly added training samples. Thus, by recognizing and processing the data to be processed through the language processing model, a more accurate preliminary processing result can be obtained. This allows for a higher accuracy in selecting the processing result from all the predicted data in the preliminary processing result, greatly improving the accuracy of natural language processing.

[0040] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0042] Figure 1A block diagram of a natural language processing system provided in an embodiment of this application is shown.

[0043] Figure 2 This illustration shows one of the flowcharts of the natural language processing method provided in the embodiments of this application.

[0044] Figure 3 This is a second schematic flowchart of the natural language processing method provided in the embodiments of this application.

[0045] Figure 4 It shows Figure 3 A flowchart illustrating some sub-steps of step S23.

[0046] Figure 5 It shows Figure 3 A flowchart illustrating some sub-steps of step S25.

[0047] Figure 6 A block diagram of a natural language processing apparatus provided in an embodiment of this application is shown.

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0049] Explanation of reference numerals in the attached figures: 1000 - Natural Language Processing System; 10 - Processing Equipment; 20 - Control Equipment; 30 - Natural Language Processing Device; 301 - Data Input Module; 302 - Data Processing Module; 303 - Result Determination Module; 40 - Electronic Equipment. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] Natural language processing models based on incremental learning algorithms improve performance by incrementally updating the model based on the relationship between prior knowledge and new data. Incremental learning algorithms utilize online learning, allowing for model updates with new data without system downtime. The advantage of this method is its ability to rapidly update the model, enabling it to adapt quickly to changing data distributions. However, while this method yields good learning results, it also suffers from poor stability, exhibiting the following drawbacks:

[0054] (1) When the model learns new data, the old data may be forgotten, which will lead to a decrease in model performance.

[0055] (2) Over time, the decision boundary of the model may drift, leading to a decline in the model’s performance on new data.

[0056] (3) Changes in data distribution may cause the model to be unable to adapt well to new data, thus affecting the performance of the model.

[0057] Based on the above considerations, this application provides a natural language processing method that can improve the problem that the learning effect and stability of models using incremental learning algorithms are poor, thus affecting the accuracy of natural language processing.

[0058] The natural language processing method provided in this application can be applied to... Figure 1 The natural language processing system 1000 shown includes a processing device 10 and a control device 20. The control device 20 can communicate with the processing device 10 via wired or wireless means.

[0059] Control device 20 is used to send new data to processing device 10 and to transmit data to be processed to processing device 10.

[0060] The processing device 10 is used to train the already trained model based on the new data to obtain a language processing model, process the data to be processed based on the language processing model to obtain preliminary processing results, and select at least one predicted data from all the predicted data as the type of the data to be processed based on the confidence level of each predicted data in the preliminary processing results.

[0061] The control device 20 may be, but is not limited to, any of the following: a personal computer, a laptop computer, a mobile terminal, a mobile phone, a tablet computer, and a wearable portable device. The processing device 10 includes, but is not limited to, a standalone server, a server cluster, and a personal computer.

[0062] In one possible implementation, this application provides a natural language processing method, referring to... Figure 2 This may include the following steps. In this embodiment, the method is applied to... Figure 1 Let's take the processing device 10 in the middle as an example.

[0063] S11, acquire the data to be processed, and input the data to be processed into the preset language processing model.

[0064] In this embodiment, the language processing model is obtained by updating the already trained model using the attention matrix corresponding to the newly added training samples. The already trained model is obtained by training the initial model using the original training samples. The attention matrix is ​​obtained by performing attention processing on the original training samples and the newly added training samples.

[0065] S13 uses a language processing model to identify and process the data to be processed, and obtains preliminary processing results.

[0066] It should be noted that the preliminary processing results include at least one predicted data point and the confidence level of each predicted data point.

[0067] S15, based on the confidence level of each predicted data, select at least one predicted data from all predicted data as the processing result of the data to be processed.

[0068] The data to be processed can be any of the following: text to be classified, text to be translated, speech data to be recognized, questions to be answered, and text from which viewpoints need to be extracted. When the data to be processed is text to be classified, the preliminary processing results may include at least one prediction type and the confidence level of each prediction type. The final processing result can be the prediction type with the highest confidence level among the preliminary processing results. The prediction type can be, but is not limited to, sentiment type and public opinion type. When the data to be processed is text to be translated, the preliminary processing results may include at least one predicted text and the confidence level of each predicted text. The final processing result can be the predicted text with the highest confidence level among the preliminary processing results.

[0069] When the control device 20 receives new data for updating the trained model, it performs segmentation processing on the new data and then sends the new data to the processing device 10. It should be understood that the new data includes new training samples and new test samples, and the content and labels of the new data may differ from the original training samples.

[0070] After receiving the new data, the processing device 10 performs attention processing on the original training samples and the new training samples to obtain an attention matrix, and uses the attention matrix to train the already trained model to obtain a language processing model.

[0071] When the control device 20 confirms that the language processing model of the processing device 10 has completed training, it sends the acquired data to be processed to the processing device 10. The processing device 10 inputs the data to be processed into the language processing model to identify and classify the data, obtaining a preliminary processing result including at least one predicted data and the confidence level of each predicted data. Then, based on the confidence level of each predicted data, the control device 20 selects at least one predicted data as the processing result from all predicted data. For example, the predicted data with the highest confidence level can be selected as the processing result, or the predicted data with a confidence level exceeding a preset threshold can be selected as the processing result; no specific limitation is made in this embodiment.

[0072] Compared to traditional natural language processing (NLP) methods, the language processing model used in the above method is obtained by updating the already trained model with the attention matrix corresponding to the new training samples. The already trained model is obtained by updating the initial model with the original training samples. The attention matrix is ​​obtained by applying attention to both the original and new training samples. This allows the language processing model to adapt to new training samples without forgetting the original training data, and reduces the shift in the decision boundary. Therefore, by using the language processing model to identify and process the data to be processed, more accurate preliminary processing results can be obtained. This leads to higher accuracy in selecting the final processing result from all the predicted data in the preliminary processing results, significantly improving the accuracy of natural language processing.

[0073] In one possible implementation, to obtain a language processing model that adapts to new training samples without forgetting the original training data, and to reduce the decision boundary of the language processing model, a method is introduced whereby the attention matrix of the new training samples is used to train the already trained model to obtain the language processing model. (Refer to...) Figure 3 The language processing model can be obtained through the following steps.

[0074] S21, extract the features of each newly added training sample, convert each feature into a feature vector, and form the new sub-matrix corresponding to the newly added training sample.

[0075] S23, based on the newly added submatrix, update the weights of multiple original submatrixes of the original training samples, and obtain the weights of the newly added submatrix.

[0076] S25, combining the weights of the newly added submatrix and multiple original submatrixes, updates the representations of each eigenvector of the newly added submatrix and the original submatrixes to obtain the attention matrix.

[0077] S27. The attention matrix is ​​used to update the trained model to obtain the language processing model.

[0078] The original training samples can include multiple sample sets, each with a corresponding original submatrix. The original submatrix of each sample set includes the feature vector of each original training sample in that sample set.

[0079] In steps S21 to S27 above, the weights of multiple original submatrices of the original training samples are updated based on the newly added submatrices, and the weights of the newly added submatrices are obtained simultaneously. Then, combining the weights of the newly added submatrices and the weights of the multiple original submatrices, the representations of each feature vector of both the newly added and original submatrices are updated to obtain the attention matrix. This ensures that the attention matrix considers both the newly added training samples and their impact on the original training samples, i.e., it does not forget old data. This improves the learning performance and stability of the language processing model trained based on the attention matrix, thereby contributing to improved data processing accuracy.

[0080] In step S21, the features extracted from the newly added training samples can be any existing feature extraction method, such as principal component analysis or MDS (Multi-Demision Scaling) algorithm, and are not specifically limited in this embodiment. Similarly, the conversion of features into feature vectors can be done using any vectorization method, such as TF-IDF text vectorization or hash vectorization, and are not specifically limited in this embodiment either.

[0081] In one possible implementation, refer to Figure 4 Step S23 can be further implemented as follows.

[0082] S231, calculate the sum of weights of all eigenvectors in the multiple original submatrices and the newly added submatrices to obtain the total weight sum.

[0083] In this embodiment, the weight of each feature vector is pre-assigned. For example, the weight of the feature vector can be copied according to the label of the sample data corresponding to the feature vector, or it can be copied according to the content category of the sample data corresponding to the feature vector. In this embodiment, no specific limitation is made.

[0084] S232, for any submatrix, calculate the sum of the weights of all eigenvectors of the submatrix to obtain the submatrix weight sum, and take the ratio of the submatrix weight to the total weight sum as the weight of the submatrix.

[0085] Based on steps S231 and S232 above, the weight calculation formula for the submatrix (including the newly added submatrix) can be expressed as: Among them, w i F represents the weight of the i-th submatrix. i d represents the set of indices (i.e., the number) of all eigenvectors in the i-th submatrix. j represents the weight of the j-th eigenvector, and N represents the total number of eigenvectors in the newly added submatrix and the original submatrix.

[0086] Through the steps S231 to S231 above, the weight of each submatrix under the attention mechanism can be calculated, thereby adjusting the contribution level (i.e. weight) of the submatrix so that the importance of each submatrix can be balanced.

[0087] In one possible implementation, refer to Figure 5 Step S25 can be further implemented as follows.

[0088] S251, combining the weights of the newly added submatrix and the weights of multiple original submatrixes, calculate the importance of each submatrix to obtain the importance score of each submatrix.

[0089] In this embodiment, the submatrix is ​​any one of the newly added submatrix and multiple original submatrixes.

[0090] S252, for each submatrix, update the representation of each eigenvector of the submatrix according to the weight of the submatrix to obtain a new submatrix.

[0091] S253, calculate the attention matrix based on the importance scores of all new submatrices and newly added submatrices.

[0092] In step S251, the method for obtaining the importance score of the newly added submatrix can be flexibly set. For example, the importance score of the submatrix can be calculated using a machine learning model, or it can be calculated according to a preset rule. In this embodiment, no specific limitation is made.

[0093] In one possible implementation, step S251 can be further implemented as follows: calculating the sum of weights of the newly added submatrix and the multiple original submatrixes, and for each submatrix, using the ratio of the weight of the submatrix to the sum of weights as the importance score of the submatrix.

[0094] In the above implementation, the importance score of the submatrix can be expressed as: Among them, S i Let w represent the importance score of the i-th submatrix, K represent the total number of newly added submatrices and the original submatrix, and w represent the total importance score of the i-th submatrix. k This represents the weight of the k-th submatrix.

[0095] By using the above steps, the proportion of the weight of the newly added submatrix to the sum of the weights of all submatrixes is used as the importance score of the newly added submatrix. This approach takes into account the influence of old training data, further improves the learning effect and stability of the model, and thus helps to improve the accuracy of natural language processing.

[0096] Furthermore, the importance score determines the contribution of each submatrix during model updates; that is, the importance score measures the significance of each submatrix to model improvement. During model updates, resources and attention can be allocated based on the importance scores of the submatrixes, with submatrixes receiving more resources and more update iterations. By adjusting resource allocation and attention mechanisms, the model can learn and improve more efficiently.

[0097] For step S252, for each eigenvector of the submatrix, the product of the eigenvectors can be used as a new representation of the eigenvector, and the matrix formed by the new representations of the eigenvectors can be used as a new submatrix of the submatrix.

[0098] The new submatrix in the above method can be represented as: Among them, M i Let x represent the new submatrix corresponding to the i submatrices. j Let represent the j-th eigenvector of the i-th submatrix.

[0099] In one possible implementation, step S253 can be further implemented as follows: using the attention matrix formula, combining the importance scores of all new submatrices and the submatrices corresponding to the new submatrices, to calculate the attention matrix.

[0100] The attention matrix formula can be expressed as: Where A represents the attention matrix, S i Let M represent the importance score of the submatrix corresponding to the i-th new submatrix, k represent the total number of new submatrixes, and M represent the importance score of the submatrix corresponding to the ith new submatrix. i Let i represent the i-th new submatrix.

[0101] By using the above method, the attention submatrix is ​​obtained by weighted averaging of each new submatrix based on its importance score. This allows the attention matrix to consider both the new training samples and the changes in data distribution caused by the impact of the new training samples on the original training samples. As a result, when retraining the language processing model, it can continue to learn new knowledge without losing the previously trained knowledge, thus obtaining a language processing model with better learning performance and greater stability.

[0102] The method of using the attention matrix to update the trained model can be any model training method; in this embodiment, no specific limitation is made.

[0103] In this application, when new data arrives, only the relevant model parameters can be updated based on its corresponding submatrix, and then the attention matrix can be recalculated to update the model. This allows for better prediction results on new data while maintaining the model's continuous learning ability, without requiring retraining the entire model. This results in a language processing model with better learning performance and higher stability, thereby improving the accuracy of natural language processing.

[0104] Let's take a dataset containing 100 samples, each with 10 features, as an example. Using current continuous learning methods, this dataset is trained with only one sample as input at a time, and 10 sub-matrices are used for training, each with 5 features.

[0105] First, the dataset is divided into a training set and a test set. The training set contains the first 80 samples, and the test set contains the last 20 samples. Then, the trained model is retrained using steps S21 to S27 and their sub-steps (which can be called agnostic continuous learning). Randomly initialized attention matrices and model parameters are used to train the model on the training set until convergence. During training, the loss function is calculated and the model parameters and attention matrix are updated for each input sample. When calculating the loss function, cross-entropy loss can be used, with a regularization term added to avoid overfitting.

[0106] Using the trained model and attention matrix, predictions were made on the test set, and the accuracy and recall of the prediction results were calculated. To compare the performance of different methods, the traditional random forest method was also used for training and testing, and its accuracy and recall were calculated.

[0107] Table 1 below shows the results of predictions on the test set using the continuous learning method based on agnostic and attention matrix and the random forest method.

[0108] Table 1

[0109] method accuracy Recall rate Unknowable continuous learning 0.95 0.92 Random Forest 0.90 0.88

[0110] As can be seen from the table above, the accuracy and recall of predictions using the agnostic continuous learning method (i.e., the model training methods provided in steps S21 to S27 and their sub-steps) are higher than those using the traditional random forest method, indicating that the model training methods provided in steps S21 to S27 and their sub-steps can effectively improve the performance and generalization ability of the model.

[0111] This application utilizes an attention matrix to learn new data in real time and update the trained model, enabling the model to continuously adapt to changing data distributions. This results in the following beneficial effects in practical applications:

[0112] (i) Higher prediction accuracy: Traditional machine learning methods require retraining the model to adapt to new data distributions, while this application can learn new data to update the weights and importance of old data to obtain a new attention matrix. The model is then updated with this attention matrix, thereby enabling the model to better adapt to the ever-changing data distribution and improve prediction accuracy.

[0113] (ii) Better data utilization efficiency: Traditional machine learning methods require retraining the model to adapt to new data distributions, while this application can learn new data and update the model in real time, making full use of all available data and improving data utilization efficiency.

[0114] (iii) Lower system overhead: Traditional machine learning methods require retraining the model to adapt to new data distributions, while this application can learn new data and update the model in real time based on the original model, reducing system overhead.

[0115] Based on the same concept as the natural language processing method described above, in one possible implementation, a natural language processing device 30 is also provided, which can be applied to... Figure 1 Processing device 10. (Refer to...) Figure 6 The natural language processing device 30 may include a data input module 301, a data processing module 302, and a result determination module 303.

[0116] The data input module 301 is used to acquire the data to be processed and input the data to be processed into a preset language processing model. The language processing model is obtained by updating the already trained model with the attention matrix corresponding to the newly added training samples. The already trained model is obtained by training the initial model with the original training samples. The attention matrix is ​​obtained by performing attention processing on the original training samples and the newly added training samples.

[0117] The data processing module 302 is used to identify and process the data to be processed through a language processing model to obtain preliminary processing results. These preliminary processing results include at least one predicted data point and the confidence level of each predicted data point.

[0118] The result determination module 303 is used to select at least one predicted data from all predicted data as the processing result of the data to be processed, based on the confidence level of each predicted data.

[0119] Furthermore, it may also include a model update module for:

[0120] Extract the features of each new training sample, convert each feature into a feature vector, and form the new sub-matrix corresponding to the new training sample;

[0121] Based on the newly added submatrix, update the weights of multiple original submatrixes of the original training samples, and obtain the weights of the newly added submatrix.

[0122] By combining the weights of the newly added submatrix and multiple original submatrixes, the representations of each eigenvector of the newly added submatrix and the original submatrixes are updated to obtain the attention matrix.

[0123] The trained model is updated using an attention matrix to obtain the language processing model.

[0124] In the aforementioned natural language processing device 30, through the collaborative action of the data input module 301, the data processing module 302, and the result determination module 303, the language processing model is obtained by updating the already trained model using the attention matrix corresponding to the newly added training samples, while the already trained model is obtained by updating the initial model using the original training samples. The attention matrix is ​​obtained by applying attention processing to both the original and newly added training samples. This allows the language processing model to adapt to newly added training samples without forgetting the original training data and reduces the shift in the decision boundary. Therefore, by using the language processing model to identify and process the data to be processed, a more accurate preliminary processing result can be obtained. This leads to a higher accuracy in selecting the processing result from all the predicted data in the preliminary processing result, significantly improving the accuracy of natural language processing.

[0125] Specific limitations regarding the natural language processing device 30 can be found in the limitations of the natural language processing method described above, and will not be repeated here. Each module in the natural language processing device 30 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the electronic device 40, or stored in software in the memory of the electronic device 40, so that the processor can call and execute the operations corresponding to each module.

[0126] In one embodiment, an electronic device 40 is provided, which may be a server, and its internal structure diagram may be as follows. Figure 7 As shown, the electronic device 40 includes a processor, memory, communication interface, and input device connected via a system bus. The processor of the electronic device 40 provides computing and control capabilities. The memory of the electronic device 40 includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the electronic device 40 is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, near-field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements the natural language processing method provided in the above embodiments.

[0127] Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device 40 to which the present invention is applied. The specific electronic device 40 may include, but is not limited to, the following: Figure 7 The diagram shows more or fewer components, or combinations of certain components, or different component arrangements.

[0128] In one embodiment, the natural language processing device 30 provided by the present invention for use in a deployed device can be implemented as a computer program, which can be implemented in, for example... Figure 7 The device operates on the electronic device 40 shown. The memory of the electronic device 40 can store various program modules that make up the natural language processing device 30, for example, Figure 6 The data input module 301, data processing module 302, and result determination module 303 are shown. The computer program, composed of these modules, causes the processor to execute the steps in the natural language processing method described in this specification.

[0129] For example, Figure 7 The electronic device 40 shown can be accessed via, for example Figure 6 The data input module 301 of the natural language processing device 30 shown executes step S11. The electronic device 40 can execute step S13 through the data processing module 302. The electronic device 40 can execute step S15 through the result determination module 303.

[0130] In one embodiment, an electronic device 40 is provided, including a memory and a processor. The memory stores machine-executable instructions, and the processor executes the machine-executable instructions to perform the following steps: acquiring data to be processed and inputting the data to be processed into a preset language processing model; recognizing and processing the data to be processed through the language processing model to obtain a preliminary processing result; and selecting at least one predicted data from all predicted data as the processing result of the data to be processed based on the confidence level of each predicted data.

[0131] In one embodiment, a storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: acquiring data to be processed and inputting the data to be processed into a preset language processing model; recognizing and processing the data to be processed through the language processing model to obtain a preliminary processing result; and selecting at least one predicted data from all predicted data as the processing result of the data to be processed based on the confidence level of each predicted data.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functions, and operations of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0134] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A natural language processing method, characterized in that, The method includes: Acquire the data to be processed and input the data to be processed into a preset language processing model; wherein the language processing model is obtained by updating the trained model with the attention matrix corresponding to the newly added training sample, the trained model is obtained by training the initial model with the original training sample, the attention matrix is ​​obtained by performing attention processing on the original training sample and the newly added training sample, and the data to be processed includes text data and / or speech data. The language processing model is used to identify and process the data to be processed to obtain preliminary processing results; wherein, the preliminary processing results include at least one predicted data and the confidence level of each predicted data. Based on the confidence level of each predicted data, at least one predicted data is selected from all predicted data as the processing result of the data to be processed; The method further includes the step of updating the trained model using the attention matrix of the newly added training samples to obtain the language processing model, including: Extract the features of each of the newly added training samples, convert each feature into a feature vector, and form a new sub-matrix corresponding to the newly added training sample; Based on the newly added submatrix, update the weights of multiple original submatrixes of the original training samples, and obtain the weights of the newly added submatrix. By combining the weights of the newly added submatrix and the multiple original submatrixes, the representations of each feature vector of the newly added submatrix and the original submatrix are updated to obtain the attention matrix; The step of updating the trained model using the attention matrix to obtain a language processing model, and updating the representations of each feature vector of the new sub-matrix and the original sub-matrix by combining the weights of the new sub-matrix and the multiple original sub-matrixes to obtain the attention matrix, includes: By combining the weights of the newly added submatrix and the weights of the plurality of original submatrixes, the importance of each submatrix is ​​calculated to obtain an importance score for each submatrix; wherein, the submatrix is ​​any one of the newly added submatrix and the plurality of original submatrixes; For each submatrix, the representation of each eigenvector of the submatrix is ​​updated according to the weight of the submatrix to obtain a new submatrix. The attention matrix is ​​calculated based on the importance scores of the submatrices corresponding to all the new submatrices. The step of updating the weights of multiple original sub-matrices of the original training samples based on the newly added sub-matrix, and obtaining the weights of the newly added sub-matrix, includes: Calculate the sum of weights of all eigenvectors in the plurality of original submatrices and the newly added submatrices to obtain the total weight sum; For any submatrix, calculate the sum of weights of all eigenvectors of the submatrix to obtain the submatrix weight sum, and use the ratio of the submatrix weight to the total weight sum as the weight of the submatrix; wherein, the submatrix is ​​any one of the newly added submatrix and the plurality of original submatrixes.

2. The natural language processing method according to claim 1, characterized in that, The step of combining the weights of the newly added submatrix and the weights of the multiple original submatrixes to calculate the importance of each submatrix and obtain the importance score of each submatrix includes: Calculate the sum of the weights of the newly added submatrix and the plurality of original submatrixes; For each submatrix, the ratio of the weight of the submatrix to the sum of the weights is used as the importance score of the submatrix.

3. The natural language processing method according to claim 1, characterized in that, The step of updating the representation of each eigenvector of the submatrix according to the weights of the submatrix to obtain a new submatrix includes: For each eigenvector of the submatrix, the product of the eigenvectors is used as a new representation of the eigenvector; The matrix formed by the new representations of each of the eigenvectors is taken as the new submatrix of the submatrix.

4. The natural language processing method according to claim 1, characterized in that, The step of calculating the attention matrix based on the importance scores of the submatrices corresponding to all the new submatrices includes: The attention matrix is ​​calculated by using the attention matrix formula and combining the importance scores of all the new submatrices and the submatrices corresponding to the new submatrices. The attention matrix formula includes: in, Represents the attention matrix. Indicates the first The importance scores of the submatrices corresponding to the new submatrices This represents the total number of new submatrices. Indicates the first A new submatrix.

5. A natural language processing device, characterized in that, The apparatus performs the method as described in any one of claims 1-4.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor to implement the natural language processing method as described in any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the natural language processing method as described in any one of claims 1 to 4.

Images