Method for generating text processing neural network model and text processing method

Abstract

The present disclosure provides a method for generating a text processing neural network model and a text processing method, and relates to the field of artificial intelligence technology, and in particular to the fields of deep learning and data processing. The specific implementation scheme is: the same sample text is input into the neural network model to be trained twice to generate the first and second prediction processing results, the first and second discard layer results, multiple task losses are calculated according to the generated results and the actual processing results corresponding to the sample text, and the comprehensive loss is calculated according to the multiple task losses, and the model is trained according to the comprehensive loss to generate a neural network model. The present disclosure generates multiple results by inputting the same sample text into the neural network model to be trained twice, calculates multiple task losses according to the multiple results, and calculates the comprehensive loss according to the calculated losses, and finally trains the model to be trained according to the comprehensive loss to generate a neural network model, thereby improving the generalization of the neural network model.

Description

Technical Field

[0001] The present disclosure relates to the fields of deep learning, data processing, etc. in the field of artificial intelligence technology, and in particular to a method for generating a text processing neural network model and a text processing method. Background Art

[0002] Currently, in the scenario of solving the overfitting problem of neural network models, a random dropout layer is added to the neural network through a dropout-based regularization method to randomly drop out a part of neurons, thereby introducing randomness.

[0003] However, the dropout-based regularization method cannot further improve the generalization of the neural network model. Summary of the invention

[0004] The present invention provides a method for generating a text processing neural network model, and a text processing method, device, electronic device and storage medium.

[0005] According to a first aspect, a method for generating a text processing neural network model is provided, comprising: obtaining a sample text and an actual processing result corresponding to the sample text; inputting the same sample text into a neural network model to be trained twice, wherein the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result, and a second discarding layer result; calculating multiple task losses of the neural network model to be trained based on the first prediction processing result, the second prediction processing result, the first discarding layer result, the second discarding layer result, and the actual processing result; calculating a comprehensive loss of the neural network model to be trained based on the multiple task losses; and training the neural network model to be trained based on the comprehensive loss to generate a neural network model.

[0006] According to the second aspect, a text processing method is provided, comprising: obtaining a text to be processed; obtaining a neural network model, wherein the neural network model is trained according to the method for generating a text processing neural network model according to the first aspect of the present disclosure; and inputting the text to be processed into the neural network model to generate a predicted processing result of the text to be processed.

[0007] According to a third aspect, a device for generating a text processing neural network model is provided, comprising: a first acquisition module for acquiring a sample text and an actual processing result corresponding to the sample text; a first generation module for inputting the same sample text into a neural network model to be trained twice, wherein the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result and a second discarding layer result; a first calculation module for calculating multiple task losses of the neural network model to be trained based on the first prediction processing result, the second prediction processing result, the first discarding layer result, the second discarding layer result and the actual processing result; a second calculation module for calculating the comprehensive loss of the neural network model to be trained based on the multiple task losses; and a training module for training the neural network model to be trained based on the comprehensive loss to generate a neural network model.

[0008] According to a fourth aspect, a text processing device is provided, comprising: a second acquisition module for acquiring a text to be processed; a third acquisition module for acquiring a neural network model, wherein the neural network model is trained according to a method for generating a text processing neural network model; and a second generation module for inputting the text to be processed into the neural network model to generate a predicted processing result of the text to be processed.

[0009] According to the fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a text processing neural network model described in the first aspect of the present disclosure or the text processing method described in the second aspect of the present disclosure.

[0010] According to the sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method for generating a text processing neural network model according to the first aspect of the present disclosure or the text processing method according to the second aspect of the present disclosure.

[0011] According to the seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method for generating a text processing neural network model according to the first aspect of the present disclosure or the steps of the text processing method according to the second aspect of the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a flowchart of a method for generating a text processing neural network model according to the first embodiment of the present disclosure;

[0015] Figure 2 It is a flowchart of the application of neural network model in text classification tasks;

[0016] Figure 3 It is a flowchart of the application of neural network model in text regression task;

[0017] Figure 4 is a flowchart of a method for generating a text processing neural network model according to a second embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of a text processing method according to the first embodiment of the present disclosure;

[0019] Figure 6 is a block diagram of a device for generating a text processing neural network model according to a first embodiment of the present disclosure;

[0020] Figure 7 is a block diagram of a device for generating a text processing neural network model according to a second embodiment of the present disclosure;

[0021] Figure 8 is a block diagram of a text processing device for implementing the first embodiment of the present disclosure;

[0022] Fig. 9 It is a block diagram of an electronic device used to implement the method for generating a text processing neural network model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Artificial Intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. At present, AI technology has the advantages of high automation, high precision and low cost, and has been widely used.

[0025] Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It learns the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. In terms of specific research content, it mainly includes neural network systems based on convolution operations, namely convolutional neural networks; autoencoding neural networks based on multi-layer neurons; deep belief networks that are pre-trained in the form of multi-layer autoencoding neural networks and then further optimize the weights of neural networks in combination with identification information. Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as audio-visual and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0026] Data processing (DP) is the collection, storage, retrieval, processing, transformation and transmission of data. The basic purpose of data processing is to extract and derive valuable and meaningful data for certain people from a large amount of data that may be disorganized and difficult to understand. Data processing is a basic link in system engineering and automatic control. Data processing runs through all areas of social production and social life. The development of data processing technology and the breadth and depth of its application have greatly affected the process of human social development.

[0027] The following describes the method, device, system, terminal, electronic device and medium for generating a text processing neural network model according to an embodiment of the present disclosure in conjunction with the accompanying drawings.

[0028] Figure 1 It is a flowchart of a method for generating a text processing neural network model according to the first embodiment of the present disclosure.

[0029] like Figure 1 As shown, the method for generating a text processing neural network model in the embodiment of the present disclosure may specifically include the following steps:

[0030] S101, obtaining sample text and actual processing results corresponding to the sample text.

[0031] Optionally, the execution subject of the method for generating a text processing neural network model of the embodiment of the present disclosure may be the generation device of the text processing neural network model provided by the embodiment of the present disclosure, and the generation device of the text processing neural network model may be a hardware device with data information processing capabilities and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc.

[0032] Specifically, the sample text is the text used to generate the text processing neural network model as a sample, and the actual processing result is the label of the sample text in the text classification task or the true value in the text regression task. The sample text and the actual processing result corresponding to the sample text are obtained for subsequent processing. For example, Figure 2 This is a flowchart of the application of neural network models in text classification tasks. Figure 3 This is a flowchart of the application of neural network models in text regression tasks, such as Figure 2 and Figure 3 As shown, in Figure 2 In the text classification task, sample_a is the sample text, and label_a is the actual processing result of the sample text. Figure 3 In the text regression task, sample_a refers to the sample text, and true value_a refers to the actual processing result of the sample text.

[0033] Optionally, when obtaining sample labels in a text classification task, the sample scores can be calculated by using a neural network model and weight parameters. For example, the sample score can be [label_0_score, label_1_score,…,label_n_score], and the subscript of the maximum score is obtained as the sample label. For example, if label_0_score is the maximum value, the sample label is 0, thereby realizing the acquisition of the sample label.

[0034] S102, inputting the same sample text twice into the neural network model to be trained, wherein the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result and a second discarding layer result.

[0035] Specifically, the neural network model to be trained includes multiple discarding layers and processing layers. The processing layer may specifically include but is not limited to a classification layer and a regression layer. The neural network model to be trained can be applied to text classification tasks and text regression tasks. In the text classification task, the processing layer is a classification layer. The first prediction processing result and the second prediction processing result are the processing results calculated as predictions by the same sample text after two classification layers. The first discarding layer result and the second discarding layer result are the discarding results calculated as predictions by the same sample text after two discarding layers. In the text regression task, the processing layer is a regression layer. The first prediction processing result and the second prediction processing result are the processing results calculated as predictions by the same sample text after two regression layers. The first discarding layer result and the second discarding layer result are the discarding results output by the same sample text after two discarding layers. The same sample text obtained in step S101 is input into the neural network model to be trained twice, and the corresponding first prediction processing result, second prediction processing result, first discarding layer result and second discarding layer result can be generated. For example, as above Figure 2 and Figure 3 As shown, the neural network model to be trained may include N network units with a discard layer, where N is a positive integer greater than 1. Here, we set N to 2 for description. In the following, N is also 2. Figure 2 In , the first prediction processing result and the second prediction processing result are prediction_1a and prediction_1b respectively, and the first discarded layer result and the second discarded layer result are the calculation results of the outputs of the upper and lower network units 1 with discarded layers respectively. Figure 3 In the figure, the first prediction processing result and the second prediction processing result are prediction_1a and prediction_1b respectively, and the first discarded layer result and the second discarded layer result are the calculation results of the outputs of the upper and lower network units 1 with discarded layers respectively.

[0036] It can be understood by those skilled in the art that when the same sample is input into a neural network model twice, due to the existence of a discard layer in the neural network model, the results calculated by each layer will be slightly different. Therefore, according to the principle of contrastive learning, the two calculation results obtained by inputting the same sample into the neural network model can become a pair of positive examples.

[0037] S103, calculating multiple task losses of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, the first discarded layer result, the second discarded layer result and the actual processing result.

[0038] Specifically, the task loss is the loss of multiple tasks of the neural network model to be trained. The multiple task losses of the neural network model to be trained are calculated according to the first prediction processing result, the second prediction processing result, the first discarded layer result and the second discarded layer result generated in step S102 and the actual processing result corresponding to the sample text obtained in step S101.

[0039] S104, calculating the comprehensive loss of the neural network model to be trained according to the multiple task losses.

[0040] Specifically, the comprehensive loss of the neural network model to be trained is calculated according to the multiple task losses of the neural network model to be trained calculated in step S103 for subsequent processing.

[0041] S105, training the neural network model to be trained according to the comprehensive loss to generate a neural network model.

[0042] Specifically, the neural network model to be trained is trained according to the comprehensive loss of the neural network model to be trained calculated in step S104 to generate a neural network model.

[0043] In summary, the method for generating a text processing neural network model of the embodiment of the present disclosure obtains a sample text and an actual processing result corresponding to the sample text, and inputs the same sample text into the neural network model to be trained twice, wherein the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result, and a second discarding layer result, and calculates multiple task losses of the neural network model to be trained based on the first prediction processing result, the second prediction processing result, the first discarding layer result, the second discarding layer result, and the actual processing result, and calculates the comprehensive loss of the neural network model to be trained based on the multiple task losses, and trains the neural network model to be trained based on the comprehensive loss to generate a neural network model. The method for generating a text processing neural network model of the present disclosure inputs the same sample text into the neural network model to be trained twice to generate multiple results, calculates multiple task losses based on the multiple results and calculates the comprehensive loss based on the calculated loss, and finally trains the neural network model to be trained based on the comprehensive loss to generate a neural network model, thereby improving the generalization of the neural network model.

[0044] Figure 4 It is a flowchart of a method for generating a text processing neural network model according to the second embodiment of the present disclosure.

[0045] like Figure 4 As shown, in Figure 1 Based on the illustrated embodiment, the method for generating a text processing neural network model in the embodiment of the present disclosure may specifically include the following steps:

[0046] S401, obtaining sample text and actual processing results corresponding to the sample text.

[0047] S402, inputting the same sample text twice into the neural network model to be trained, wherein the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result and a second discarding layer result.

[0048] It should be noted that steps S401-S402 in this embodiment are the same as steps S101-S102 in the above embodiment, and will not be described in detail here.

[0049] In the above embodiment, step S103 "calculating multiple task losses of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, the first discarded layer result, the second discarded layer result and the actual processing result" may specifically include the following steps S403-S405:

[0050] S403: Calculate the first task loss of the neural network model to be trained according to the first prediction processing result and the second prediction processing result.

[0051] Specifically, the first task loss of the neural network model to be trained is calculated according to the first prediction processing result and the second prediction processing result. When the neural network model to be trained is applied to the text classification task, the first task loss is the divergence loss, and when the neural network model to be trained is applied to the text regression task, the first task loss is the mean square error loss. For example, as above Figure 2 and Figure 3 As shown, in Figure 2 In , the first task loss is the loss of the KL divergence calculated by the two category probabilities output by the same sample text twice, namely prediction_1a and prediction_1b, that is, Loss(KL). Figure 3 In the figure, the first task loss is the loss of the mean square error prediction value calculated by the two category probabilities output by the same sample text after two regression layers, namely prediction_1a and prediction_1b, that is, Loss(MSE_predict).

[0052] S404, calculating a second task loss of the neural network model to be trained according to the first discard layer result and the second discard layer result.

[0053] Specifically, the second task loss of the neural network model to be trained is calculated based on the first discard layer result and the second discard layer result. Wherein, when the neural network model to be trained is applied to text classification tasks and text regression tasks, the second task loss is the mean square error loss. It can be understood by those skilled in the art that, since the output results of the discard layer are often close, the first discard layer result and the second discard layer result are also close. It should be noted that the present disclosure does not specifically limit the calculation method of the second task loss, and it can be set according to actual conditions.

[0054] As a possible implementation, the second task loss is calculated based on the multiple first dropout layer results output by the multiple dropout layers and the multiple second dropout layer results output by the multiple dropout layers. Figure 2 and Figure 3 As shown, in Figure 2 In the example, the same sample text passes through the upper and lower two network units with a drop layer 1 to calculate the corresponding mean square error, which is the first drop layer result Net1 (MSE). The same sample text passes through the upper and lower two network units with a drop layer N to calculate the corresponding mean square error, which is the second drop layer result Net N (MSE), add the first dropout layer result and the second dropout layer result to get the corresponding second task loss, that is, Loss(MSE)=Net1(MSE) + … + Net N (MSE), in Figure 3 In the above example, the same sample text passes through the upper and lower two network units 1 with dropout layers to calculate the corresponding mean square error, that is, the first dropout layer result Net1 (MSE_dropout_lay). The same sample text passes through the upper and lower two network units N with dropout layers to calculate the corresponding mean square error, that is, the second dropout layer result Net1 (MSE_dropout_lay). The sum of the first dropout layer result and the second dropout layer result is the corresponding second task loss, that is, Loss(MSE dropout_lay)=Net1(MSE dropout_lay) + … + Net N (MSE dropout_lay).

[0055] Optionally, the second task loss can be solved by random selection or arithmetic selection method. In the random selection method, the number of random selections can be set to m, and then m layers of network units 1-N with discarded layers are randomly selected. The second task loss is the sum of the mean square errors of the m network units. In the arithmetic selection method, the network units can be selected from 1-N according to a step size of m, and the network units are selected arithmetic with an initial number of k, specifically k, k+m, k+2m…, k+i*m (k+i*m<= N and k+i*m + k>N), etc., and the second task loss is the sum of the mean square errors of the calculation results of i network units. It should be noted here that the specific selection strategy in the random selection or arithmetic selection method can be determined according to the task effect.

[0056] S405, calculating the third task loss of the neural network model to be trained according to the first prediction processing result, the second prediction processing result and the actual processing result.

[0057] Specifically, the first prediction processing result, the second prediction processing result and the actual processing result are used to train the third task loss of the neural network model to be trained. When the neural network model to be trained is applied to the text classification task, the third task loss is the cross entropy loss; when the neural network model to be trained is applied to the text regression task, the third task loss is the mean square error loss. For example, as above Figure 2 and Figure 3 As shown, in Figure 2 In , the cross entropy loss between the first prediction processing result, prediction_1a, the second prediction processing result, prediction_1b, and the actual processing result, label_a, is the third task loss, namely Loss (CE). Figure 3 In , the cross entropy loss between the first prediction processing result, i.e. prediction_1a, the second prediction processing result, i.e. prediction_1b, and the actual processing result, i.e. true value_a, is the third task loss, i.e. Loss(MSE_label).

[0058] In the above embodiment, step S104 “calculating the comprehensive loss of the neural network model to be trained according to multiple task losses” may specifically include the following steps S406-S407:

[0059] S406, obtaining the weight corresponding to the task loss.

[0060] Specifically, the task losses and weights correspond one to one, each task loss has a corresponding weight, and multiple weights corresponding to the multiple task losses calculated in steps S403-S405 are obtained for subsequent processing. For example, as mentioned in the above example, in the text classification task, the first task loss is the divergence loss Loss(KL), the second task loss is the mean square error loss Loss(MSE), and the third task loss is the cross entropy loss Loss(CE). At this time, the weight of Loss(KL) can be a, the weight of Loss(MSE) can be b, and the weight of Loss(CE) can be 1, where a and b are less than 1, and a and b are empirical parameters. After testing in the text classification task, it is found that the task effect is better when a and b are 0.1, which can be obtained by setting hyperparameter traversal; in the text regression task, the first task loss is the mean square error loss Loss(MSE_predict), the second task loss is the mean square error loss and the mean square error loss Loss(MSE_dropout_lay), and the third task loss is the cross entropy loss mean square error loss Loss(MSE_label). At this time, the weight of Loss(MSE_predict) can be a, The weight of Loss (MSE_dropout_lay) can be b, and the weight of Loss (MSE_label) can be 1. The value range and acquisition method of a and b are the same as those described in the text classification task.

[0061] S407, calculating a comprehensive loss according to the multiple task losses and the corresponding multiple weights.

[0062] Specifically, the comprehensive loss of the neural network model to be trained is calculated according to the multiple task losses calculated in steps S403-S405 and the multiple weights corresponding to the multiple task losses obtained in step S406. For example, as described in the above example, in the text classification task, the divergence loss Loss(KL), the mean square error loss Loss(MSE) and the cross entropy loss Loss(CE) and the corresponding weights can be calculated to obtain the comprehensive loss, that is, Loss = Loss(CE) + a* Loss(KL) + b*Loss(MSE); in the text regression task, the mean square error loss Loss(MSE_predict), the mean square error loss Loss(MSE_dropout_lay) and the mean square error loss Loss(MSE_label) and the corresponding weights can be calculated to obtain the comprehensive loss, that is, Loss=Loss(MSE_label) + a*Loss(MSE_predict) + b*Loss(MSE_dropout_lay).

[0063] S408, training the neural network model to be trained according to the comprehensive loss to generate a neural network model.

[0064] Optionally, the neural network model to be trained can be trained in batches, where a batch neural network is calculated twice. In the text classification task, Loss(CE), Loss(MSE), and Loss(KL) are calculated. The overall Loss is obtained according to Loss = Loss(CE) + a* Loss(KL) + b* Loss(MSE) and back-propagated. The gradient dt(w) of Loss with respect to the parameter wi is calculated. The parameters are updated using the stochastic gradient descent algorithm. The wi update logic is w= w-lr*dt(w), where lr is the learning rate. The neural network parameters are updated in sequence until the loss converges, thereby realizing the training of the neural network model. The training process in the text regression task is similar to that in the text classification task, and will not be repeated here.

[0065] It should be noted that step S408 in this embodiment is the same as step S105 in the above embodiment and will not be described again here.

[0066] In summary, the method for generating a text processing neural network model of the embodiment of the present disclosure obtains a sample text and an actual processing result corresponding to the sample text, inputs the same sample text into the neural network model to be trained twice, and the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result, and a second discarding layer result, and calculates the first task loss of the neural network model to be trained according to the first prediction processing result and the second prediction processing result, and calculates the second task loss of the neural network model to be trained according to the first discarding layer result and the second discarding layer result, and calculates the third task loss of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, and the actual processing result, and obtains the weight corresponding to the task loss, and calculates the comprehensive loss according to the multiple task losses and the corresponding multiple weights, and trains the neural network model to be trained according to the comprehensive loss to generate a neural network model. The method for generating a text processing neural network model of the present disclosure inputs the same sample text into the neural network model to be trained twice to generate multiple results, calculates multiple task losses according to the multiple results and calculates the comprehensive loss according to the calculated loss, and finally trains the neural network model to be trained according to the comprehensive loss to generate a neural network model, thereby improving the generalization of the neural network model. At the same time, by obtaining the weights corresponding to the task loss to calculate the comprehensive loss, the output of the same sample text under different discard layers is basically the same, which improves the performance and stability of the neural network model.

[0067] Figure 5 A flowchart of a text processing method according to the first embodiment of the present disclosure. Figure 5 As shown, the text processing method of the embodiment of the present disclosure may specifically include the following steps:

[0068] S501, obtaining the text to be processed.

[0069] Specifically, the text to be processed is the target text waiting for text processing, and the text to be processed is obtained for subsequent processing.

[0070] S502, obtaining a neural network model.

[0071] Specifically, the neural network model is obtained by training the neural network model to be trained according to the method for generating the text processing neural network model in the above embodiment, and the neural network model is acquired for subsequent processing.

[0072] S503, inputting the text to be processed into the neural network model to generate a prediction processing result of the text to be processed.

[0073] Specifically, the text to be processed obtained in step S501 is input into the neural network model obtained in step S502 to generate a prediction processing result of the text to be processed.

[0074] It should be noted that the above explanation of the embodiment of the method for generating a text processing neural network model is also applicable to the text processing method of the embodiment of the present disclosure, and the specific process will not be repeated here.

[0075] In summary, the text processing method of the embodiment of the present disclosure obtains the text to be processed and the neural network model, and inputs the text to be processed into the neural network model to generate a predicted processing result of the text to be processed. The text processing method of the present disclosure improves the accuracy of text processing by inputting the text to be processed into the neural network model to generate a predicted processing result of the text to be processed.

[0076] Figure 6 It is a block diagram of a device for generating a text processing neural network model according to the first embodiment of the present disclosure.

[0077] like Figure 6 As shown, the device 600 for generating a text processing neural network model according to an embodiment of the present disclosure includes: a first acquisition module 601, a first generation module 602, a first calculation module 603, a second calculation module 604 and a training module 605.

[0078] The first acquisition module 601 is used to acquire sample text and actual processing results corresponding to the sample text.

[0079] The first generation module 602 is used to input the same sample text twice into the neural network model to be trained, and the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result and a second discarding layer result.

[0080] The first calculation module 603 is used to calculate multiple task losses of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, the first discarded layer result, the second discarded layer result and the actual processing result.

[0081] The second calculation module 604 is used to calculate the comprehensive loss of the neural network model to be trained according to multiple task losses.

[0082] The training module 605 is used to train the neural network model to be trained according to the comprehensive loss to generate a neural network model.

[0083] It should be noted that the above explanation of the embodiment of the method for generating a text processing neural network model is also applicable to the device for generating a text processing neural network model in the embodiment of the present disclosure, and the specific process will not be repeated here.

[0084] In summary, the device for generating a text processing neural network model of the embodiment of the present disclosure obtains a sample text and an actual processing result corresponding to the sample text, and inputs the same sample text into the neural network model to be trained twice, wherein the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result, and a second discarding layer result, and calculates multiple task losses of the neural network model to be trained based on the first prediction processing result, the second prediction processing result, the first discarding layer result, the second discarding layer result, and the actual processing result, and calculates the comprehensive loss of the neural network model to be trained based on the multiple task losses, and trains the neural network model to be trained based on the comprehensive loss to generate a neural network model. The device for generating a text processing neural network model of the present disclosure inputs the same sample text into the neural network model to be trained twice to generate multiple results, calculates multiple task losses based on the multiple results and calculates the comprehensive loss based on the calculated loss, and finally trains the neural network model to be trained based on the comprehensive loss to generate a neural network model, thereby improving the generalization of the neural network model.

[0085] Figure 7 It is a block diagram of a device for generating a text processing neural network model according to a second embodiment of the present disclosure.

[0086] like Figure 7 As shown, the device 700 for generating a text processing neural network model according to an embodiment of the present disclosure includes: a first acquisition module 701, a first generation module 702, a first calculation module 703, a second calculation module 704 and a training module 705.

[0087] Among them, the first acquisition module 701 has the same structure and function as the first acquisition module 601 in the previous embodiment, the first generation module 702 has the same structure and function as the first generation module 602 in the previous embodiment, the first calculation module 703 has the same structure and function as the first calculation module 603 in the previous embodiment, the second calculation module 704 has the same structure and function as the second calculation module 604 in the previous embodiment, and the training module 705 has the same structure and function as the training module 605 in the previous embodiment.

[0088] Furthermore, the first calculation module 703 may specifically include: a first calculation unit 7031, used to calculate the first task loss of the neural network model to be trained according to the first prediction processing result and the second prediction processing result; a second calculation unit 7032, used to calculate the second task loss of the neural network model to be trained according to the first discarded layer result and the second discarded layer result; and a third calculation unit 7033, used to calculate the third task loss of the neural network model to be trained according to the first prediction processing result, the second prediction processing result and the actual processing result.

[0089] Furthermore, when the neural network model to be trained is applied to a text classification task, the first task loss is the divergence loss, the second task loss is the mean square error loss, and the third task loss is the cross entropy loss.

[0090] Furthermore, the neural network model to be trained is applied to the text regression task, and the first task loss, the second task loss and the third task loss are all mean square error losses.

[0091] Furthermore, the second calculation unit 7032 includes: a calculation subunit, which is used to calculate the second task loss according to multiple first discarding layer results output by multiple discarding layers and multiple second discarding layer results output by multiple discarding layers.

[0092] Furthermore, the second calculation module 704 includes: an acquisition unit, used to acquire the weight corresponding to the task loss; and a fourth calculation unit, used to calculate the comprehensive loss according to multiple task losses and corresponding multiple weights.

[0093] It should be noted that the above explanation of the embodiment of the method for generating a text processing neural network model is also applicable to the device for generating a text processing neural network model in the embodiment of the present disclosure, and the specific process will not be repeated here.

[0094] In summary, the device for generating a text processing neural network model of the embodiment of the present disclosure obtains a sample text and an actual processing result corresponding to the sample text, inputs the same sample text into the neural network model to be trained twice, and the neural network model to be trained includes multiple discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result, and a second discarding layer result, and calculates the first task loss of the neural network model to be trained according to the first prediction processing result and the second prediction processing result, and calculates the second task loss of the neural network model to be trained according to the first discarding layer result and the second discarding layer result, and calculates the third task loss of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, and the actual processing result, and obtains the weight corresponding to the task loss, and calculates the comprehensive loss according to the multiple task losses and the corresponding multiple weights, and trains the neural network model to be trained according to the comprehensive loss to generate a neural network model. The device for generating a text processing neural network model of the present disclosure inputs the same sample text into the neural network model to be trained twice to generate multiple results, calculates multiple task losses according to the multiple results and calculates the comprehensive loss according to the calculated loss, and finally trains the neural network model to be trained according to the comprehensive loss to generate a neural network model, thereby improving the generalization of the neural network model. At the same time, by obtaining the weights corresponding to the task loss to calculate the comprehensive loss, the output of the same sample text under different discard layers is basically the same, which improves the performance and stability of the neural network model.

[0095] Figure 8 is a block diagram of a text processing apparatus according to a first embodiment of the present disclosure.

[0096] like Figure 8 As shown, the text processing device 800 of the embodiment of the present disclosure includes: a second acquisition module 801, a third acquisition module 802 and a second generation module 803.

[0097] The second acquisition module 801 is used to acquire the text to be processed.

[0098] The third acquisition module 802 is used to acquire a neural network model, where the neural network model is trained according to a method for generating a text processing neural network model.

[0099] The second generating module 803 is used to input the text to be processed into the neural network model to generate a prediction processing result of the text to be processed.

[0100] It should be noted that the above explanation of the text processing method embodiment is also applicable to the text processing device of the embodiment of the present disclosure, and the specific process will not be repeated here.

[0101] In summary, the text processing device of the embodiment of the present disclosure obtains the text to be processed and the neural network model, and inputs the text to be processed into the neural network model to generate a predicted processing result of the text to be processed. The text processing method of the present disclosure improves the accuracy of text processing by inputting the text to be processed into the neural network model to generate a predicted processing result of the text to be processed.

[0102] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0103] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0104] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0105] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0106] A number of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as Figures 1 to 4 The method for generating a text processing neural network model as shown or as Figure 5 The text processing method shown. For example, in some embodiments, the method for generating a text processing neural network model or the text processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method for generating a text processing neural network model or the text processing method described above may be executed. Alternatively, in other embodiments, the computing unit 901 may be used to execute the method for generating a text processing neural network model or the text processing method in any other appropriate manner (e.g., by means of firmware).

[0108] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable text processing neural network model generation device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0113] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the method for generating a text processing neural network model according to the above embodiment of the present disclosure or implements the steps of the text processing method according to the above embodiment of the present disclosure.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a text processing neural network model, comprising: Obtaining sample text and actual processing results corresponding to the sample text; Inputting the same sample text twice into a neural network model to be trained, wherein the neural network model to be trained includes a plurality of discarding layers and processing layers to generate a first prediction processing result, a second prediction processing result, a first discarding layer result, and a second discarding layer result, wherein the processing layer includes a classification layer and a regression layer; Calculating multiple task losses of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, the first discarded layer result, the second discarded layer result and the actual processing result; Calculating the comprehensive loss of the neural network model to be trained according to the multiple task losses; and Training the neural network model to be trained according to the comprehensive loss to generate a neural network model; The step of calculating the multiple task losses of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, the first discarded layer result, the second discarded layer result and the actual processing result includes: Calculating a first task loss of the neural network model to be trained according to the first prediction processing result and the second prediction processing result; Calculating a second task loss of the neural network model to be trained according to the first discarded layer result and the second discarded layer result; and According to the first prediction processing result, the second prediction processing result and the actual processing result, the third task loss of the neural network model to be trained is calculated.

2. The generation method according to claim 1, wherein: The neural network model to be trained is applied to a text classification task, then the first task loss is a divergence loss, the second task loss is a mean square error loss, and the third task loss is a cross entropy loss.

3. The generation method according to claim 1, wherein: The neural network model to be trained is applied to a text regression task, and the first task loss, the second task loss and the third task loss are all mean square error losses.

4. The generation method according to claim 1, wherein: The calculating the second task loss of the neural network model to be trained according to the first discarded layer result and the second discarded layer result includes: The second task loss is calculated according to the multiple first discarding layer results output by the multiple discarding layers and the multiple second discarding layer results output by the multiple discarding layers.

5. The generation method according to claim 1, wherein: The calculating the comprehensive loss of the neural network model to be trained according to the multiple task losses includes: Obtaining a weight corresponding to the task loss; and The comprehensive loss is calculated according to the multiple task losses and the corresponding multiple weights.

6. A text processing method, comprising: Get the text to be processed; Obtain a neural network model, wherein the neural network model is trained by the generation method according to any one of claims 1 to 5; The text to be processed is input into the neural network model to generate a predicted processing result of the text to be processed.

7. A device for generating a text processing neural network model, comprising: A first acquisition module, used to acquire a sample text and an actual processing result corresponding to the sample text; A first generating module, used for inputting the same sample text into a neural network model to be trained twice, wherein the neural network model to be trained includes a plurality of discarding layers and processing layers, so as to generate a first prediction processing result, a second prediction processing result, a first discarding layer result and a second discarding layer result, wherein the processing layer includes a classification layer and a regression layer; A first calculation module, used for calculating multiple task losses of the neural network model to be trained according to the first prediction processing result, the second prediction processing result, the first discarded layer result, the second discarded layer result and the actual processing result; a second calculation module, configured to calculate a comprehensive loss of the neural network model to be trained according to the multiple task losses; and A training module, used for training the neural network model to be trained according to the comprehensive loss to generate a neural network model; Wherein, the first calculation module includes: A first calculation unit, used for calculating a first task loss of the neural network model to be trained according to the first prediction processing result and the second prediction processing result; A second calculation unit is used to calculate a second task loss of the neural network model to be trained according to the first discarded layer result and the second discarded layer result; and A third calculation unit is used to calculate the third task loss of the neural network model to be trained according to the first prediction processing result, the second prediction processing result and the actual processing result.

8. The generating device according to claim 7, wherein: The neural network model to be trained is applied to a text classification task, then the first task loss is a divergence loss, the second task loss is a mean square error loss, and the third task loss is a cross entropy loss.

9. The generating device according to claim 7, wherein: The neural network model to be trained is applied to a text regression task, and the first task loss, the second task loss and the third task loss are all mean square error losses.

10. The generating device according to claim 7, wherein: The second computing unit comprises: A calculation subunit is used to calculate the second task loss according to the multiple first discarding layer results output by the multiple discarding layers and the multiple second discarding layer results output by the multiple discarding layers.

11. The generating device according to claim 7, wherein: The second calculation module includes: An acquisition unit, used to acquire a weight corresponding to the task loss; and A fourth calculation unit is used to calculate the comprehensive loss according to the multiple task losses and the corresponding multiple weights.

12. A text processing device, comprising: The second acquisition module is used to acquire the text to be processed; A third acquisition module, used to acquire a neural network model, wherein the neural network model is trained by the generation device according to any one of claims 7 to 11; The second generating module is used to input the text to be processed into the neural network model to generate a prediction processing result of the text to be processed.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 5, or execute the method according to any one of claim 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 5, or to execute the method according to any one of claims 6.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6.

Images