Model training, text information processing method, system, device and storage medium

Abstract

The application discloses a text information processing model training method, a text information processing method, a system, a device and a storage medium, which can be applied in the field of artificial intelligence. The training method comprises the following steps: encoding the first training text to obtain first semantic information; encoding the first candidate word to obtain second semantic information; predicting the classification of the first training text according to the first semantic information to obtain a first classification result; predicting the weight of the first candidate word according to the first semantic information and the second semantic information; determining a first loss value according to the first classification result and the type label of the first training text; determining a second loss value according to the weight of the first candidate word and the weight label of the first candidate word; and training the text information processing model according to the first loss value and the second loss value. The application can reduce the training cost or obtain higher model precision under the condition of the same training cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text information processing model training method, a text information processing method, a system, a device and a storage medium. BACKGROUND

[0002] At present, the word weight task is mostly completed by using two ways of statistics and text classification, wherein the statistics is an unsupervised way, and representative ways are TF-IDF (term frequency-inverse document frequency, a commonly used weighting technique for information retrieval and data mining) and mutual information MI (mutual information, a commonly used information measure in information theory). The supervised way is represented by deep learning. Compared with the statistical way, the deep learning model can learn the relationship between the key words and the sentences, and thus is more accurate in the task of key word extraction.

[0003] However, a large number of training samples are needed to train the deep learning model, and these training samples need to be manually labeled. For key word labeling, the difficulty is relatively large, the labeling efficiency is relatively low, and thus the cost is relatively high. The key degree of the same word is different in different contexts. Therefore, if a model that can accurately predict the weight of the word in the sentence in different contexts is needed, a large number of training samples are needed, and the cost is relatively high. SUMMARY

[0004] To solve at least one of the above technical problems, the present application provides a text information processing model training method, a text information processing method, a system, a device and a storage medium to reduce the training cost or make the model have higher accuracy under the condition of the same training cost.

[0005] According to an aspect of the present application, a text information processing model training method is provided, comprising the following steps:

[0006] obtaining a first training text, a first candidate word in the first training text, a type label of the first training text and a weight label of the first candidate word;

[0007] encoding the first training text to obtain first semantic information;

[0008] encoding the first candidate word to obtain second semantic information;

[0009] predicting the classification of the first training text according to the first semantic information to obtain a first classification result;

[0010] predict a weight of the first candidate word according to the first semantic information and the second semantic information;

[0011] determine a first loss value according to the first classification result and a type label of the first training text;

[0012] determine a second loss value according to the weight of the first candidate word and a weight label of the first candidate word;

[0013] train the text information processing model according to the first loss value and the second loss value.

[0014] According to another aspect of the present application, a training method of a text information processing model is provided, the text information processing model comprising a first encoder, a second encoder, a classifier, a feature fusion unit and an activation unit, the method comprising the following steps:

[0015] obtaining a second training text, a second candidate word in the second training text, a type label of the second training text and a weight label of the second candidate word;

[0016] encoding the second training text by the first encoder to obtain third semantic information;

[0017] encoding the second candidate word by the second encoder to obtain fourth semantic information;

[0018] classifying and predicting the third semantic information by the classifier to obtain a second classification result of the second training text;

[0019] extracting features of the third semantic information and the fourth semantic information by the feature fusion unit to obtain fifth semantic information;

[0020] activating the fifth semantic information by the activation unit to obtain a weight of the second candidate word;

[0021] determining a third loss value according to the second classification result and the type label of the second training text;

[0022] determining a fourth loss value according to the weight of the second candidate word and the weight label of the second candidate word;

[0023] updating parameters of the first encoder, the second encoder and the feature fusion unit according to the third loss value and the fourth loss value.

[0024] According to another aspect of the present application, a text information processing method is provided, comprising the following steps:

[0025] obtaining an input text and a third candidate word in the input text;

[0026] inputting the input text and the third candidate word input text information processing model to obtain the weight of the third candidate word in the input text; wherein the text information processing model is trained by the method described above.

[0027] According to another aspect of the present application, a training system of a text information processing model is provided, comprising:

[0028] a first obtaining unit configured to obtain a first training text, a first candidate word in the first training text, a type label of the first training text, and a weight label of the first candidate word;

[0029] a first encoding unit configured to encode the first training text to obtain first semantic information;

[0030] a second encoding unit configured to encode the first candidate word to obtain second semantic information;

[0031] a classification unit configured to predict the classification of the first training text according to the first semantic information to obtain a first classification result;

[0032] a word weight prediction unit configured to predict the weight of the first candidate word according to the first semantic information and the second semantic information;

[0033] a first loss value determination unit configured to determine a first loss value according to the first classification result and the type label of the first training text;

[0034] a second loss value determination unit configured to determine a second loss value according to the weight of the first candidate word and the weight label of the first candidate word;

[0035] a training unit configured to train the text information processing model according to the first loss value and the second loss value.

[0036] According to another aspect of the present application, a training system of a text information processing model is provided, the text information processing model comprising a first encoder, a second encoder, a classifier, a feature fusion unit, and an activation unit, the system comprising:

[0037] a third obtaining unit configured to obtain a second training text, a second candidate word in the second training text, a type label of the second training text, and a weight label of the second candidate word;

[0038] a third encoding unit configured to encode the second training text by the first encoder to obtain third semantic information;

[0039] a fourth encoding unit, configured to encode the second candidate word by using a second encoder to obtain fourth semantic information;

[0040] a text classification unit, configured to perform classification prediction on the third semantic information by using the classifier to obtain a second classification result of the second training text;

[0041] a fusion feature extraction unit, configured to perform feature extraction on the third semantic information and the fourth semantic information by using the feature fusion unit to obtain fifth semantic information;

[0042] an activation processing unit, configured to perform activation on the fifth semantic information by using the activation unit to obtain a weight of the second candidate word;

[0043] a third loss determination unit, configured to determine a third loss value according to the second classification result and a type label of the second training text;

[0044] a fourth loss determination unit, configured to determine a fourth loss value according to the weight of the second candidate word and a weight label of the second candidate word;

[0045] a parameter training unit, configured to update parameters of the first encoder, the second encoder and the feature fusion unit according to the third loss value and the fourth loss value.

[0046] According to an aspect of the present application, a text information processing system is provided, comprising:

[0047] a second acquisition unit, configured to acquire an input text and a third candidate word in the input text;

[0048] a weight prediction unit, configured to input the input text and the third candidate word into a text information processing model to obtain a weight of the third candidate word in the input text; wherein the text information processing model is trained by the text information processing method.

[0049] According to another aspect of the present application, a data processing apparatus is provided, comprising:

[0050] a program;

[0051] a memory, configured to store the program;

[0052] a processor, configured to load the program to execute the text information processing method or the training method.

[0053] According to another aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores a program, and the program is executed by a processor to implement the text information processing method.

[0054] The beneficial effect of the embodiments of the present application is that, in the training stage, the model is trained on two tasks, text classification and word weight, using the type label of the text and the weight label of the candidate word, and the loss of the two tasks is used to update the model, so that the model can not only learn the classification features of the text, but also learn the weight features of the text under the classification. Compared with the prior art method of training only using keyword labels, the embodiments of the present application can enable the model to learn the relationship between the weight of the keyword and the text type more quickly. Therefore, while achieving similar accuracy, fewer training samples are required. Since the annotator needs to read the text when annotating keywords, determining the text type does not lead to a significant increase in the annotator's annotation time. Therefore, overall, the training cost is reduced. Similarly, under the same training cost, the model trained by the embodiments of the present application has higher accuracy than the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a method for training a text information processing model according to an embodiment of the present application;

[0056] Figure 2 A block diagram of a system module provided according to an embodiment of the present application and applied to each embodiment of the present application;

[0057] Figure 3 A structural block diagram of a text information processing model provided according to an embodiment of the present application;

[0058] Figure 4a A structural block diagram of another text information processing model provided according to an embodiment of the present application; Figure 4b A structural block diagram of another text information processing model provided according to an embodiment of the present application;

[0059] Figure 5 A flowchart of sub-steps of step 150 provided according to an embodiment of the present application;

[0060] Figure 6 A flowchart of a sub-step of step 120 provided according to an embodiment of the present application;

[0061] Figure 7 This is a structural block diagram of a first encoder provided according to an embodiment of the present application;

[0062] Figure 8 A schematic diagram of the processing flow of a self-attention layer provided according to an embodiment of the present application.

[0063] Figure 9 This is another sub-step flowchart of step 120 provided according to an embodiment of the present application;

[0064] Figure 10 A structural schematic diagram of another first encoder provided according to an embodiment of the application is provided.

[0065] Figure 11 A sub-step flowchart of step 920 provided according to an embodiment of the application is provided.

[0066] Figure 12 A flowchart of a training method of a text information processing model provided according to an embodiment of the application is provided.

[0067] Figure 13 A flowchart of a text information processing method provided according to an embodiment of the application is provided.

[0068] Figure 14a An interface schematic diagram of a communication software discovery page provided according to an embodiment of the application is provided. Figure 14b A schematic diagram of a communication software video number interface provided according to an embodiment of the application is provided.

[0069] Figure 15 A schematic diagram of a video search result page provided according to an embodiment of the application is provided.

[0070] Figure 16a 、 Figure 16b A schematic diagram of another video search result interface provided according to an embodiment of the application is provided.

[0071] Figure 17 A module block diagram of a training system of a text information processing model provided according to an embodiment of the application is provided.

[0072] Figure 18 A module block diagram of a text information processing system provided according to an embodiment of the application is provided.

[0073] Figure 19 A module block diagram of a server provided according to an embodiment of the application is provided.

[0074] Figure 20 A module block diagram of a mobile phone provided according to an embodiment of the application is provided.

[0075] Figure 21 A module block diagram of another training system of a text information processing model provided according to an embodiment of the application is provided. DETAILED DESCRIPTION

[0076] In the following description, “some embodiments” are referred to, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0078] Before the embodiments of the present application are further described, the terms and names involved in the embodiments of the present application are explained, and the terms and names involved in the embodiments of the present application are applicable to the following explanations.

[0079] Artificial Intelligence (AI) is the use of digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other major directions.

[0080] Nature Language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it has a close relationship with the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies.

[0081] Long Short-Term Memory (LSTM), Bidirectional Encoder Representation from Transformer (BERT), word2vec, OpenAI-GPT, Recurrent Neural Networks (RNN), Embeddings from Language Models (ELMo) are models in the field of natural language technology.

[0082] Attention Mechanism is essentially derived from human visual attention mechanism. When people perceive things, they usually do not look at everything from beginning to end every time, but often observe a specific part according to the needs. Moreover, when people find that a scene often has the things they want to observe in a certain part, people will learn to put their attention on that part in the future when similar scenes appear. Self-attention Mechanism is an improved type of Attention Mechanism, which does not rely on time sequence when processing attention weight, can be processed in parallel, and is more efficient.

[0083] Semantic space: a world representing the meaning of language. In natural language processing technology, the meaning of a word, a sentence or a text in the semantic space is usually represented by a multi-dimensional vector. In the semantic space, the meanings expressed by two vectors with closer distance are more similar.

[0084] Text refers to the form of written language, which is usually a complete and systematic sentence or a combination of multiple sentences. A text can be a sentence, a paragraph or a chapter. In this application, a text can also be understood as a sentence fragment composed of multiple words.

[0085] In related art, to identify keywords in a sentence, the sentence is broken down into multiple candidate words. A trained neural network model is then used to predict the weight of each candidate word in the sentence. The weight of each word is then used to determine whether it is a keyword. Related art uses sentences labeled with keywords for training. This training method allows the neural network model to learn which candidate words have the highest weight in the sentence. However, in real-world applications, the same word may have different levels of importance in different contexts. For example, the word "mobile phone" may be more important in the digital product category than in the game category. However, it is very difficult for the model to learn the category of a sentence based on the keyword labels in the sentence. Therefore, in order for the model to distinguish the importance of candidate words in different contexts, a large amount of training data is required. This training data is often manually annotated, and the task of annotating keywords in sentences is relatively complex. Data workers need to understand the sentence and annotate each word individually, which is time-consuming and incurs significant labor costs. At the same time, as the number of training samples in the training set increases, the time and computation required to train the model also increase successively. Therefore, the required computing cost (graphics card equipment, etc.) and time cost are relatively high.

[0086] Therefore, the embodiment of the present application proposes a training method for a text information processing model, which utilizes the type label of the text and the keyword label of the text to jointly train the model. Since the type label of the text is relatively easy to obtain, when the data worker annotates the keywords of the text, the classification of the text can basically be determined, so the annotation time of the data worker will basically not be increased. Therefore, the model is trained on multiple tasks using the type label of the text and the weight label of the candidate word, so that the model can not only learn the classification features of the text, but also learn the weight features of the words under the text classification. Compared with the related art, the model can learn the relationship between the weight of the keyword and the text type more quickly. Therefore, when achieving similar accuracy, fewer training samples are required. Since the annotator needs to read the text when annotating the keyword, determining the text type does not lead to a significant increase in the annotation time of the annotator. Therefore, overall, the training cost is reduced. Similarly, under the same training cost, the model trained by the embodiment of the present application has higher accuracy than the prior art.

[0087] Reference Figure 1 This embodiment discloses a training method for a text information processing model. The method in this embodiment can be used in Figure 2 In the system shown, Figure 2The system includes a first server 210, a second server 220 and a user terminal 230. The training method of the embodiment is implemented in the first server 210, and the model trained can be deployed in the second server 220 for access and calling by the user terminal 230. Of course, the model trained by the server 210 can also be deployed in the user terminal 230, and the user terminal 230 processes the text information by using the trained model, and then sends the processing result to the second server 220 for data query and other processing. It should be understood that the first server 210 and the server 220 can be two functional modules of the same server. For either of the first server 210 and the server 220, it can be an independent physical server or a distributed server. The user terminal 230 can be a device such as a mobile phone, a personal computer, a notebook computer, a tablet computer or a vehicle-mounted terminal. In the system, Figure 2 In the system, the devices can communicate through the Internet or a local area network.

[0088] The method of the embodiment includes the following steps:

[0089] Step 110, obtaining a first training text, a first candidate word in the first training text, a type label of the first training text and a weight label of the first candidate word.

[0090] Before starting the training, first obtain a training sample set, and the training sample set includes multiple groups of training samples. Each group of training samples includes a training text, a candidate word in the training text, a type label of the training text and a weight label of the candidate word. For example, a certain group of training samples is {“today the weather is fine”, “fine”, 3, 1}. In this example, “today the weather is fine” is the training text, “fine” is the candidate word, 3 represents the type of the training text (for example, representing the weather classification), and 1 represents that the word is a key word (0 represents a non-key word). In this step, a group of training samples is obtained to train the model.

[0091] Step 120, encoding the first training text to obtain first semantic information.

[0092] In this embodiment, the training text is encoded by an encoder, which can map the first training text into a semantic space to form first semantic information of the first training text in the semantic space. The first semantic information can be represented in the form of a vector, a matrix or a tensor, etc. It should be understood that the encoder itself can also be a neural network, and the semantic information extracted by the encoder can be adjusted by training the encoder. In this embodiment, since the lengths of the training texts can be different, the encoder can be padded with meaningless characters at the input layer to make the lengths of the information entering the encoder the same, without affecting the semantic information of the training text. Of course, whether the training text needs to be padded depends on the structure of the encoder. In the encoder of this step, the length of the output semantic information is fixed. In this embodiment, the encoder can be implemented by using an LSTM model, a BSRT model, an OpenAI-GPT model, an RNN model or an ELMo model, etc.

[0093] Step 130, encoding the first candidate word to obtain second semantic information.

[0094] In this step, the encoder is used to encode the candidate word to extract the second semantic information of the first candidate word in the semantic space. It should be understood that the encoder in this step can be the same as the encoder in step 120 or can be different. When the encoder in this step and the encoder in step 120 are the same, the first training text and the first candidate word can be input side by side. It can be understood that the first input text and the first candidate text can also be input in a time-sharing manner. When the encoder in this step and the encoder in step 120 are different, the model of this embodiment is as shown in Figure 3 Figure 3 In the model, the first training text is encoded by a first encoder 310, and the candidate word is encoded by a second encoder 320. The first encoder 310 and the second encoder 320 can be the same type of encoder or different types of encoder. For example, the first encoder 310 can be a BERT model, and the second encoder 320 can be a general CNN model. For the encoding of the training text, the encoding complexity of the sentence is higher, and the information of each word in the sentence and the relationship information between the words need to be extracted, so the first encoder can select a model with a relatively complex network structure, and for the extraction of the semantic information of a single word, a relatively simple network model can be used, which is conducive to simplifying the network and reducing the training cost.

[0095] When the encoder in this step and the encoder in step 120 are the same, the model structure can be as shown in Figure 4a or​Figure 4b As shown. Among them, for Figure 4a For the model in , the length of the input layer of the third encoder 410 is equal to the sum of the maximum length of the text and the length of the word. For example, if the upper limit of the sentence is 100 characters and the candidate word is 3 characters, then the input length of the third encoder 410 is 103 characters. The output of the third encoder 410 consists of two parts, one for representing the first semantic information and the other for representing the second semantic information. Figure 4b As shown, the length of the input layer of the fourth encoder 420 is equal to the maximum length limit of the text. In this model, the fourth encoder is used to encode the sentence and candidate words separately through time-division multiplexing, successively obtaining the first semantic information and the second semantic information. The first and second semantic information are then input into the next level network for processing.

[0096] Step 140: Predict the classification of the first training text according to the first semantic information to obtain a first classification result.

[0097] Reference Figure 3 In this step, the first semantic information can be classified by a classifier 350, such as a softmax classifier, to obtain a first classification result. It should be understood that the length of the first semantic information can be the same as the input length of the softmax classifier. When the length of the first semantic information is different from the input length of the classifier, a fully connected layer can be set between the encoder and the classifier to adjust the length of the information input to the classifier. Taking the softmax classifier as an example, the output result is the probability that the first training text belongs to each category. For example, the softmax classifier outputs a three-dimensional vector to represent the classification results of the text under the three categories of {weather, shopping, food}. The value of each dimension in the vector represents the probability that the text belongs to a category. Assuming that the output result is {0.12, 0.68, 0.2}, it means that the probability that the text belongs to the weather category is 0.12, the probability that it belongs to the shopping category is 0.68, and the probability that it belongs to the food category is 0.2.

[0098] Step 150: predict the weight of the first candidate word according to the first semantic information and the second semantic information.

[0099] Reference Figure 3In this embodiment, the first semantic information and the second semantic information are extracted by a feature fusion layer 330, the weight information of the first candidate word in the first training text is extracted, then an activation unit 340 is used for activation, and finally a value representing the weight of the first candidate word in the first training text is output. In this embodiment, a multi-layer neural network can be used as the feature fusion layer 330 for feature extraction, and a Sigmoid function can be used as the activation unit 340. The Sigmoid function outputs a value between 0 and 1, and thus can be used to represent the weight of the candidate word in the text. It should be understood that the first semantic information and the second semantic information can be input to the feature fusion layer 330 in a concatenation manner or after being processed by weighting.

[0100] Step 160: determining a first loss value according to the first classification result and the type label of the first training text.

[0101] In this embodiment, taking the classification task as {weather, shopping, food} as an example, it is assumed that the text type label of the first training text is weather, and the label is represented as {1, 0, 0}. It is assumed that the first classification result obtained by step 150 is {0.8, 0.1, 0.1}. The loss is determined by calculating the difference between the label and the first classification result. For example, the first loss value is wherein, represents the n-th dimension value of the classifier output vector, y n represents the n-th dimension value of the label. In the above example, L1=a*[0.2 2 +(-0.1) 2 +(-0.1) 2 Of course, the loss function used in this step can also be a log-likelihood loss function, a cross-entropy loss function, or an exponential loss function, etc.

[0102] Step 170: determining a second loss value according to the weight of the first candidate word and the weight label of the first candidate word.

[0103] In this step, the weight label of the first candidate word can be represented in the following manner: if the word is a key word in the sentence, the weight label is 1, and if it is not a key word, the weight label is 0. In this embodiment, the absolute value of the difference between the weight of the first candidate word and the label can be used as the loss value. Of course, the loss function used in this step can also be a log-likelihood loss function, a cross-entropy loss function, or an exponential loss function, etc.

[0104] Step 180: training the text information processing model according to the first loss value and the second loss value.

[0105] In this step, the total loss is calculated according to the two loss values, and then the network parameters of the model are updated using the backpropagation (BP) algorithm according to the total loss. Through the training of multiple sets of training samples, the final model can reach a certain condition. The condition can be convergence on the validation set or the number of training times reaching a predetermined number.

[0106] This embodiment takes the loss function for calculating the first loss value as the log-likelihood loss function and the loss function for calculating the second loss value as the cross-entropy loss function as an example to describe the calculation method of the total loss. In this example, the total loss is: Loss = λ1loss cls + λ2loss term-weights , wherein, λ1 and λ2 are hyperparameters, y i represents the label value of the i-th dimension, a i represents the i-th dimension prediction value. For example, the output vector is a 3-dimensional vector, so the value range of i is {1, 2, 3}, n = 3. If the output result is only one dimension, i = n = 1. It should be noted that loss cls is used to calculate the loss value of the first classification result, loss term-weights is used to calculate the loss value of the candidate word weight.

[0107] Based on the analysis of the above embodiments, it can be known that the model is trained by the type label of the text and the keyword label of the text in this embodiment, so that the model can learn not only the classification features of the text, but also the weight features of the text under the classification. Therefore, the model can learn the relationship between the type of the text and the weight of the candidate word more quickly. Under the condition of reaching similar accuracy, fewer training samples are required. When labeling samples, the text type is easier to obtain, so the overall labeling cost is lower. At the same time, the training sample is less, which can reduce the training time of the model and occupy less computing resources. Similarly, it can be known that the model accuracy obtained by the above embodiments is higher under the same training cost.

[0108] Referring to Figure 5 In this embodiment, the first semantic information is represented by a first vector V1, and the second semantic information is represented by a second vector V2.

[0109] Step 150, predicting the weight of the first candidate word according to the first semantic information and the second semantic information, comprising:

[0110] Step 510, splicing the first vector V1 and the second vector V2 to obtain a third vector V3.

[0111] Taking the BERT model of the first encoder as an example, the output first vector V1 is a 768-dimensional vector, which is used to represent the semantic information of the text. The length of the second vector V2 can also be determined according to the network structure of the selected encoder, for example, in this embodiment, a 100-dimensional vector can be selected to represent the second vector V2. In this embodiment, the two vectors are combined into a third vector V3 in a splicing manner, and in this example, the length of the third vector V3 is 100+768=868.

[0112] Step 520, according to the third vector, predicting the probability that the first candidate word is the key word of the first training text.

[0113] Referring to Figure 3 In this embodiment, the third vector is extracted by the feature fusion layer 330, and then activated by the activation unit 340 (Sigmoid function is used in this embodiment) to output a value between 0 and 1, which represents the probability that the first candidate word is the key word of the first training text.

[0114] Step 530, determining the probability in step 520 as the weight of the first candidate word.

[0115] For example, the result output by the Sigmoid function is 0.88, and the weight of the first candidate word is 0.88. Of course, in some embodiments, the result output by the Sigmoid function can be multiplied by a certain coefficient to serve as the weight of the first candidate word.

[0116] Referring to Figure 6 , Figure 7 and Figure 8 In this embodiment, a structure of the first encoder 310 and a process of internal processing of the first encoder 310 are disclosed. Figure 7 As can be seen from Figure 7 , the first encoder 310 of this embodiment is composed of a self-attention layer 710 and a forward neural network layer 720.

[0117] In this embodiment, step 120, encoding the first training text to obtain first semantic information, includes:

[0118] Step 610, encoding each character in the first training text to obtain first encoding information of each character.

[0119] As shown in Figure 8 , first, each character in the first training text is encoded, for example, in this embodiment, "today the weather is really good", six characters, are encoded into e1, e2, e3, e4, e5 and e6 six vectors in turn. It should be understood that the encoding information in this embodiment can be a vector, a tensor or a matrix, and a vector is used as an example for description.

[0120] Step 620, determining the attention weight of each character according to the first encoding information of each character, the attention weight of the character being determined according to the similarity between the first encoding information of the character and the first encoding information of each other character.

[0121] Referring to Figure 8 Taking the attention weight of e1 as an example, first, the vector e1 and the six vectors including itself (i.e., e1-e6) are subjected to similarity calculation, referring to the dashed lines between e1 and e1-e6. The similarity calculation can be calculated by using the cosine similarity algorithm and the like. Then, the weight is calculated according to the calculated similarities, wherein each character corresponds to a weight value with each character. In this embodiment, the following formula can be used to calculate the weight, wherein a i represents the weight of the character e n . i For the character e T , Q n K represents the dot product of e T and e i , e n T and e i correspond to Q (query) and K (key) in the QKV model. is a constant. Of course, there are many formulas for calculating the attention weight, and the main idea is to analyze the similarity between characters, and then determine the contribution degree of a character to another character according to the size of the similarity, so as to obtain more accurate encoding information in the next processing step.

[0122] Step 630, determining the second encoding information of each character, the second encoding information of the character being determined according to the attention weight of the character and the first encoding information of all characters.

[0123] In this step, the second encoding information of the character is represented by Z wi , and Z wi =∑Z i =∑a i e i . Referring to Figure 8 Taking the character corresponding to e1 as an example, first, the weight of e1 corresponding to e1-e6 is calculated, which is denoted as a1-a6, then Z1-Z6 is calculated by using a1-a6. Finally, the second encoding information of the character corresponding to e1 is obtained by accumulating Z1-Z6, which means that the self-attention weight of each character is a vector. By encoding each character in the above manner, the global information of the text can be extracted, so that the final sentence encoding information is more accurate. At the same time, this method does not depend on the time sequence, and each character can be encoded synchronously, so the encoding efficiency is high.

[0124] Step 640: Determine a first vector according to the second encoding information of all characters.

[0125] Reference Figure 7 The second encoding information of each character output from the attention layer 710 is encoded into a first vector through the forward neural network 720. The first vector contains the semantic information of the sentence in the semantic space.

[0126] Reference Figure 9 and Figure 10 This embodiment discloses another structure of the first encoder 310 and another internal working mode of the first encoder 310.

[0127] In this embodiment, step 120, encoding the first training text to obtain first semantic information, includes:

[0128] Step 910: Segment the first training text to obtain each word in the first training text.

[0129] In this embodiment, words are encoded, so a word segmentation tool such as HanLP can be used to first segment the first training text. For example, "The weather is good and sunny today" becomes "Today / weather / good / sunny" after segmentation.

[0130] Step 920: Encode each word in the first training text to obtain third encoding information of each word and encoding information of the first training text.

[0131] After word segmentation, the words in the first training text enter the input layer and are denoted as X1~X n , and then embed each word in the word embedding layer, which is equivalent to mapping each word into the semantic space. Then, each word is encoded through the LSTM layer to output the third encoding information h of each word i And the encoding information of the first training text, i represents the i-th word. Figure 10 In the example, bidirectional LSTM is used for encoding, so the encoding information of the last word in both directions can be averaged as the encoding information of the first training text. For ordinary LSTM networks, the encoding information of the last word can be selected as the encoding information of the first training text.

[0132] Step 930: Calculate the attention weights of all words. The attention weights of words are determined by the similarity between the third encoding information of the words and the encoding information of the first training text.

[0133] The similarity of two encoding information can be determined by cosine similarity or other calculation methods in the embodiment. Then the sum of the similarity of the third encoding information of each word and the encoding information of the training text is taken as the denominator, and the similarity corresponding to each word is taken as the numerator to calculate the attention weight of each word. For example, the similarity of each word to the sentence in the sentence "today / weather / good / sunny" is 0.1, 0.2, 0.4, and 0.3, respectively, and the attention weight corresponding to "today" is 0.1 / (0.1+0.2+0.4+0.3)=0.1.

[0134] Step 940, calculating the fourth encoding information of each word, which is determined by the product of the third encoding information of the word and the attention weight.

[0135] In this step, the third encoding information of the word is multiplied by the corresponding attention weight to obtain the fourth encoding information.

[0136] For example, the third encoding information corresponding to "today" is (1, 2, …, 3), and the attention weight is 0.1, so the fourth encoding information of "today" is (0.1, 0.2, …, 0.3).

[0137] Step 950, determining the first vector according to the fourth encoding information of each word.

[0138] In this step, the first vector can be obtained by adding each fourth encoding information. For example, the fourth encoding information of "today" is (0.1, 0.2, …, 0.3), the fourth encoding information of "weather" is (0, 0.1, …, 0.3), the fourth encoding information of "good" is (1, 0.1, …, 0.3), and the fourth encoding information of "sunny" is (0.1, 0.1, …, 0.1), so the first vector is (1.2, 0.5, …, 1).

[0139] As shown in Figure 10 and Figure 11 In order to improve the accuracy of extracting semantic information of the sentence in the embodiment, a bi-LSTM network structure is used for encoding. Specifically, step 920, encoding each word in the first training text to obtain the third encoding information of each word and the encoding information of the first training text, including:

[0140] Step 1110, encoding each word in the first training text in the forward order to obtain the fifth encoding information of each word. That is, obtaining

[0141] Step 1120, encoding each word in the first training text in the reverse order to obtain the sixth encoding information of each word. That is, obtaining

[0142] Step 1130, calculating the third encoding information of each word; the third encoding information of the word is weighted by the fifth encoding information and the sixth encoding information of the word.

[0143] In this embodiment, the selected LSTM layer is a bi-LSTM network, which is characterized by bidirectional encoding, and then the encoding of the two direction outputs is averaged to obtain the output third encoding information. For example, the third encoding corresponding to X1 is h1, and h1 is determined by the average of the fifth encoding information and the sixth encoding information .

[0144] Step 1140, weighting the third encoding information of the first word and the third encoding information of the last word in the first training text to obtain the encoding information of the first training text.

[0145] From the encoding mode of LSTM, the encoding of the last word in the sequence contains the information of the sentence, so the average of and can be used as the encoding information of the sentence. Through the bi-LSTM network processing, more relationships between words are mined, and the overall information of the sentence is strengthened.

[0146] In some embodiments, step 180, training the text information processing model according to the first loss value and the second loss value, comprising:

[0147] Determining the total loss according to the weighting of the first loss value and the second loss value, and training the text information processing model according to the total loss.

[0148] In this embodiment, Loss = λ1loss cls + λ2loss term-weights , wherein, λ1 and λ2 are hyperparameters, y i represents the label value of the i-th dimension, and a i represents the i-th dimension prediction value. In this way, the model can quickly learn the type of text and the relationship between words. Therefore, using this embodiment can train a more accurate model under the same cost conditions or save training costs under the condition of training a model close to the accuracy.

[0149] Referring to Figure 3 and Figure 12 , this embodiment discloses a method for training a text information processing model, which includes a first encoder 310, a second encoder 320, a classifier 350, a feature fusion unit 330, and an activation unit 340. The method of this embodiment includes the following steps:

[0150] Step 1210, obtaining a second training text, a second candidate word in the second training text, a type label of the second training text, and a weight label of the second candidate word.

[0151] Step 1220, encoding the second training text by the first encoder to obtain third semantic information.

[0152] Step 1230, encoding the second candidate word by the second encoder to obtain fourth semantic information.

[0153] Step 1240, classifying and predicting the third semantic information by the classifier to obtain a second classification result of the second training text.

[0154] Step 1250, extracting features of the third semantic information and the fourth semantic information by the feature fusion unit to obtain fifth semantic information.

[0155] Step 1260, activating the fifth semantic information by the activation unit to obtain a weight of the second candidate word.

[0156] Step 1270, determining a third loss value according to the second classification result and the type label of the second training text.

[0157] Step 1280, determining a fourth loss value according to the weight of the second candidate word and the weight label of the second candidate word.

[0158] Step 1290, updating parameters of the first encoder, the second encoder, and the feature fusion unit according to the third loss value and the fourth loss value.

[0159] According to the description of the embodiments of the corresponding Figure 1 , the technical effects that can be achieved by the embodiments are the same as those of the embodiments in Figure 1 . In addition, the embodiments use different encoders to encode the text of the candidate word, so the encoder can be trained in real time and individually, so that the training samples of the model are easier to obtain, further reducing the training cost.

[0160] The first encoder is obtained by the following steps:

[0161] A first training set is obtained, and the first training set includes a plurality of third training texts and type labels corresponding to the third training texts. The first encoder is iteratively trained by the first training set until a first training condition is met.

[0162] In this embodiment, the first encoder can be trained separately using only text classification label data until the first encoder meets the convergence condition or the number of training times reaches a certain number. In this way, the model of the present embodiment can be pre-trained by reusing the data of other tasks, so the cost of obtaining training samples is low.

[0163] The model trained by the training method of the above embodiment can be used to predict the weight of the candidate word in the input text and the classification of the text, as shown in Figure 3 、 Figure 4a or Figure 4b .

[0164] Referring to Figure 13 , the embodiment discloses a text information processing method, which can run in a system as shown in Figure 2 , wherein the text information processing model is deployed on the server 220 or the user terminal 230, and the embodiment comprises the following steps:

[0165] Step 1310, obtaining an input text and a third candidate word in the input text.

[0166] For example, the input text "today the weather is fine" and the third candidate word "fine" are obtained.

[0167] In order to make the processing process more efficient, the input text can be segmented, and then each word is used as a candidate word for prediction. Therefore, the third candidate word can be obtained by segmenting the input text to obtain a plurality of words in the input text, and one of the words is used as a candidate word. For example, in the example of "today the weather is fine", "today", "weather", "good" and "fine" can be used as the third candidate word in turn.

[0168] Step 1320, inputting the input text and the third candidate word into the text information processing model to obtain the weight of the third candidate word in the input text; wherein the text information processing model is trained by the above training method.

[0169] Specifically, the processing of the text information processing model on the input text and the third candidate word comprises the following steps:

[0170] Step 1321, encoding the input text to obtain sixth semantic information.

[0171] Step 1322, encoding the third candidate word to obtain seventh semantic information.

[0172] Step 1323, predicting the weight of the third candidate word according to the seventh semantic information and the sixth semantic information.

[0173] Step 1324, predicting the classification of the input text according to the sixth semantic information to obtain a first classification result.

[0174] As can be seen from the description of the embodiment, the model obtained by the above training method can predict the classification of the input text, and can also predict the weight of the third candidate word in the input text.

[0175] For example, inputting the input text "Today the weather is fine" and the third candidate word "fine" into the text information processing model for processing can obtain an output vector (0.1, 0.2, 0.7, 0.88). The first three positions of the output result represent the probabilities of the sentence belonging to the three categories, and the fourth position output represents the weight of "fine" in the sentence.

[0176] When the weight of the word in the input text and the type of the input text are obtained, these data can play a role in many occasions. For example, the classification information can be used to label the theme of an article, a picture or a video as the basis for indexing. The weight of the keyword can also be used as the basis for search sorting.

[0177] For example, using the classification information of the text to add labels to the theme of a video, these labels can be classification labels such as "music", "food", "shopping", etc. The weight information of the theme of the video can also be used as the basis for sorting, for example, there are three article titles, "Today the weather is fine", "The fine weather is really good" and "Will the weather be fine tomorrow". Among them, the weights of "fine" in the three titles are 0.66, 0.7 and 0.68 respectively, so when the user searches by "fine", the sorting result is arranged in descending order of weight, that is, in the example, the order is "The fine weather is really good", "Will the weather be fine tomorrow", "Today the weather is fine".

[0178] Referring to Figure 14a and Figure 14b The embodiment provides a video function interface embedded in a chat software, wherein in the discovery page 1410 of the chat software, the menu video number 1411 can be clicked to enter the interface 1420 of the video number. In the interface 1420 of the video number, the video content is displayed in the form of an information waterfall, wherein the area 1421 is a video display area, according to the user's setting, the area can directly play the video or display the cover of the video. The title, label and video link of the video are displayed in the area 1423. For example, in this example, "Valentine's Day, the most loved gift of boys is actually milk tea" is the title, and "#Valentine's Day #gift #milk tea #" is the label. In addition, the interface 1420 of the video number also includes a search bar 1422 of the video. The user can search for videos through the search bar 1422, and the basic data for searching is the data obtained by processing the title of the video by using the above-mentioned text information processing method. Then it means that the background server will process the text information according to the title uploaded by the video publisher every time a video is uploaded, and extract the classification of the video title and the weight of each word in the title as the basic data for searching.

[0179] Referring to Figure 15 from Figure 14bThe search bar is inputted with a search word, such as "milk tea", and a search result interface 1500 is popped up, which includes a plurality of search results 1510 composed of video covers and titles. In this example, there is only one search word, so the sorting can be directly based on the weight of the word in the title. For example, the weight of the word "milk tea" in "milk tea is delicious", "oolong milk tea", "milk tea tower defense strategy", and "milk tea net cafe" is 0.9, 0.8, 0.77, and 0.62, respectively, so the sorting result is shown in interface 1500, which is in descending order according to the weight. Of course, in some examples, the user inputted key word can be a sentence or have multiple key words. When the traditional key word search method cannot search out the result, a fuzzy search is needed through semantic retrieval. At this time, the semantic information of the sentence can be reconstructed by using the weight of the key word in the sentence, for example, the words in the sentence are re-encoded, and then the weight of each word is weighted as the encoding information of the sentence, so that the key information of the sentence can be better expressed, and the search result is more accurate.

[0180] It should be understood that the text type and the weight of the candidate word extracted from the text in the above examples can be applied in the classified search. Referring to Figure 16a and Figure 16b In this example, the search result interface 1610 can be entered from the interface 1420 shown in Figure 14b . The interface 1610 is provided with a classification option bar 1620, as shown in Figure 16a When the user selects the "food broadcast" option in the classification option bar 1620, the topics appearing in the dynamic area are all related to food broadcast, such as the video with "milk tea is delicious" as the topic. When the user inputted key word is inputted into the background server, the background server retrieves according to the key word, filters out the topics containing the key word, and then sorts according to the weight of the key word in the content subject. At the same time, according to the topic classification selected by the user, the classification information extracted by the text information processing method of the above examples is used to filter the content. For example, the content "match champion sends milk tea" in Figure 16b belongs to game content rather than food broadcast content, so its classification label is "game", and when the user selects the topic classification as "game", the content without the game label is filtered out, and then sorted according to the weight of the key word. The above example is only illustrative, and the text classification and key word weight extracted by the above text information processing method can be used in combination with current search technologies.

[0181] Referring to Figure 17 , the embodiment discloses a training system of a text information processing model, which comprises:

[0182] The first obtaining unit 1710 is configured to obtain a first training text, a first candidate word in the first training text, a type label of the first training text, and a weight label of the first candidate word.

[0183] The first encoding unit 1720 is configured to encode the first training text to obtain first semantic information.

[0184] The second encoding unit 1730 is configured to encode the first candidate word to obtain second semantic information.

[0185] The classification unit 1740 is configured to predict a classification of the first training text according to the first semantic information, to obtain a first classification result.

[0186] The word weight prediction unit 1750 is configured to predict a weight of the first candidate word according to the first semantic information and the second semantic information.

[0187] The first loss value determination unit 1760 is configured to determine a first loss value according to the first classification result and the type label of the first training text.

[0188] The second loss value determination unit 1770 is configured to determine a second loss value according to the weight of the first candidate word and the weight label of the first candidate word.

[0189] The training unit 1780 is configured to train the text information processing model according to the first loss value and the second loss value.

[0190] With reference to Figure 21 The embodiment provides a text information processing model training system, the text information processing model comprising a first encoder, a second encoder, a classifier, a feature fusion unit and an activation unit, the system comprising:

[0191] The third obtaining unit 2110 is configured to obtain a second training text, a second candidate word in the second training text, a type label of the second training text, and a weight label of the second candidate word.

[0192] The third encoding unit 2120 is configured to encode the second training text by the first encoder to obtain third semantic information.

[0193] The fourth encoding unit 2130 is configured to encode the second candidate word by the second encoder to obtain fourth semantic information.

[0194] The text classification unit 2140 is configured to predict a classification of the third semantic information by the classifier to obtain a second classification result of the second training text.

[0195] The fusion feature extraction unit 2150 is configured to perform feature extraction on the third semantic information and the fourth semantic information by the feature fusion unit to obtain fifth semantic information.

[0196] The activation processing unit 2160 is configured to activate the fifth semantic information by the activation unit to obtain a weight of the second candidate word.

[0197] The third loss determination unit 2170 is configured to determine a third loss value according to the second classification result and a type label of the second training text.

[0198] The fourth loss determination unit 2180 is configured to determine a fourth loss value according to the weight of the second candidate word and a weight label of the second candidate word.

[0199] The parameter training unit 2190 is configured to update parameters of the first encoder, the second encoder and the feature fusion unit according to the third loss value and the fourth loss value.

[0200] Reference Figure 18 The embodiment of the present application discloses a text information processing system, comprising:

[0201] The second acquisition unit 1810 is configured to acquire an input text and a third candidate word in the input text.

[0202] The weight prediction unit 1820 is configured to input the input text and the third candidate word into a text information processing model to obtain a weight of the third candidate word in the input text; wherein the text information processing model is trained by the model training method.

[0203] Figure 19 The server 1900 provided by the embodiment of the present application has a module block diagram, and the server 1900 can have great difference due to different configurations or performances, and can include one or more central processing units (CPU, Central Processing Units) 1922 (for example, one or more processors) and a memory 1932, one or more storage media 1930 (for example, one or more mass storage devices) for storing application programs 1942 or data 1944. The memory 1932 and the storage medium 1930 can be temporary storage or persistent storage. The programs stored in the storage medium 1930 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the central processing unit 1922 can be configured to communicate with the storage medium 1930, and execute a series of instruction operations in the storage medium 1930 on the server 1900 to realize the above-mentioned method.

[0204] The server 1900 can also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input / output interfaces 1958, and / or one or more operating systems 1941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0205] The embodiment of the present application further provides a device, which can execute the model training method or the text information processing method described above. The device is introduced below in combination with the drawings. Please refer to Figure 20 The embodiment of the present application provides a device, which can also be a terminal device. The terminal device can be any intelligent terminal, including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a vehicle-mounted computer, etc. The terminal device is taken as a mobile phone for example:

[0206] Figure 20 A block diagram of part of the structure of the mobile phone related to the terminal device provided by the embodiment of the present application is shown. Please refer to Figure 20 The mobile phone includes a radio frequency (RF) circuit 2010, a memory 2020, an input unit 2030, a display unit 2040, a sensor 2050, an audio circuit 2060, a wireless fidelity (WiFi) module 2070, a processor 2080, and a power supply 2090, etc. Those skilled in the art can understand that Figure 20 The structure of the mobile phone shown in the embodiment of the present application does not constitute a limitation on the mobile phone, and the mobile phone can include more or less components than those shown in the figure, or combine some components, or different arrangement of components. The various constituent components of the mobile phone are introduced below in combination with Figure 20 The RF circuit 2010 can be used for receiving and sending signals in the process of information transmission or call, especially, receiving the downlink information of the base station and processing by the processor 2080. In addition, the data designed to go up is sent to the base station. Usually, the RF circuit 2010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 2010 can also communicate with the network and other devices through wireless communication.

[0207] The memory 2020 can be used to store software programs and modules, and the processor 2080 can execute various function applications and data processing of the mobile phone by running the software programs and modules stored in the memory 2020. The memory 2020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 2020 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0208] The input unit 2030 can be used to receive inputted digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 2030 can include a touch panel 2031 and other input devices 2032. The touch panel 2031, also known as a touch screen, can collect a user's touch operation (such as the user's operation on or near the touch panel 2031 using a finger, a stylus, or any suitable object or accessory) and drive the corresponding connection device according to the pre-set program. Optionally, the touch panel 2031 can include two parts of a touch detection device and a touch controller. The touch detection device detects the user's touch position and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 2080, and can also receive the command from the processor 2080 and execute it. In addition, the touch panel 2031 can be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel 2031, the input unit 2030 can also include other input devices 2032. Specifically, the other input devices 2032 can include one or more of a physical keyboard, a function key (such as a volume control key, an on-off key, etc.), a trackball, a mouse, a joystick, and the like.

[0209] The display unit 2040 can be used to display information input by a user or information provided to the user as well as various menus of the phone. The display unit 2040 can include a display panel 2041, which can be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, a touch panel 2031 can be overlaid on the display panel 2041, which, when detecting a touch operation thereon or thereabout, transmits to the processor 2080 to determine the type of touch event, and then the processor 2080 provides corresponding visual output on the display panel 2041 according to the type of touch event. Although in the above description, the touch panel 2031 and the display panel 2041 are realized as two independent components to realize the input and output functions of the phone, in some embodiments, the touch panel 2031 and the display panel 2041 can be integrated to realize the input and output functions of the phone. Figure 20

[0210] The phone can further include at least one sensor 2050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 2041 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 2041 and / or the backlight when the phone is moved to the ear.

[0211] The audio circuit 2060, the speaker 2061, and the microphone 2062 can provide an audio interface between the user and the phone. The audio circuit 2060 can convert the received audio data into an electrical signal, which is transmitted to the speaker 2061 to be converted into a sound signal for output. On the other hand, the microphone 2062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 2060 to be converted into audio data. The audio data is output to the processor 2080 for processing, and then transmitted to another phone via the RF circuit 2010, or output to the memory 2020 for further processing. WiFi is a short-range wireless transmission technology, and the WiFi module 2070 can help the user to send and receive emails, browse web pages, and access streaming media, etc., which provides the user with wireless broadband Internet access. Although the WiFi module 2070 is shown, it is understood that it does not belong to the essential components of the phone, and can be omitted as needed without changing the essence of the application. Figure 20

[0212] ​​The processor 2080 is the control center of the mobile phone, which connects all parts of the mobile phone through various interfaces and lines, and performs various functions and processes data of the mobile phone by running or executing software programs and / or modules stored in the memory 2020 and calling data stored in the memory 2020, thereby monitoring the whole mobile phone. Optionally, the processor 2080 can include one or more processing units, and the mobile phone further includes a power supply 2090 for supplying power to each component.

[0213] In the embodiment, the processor 2080 included in the terminal device has a function of executing the model training method or the text information processing method of the above embodiment.

[0214] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0215] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0216] The integrated unit described in the present application, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that makes a contribution to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0217] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for training a text information processing model, characterized in that, The method comprises the following steps: obtaining first training text, a first candidate word in the first training text, a type label of the first training text, and a weight label of the first candidate word; encoding the first training text to obtain first semantic information, the first semantic information being represented by a first vector; the encoding of the first training text to obtain the first semantic information comprises: encoding each character in the first training text to obtain first encoding information of each character; determining an attention weight of each character according to the first encoding information of the character and the first encoding information of each other character, the attention weight of the character being determined according to a similarity between the first encoding information of the character and the first encoding information of each other character; determining second encoding information of each character according to the attention weight of the character and the first encoding information of all characters, the second encoding information of the character being determined according to the attention weight of the character and the first encoding information of all characters; determining the first vector according to the second encoding information of all characters; or the encoding of the first training text to obtain the first semantic information comprises: segmenting the first training text to obtain each word in the first training text; encoding each word in the first training text to obtain third encoding information of each word and encoding information of the first training text; calculating an attention weight of each word, the attention weight of each word being determined according to a similarity between the third encoding information of the word and the encoding information of the first training text; calculating fourth encoding information of each word, the fourth encoding information of each word being determined according to a product of the third encoding information of the word and the attention weight; determining the first vector according to the fourth encoding information of each word; encoding the first candidate word to obtain second semantic information; predicting a classification of the first training text according to the first semantic information to obtain a first classification result; predicting a weight of the first candidate word according to the first semantic information and the second semantic information; determining a first loss value according to the first classification result and the type label of the first training text; determining a second loss value according to the weight of the first candidate word and the weight label of the first candidate word; training the text information processing model according to the first loss value and the second loss value. 2.The method of claim 1, wherein, the second semantic information is represented by a second vector; the prediction of the weight of the first candidate word according to the first semantic information and the second semantic information comprises: splicing the first vector and the second vector to obtain a third vector; predicting a probability that the first candidate word is a key word of the first training text according to the third vector; determining the probability as the weight of the first candidate word. 3.The method of claim 1, wherein, the encoding of each word in the first training text to obtain third encoding information of each word and encoding information of the first training text comprises: encoding each word in the first training text in a forward order of the first training text to obtain fifth encoding information of each word; encoding information of each word in the first training text is obtained by encoding each word in the first training text in reverse order of the first training text; third encoding information of each word is calculated; the third encoding information of each word is obtained by weighting the fifth encoding information and the sixth encoding information of the word; the third encoding information of the first word in the first training text and the third encoding information of the last word in the first training text are weighted to obtain the encoding information of the first training text. 4.The method of claim 1, wherein, The information processing model is trained according to the first loss value and the second loss value, including: determining a total loss according to the weighting of the first loss value and the second loss value; training the information processing model according to the total loss.

5. A method for training a text information processing model, characterized by, The text information processing model includes a first encoder, a second encoder, a classifier, a feature fusion unit and an activation unit, and the method includes the following steps: obtaining a second training text, a second candidate word in the second training text, a type label of the second training text and a weight label of the second candidate word; encoding the second training text by the first encoder to obtain third semantic information, the third semantic information being represented by a third vector; encoding the second training text by the first encoder to obtain third semantic information, including: encoding each character in the second training text by the first encoder to obtain fifth encoding information of each character; determining an attention weight of each character according to the fifth encoding information of the character, the attention weight of the character being determined according to the similarity between the fifth encoding information of the character and the fifth encoding information of each other character; determining sixth encoding information of each character, the sixth encoding information of the character being determined according to the attention weight of the character and the fifth encoding information of all characters; determining the third vector according to the sixth encoding information of all characters; or the encoding of the second training text by the first encoder to obtain third semantic information includes: segmenting the second training text to obtain each word in the second training text; encoding each word in the second training text by the first encoder to obtain seventh encoding information of each word and encoding information of the second training text; calculating attention weights of all words, the attention weight of each word being determined by the similarity between the seventh encoding information of the word and the encoding information of the second training text; calculating eighth encoding information of each word, the eighth encoding information of each word being determined by the product of the seventh encoding information of the word and the attention weight; determining the third vector according to the eighth encoding information of each word; encoding the second candidate word by the second encoder to obtain fourth semantic information; classifying and predicting the third semantic information by the classifier to obtain a second classification result of the second training text; extracting features of the third semantic information and the fourth semantic information by the feature fusion unit to obtain fifth semantic information; activating the fifth semantic information by the activating unit to obtain a weight of the second candidate word; determining a third loss value according to the second classification result and a type label of the second training text; determining a fourth loss value according to the weight of the second candidate word and a weight label of the second candidate word; updating parameters of the first encoder, the second encoder and the feature fusion unit according to the third loss value and the fourth loss value. 6.The method of Claim 5, wherein, The first encoder obtains by the following steps: obtaining a first training set, the first training set comprising a plurality of third training texts and type labels corresponding to each of the third training texts; iteratively training the first encoder by the first training set until a first training condition is met.

7. A method of processing text information, characterized by, comprising the following steps: obtaining an input text and a third candidate word in the input text; inputting the input text and the third candidate word into a text information processing model to obtain a weight of the third candidate word in the input text; wherein the text information processing model is trained by the method of any one of claims 1-6.

8. The text information processing method according to claim 7, characterized in that, The method further comprises: inputting the input text and the third candidate word into a text information processing model to obtain a classification result of the input text.

9. The text information processing method according to claim 7, characterized in that, The method further comprises: segmenting the input text to obtain a plurality of words in the input text; taking one of the words as a third candidate word.

10. A training system for a text information processing model, characterized in that: comprising: a first obtaining unit configured to obtain a first training text, a first candidate word in the first training text, a type label of the first training text and a weight label of the first candidate word; a first encoding unit configured to encode the first training text to obtain first semantic information, the first semantic information being represented by a first vector; The first encoding unit is configured to: encode each character in the first training text to obtain first encoding information of each character; determine an attention weight of each character according to the first encoding information of the character and the first encoding information of each other character, the attention weight of the character being determined according to a similarity between the first encoding information of the character and the first encoding information of each other character; determine second encoding information of each character, the second encoding information of the character being determined according to the attention weight of the character and the first encoding information of all characters; determine the first vector according to the second encoding information of all characters; Or the first encoding unit is configured to: segment the first training text to obtain each word in the first training text; encode each word in the first training text to obtain third encoding information of each word and encoding information of the first training text; calculate an attention weight of each word, the attention weight of the word being determined by a similarity between the third encoding information of the word and the encoding information of the first training text; calculate fourth encoding information of each word, the fourth encoding information of the word being determined by a product of the third encoding information of the word and the attention weight; determine the first vector according to the fourth encoding information of each word. a second encoding unit, configured to encode the first candidate word to obtain second semantic information; a classification unit, configured to predict a classification of the first training text according to the first semantic information to obtain a first classification result; a word weight prediction unit, configured to predict a weight of the first candidate word according to the first semantic information and the second semantic information; a first loss value determination unit, configured to determine a first loss value according to the first classification result and a type label of the first training text; a second loss value determination unit, configured to determine a second loss value according to the weight of the first candidate word and a weight label of the first candidate word; a training unit, configured to train the text information processing model according to the first loss value and the second loss value.

11. A text information processing system characterized by comprising: comprising: a second obtaining unit, configured to obtain an input text and a third candidate word in the input text; a weight prediction unit, configured to input the input text and the third candidate word into a text information processing model to obtain a weight of the third candidate word in the input text; wherein the text information processing model is trained by the method of any one of claims 1-6.

12. A data processing apparatus, characterized by comprising: a program; a memory, configured to store the program; a processor, configured to load the program to execute the method of any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement the method of any one of claims 1-9. The computer readable storage medium stores a program, and the program is executed by the processor to implement the method of any one of claims 1-9.

Images