Semantic understanding model training method, device, computer equipment and storage medium

By using the encoder-attention-decoder model and mask sequence training method in the natural language understanding model, the problem of poor generalization ability of existing models is solved, and the understanding and coding ability of natural language is improved.

CN111444311BActive Publication Date: 2024-11-01PING AN TECH (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202010120133.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-02-26
Publication Date
2024-11-01
Estimated Expiration
2040-02-26

AI Technical Summary

Technical Problem

The poor generalization ability of existing natural language understanding models leads to a decrease in understanding ability when the content or format of natural language text changes.

Method used

The semantic understanding model training method is adopted. By randomly selecting continuous word vectors from the total word sequence of the training text and replacing them with a mask sequence, input them to the encoder-attention-decoder model for training, adjusting the model parameters to reduce the prediction output and test The difference between outputs until the preset training stop condition is met.

Benefits of technology

The model's understanding and coding ability of natural language is improved, the joint training of the encoder-attention-decoder model is promoted, and the understanding ability of natural language is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111444311B_ABST
    Figure CN111444311B_ABST
Patent Text Reader

Abstract

The present application discloses a method, device, computer equipment and storage medium for training a semantic understanding model, wherein the method includes obtaining a total word sequence corresponding to a training text from a training set; randomly selecting a preset number of continuous word vectors from the total word sequence and replacing them with a mask sequence to obtain an input word sequence, and using the preset number of continuous word vectors as a test output word sequence; inputting the input word sequence into an encoder-attention-decoder model for training to obtain a predicted output word sequence; adjusting the model parameters of the encoder-attention-decoder model according to the difference between the predicted output word sequence and the test output word sequence to reduce the difference; returning to the step of inputting the input word sequence into the encoder-attention-decoder model for training to obtain a predicted output word sequence and continuing training until a preset training stop condition is met to obtain a semantic understanding model. The present application improves the accuracy of a computer's understanding of natural language.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a semantic understanding model training method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of artificial intelligence, computers need to understand a large amount of natural language when performing human-computer interaction. The study of natural language models is a hot topic in the current Internet and artificial intelligence fields. At present, mainstream natural language understanding mainly adopts a statistical rule-based learning method to obtain language statistical knowledge from a large amount of language sample data to establish a language understanding model. However, in actual use, due to the poor generalization of statistical rules, when the content or format of the natural language text changes, the corresponding statistical rules will become invalid, and the understanding ability of the language understanding model established based on statistical rules may become very poor. Summary of the invention

[0003] The main purpose of this application is to provide a semantic understanding model training method, device, computer equipment and storage medium, aiming to solve the problem in the prior art that the computer's understanding of natural language is not accurate enough.

[0004] This application proposes a semantic understanding model training method, including:

[0005] Obtaining a total word sequence corresponding to the training text from the training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence;

[0006] From the total word sequence, a preset number of continuous word vectors are randomly selected and replaced with a mask sequence to obtain an input word sequence, and the preset number of continuous word vectors are used as a test output word sequence;

[0007] Inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model on the mask sequence in the input word sequence;

[0008] According to the difference between the predicted output word sequence and the test output word sequence, a preset optimizer is used to adjust the model parameters of the encoder-attention-decoder model to reduce the difference;

[0009] Return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continue training to obtain the predicted output word sequence until the preset training stop condition is met, then stop training to obtain the semantic understanding model.

[0010] Further, the mask sequence includes at least one mask vector, and the encoder-attention-decoder model includes an encoder and a decoder;

[0011] The step of inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence comprises:

[0012] Inputting the input word sequence into the encoder, semantically encoding the input word sequence, and obtaining an intermediate semantic vector sequence; wherein the intermediate semantic vector sequence includes the intermediate semantic vectors corresponding to each of the mask vectors;

[0013] The decoder generates the predicted output word sequence according to the intermediate semantic vector sequence and the input word sequence; wherein the predicted output word sequence includes the predicted output words corresponding to each of the mask vectors.

[0014] Furthermore, the encoder includes a multi-layer neural network, the multi-layer neural network includes an input layer, an intermediate hidden layer and an output layer, the number of the intermediate hidden layers includes multiple layers, and the step of inputting the input word sequence into the encoder, semantically encoding the input word sequence, and obtaining an intermediate semantic vector sequence includes:

[0015] Input the input word sequence into the first intermediate hidden layer of the encoder through the input layer of the encoder;

[0016] In each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed on the input word sequence input by the previous neural network layer layer by layer, and the hidden layer semantic vectors output by each intermediate hidden layer are obtained respectively, until the hidden layer semantic vector output by the last intermediate hidden layer is obtained;

[0017] The hidden layer semantic vector output by the last intermediate hidden layer is used as the intermediate semantic vector sequence, and is output to the decoder through the output layer of the encoder.

[0018] Furthermore, in each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed on the input word sequence input by the previous neural network layer by layer, and the steps of obtaining the hidden layer semantic vectors output by each intermediate hidden layer include:

[0019] In the middle hidden layer of the current layer, respectively obtaining the attention weights of each word vector in the input word sequence input by the previous neural network layer to each mask vector;

[0020] According to each of the attention weights, weighted summation is performed on each of the word vectors to obtain attention vectors corresponding to each of the mask vectors;

[0021] The attention vectors corresponding to the mask vectors in the intermediate hidden layer of the current layer are concatenated to obtain the hidden layer semantic vector output by the intermediate hidden layer of the current layer.

[0022] Furthermore, the step of obtaining the total word sequence corresponding to the training text from the training set includes:

[0023] Obtaining training text from the training set;

[0024] Segmenting the training text into multiple word segments according to a preset word segmentation method;

[0025] Convert each of the word fragments into a word vector respectively;

[0026] The total word sequence corresponding to the training text is constructed according to the word vectors arranged in sequence.

[0027] Furthermore, the step of randomly selecting a preset number of continuous word vectors from the total word sequence and replacing them with a mask sequence to obtain an input word sequence, and using the preset number of continuous word vectors as a test output word sequence includes:

[0028] Randomly select one of the word vectors from the total word sequence as a starting point;

[0029] Starting from the starting point, the preset number of continuous word vectors are replaced by the preset number of corresponding mask vectors, and the preset number of mask vectors constitute the mask sequence;

[0030] The total word sequence after being replaced by the mask sequence is used as the input word sequence, and the preset number of continuous word vectors is used as the test output word sequence.

[0031] Furthermore, the step of determining whether a preset training stop condition is met includes:

[0032] Determine whether the perplexity value of the above semantic understanding model is less than the preset perplexity value; wherein the perplexity calculation formula is:

[0033]

[0034] Among them, PP represents perplexity; S represents the sentence input to the model; N is the length of the sentence; p(w i ) is the probability of the i-th word appearing given the first i-1 words of a sentence;

[0035] If so, it is determined that the preset training stop condition is met.

[0036] This application also proposes a semantic understanding model training device, including:

[0037] An acquisition unit, used to acquire a total word sequence corresponding to a training text from a training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence;

[0038] A masking unit, used to randomly select a preset number of continuous word vectors from the total word sequence and replace them with a masking sequence to obtain an input word sequence, and use the preset number of continuous word vectors as a test output word sequence;

[0039] A first training unit is used to input the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model for the mask sequence in the input word sequence;

[0040] an adjusting unit, configured to adjust model parameters of the encoder-attention-decoder model using a preset optimizer according to a difference between the predicted output word sequence and the test output word sequence, so as to reduce the difference;

[0041] The second training unit is used to return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continuing the training to obtain the predicted output word sequence until a preset training stop condition is met, then stop the training to obtain the semantic understanding model.

[0042] The present application also proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0043] The present application also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0044] Beneficial effects of this application:

[0045] The semantic understanding model training method, device, computer equipment and storage medium of the present application first obtain the total word sequence corresponding to the training text from the training set; then randomly select a preset number of continuous word vectors from the total word sequence and replace them with a mask sequence to obtain an input word sequence, and use the preset number of continuous word vectors as a test output word sequence; then input the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence; then, according to the difference between the predicted output word sequence and the test output word sequence, use a preset optimizer to adjust the model parameters of the encoder-attention-decoder model to reduce the difference; then return to continue training the encoder-attention-decoder model until the preset training stop condition is met to stop training and obtain a semantic understanding model. The present application uses an encoder-attention-decoder model to predict the masked part of a sentence, thereby improving the encoder's understanding and encoding ability of natural language, and encourages the decoder to extract information from the encoder to help predict the mask sequence, promote the joint training of the encoder-attention-decoder model, and improve the model's understanding of natural language. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of a semantic understanding model training method according to an embodiment of the present application;

[0047] Figure 2 It is a schematic block diagram of the structure of a semantic understanding model training device according to an embodiment of the present application;

[0048] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] Reference Figure 1 In one embodiment of the present application, a semantic understanding model training method is provided, comprising:

[0052] S1. Obtain a total word sequence corresponding to a training text from a training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence;

[0053] S2. Randomly select a preset number of continuous word vectors from the total word sequence and replace them with a mask sequence to obtain an input word sequence, and use the preset number of continuous word vectors as a test output word sequence;

[0054] S3, inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model for the mask sequence in the input word sequence;

[0055] S4. According to the difference between the predicted output word sequence and the test output word sequence, a preset optimizer is used to adjust the model parameters of the encoder-attention-decoder model to reduce the difference;

[0056] S5. Return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continue training to obtain a predicted output word sequence until a preset training stop condition is met, then stop training to obtain the semantic understanding model.

[0057] In this embodiment, the above-mentioned semantic understanding model can enhance the computer's ability to understand natural language. The above-mentioned semantic understanding model can be used in a variety of scenarios, such as human-computer question and answer, automatic translation, intelligent interviews, etc., thereby improving the accuracy of feedback information generated by the computer based on user input.

[0058] In the above step S1, the training texts in the above training set can be collected according to the specific application scenarios of the semantic understanding model. For example, when the above semantic understanding model is used for intelligent interviews, a large number of texts used in the user interview process can be obtained from the historical interview database or the Internet. The above training text can be a paragraph of a certain length, which can be divided into multiple word fragments, and the total sequence corresponding to the training text is constructed according to each word fragment. When the above training text is Chinese text, a dictionary-based or statistical-based word segmentation method can be adopted; when the above training text is English text, a space-based word segmentation method can be adopted; or an existing word segmentation model can be used to segment the above training text. For example, the total word sequence obtained by step S1 is ([x1], [x2], [x3], [x4], [x5]).

[0059] In the above step S2, the above preset number can be obtained by multiplying the number of word vectors contained in the above total word sequence by a specified percentage, wherein the value of the specified percentage can be adjusted according to the prediction accuracy of the above semantic understanding model. A preset number of continuous word vectors are randomly selected from the above total word sequence. For example, for the total word sequence ([x1], [x2], [x3], [x4], [x5]) of step S1, 2 continuous word sequences (e.g., [x2][x3]) are randomly selected and replaced with a mask sequence, and the word sequence replaced with the mask sequence is used as the input word sequence, that is, the input word sequence is ([x1], [-], [-], [x4], [x5]), and the above preset number of continuous word vectors are used as the test output word sequence, that is, the test output word sequence is ([-], [x2], [x3], [-], [-]).

[0060] In the above step S3, the above input word sequence is input into the above encoder-attention-decoder model, and the mask sequence in the above input word sequence is predicted. The above encoder-attention-decoder model includes an encoder and a decoder. The encoder is a multi-layer neural network, for example, it can include a 12-layer neural network. Each layer of the neural network includes an attention layer and a fully connected layer. The attention layer is used to learn the dependency relationship between each word vector in the above input word sequence, and allocate the attention weights of each word vector in each input word sequence to each word vector in the mask sequence according to the dependency relationship. The fully connected layer includes a residual connection layer and a normalization layer, wherein the residual connection layer is used to connect the output of the previous layer of the neural network to the input of the current layer of the neural network. The residual connection layer is considered to have created an identity mapping, so that the output of the entire encoder converges in the direction of the identity mapping, ensuring that the final error rate will not become worse and worse due to the increase in the number of layers of the encoder's neural network; the normalization layer is used to normalize the data output by each layer of the neural network, which can solve the problem of disappearance or explosion of data gradients caused by the increase in the number of neural network layers. The structure of the decoder is similar to that of the encoder, which is a multi-layer neural network. For example, it may include 12 layers of neural networks, each of which includes an attention layer and a fully connected layer.

[0061] In the above step S4, when there is a large difference between the above predicted output word sequence and the test output word sequence, a preset optimizer can be used to gradually reduce the difference gradient between the predicted output word sequence and the test output word sequence. There are many algorithm principles used by the above preset optimizer, such as standard gradient descent, stochastic gradient descent, batch gradient descent, momentum stochastic gradient descent, Newton accelerated gradient algorithm, AdaGrad algorithm, RMSProp algorithm, Adam algorithm, AdaDelta algorithm, etc. Preferably, the encoder-attention-decoder model is optimized by a neural network optimizer LAMB (Layer-wise Adaptive Momentsoptimizer for Batch training), which uses an adaptive learning rate optimization algorithm, which can greatly reduce the number of iterations and reduce the training time of the model. If the difference between the above predicted output word sequence and the above test output word sequence is large, it indicates that the accuracy of the above encoder-attention-decoder model needs to be improved. The model parameters of the above encoder-attention-decoder model specifically include parameters such as the attention weights contained in the above encoder and decoder. By adopting the above-mentioned preset optimizer, by adjusting model parameters such as attention weights, etc., the difference between the predicted output word sequence and the test output word sequence is continuously reduced, and the encoder-attention-decoder model with adjusted model parameters is repeatedly trained until the parameter optimization is stopped when the preset training stop condition is met.

[0062] In the above step S5, the above preset training stop condition can be a preset prediction accuracy rate. The above semantic understanding model is used to predict the mask sequence of the input word sequence. When the prediction accuracy rate is greater than the preset prediction accuracy rate, it indicates that the preset training stop condition is met and the above semantic understanding model can meet the usage requirements. The above preset training stop condition can also be that the perplexity value of the above semantic understanding model is less than the preset perplexity value. The perplexity can be calculated by the following formula (1):

[0063]

[0064] Among them, PP stands for Perplexity, which is the perplexity; S stands for Stentence, which is the sentence input to the model; N is the sentence length; p(w i ) is the probability of the i-th word appearing, that is, given the first i-1 words of a sentence, the model predicts the probability of the i-th word appearing. When the perplexity value is lower, p(w i ) is larger, the higher the accuracy of the semantic understanding model prediction is.

[0065] The semantic understanding model training method of the present application first obtains the total word sequence corresponding to the training text from the training set; then, from the total word sequence, randomly selects a preset number of continuous word vectors and replaces them with a mask sequence to obtain an input word sequence, and uses the preset number of continuous word vectors as a test output word sequence; then, the input word sequence is input into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence; then, according to the difference between the predicted output word sequence and the test output word sequence, the model parameters of the encoder-attention-decoder model are adjusted to reduce the difference; then, the encoder-attention-decoder model is returned to continue training until the preset training stop condition is met, and the semantic understanding model is obtained. The present application uses an encoder-attention-decoder model to predict the masked part of a sentence, thereby improving the encoder's understanding and encoding ability of natural language, and encourages the decoder to extract information from the encoder to help predict the mask sequence, promote the joint training of the encoder-attention-decoder model, and improve the model's understanding of natural language.

[0066] In one embodiment, the mask sequence includes at least one mask vector, and the encoder-attention-decoder model includes an encoder and a decoder;

[0067] The step S3 of inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence comprises:

[0068] S31, inputting the input word sequence into the encoder, performing semantic encoding on the input word sequence, and obtaining an intermediate semantic vector sequence; wherein the intermediate semantic vector sequence includes the intermediate semantic vectors corresponding to each of the mask vectors;

[0069] S32. Generate the predicted output word sequence through the decoder according to the intermediate semantic vector sequence and the input word sequence; wherein the predicted output word sequence includes the predicted output words corresponding to each of the mask vectors.

[0070] In this embodiment, the above input word sequence is encoded by the encoder into an intermediate semantic vector sequence, and then decoded by the decoder.

[0071] In the above step S31, an intermediate semantic vector corresponding to each mask vector is generated according to each word vector in the input word sequence. When generating the intermediate semantic vector, it is generated according to the context information of the input word sequence and the relevant information of the word vectors before and after each mask vector. For example, for the above input word sequence of ([x1], [-], [-], [x4], [x5]), the encoder obtains the intermediate semantic vector C1 corresponding to the first mask vector and the intermediate semantic vector C2 corresponding to the second mask vector by encoding. The above intermediate semantic vector sequence is (C1, C2). Since each word vector has different attention weights for different mask vectors, the values ​​of C1 and C2 are different. By generating corresponding intermediate semantic vectors for different mask vectors, the accuracy of the predicted output words corresponding to each mask vector in the sentence is improved.

[0072] In the above step S32, the mask sequence in the input word sequence is predicted according to the intermediate semantic vector sequence generated by the above decoder to obtain the predicted output words corresponding to each vector in the mask sequence. For example, for the above input word sequence ([x1], [-], [-], [x4], [x5]), the decoder generates predicted output words ([-], [x2], [x3], [-], [-]) corresponding to two of the mask vectors, and the words in the input word sequence that are not hidden by the mask do not need to be predicted. Predicting the mask sequence through the decoder improves the encoder's ability to understand and encode natural language, and encourages the decoder to extract information from the encoder to help predict the mask sequence, thereby improving the model's ability to understand natural language.

[0073] In one embodiment, the encoder includes a multi-layer neural network, the multi-layer neural network includes an input layer, an intermediate hidden layer and an output layer, the number of the intermediate hidden layers includes multiple layers, the step S31 of inputting the input word sequence into the encoder, semantically encoding the input word sequence, and obtaining an intermediate semantic vector sequence includes:

[0074] S311, inputting the input word sequence into the first intermediate hidden layer of the encoder through the input layer of the encoder;

[0075] S312, in each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed on the input word sequence input by the previous neural network layer layer by layer, and the hidden layer semantic vectors output by each intermediate hidden layer are obtained respectively, until the hidden layer semantic vector output by the last intermediate hidden layer is obtained;

[0076] S313: Use the hidden layer semantic vector output by the last intermediate hidden layer as the intermediate semantic vector sequence, and output it to the decoder through the output layer of the encoder.

[0077] In this embodiment, the encoder includes a multi-layer neural network, for example, a 12-layer neural network, the first layer is an input layer, the last layer is an output layer, and the intermediate layers are intermediate hidden layers of each layer, and the number of intermediate hidden layers can be set as needed. Each layer of the neural network includes an attention layer and a fully connected layer. The attention layer is used to learn the dependency relationship between each word vector in the above-mentioned input word sequence, and allocate the attention weights of each word vector in each input word sequence to each word vector in the mask sequence according to the dependency relationship. The fully connected layer includes a residual connection layer and a normalization layer, wherein the residual connection layer is used to connect the output of the previous layer of the neural network to the input of the current layer of the neural network, and the residual connection layer is artificially created to make the output of the entire encoder converge in the direction of the identity mapping, ensuring that the final error rate will not become worse and worse due to the increase in the number of layers of the neural network of the encoder; the normalization layer is used to normalize the data output by each layer of the neural network, so that the problem of disappearance or explosion of data gradients caused by the increase in the number of neural network layers can be solved. The input word sequence is encoded through a multi-layer neural network. As the number of neural network layers increases, each layer has a deeper abstract representation of the previous layer, and the function simulation ability for generating intermediate semantic vector sequences is stronger, which is conducive to improving the accuracy of model prediction.

[0078] In one embodiment, in each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed layer by layer on the input word sequence input by the previous neural network to obtain the hidden layer semantic vector output by each intermediate hidden layer, step S312, includes:

[0079] S3121. In the middle hidden layer of the current layer, respectively obtain the attention weights of each word vector in the input word sequence input by the previous layer of the neural network to each mask vector;

[0080] S3122. Perform weighted summation on each of the word vectors according to each of the attention weights to obtain an attention vector corresponding to each of the mask vectors;

[0081] S3123. Concatenate the attention vectors corresponding to the mask vectors in the intermediate hidden layer of the current layer to obtain the hidden layer semantic vector output by the intermediate hidden layer of the current layer.

[0082] In this embodiment, for the input word sequence input by the previous layer of neural network, for example ([x1], [-], [-], [x4], [x5]), the attention mechanism is used to assign the attention weight of each word vector to the first mask vector [-], and the attention weight of each word vector to the second mask vector [-]. The attention vector is calculated by formula (2):

[0083]

[0084] Among them, L x Represents the length of the input word sequence; a ij represents the attention weight of the jth word vector in the input word sequence when predicting the i-th masked word vector; h j is the jth word vector in the input word sequence.

[0085] By introducing the attention mechanism, the model can more easily capture the interdependence between word vectors in the input word sequence, which is conducive to the use of dependencies between long-distance word vectors, thus greatly improving the accuracy of model prediction.

[0086] In one embodiment, the step S1 of obtaining the total word sequence corresponding to the training text from the training set includes:

[0087] S11, obtaining training text from the training set;

[0088] S12, dividing the training text into a plurality of word segments according to a preset word segmentation method;

[0089] S13, converting each of the word fragments into a word vector respectively;

[0090] S14. Construct the total word sequence corresponding to the training text according to the word vectors arranged in sequence.

[0091] In this embodiment, in the above step S11, the training text in the above training set can be collected according to the specific application scenario of the semantic understanding model. For example, when the above semantic understanding model is used for intelligent interviews, a large amount of text used in the user interview process can be obtained from the historical interview database or the Internet.

[0092] In the above step S12, the above preset segmented word segments can be manually segmented; when the above training text is Chinese text, a dictionary-based or statistical-based word segmentation method can be adopted; when the above training text is English text, a space-based word segmentation method can be adopted; or an existing word segmentation model can be used to segment the above training text.

[0093] In the above steps S13 to S14, the word vectors corresponding to the word fragments are generated according to the order of the word fragments in the training text, and the total word sequence corresponding to the training text is constructed according to the word vectors arranged in sequence. For example, the total word sequence obtained by step S1 is ([x1], [x2], [x3], [x4], [x5]).

[0094] In one embodiment, the step S2 of randomly selecting a preset number of continuous word vectors from the total word sequence and replacing them with a mask sequence to obtain an input word sequence, and using the preset number of continuous word vectors as a test output word sequence includes:

[0095] S21, randomly selecting one of the word vectors from the total word sequence as a starting point;

[0096] S22, starting from the starting point, replacing the preset number of continuous word vectors with the preset number of corresponding mask vectors, the preset number of mask vectors forming the mask sequence;

[0097] S23. Use the total word sequence after being replaced by the mask sequence as the input word sequence, and use the preset number of continuous word vectors as the test output word sequence.

[0098] In this embodiment, a preset number of continuous word vectors are randomly selected from the total word sequence. For example, for the total word sequence ([x1], [x2], [x3], [x4], [x5]) of step S14, two continuous word sequences (for example, [x2][x3]) are randomly selected and replaced with a mask sequence, and the word sequence replaced with the mask sequence is used as the input word sequence, that is, the input word sequence is ([x1], [-], [-], [x4], [x5]), and the preset number of continuous word vectors are used as the test output word sequence, that is, the test output word sequence is ([-], [x2], [x3], [-], [-]).

[0099] In one embodiment, the above-mentioned step S5 of determining whether the preset training stop condition is met includes:

[0100] S51, judging whether the perplexity value of the above semantic understanding model is less than a preset perplexity value; wherein the perplexity calculation formula (3) is:

[0101]

[0102] Among them, PP represents perplexity; S represents the sentence input to the model; N is the length of the sentence; p(w i ) is the probability of the i-th word appearing given the first i-1 words of a sentence;

[0103] S52: If yes, determine that the preset training stop condition is met.

[0104] In this embodiment, the lower the perplexity value, the higher the p(w i ) is larger, the higher the accuracy of the semantic understanding model prediction is.

[0105] In one embodiment, the ratio of the preset number to the number of word vectors included in the total word sequence is 50% to 70%.

[0106] In this embodiment, the preset number can be obtained by multiplying the number of word vectors contained in the total word sequence by a specified percentage, wherein the value of the specified percentage can be adjusted according to the prediction accuracy of the semantic understanding model. According to experimental tests, when the above ratio is within the range of 50% to 70%, the accuracy of model prediction is higher and the time required for model training is shorter.

[0107] Reference Figure 2 In one embodiment of the present application, a semantic understanding model training device is provided, comprising:

[0108] An acquisition unit 10 is used to acquire a total word sequence corresponding to a training text from a training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence;

[0109] The masking unit 20 is used to randomly select a preset number of continuous word vectors from the total word sequence and replace them with the masking sequence to obtain an input word sequence, and use the preset number of continuous word vectors as a test output word sequence;

[0110] A first training unit 30 is used to input the input word sequence into the encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model for the mask sequence in the input word sequence;

[0111] An adjustment unit 40, configured to adjust model parameters of the encoder-attention-decoder model using a preset optimizer according to a difference between the predicted output word sequence and the test output word sequence to reduce the difference;

[0112] The second training unit 50 is used to return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continuing the training to obtain the predicted output word sequence until a preset training stop condition is met, then stop the training to obtain the semantic understanding model.

[0113] In this embodiment, the implementation process of the functions and effects of the acquisition unit 10, mask unit 20, first training unit 30, adjustment unit 40 and second training unit 50 in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S1-S5 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0114] The semantic understanding model training device of the present application first obtains the total word sequence corresponding to the training text from the training set; then randomly selects a preset number of continuous word vectors from the total word sequence and replaces them with a mask sequence to obtain an input word sequence, and uses the preset number of continuous word vectors as a test output word sequence; then the input word sequence is input into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence; then, according to the difference between the predicted output word sequence and the test output word sequence, the model parameters of the encoder-attention-decoder model are adjusted to reduce the difference; then the encoder-attention-decoder model is returned to continue training until the preset training stop condition is met, and the semantic understanding model is obtained. The present application uses an encoder-attention-decoder model to predict the masked part of a sentence, thereby improving the encoder's understanding and encoding ability of natural language, and encourages the decoder to extract information from the encoder to help predict the mask sequence, promote the joint training of the encoder-attention-decoder model, and improve the model's understanding of natural language.

[0115] In one embodiment, the mask sequence includes at least one mask vector, and the encoder-attention-decoder model includes an encoder and a decoder;

[0116] The first training unit 30 comprises:

[0117] An encoding subunit, used for inputting the input word sequence into the encoder, performing semantic encoding on the input word sequence, and obtaining an intermediate semantic vector sequence; wherein the intermediate semantic vector sequence includes the intermediate semantic vectors corresponding to each of the mask vectors;

[0118] The decoding subunit is used to generate the predicted output word sequence through the decoder according to the intermediate semantic vector sequence and the input word sequence; wherein the predicted output word sequence includes the predicted output words corresponding to each of the mask vectors.

[0119] In this embodiment, the implementation process of the functions and effects of the encoding subunit and the decoding subunit in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S31-S32 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0120] In one embodiment, the encoder includes a multi-layer neural network, the multi-layer neural network includes an input layer, an intermediate hidden layer and an output layer, the number of the intermediate hidden layers includes multiple layers, and the encoding subunit includes:

[0121] An input module, used for inputting the input word sequence into the first intermediate hidden layer of the encoder through the input layer of the encoder;

[0122] An intermediate encoding module is used to semantically encode the input word sequence input by the previous neural network layer layer by layer in each intermediate hidden layer of the encoder according to the attention mechanism, and obtain the hidden layer semantic vector output by each intermediate hidden layer, until the hidden layer semantic vector output by the last intermediate hidden layer is obtained;

[0123] The output module is used to output the hidden layer semantic vector output by the last intermediate hidden layer as the intermediate semantic vector sequence to the decoder through the output layer of the encoder.

[0124] In this embodiment, the implementation process of the functions and effects of the input module, intermediate encoding module and output module in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S311-S313 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0125] In one embodiment, the intermediate encoding module includes:

[0126] A weight acquisition submodule, used to obtain, in the middle hidden layer of the current layer, the attention weights of each word vector in the input word sequence input by the previous layer of the neural network to each of the mask vectors;

[0127] A weighted summation submodule, used to perform weighted summation on each of the word vectors according to each of the attention weights, to obtain an attention vector corresponding to each of the mask vectors;

[0128] The vector generation submodule is used to concatenate the attention vectors corresponding to the mask vectors in the intermediate hidden layer of the current layer to obtain the hidden layer semantic vector output by the intermediate hidden layer of the current layer.

[0129] In this embodiment, the implementation process of the functions and effects of the weight acquisition submodule, weighted summation submodule and vector generation submodule in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S3121-S3123 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0130] In one embodiment, the acquisition unit 10 includes:

[0131] An acquisition subunit, used for acquiring training text from the training set;

[0132] A segmentation subunit, used to segment the training text into a plurality of word segments according to a preset word segmentation method;

[0133] A conversion subunit, used to convert each of the word fragments into a word vector;

[0134] The construction subunit is used to construct the total word sequence corresponding to the training text according to the word vectors arranged in sequence.

[0135] In this embodiment, the implementation process of the functions and effects of the acquisition subunit, segmentation subunit, transformation subunit and construction subunit in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S11-S14 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0136] In one embodiment, the mask unit 20 includes:

[0137] A selection subunit, used for randomly selecting one of the word vectors as a starting point from the total word sequence;

[0138] A mask subunit, configured to replace the preset number of continuous word vectors with the preset number of corresponding mask vectors starting from the starting point, wherein the preset number of mask vectors constitute the mask sequence;

[0139] A generating subunit uses the total word sequence after being replaced by the mask sequence as the input word sequence, and uses the preset number of continuous word vectors as the test output word sequence.

[0140] In this embodiment, the implementation process of the functions and effects of the selection subunit, mask subunit and generation subunit in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S21-S23 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0141] In one embodiment, the second training unit 50 includes:

[0142] The judgment subunit is used to judge whether the perplexity value of the above semantic understanding model is less than the preset perplexity value; wherein the perplexity calculation formula (3) is:

[0143]

[0144] Among them, PP represents perplexity; S represents the sentence input to the model; N is the length of the sentence; p(w i ) is the probability of the i-th word appearing given the first i-1 words of a sentence;

[0145] The determination subunit is used to determine whether the preset training stop condition is met if the perplexity value is less than a preset perplexity value.

[0146] In this embodiment, the implementation process of the functions and effects of the judgment subunit and the determination subunit in the above-mentioned semantic understanding model training device is specifically described in the implementation process of the corresponding steps S51-S52 in the above-mentioned semantic understanding model training method, and will not be repeated here.

[0147] In one embodiment, the ratio of the preset number to the number of word vectors included in the total word sequence is 50% to 70%.

[0148] In this embodiment, the preset number can be obtained by multiplying the number of word vectors contained in the total word sequence by a specified percentage, wherein the value of the specified percentage can be adjusted according to the prediction accuracy of the semantic understanding model. According to experimental tests, when the above ratio is within the range of 50% to 70%, the accuracy of model prediction is higher and the time required for model training is shorter.

[0149] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training texts. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a semantic understanding model training method is implemented.

[0150] The processor executes the semantic understanding model training method, including:

[0151] Obtaining a total word sequence corresponding to the training text from the training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence;

[0152] From the total word sequence, a preset number of continuous word vectors are randomly selected and replaced with a mask sequence to obtain an input word sequence, and the preset number of continuous word vectors are used as a test output word sequence;

[0153] Inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model on the mask sequence in the input word sequence;

[0154] According to the difference between the predicted output word sequence and the test output word sequence, a preset optimizer is used to adjust the model parameters of the encoder-attention-decoder model to reduce the difference;

[0155] Return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continue training to obtain the predicted output word sequence until the preset training stop condition is met, then stop training to obtain the semantic understanding model.

[0156] In one embodiment, the mask sequence includes at least one mask vector, and the encoder-attention-decoder model includes an encoder and a decoder;

[0157] The processor inputs the input word sequence into a preset encoder-attention-decoder model for training to obtain a step of predicting an output word sequence, including:

[0158] Inputting the input word sequence into the encoder, semantically encoding the input word sequence, and obtaining an intermediate semantic vector sequence; wherein the intermediate semantic vector sequence includes the intermediate semantic vectors corresponding to each of the mask vectors;

[0159] The decoder generates the predicted output word sequence according to the intermediate semantic vector sequence and the input word sequence; wherein the predicted output word sequence includes the predicted output words corresponding to each of the mask vectors.

[0160] In one embodiment, the encoder includes a multi-layer neural network, the multi-layer neural network includes an input layer, an intermediate hidden layer and an output layer, the number of the intermediate hidden layers includes multiple layers, and the processor inputs the input word sequence into the encoder, performs semantic encoding on the input word sequence, and obtains the step of an intermediate semantic vector sequence, including:

[0161] Input the input word sequence into the first intermediate hidden layer of the encoder through the input layer of the encoder;

[0162] In each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed on the input word sequence input by the previous neural network layer layer by layer, and the hidden layer semantic vectors output by each intermediate hidden layer are obtained respectively, until the hidden layer semantic vector output by the last intermediate hidden layer is obtained;

[0163] The hidden layer semantic vector output by the last intermediate hidden layer is used as the intermediate semantic vector sequence, and is output to the decoder through the output layer of the encoder.

[0164] In one embodiment, the processor performs semantic encoding on the input word sequence input by the previous neural network layer layer by layer in each intermediate hidden layer of the encoder according to the attention mechanism, and obtains the hidden layer semantic vector output by each intermediate hidden layer, including:

[0165] In the middle hidden layer of the current layer, respectively obtaining the attention weights of each word vector in the input word sequence input by the previous neural network layer to each mask vector;

[0166] According to each of the attention weights, weighted summation is performed on each of the word vectors to obtain attention vectors corresponding to each of the mask vectors;

[0167] The attention vectors corresponding to the mask vectors in the intermediate hidden layer of the current layer are concatenated to obtain the hidden layer semantic vector output by the intermediate hidden layer of the current layer.

[0168] In one embodiment, the processor obtains the total word sequence corresponding to the training text from the training set, comprising:

[0169] Obtaining training text from the training set;

[0170] Segmenting the training text into multiple word segments according to a preset word segmentation method;

[0171] Convert each of the word fragments into a word vector respectively;

[0172] The total word sequence corresponding to the training text is constructed according to the word vectors arranged in sequence.

[0173] In one embodiment, the processor randomly selects a preset number of continuous word vectors from the total word sequence and replaces them with a mask sequence to obtain an input word sequence, and uses the preset number of continuous word vectors as a test output word sequence, comprising:

[0174] Randomly select one of the word vectors from the total word sequence as a starting point;

[0175] Starting from the starting point, the preset number of continuous word vectors are replaced by the preset number of corresponding mask vectors, and the preset number of mask vectors constitute the mask sequence;

[0176] The total word sequence after being replaced by the mask sequence is used as the input word sequence, and the preset number of continuous word vectors is used as the test output word sequence.

[0177] In one embodiment, the processor performs a step of determining whether a preset training stop condition is met, including:

[0178] Determine whether the perplexity value of the above semantic understanding model is less than the preset perplexity value; wherein the perplexity calculation formula is:

[0179]

[0180] Among them, PP represents perplexity; S represents the sentence input to the model; N is the length of the sentence; p(w i ) is the probability of the i-th word appearing given the first i-1 words of a sentence;

[0181] If so, it is determined that the preset training stop condition is met.

[0182] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0183] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a semantic understanding model training method is implemented, specifically:

[0184] Obtaining a total word sequence corresponding to the training text from the training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence;

[0185] From the total word sequence, a preset number of continuous word vectors are randomly selected and replaced with a mask sequence to obtain an input word sequence, and the preset number of continuous word vectors are used as a test output word sequence;

[0186] Inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model on the mask sequence in the input word sequence;

[0187] According to the difference between the predicted output word sequence and the test output word sequence, a preset optimizer is used to adjust the model parameters of the encoder-attention-decoder model to reduce the difference;

[0188] Return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continue training to obtain the predicted output word sequence until the preset training stop condition is met, then stop training to obtain the semantic understanding model.

[0189] In one embodiment, the mask sequence includes at least one mask vector, and the encoder-attention-decoder model includes an encoder and a decoder;

[0190] The processor inputs the input word sequence into a preset encoder-attention-decoder model for training to obtain a step of predicting an output word sequence, including:

[0191] Inputting the input word sequence into the encoder, semantically encoding the input word sequence, and obtaining an intermediate semantic vector sequence; wherein the intermediate semantic vector sequence includes the intermediate semantic vectors corresponding to each of the mask vectors;

[0192] The decoder generates the predicted output word sequence according to the intermediate semantic vector sequence and the input word sequence; wherein the predicted output word sequence includes the predicted output words corresponding to each of the mask vectors.

[0193] In one embodiment, the encoder includes a multi-layer neural network, the multi-layer neural network includes an input layer, an intermediate hidden layer and an output layer, the number of the intermediate hidden layers includes multiple layers, and the processor inputs the input word sequence into the encoder, performs semantic encoding on the input word sequence, and obtains the step of an intermediate semantic vector sequence, including:

[0194] Input the input word sequence into the first intermediate hidden layer of the encoder through the input layer of the encoder;

[0195] In each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed on the input word sequence input by the previous neural network layer layer by layer, and the hidden layer semantic vectors output by each intermediate hidden layer are obtained respectively, until the hidden layer semantic vector output by the last intermediate hidden layer is obtained;

[0196] The hidden layer semantic vector output by the last intermediate hidden layer is used as the intermediate semantic vector sequence, and is output to the decoder through the output layer of the encoder.

[0197] In one embodiment, the processor performs semantic encoding on the input word sequence input by the previous neural network layer layer by layer in each intermediate hidden layer of the encoder according to the attention mechanism, and obtains the hidden layer semantic vector output by each intermediate hidden layer, including:

[0198] In the middle hidden layer of the current layer, respectively obtaining the attention weights of each word vector in the input word sequence input by the previous neural network layer to each mask vector;

[0199] According to each of the attention weights, weighted summation is performed on each of the word vectors to obtain attention vectors corresponding to each of the mask vectors;

[0200] The attention vectors corresponding to the mask vectors in the intermediate hidden layer of the current layer are concatenated to obtain the hidden layer semantic vector output by the intermediate hidden layer of the current layer.

[0201] In one embodiment, the processor obtains the total word sequence corresponding to the training text from the training set, comprising:

[0202] Obtaining training text from the training set;

[0203] Segmenting the training text into multiple word segments according to a preset word segmentation method;

[0204] Convert each of the word fragments into a word vector respectively;

[0205] The total word sequence corresponding to the training text is constructed according to the word vectors arranged in sequence.

[0206] In one embodiment, the processor randomly selects a preset number of continuous word vectors from the total word sequence and replaces them with a mask sequence to obtain an input word sequence, and uses the preset number of continuous word vectors as a test output word sequence, comprising:

[0207] Randomly select one of the word vectors from the total word sequence as a starting point;

[0208] Starting from the starting point, the preset number of continuous word vectors are replaced by the preset number of corresponding mask vectors, and the preset number of mask vectors constitute the mask sequence;

[0209] The total word sequence after being replaced by the mask sequence is used as the input word sequence, and the preset number of continuous word vectors is used as the test output word sequence.

[0210] In one embodiment, the processor performs a step of determining whether a preset training stop condition is met, including:

[0211] Determine whether the perplexity value of the above semantic understanding model is less than the preset perplexity value; wherein the perplexity calculation formula is:

[0212]

[0213] Among them, PP represents perplexity; S represents the sentence input to the model; N is the length of the sentence; p(w i ) is the probability of the i-th word appearing given the first i-1 words of a sentence;

[0214] If so, it is determined that the preset training stop condition is met.

[0215] In summary, for the semantic understanding model training method, device, computer equipment and storage medium of the present application, first obtain the total word sequence corresponding to the training text from the training set; then randomly select a preset number of continuous word vectors from the total word sequence and replace them with a mask sequence to obtain an input word sequence, and use the preset number of continuous word vectors as a test output word sequence; then input the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence; then adjust the model parameters of the encoder-attention-decoder model according to the difference between the predicted output word sequence and the test output word sequence to reduce the difference; then return to continue training the encoder-attention-decoder model until the preset training stop condition is met to stop training and obtain a semantic understanding model. The present application uses an encoder-attention-decoder model to predict the masked part of a sentence, thereby improving the encoder's understanding and encoding ability of natural language, and encourages the decoder to extract information from the encoder to help predict the mask sequence, promote the joint training of the encoder-attention-decoder model, and improve the model's understanding of natural language.

[0216] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0217] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A semantic understanding model training method, characterized in that: include: Obtaining a total word sequence corresponding to the training text from the training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence; From the total word sequence, a preset number of continuous word vectors are randomly selected and replaced with a mask sequence to obtain an input word sequence, and the preset number of continuous word vectors are used as a test output word sequence; Inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model on the mask sequence in the input word sequence; According to the difference between the predicted output word sequence and the test output word sequence, a preset optimizer is used to adjust the model parameters of the encoder-attention-decoder model to reduce the difference; Returning to the step of inputting the input word sequence into the encoder-attention-decoder model for training to obtain a predicted output word sequence, and continuing training until a preset training stop condition is met, then stopping training to obtain the semantic understanding model; The mask sequence includes at least one mask vector, the encoder-attention-decoder model includes an encoder and a decoder; the step of inputting the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence includes: Inputting the input word sequence into the encoder, semantically encoding the input word sequence, and obtaining an intermediate semantic vector sequence; wherein the intermediate semantic vector sequence includes the intermediate semantic vectors corresponding to each of the mask vectors; Generate the predicted output word sequence according to the intermediate semantic vector sequence and the input word sequence through the decoder; wherein the predicted output word sequence includes the predicted output words corresponding to each of the mask vectors; The encoder includes a multi-layer neural network, the multi-layer neural network includes an input layer, an intermediate hidden layer and an output layer, the number of the intermediate hidden layers includes multiple layers, the step of inputting the input word sequence into the encoder, performing semantic encoding on the input word sequence, and obtaining an intermediate semantic vector sequence includes: Input the input word sequence into the first intermediate hidden layer of the encoder through the input layer of the encoder; In each intermediate hidden layer of the encoder, according to the attention mechanism, semantic encoding is performed on the input word sequence input by the previous neural network layer layer by layer, and the hidden layer semantic vectors output by each intermediate hidden layer are obtained respectively, until the hidden layer semantic vector output by the last intermediate hidden layer is obtained; The hidden layer semantic vector output by the last intermediate hidden layer is used as the intermediate semantic vector sequence, and is output to the decoder through the output layer of the encoder.

2. The semantic understanding model training method according to claim 1, characterized in that: The step of semantically encoding the input word sequence input by the previous neural network layer layer by layer in each intermediate hidden layer of the encoder according to the attention mechanism to obtain the hidden layer semantic vector output by each intermediate hidden layer includes: In the middle hidden layer of the current layer, respectively obtaining the attention weights of each word vector in the input word sequence input by the previous neural network layer to each mask vector; According to each of the attention weights, weighted summation is performed on each of the word vectors to obtain attention vectors corresponding to each of the mask vectors; The attention vectors corresponding to the mask vectors in the intermediate hidden layer of the current layer are concatenated to obtain the hidden layer semantic vector output by the intermediate hidden layer of the current layer.

3. The semantic understanding model training method according to claim 1, characterized in that: The step of obtaining the total word sequence corresponding to the training text from the training set includes: Obtaining training text from the training set; Segmenting the training text into multiple word segments according to a preset word segmentation method; Convert each of the word fragments into a word vector respectively; The total word sequence corresponding to the training text is constructed according to the word vectors arranged in sequence.

4. The semantic understanding model training method according to claim 3, characterized in that: The step of randomly selecting a preset number of continuous word vectors from the total word sequence and replacing them with a mask sequence to obtain an input word sequence, and using the preset number of continuous word vectors as a test output word sequence includes: Randomly select one of the word vectors from the total word sequence as a starting point; Starting from the starting point, the preset number of continuous word vectors are replaced by the preset number of corresponding mask vectors, and the preset number of mask vectors constitute the mask sequence; The total word sequence after being replaced by the mask sequence is used as the input word sequence, and the preset number of continuous word vectors is used as the test output word sequence.

5. The semantic understanding model training method according to claim 1, characterized in that: The step of determining whether a preset training stop condition is met comprises: Determining whether the perplexity value of the semantic understanding model is less than a preset perplexity value; If so, it is determined that the preset training stop condition is met; The calculation formula of perplexity is: PP(S)= = = Among them, PP represents perplexity; S represents the sentence input to the model; N is the length of the sentence; p(w i ) is the probability of the i-th word appearing given the first i-1 words of a sentence.

6. A semantic understanding model training device, characterized in that: A method for implementing a semantic understanding model training method as claimed in any one of claims 1 to 5, comprising: An acquisition unit, used to acquire a total word sequence corresponding to a training text from a training set, wherein the total word sequence includes a plurality of word vectors arranged in sequence; A masking unit, used to randomly select a preset number of continuous word vectors from the total word sequence and replace them with a masking sequence to obtain an input word sequence, and use the preset number of continuous word vectors as a test output word sequence; A first training unit is used to input the input word sequence into a preset encoder-attention-decoder model for training to obtain a predicted output word sequence, wherein the predicted output word sequence is a prediction result of the encoder-attention-decoder model for the mask sequence in the input word sequence; an adjusting unit, configured to adjust model parameters of the encoder-attention-decoder model using a preset optimizer according to a difference between the predicted output word sequence and the test output word sequence, so as to reduce the difference; The second training unit is used to return to the step of inputting the input word sequence into the encoder-attention-decoder model for training, and continuing the training to obtain the predicted output word sequence until a preset training stop condition is met, then stop the training to obtain the semantic understanding model.

7. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Semantic comprehension training method and system

    CN108417205A

  • Chinese language processing model and method based on deep neural network

    CN110188348A

  • Language processing method and apparatus

    US20170372696A1