Model pre-training method and device, text analysis method and device, and storage medium

Abstract

The present disclosure provides a model pre-training method and device, and a text analysis method and device, and a storage medium, and relates to the technical field of machine learning. The model pre-training method comprises: obtaining a matching sentence corresponding to an initial sentence in a text; obtaining training data according to a predetermined phrase masking strategy; obtaining a first encoding and a first vector representation according to the training data, a BERT model and a convolutional encoder; obtaining a noise encoding and a second vector representation according to the first encoding, a normal distribution and the convolutional encoder; obtaining a second encoding based on a decoder and the noise encoding, and determining a first loss value according to the first vector representation and the second vector representation; obtaining a third encoding according to the first encoding and the second encoding, and obtaining a second loss value; determining the type of encoding in the third encoding by a discriminator, and determining a third loss value; and adjusting the parameters of a target model according to the first loss value, the second loss value and the third loss value until the training is completed.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and in particular to a model pre-training method, apparatus, and text analysis method, apparatus, and storage medium. Background Technology

[0002] Deep learning for semantic similarity has applications in many important scenarios, such as retrieval systems and intelligent question answering, where it is used for semantic recall or fine-grained ranking features. BERT-based semantic similarity analysis technology is a widely adopted approach in related technologies and occupies an important position in semantic retrieval and intelligent question answering in the industry.

[0003] Pre-training based on the BERT model structure plays a crucial role in enhancing the performance of downstream task-oriented models. This is especially true in industry, where a large amount of technical and business-related documentation and corresponding technical standard libraries are generated. Searching for this knowledge and intelligent question answering based on technical documents requires a large number of semantic tags. However, building a large number of semantic tags is costly in industry due to limitations imposed by specialized vocabulary and confidentiality. Summary of the Invention

[0004] One purpose of this disclosure is to avoid reliance on manually labeled data and improve the accuracy and efficiency of model training.

[0005] According to one aspect of some embodiments of this disclosure, a model pre-training method is proposed, comprising: obtaining an initial sentence in a text corresponding to a matching sentence; masking phrases in the initial sentence and the matching sentence respectively according to a predetermined phrase masking strategy to obtain training data; obtaining a first encoding output by a preset BERT model based on the training data and a BERT model, and obtaining a first vector representation based on the first encoding and a convolutional encoder; obtaining a noise encoding based on the first encoding and a normal distribution, and obtaining a second vector representation based on the noise encoding and the convolutional encoder; decoding the noise encoding based on a decoder to obtain a second encoding, and determining a first loss value based on the first vector representation and the second vector representation; obtaining a third encoding based on the first encoding and the second encoding, determining the word probability and prediction loss value of each element position in the third encoding, and obtaining a second loss value; determining the type of encoding in the third encoding by a discriminator, and determining a third loss value; adjusting the parameters of a target model based on the first loss value, the second loss value, and the third loss value until training is completed, wherein the target model includes a BERT model, a convolutional encoder, a decoder, and a discriminator.

[0006] In some embodiments, according to a predetermined phrase masking strategy, masking phrases in the initial statement and the matching statement respectively to obtain training data includes: determining the same phrases and different phrases in the initial statement and the corresponding matching statement; masking a predetermined first proportion of the same phrases and masking a predetermined second proportion of the different phrases in the initial statement and the matching statement respectively to obtain training data, wherein the masked same phrases in the initial statement and the matching statement are the same.

[0007] In some embodiments, obtaining the matching statement corresponding to the initial statement in the text includes: segmenting a predetermined text to obtain a first statement set, wherein the first statement set contains multiple initial statements of the predetermined text; obtaining a predetermined first number of similar statements for each initial statement using a predetermined text search tool, wherein the predetermined first number is greater than 1; and determining the matching statement among the similar statements based on semantic similarity according to the initial statement and the corresponding similar statements.

[0008] In some embodiments, obtaining a predetermined first number of similar statements for each initial statement according to a predetermined text search tool includes: obtaining a set of similar statements based on the initial statement using a predetermined search engine; obtaining a predetermined first number of similar statements in descending order of similarity rate among the statements in the set of similar statements; determining a second set, the second set containing the initial statement, similar statements, and the correspondence between each initial statement and a similar statement; determining a matching statement among the similar statements based on semantic similarity according to the initial statement and the corresponding similar statement includes: obtaining vectors of statements in the second set based on the second set and a word vector conversion algorithm; determining the most similar similar statement as the matching statement based on the cosine distance between the vector of the initial statement and the vector of the corresponding similar statement.

[0009] In some embodiments, the first encoding includes the encoding of the masked initial statement in the training data and the encoding of the masked matching statement in the training data.

[0010] In some embodiments, the first code occupies a predetermined third proportion in the third code, the second code occupies a predetermined fourth proportion, and the sum of the predetermined third proportion and the predetermined fourth proportion is 1.

[0011] In some embodiments, determining the word probability and predicted loss value for each element position in the third encoding, and obtaining the second loss value includes: obtaining an unmasked loss value based on the predicted loss value of the unmasked element position and a predetermined first weight; obtaining a masked loss value based on the predicted loss value of the masked element position and a predetermined second weight; and obtaining the second loss value based on the unmasked loss value and the masked loss value.

[0012] In some embodiments, determining the third loss value includes: determining a first sub-loss value of the discrimination result, wherein a true discrimination result corresponds to the code belonging to the first code, and a false discrimination result corresponds to the code belonging to the noise code; determining a second sub-loss value of whether the discriminant predicted feature is a common word of the sentence pair; and determining the third loss value based on the first sub-loss value and the second sub-loss value.

[0013] In some embodiments, obtaining a first vector representation based on a first encoder and a convolutional encoder includes: inputting the first encoder into a convolutional encoder and obtaining a first vector representation through multiple layers of convolution and max pooling.

[0014] In some embodiments, obtaining a second vector representation based on a noise encoder and a convolutional encoder includes: inputting the noise code into the convolutional encoder and obtaining the second vector representation through multiple convolutions and max pooling.

[0015] In some embodiments, determining the first loss value based on the first vector representation and the second vector representation includes: obtaining the square of the difference between the first vector representation and the second vector representation as the first loss value.

[0016] In some embodiments, a predetermined first weight is less than a predetermined second weight.

[0017] In some embodiments, the predetermined text is unstructured text from a vertical domain.

[0018] According to one aspect of some embodiments of this disclosure, a question-answering service method is proposed, comprising: obtaining a statement to be analyzed; determining a predetermined first statement corresponding to the statement to be analyzed based on a statement analysis model, wherein the statement analysis model is generated by training according to any of the model pre-training methods mentioned above; determining a target statement in the predetermined second statement according to the correspondence between the predetermined first statement and the predetermined second statement; and feeding back the target statement.

[0019] According to one aspect of some embodiments of this disclosure, a model pre-training apparatus is proposed, comprising: a statement determination unit configured to acquire a matching statement corresponding to an initial statement in a text; a training data acquisition unit configured to acquire training data by masking phrases in the initial statement and the matching statement respectively according to a predetermined phrase masking strategy; and a training unit configured to: acquire a first encoding output by a preset BERT model based on the training data and a BERT model, and acquire a first vector representation based on the first encoding and a convolutional encoder; acquire a noise encoding based on the first encoding and a normal distribution, and acquire a second vector representation based on the noise encoding and the convolutional encoder; decode the noise encoding based on a decoder to acquire the second encoding, and determine a first loss value based on the first vector representation and the second vector representation; acquire a third encoding based on the first encoding and the second encoding, determine the word probability and prediction loss value of each element position in the third encoding, and acquire a second loss value; determine the type of encoding in the third encoding by a discriminator, and determine a third loss value; and adjust the parameters of a target model based on the first loss value, the second loss value, and the third loss value until training is completed, wherein the target model includes a BERT model, a convolutional encoder, a decoder, and a discriminator.

[0020] According to one aspect of some embodiments of this disclosure, a question-answering service apparatus is proposed, comprising: an input unit configured to acquire a statement to be analyzed; a statement analysis unit configured to: determine a predetermined first statement corresponding to the statement to be analyzed based on a statement analysis model, wherein the statement analysis model is generated by training according to any of the model pre-training methods mentioned above; determine a target statement in the predetermined second statement according to the correspondence between the predetermined first statement and the predetermined second statement; and an output unit configured to provide feedback on the target statement.

[0021] According to one aspect of some embodiments of the present disclosure, a data processing apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any of the methods described above based on instructions stored in the memory.

[0022] According to one aspect of some embodiments of this disclosure, a non-transitory computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of any of the methods mentioned above. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0024] Figure 1 Flowcharts are shown for some embodiments of the model pre-training method disclosed herein.

[0025] Figure 2A Flowcharts are shown for some other embodiments of the model pre-training method disclosed herein.

[0026] Figure 2B This is a schematic diagram of some embodiments of the statement pair generation process in the model pre-training method of this disclosure.

[0027] Figure 2C This is a schematic diagram of some embodiments of the training data generation process in the model training method disclosed herein.

[0028] Figure 3A This diagram illustrates some embodiments of the model training method disclosed herein, which uses the GAN (Generative Adversarial Networks) concept for predicting similar and dissimilar phrases.

[0029] Figure 3B These are schematic diagrams illustrating other embodiments of the model training method disclosed herein.

[0030] Figure 4 These are schematic diagrams illustrating some embodiments of the question-and-answer service method disclosed herein.

[0031] Figure 5 These are schematic diagrams of some embodiments of the model pre-training apparatus disclosed herein.

[0032] Figure 6 These are schematic diagrams illustrating some embodiments of the question-and-answer service apparatus disclosed herein.

[0033] Figure 7 These are schematic diagrams illustrating some embodiments of the data processing apparatus of this disclosure.

[0034] Figure 8 These are schematic diagrams of other embodiments of the data processing apparatus of this disclosure. Detailed Implementation

[0035] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] The inventors discovered that related text semantic methods are all based on pre-training on public or vertical domain data, and then trained with a large amount of downstream labeled data to achieve a certain degree of applicability in the vertical domain. However, vertical domain vocabulary is sparse in public data, so current methods require a large amount of domain-labeled data. In addition, the semantic pre-training methods in related technologies have low accuracy and long training cycles.

[0037] To address the aforementioned issues, this disclosure proposes a model pre-training method and apparatus, as well as a method and apparatus for providing services using the pre-trained model, thereby improving the accuracy and efficiency of model training.

[0038] Flowcharts of some embodiments of the model pre-training method disclosed herein are as follows: Figure 1 As shown.

[0039] In step 101, the matching statement corresponding to the initial statement in the text is obtained. In some embodiments, there is a one-to-one correspondence between the initial statement and the matching statement.

[0040] In some embodiments, the text can be first segmented into sentences to obtain a series of initial sentences, and then matching sentences can be obtained for each initial sentence. In some embodiments, the matching sentences are those that are literally similar to the initial sentences, or those that have a close vector distance after the sentences are converted into vectors.

[0041] In some embodiments, the initial statement can be retrieved using text search tools in related technologies, such as the ES distributed search and analysis engine, to obtain literally similar statements. This method enables the rapid acquisition of matching statements. Although the similarity in meaning between the searched similar or matching statements and the initial statement is questionable due to the limitations of search engines, this possibility of dissimilarity is permissible in the model pre-training method of this disclosure and will not affect the model training effect.

[0042] In step 102, phrases are masked in the initial sentence and the matched sentence according to the predetermined phrase masking (MASK) strategy to obtain training data.

[0043] In some embodiments, the predetermined phrase masking strategy can be any of the related technologies, such as the MASK method used in the training of the BERT model, which randomly masks some words or phrases in a sentence, or selects words or phrases with certain attributes (such as nouns, verbs, subjects, predicates, modifiers, etc.) for masking.

[0044] In some embodiments, the identical and different phrases in the initial statement and the corresponding matching statement can be determined first; then, in the initial statement and the matching statement, a portion of the identical phrases and a portion of the different phrases are masked respectively to obtain training data. In some embodiments, the masked identical phrases in the initial statement and the matching statement are the same. In some embodiments, training data can be obtained by masking identical phrases according to a preset ratio, such as a predetermined first ratio and a predetermined second ratio. In some embodiments, the predetermined first ratio can be 5%, and the predetermined second ratio can be 10%, and the specific values ​​can be set and adjusted as needed. This method allows the model to better focus on semantic information.

[0045] In step 103, based on the training data and the BERT model, the first encoding of the preset BERT model output is obtained, and the first vector representation is obtained based on the first encoding and the convolutional encoder.

[0046] In some embodiments, the input can be based on the training data and the pre-training method of the BERT model in related technologies to obtain the output of the BERT model, which serves as the first encoding. In some embodiments, the first encoding includes the encoding of the masked initial statement in the training data and the encoding of the masked matching statement in the training data.

[0047] In some embodiments, the model input mainly includes three parts:

[0048] (1) The word embedding tensor of the first encoding;

[0049] (2) The first encoding position embedding tensor: position embeddings;

[0050] (3) Segment embedding of the first encoded statement tensor.

[0051] In some embodiments, the first encoding is input into the convolution encoder, and the convolution formula is:

[0052]

[0053] This convolutional encoder uses multi-layer convolution and max-pooling processing (the processing of the convolutional encoder is represented as y). i =maxpooling(conv(x) i ))), obtain the first vector representation.

[0054] The inventors discovered that many related technologies use the average or maximum value of BERT output as the comprehensive semantic information of a sentence, which results in a significant loss of lexical information. To address this issue, this disclosure employs a convolutional autoencoder method to obtain comprehensive sentence information, reducing the amount of information loss and improving the accuracy and efficiency of training.

[0055] In step 104, a noise code is obtained based on the first code and the normal distribution, and a second vector representation is obtained based on the noise code and the convolutional encoder.

[0056] In some embodiments, a normal distribution can be used to generate noise codes, with the first code serving as input to the normal distribution. The high-dimensional formula for the normal distribution is:

[0057]

[0058] Where u is the average value of the input, and the final sampling yields the noise code.

[0059] In some embodiments, noise can be encoded and input into a convolutional encoder to obtain a second vector representation through multiple layers of convolution and max pooling. This convolutional encoder is the same as or similar to the convolutional encoder in step 103.

[0060] In step 105, the noise encoding is decoded based on the decoder, and a first loss value is determined according to the first vector representation and the second vector representation. By using the intermediate output as a comprehensive representation of the sentence to calculate the loss value, this method further reduces the loss of lexical information in the sentence, improving the accuracy and efficiency of training.

[0061] In some embodiments, the decoder is obtained by upsampling and augmented convolution operations, and the decoding process is expressed by formula y. i =upsampling(conv(x) i The decoder decodes the noise code, and the decoded result is called the second code.

[0062] In some embodiments, the square of the difference between the first vector representation and the second vector representation can be obtained as the first loss value.

[0063] In step 106, a third code is obtained based on the first code and the second code, the word probability and prediction loss value of each element position in the second code are determined, and the second loss value is obtained.

[0064] In some embodiments, the first code occupies a predetermined third proportion, the second code occupies a predetermined fourth proportion, and the sum of the predetermined third proportion and the predetermined fourth proportion is 1. In some embodiments, the predetermined third proportion can be 70%, and the predetermined fourth proportion can be 30%, and the specific values ​​can be set or adjusted as needed.

[0065] In some embodiments, MLP is used to predict the probability of words at all corresponding positions, where the dimension of each element is vocab_size, which is the size of the vocabulary.

[0066] In some embodiments, the prediction loss can be calculated using cross-entropy loss, and weights can be calculated for both masked and non-masked elements. For example, the unmasked loss value can be obtained based on the prediction loss value of the unmasked element position and a predetermined first weight; the masked loss value can be obtained based on the prediction loss value of the masked element position and a predetermined second weight; and a second loss value can be obtained based on the unmasked loss value and the masked loss value.

[0067] In some embodiments, the predetermined first weight is less than the predetermined second weight, for example, the predetermined first weight is 0.05 and the predetermined second weight is 0.95, then the third loss value Vloss =0.05*Vactual loss +0.95*Mask loss The specific weight allocation can be set or adjusted as needed.

[0068] In step 107, the discriminator determines the type of encoding in the third encoding and determines the third loss value.

[0069] In some embodiments, the features of the hybrid third encoding can be used by a discriminator to determine whether the input features are true (derived from the first encoding) or false (derived from the second encoding).

[0070] In some embodiments, determining the third loss value includes: determining a first sub-loss value of the discrimination result, wherein a true discrimination result corresponds to the code belonging to a first code, and a false discrimination result corresponds to the code belonging to a second code; determining a second sub-loss value to determine whether the predicted feature is a common word of the sentence pair; and determining the third loss value based on the first sub-loss value and the second sub-loss value. In some embodiments, the third loss value is represented by D using binary cross-loss. loss =DTF loss +Dco loss Among them, DTF loss For the first sub-loss value mentioned above, Dco loss This refers to the second type of loss value mentioned above.

[0071] In step 108, the parameters of the target model are adjusted based on the first loss value, the second loss value, and the third loss value until training is complete. The target model includes a BERT model, a convolutional encoder, a decoder, and a discriminator. In some embodiments, the comprehensive loss value can be expressed as: All loss =MSE loss +V loss +D loss .

[0072] The method described in the above embodiments eliminates the need for training based on labeled sample information, reducing reliance on a large amount of semantic annotation and expanding the scope of application. During model pre-training, sentence comprehensive information is obtained through convolutional autoencoders, and a loss value is determined based on the sentence comprehensive information, reducing the amount of sentence information loss and improving the accuracy and efficiency of training.

[0073] In some embodiments, the method for obtaining the initial statement and its corresponding matching statement based on the text in step 101 above can be as follows: Figure 2A As shown.

[0074] In step 211, a first set of statements is obtained by text segmentation of the predetermined text. The first set of statements contains multiple initial statements of the predetermined text.

[0075] In some embodiments, the predetermined text is one or more unstructured texts from a vertical domain of the target domain. For example, analysis targeting the mechanical domain may employ unstructured text from the mechanical domain. In some embodiments, a large number of documents from the target domain may be collected in advance as the predetermined text.

[0076] In some embodiments, the predetermined text is segmented, for example, by segmenting the text according to a terminator (such as a period, exclamation mark, etc.) to obtain the initial statement.

[0077] In some embodiments, such as Figure 2B As shown, multiple text files are represented by DOC = [d1, d2, ..., d...]. n ] indicates that d i Let i represent a file, where i is an integer between 1 and n, and n is the number of text files, where n is a positive integer. Each text file is split according to an end-of-file character, such as "." or "!", forming the initial set of intermediate output statements: SENT = [s1, s2, ..., s...]. m ], where m is a positive integer not less than n.

[0078] In step 212, a predetermined first number of similar statements are obtained for each initial statement using a predetermined text search tool, wherein the predetermined first number is greater than 1.

[0079] In some embodiments, a set of similar statements can be obtained based on an initial statement using a predetermined search engine; then, a predetermined first number of similar statements can be obtained in descending order of similarity rate among the statements in the set of similar statements, and a second set can be determined. The second set includes the initial statement, similar statements, and the correspondence between each initial statement and a similar statement.

[0080] In some embodiments, such as Figure 2B As shown, using the Elasticsearch search engine to index data, for each initial statement data, the Elasticsearch index is used to obtain literal statistical similarity results to form a dictionary set. We take the first predetermined number of statements (e.g., 3) as candidate similar statements, and the result is represented as the set of the initial statement and its corresponding similar statements:

[0081] CAN Dict ={s1:[c 11 ,c 12, c 13 ],s2:[c 21 ,c 22 ,c 23 ],…,s m :[c m1 ,c m2 ,c m3 ]}.

[0082] In step 213, based on the initial statement and the corresponding similar statements, the matching statement is determined from the similar statements based on semantic similarity.

[0083] In some embodiments, vectors of statements in the second set are obtained based on the second set and the word vector conversion algorithm; the most similar statement is determined as the matching statement based on the cosine distance between the vector of the initial statement and the vector of the corresponding similar statement.

[0084] In some embodiments, such as Figure 2B As shown, a vertical domain word2vec algorithm can be trained using the SENT set, representing a sentence s using word vectors. i =[v1,v2,…,v q ], where v i ∈R 1*d d represents the dimension of the word vector. Sentence s i The sentence vector is represented by the average of the word vectors, which is...

[0085] Furthermore, calculate c separately. i1 ,c i2 ,c i3 With s i The similarity. In some embodiments, it can be calculated using the following formula:

[0086] s i _similar=max(cos(s i ,c i1 ),cos(s i ,c i2 ),cos(s i ,c i3 ))

[0087] Identify similar statements corresponding to the maximum value and use them as matching statements.

[0088] Using the methods described in the above embodiments, a statement pair consisting of the initial statement and the matched statement is obtained (train). set =[(s1,c1),(s2,c2),…,(s m ,c m )).

[0089] The methods described in the embodiments above enable the automatic generation of sentence pairs from unannotated text in the corresponding domain. These pairs serve as the basis for subsequent data processing, reducing the limitations imposed by the number of samples on model preprocessing and facilitating the provision of sufficient samples for pre-training, thus promoting wider application. In some embodiments, since the automated processing cannot guarantee that the initial sentences and the matched sentences are semantically similar, they can be referred to as pseudo-semantic similarity pairs.

[0090] In some embodiments, the pseudo-semantic similarity pairs generated in step 213 above are used for training. set Test data is generated. In some embodiments, a masking strategy is applied to identical phrases and different phrases in the sentence pair, which allows the model to better focus on semantic information. For example... Figure 2C As shown, the specific steps are as follows:

[0091] First, based on training set Taking one statement pair as an example, let's consider the statement pair (s) i ,c i ) is represented as:

[0092] sent oair =(s i =[w1,w2,…,w q1 [,c i =[w′1,w′2,…,w′ q2 ]),

[0093] Here, w represents the words that make up the sentence.

[0094] The `differ` function is used to extract the identical and different phrases between two sentences. In some embodiments, a label `label_SD` can also be generated, which identifies whether the words or phrases in the sentence pair are the same. For example, `label_SD[0]` identifies whether `w1` and `w′1` are the same. In some embodiments, the label value for the position corresponding to the same word or phrase is 1, and the label value for the different position is 0. In some embodiments, the label value for the same position can also be set to 0, and the label value for the different position can be set to 1.

[0095] In some embodiments, 5% of the identical phrases are masked, and 10% of the different phrases are masked, allowing the model to capture more of the shared and differing information in sentence pairs. In some embodiments, the specific masking strategy for both parts is similar to the masking method in BERT related technologies, except that when masking identical phrases, identical phrases corresponding to both sentences are masked simultaneously.

[0096] The final training data is represented as follows:

[0097] sent pair_mask =

[0098] (s i =[w1,w2,[MASK] same1 [MASK] same2 ,…,[MASK] differ1 ,w q1 ],

[0099] c i =[w′1,w′2,[MASK] same1 [MASK] same2 ,…,w′ i ,[MASK]′ differ1 ,w′ q2 ])

[0100] Here, q1 and q2 represent the length of the sentence, and they are not necessarily equal. [MASK] samei The i-th identical phrase (or word) that is masked. differi The i-th phrase (or word) is the number of different phrases (or words) of the corresponding category that are masked. i is a positive integer, ranging from 1 to the number of times the phrase is masked.

[0101] The final training data is represented as follows:

[0102] train set =[sent pair_mask1 ,sent pair_mask2 ,…,sent pair_maskm ].

[0103] By using the method described in the above embodiments, the model can acquire more identical and different information in sentence pairs by separately masking identical and different words, which is beneficial to improving the efficiency and accuracy of subsequent training.

[0104] Schematic diagrams of some embodiments of the model training method disclosed herein, which uses GAN concepts for phrase prediction, are shown below. Figure 3A As shown.

[0105] 1. The training data obtained in the previous steps is sent pair_mask Building upon this, the method of inputting training data into the BERT model is the same as in related techniques. The model input mainly consists of three parts:

[0106] (1)sent pair_mask vocabulary;

[0107] (2)sent pair_mask Positional embedding;

[0108] (3) sent pair_mask Segment embedding;

[0109] The first encoding output by the BERT model is represented as:

[0110] sent pair_encoder =

[0111]

[0112] Among them, sv i With CV i ∈R 1*d sv i , for outputting words or phrases that were not masked in the initial statement, cv i This outputs words or phrases that were not masked in the initial statement. `smaskv` samej To output the same words or phrases that are masked in the initial statement, cmaskv' samej To output identical words or phrases that are masked in the matched statement, smaskv differj cmaskv outputs different words or phrases masked in the initial statement. differj Output the different words or phrases that are masked in the matching statement. j is a positive integer, ranging from 1 to the number of phrases of the corresponding category that are masked. Figure 3A In the middle, the sent output of the BERT model pair_encoder Represented as v1~vk.

[0113] 2. Furthermore, convolutional autoencoders are used to obtain comprehensive sentence information, such as... Figure 3A As shown in the conv diagram. Based on sent... pair_encoder The encoded information of the two sentences is input into the convolutional encoder, where the convolution formula is expressed as:

[0114]

[0115] The operation of the convolution encoder uses y i =maxpooling(conv(x) i ))express.

[0116] By utilizing multiple convolutional layers and max pooling, an intermediate representation is finally obtained, such as Figure 3A As shown in V_m, this intermediate representation is used as the first vector representation. The first vector representation can be expressed as:

[0117] ymiddle =([v 11 ,v 12 ,…,v 1k ],[v 21 ,v 22 ,…,v 2k ])

[0118] Where k represents the vector dimension, which is typically 768.

[0119] 3. For example Figure 3A As shown, combining the principles of convolutional autoencoders and GAN concepts, a noise code `sent_noise` is generated using a normal distribution, such as... Figure 3A The sampler shows a normal distribution. The first encoding sent... pair_encoder As input to a normal distribution, the output is sampled to obtain the noise code.

[0120]

[0121] Furthermore, by utilizing the convolution operation y i =maxpooling(conv(x) i The noise coding is processed to obtain the comprehensive noise coding information, which is used as the second vector representation, and can be expressed as follows:

[0122] y′ middle =([v′) 11 ,v′ 12 ,…,v′ 1k ],[v′ 21 ,v′ 22 ,…,v′ 11 ])

[0123] like Figure 3A As shown in the Generator diagram, upsampling and augmented convolution operations are used to obtain the decoder. The decoding operation can be performed using the formula y. i =upsampling(conv(x) i )) indicates that the decoded result is the second encoding, represented as: sent pairnoise_decoder .

[0124] We train this part using the mean squared error loss, but unlike GANs, we utilize the loss of intermediate integrated information instead of the generated sent. pairnoise_decoder with sent pair_encoder The loss is represented by intermediate output as a comprehensive statement. The mean squared error loss is expressed as: MSE loss =(y middle -y′ middle ) 2 .

[0125] 4. Based on the second encoding obtained from the generator above and the first encoding output by BERT, a third encoding is formed by selecting the first encoding with a 70% probability and selecting the noise encoding with a 30% probability. In some embodiments, the third encoding is represented as:

[0126]

[0127] Then, MLP is used to predict the word probabilities at all corresponding positions. pre =[s pro1 ,s pro2 ,…,s proq1 ,c pro1 ,c pro2 ,…,c proq2 ], where s pro1 ~s proq1 c represents the word probability corresponding to the words in the initial statement. pro1 ~c proq2 Each element has a dimension of vocab_size, which represents the vocabulary probability corresponding to the words in the matched statement.

[0128] The prediction loss is calculated using cross-entropy loss, and a second loss value is obtained by weighting masked and non-masked objects. The formula for the second loss value can be expressed as:

[0129] V loss =0.05*Vactual loss +0.95*Mask loss

[0130] Among them, Vactual loss For loss values ​​targeting non-masked words or phrases, Mask loss This represents the loss value for the words or phrases in the mask.

[0131] 5. For the final encoded features obtained through mixing, a discriminator is used to determine whether the input features are true (from BERT) or false (from the generator), such as... Figure 3A The Discriminator is shown in the diagram. It sets up a phrase co-occurrence judgment, determining whether each predicted feature is a word co-occurring in two sentences; it obtains a binary cross-classification loss, which is used as the third loss value, expressed as:

[0132] D loss =DTF loss +Dco loss ,

[0133] Among them, DTF loss To determine the loss for distinguishing true from false features, Dco lossThe loss is due to whether or not the words are co-occurring.

[0134] The overall model is trained by combining the above loss modules, as expressed by the formula: All loss =MSE loss +V loss +D loss .

[0135] Such a model covers the similarities and differences between sentence pairs, which is more conducive to the representation of semantic similarity; it solves the problem that a large amount of data annotation is required due to the sparsity of vertical domain vocabulary in public data, improves the accuracy and efficiency of pre-training, and is conducive to the promotion and application.

[0136] [A model pre-training method using a text-based pre-training scheme for the mechanical field as an example]

[0137] The data and files in this embodiment are for illustrative purposes only and do not constitute an undue limitation on this application.

[0138] like Figure 3B As shown, the model pre-training method, taking a text-based pre-training scheme for the mechanical field as an example, mainly includes the following steps:

[0139] Step 1: Construct pseudo-similar question pairs.

[0140] Taking open national standards data for machinery as an example, we collect a document collection (DOC).

[0141] DOC = [Hydraulic Excavator Technical Specifications.pdf]

[0142] General Bridge Crane.pdf

[0143] ……,

[0144] [National Standards for Tire Rollers.pdf]

[0145] The text is divided into sentences according to symbols, paragraphs, and chapters, for example, the segmentation marker is ".5.1.3", which ultimately forms the phrase set SENT.

[0146] SENT = [

[0147] All components requiring lubrication should be equipped with reliable and easy-to-maintain lubrication devices;

[0148] What are the requirements for compaction by a road roller?

[0149] …;

[0150] When the road roller is towed after the engine is turned off, the steering mechanism should be able to steer. ]

[0152] The indexes of the segmented sentences are added to Elasticsearch (ES). ES statistical features are used to rank each sentence and select the top 3, resulting in a candidate set called CAK. Dict .

[0153] CAN Dict ={

[0154] All components requiring lubrication should be equipped with reliable and easily maintained lubrication devices.

[0155] The lubrication devices installed on components requiring lubrication are maintainable;

[0156] Lubrication methods and lubrication devices: classification, characteristics, and applications;

[0157] [The number of parts requiring lubrication is increasing.]

[0158] What are the requirements for compaction by a road roller?

[0159] What are the compaction design requirements for a stamping roller?

[0160] Construction requirements and key technical points for road rollers;

[0161] What qualifications are needed to operate a road roller?

[0162] ……,

[0163] When the road roller is towed after the engine is turned off, the steering mechanism should be able to steer:

[0164] [Engine stopped, power steering low;]

[0165] Self-steering and traction steering axle mechanism;

[0166] Composition of the steering mechanism

[0167] Then, the word2vec method is used to map sentences to vectors of corresponding words. This process can be implemented using any of the relevant techniques that use word2vec to map sentences to vectors of corresponding words. The resulting set of similar question pairs is as follows:

[0168] train set =[

[0169] (All components requiring lubrication should be equipped with reliable and easily maintainable lubrication devices, and the lubrication devices installed on the components requiring lubrication should be maintainable.)

[0170] (What are the requirements for compaction by road rollers? What are the design requirements for compaction by impact road rollers?)

[0171] …,

[0172] (When the road roller is towed after the engine is turned off, the steering mechanism should be able to turn; a self-steering and traction steering axle mechanism) ]

[0174] Step 2: Comparing similar and dissimilar statements for pseudo-similar statements mainly yields two aspects.

[0175] 1. Similarity and difference statement tags.

[0176] With training set Taking the first element as an example: here train set [0] = (All parts requiring lubrication should be equipped with reliable and easily maintained lubrication devices, and the lubrication devices installed on the parts requiring lubrication should be maintainable.)

[0177] After comparing similarities and differences, we obtain same_set = [need, lubrication, assembly, component, lubrication device, maintenance], and difference_set = [required, need, zero, ...].

[0178] Therefore, we obtain the similarity and difference label as label_SD = [0,1,1,1,1,0,1,1,0,0,1,0,0,0,0,0,1,1,1,1,1,1,1;1,0,1,1,1,1,1,1,0,1,0,1,1,1,1,0,0,1,1,1,1,0,0,1,1,1,1].

[0179] 2. Training Data. Using a similarity-difference masking strategy, 20% of the total length is masked, with 5% masking identical phrases and 10% masking different words, to obtain training data. For example, train... set [0] = (All required [MASK][MASK][MASK] components are equipped with reliable and easy-to-maintain lubrication devices; the lubrication devices installed on the required [MASK][MASK][MASK] components are maintainable.)

[0180] The above completes the core process of step two.

[0181] Step 3: Input the training data into BERT. The BERT model outputs the first encoding of the MASK statement pair, denoted as bert_encoder, for example,

[0182] bert_encoder = [[0.0568,-0.125,…,0.854],…,[0.547,0.002,-0.985]], corresponding to dimensions [45,768], where the first number represents the length of the statement pair and the second represents the vector dimension.

[0183] Step 4: As shown in Figure 3, the BERT output is first processed through multiple convolutional and max-pooling operations to obtain the comprehensive statement vector, represented as y. middle For example, y middile = ([0.645,0.213,...,-0.021],[0.789,0.345,...,0.006]), with dimensions [2,768], where 2 represents a two-statement pair and 768 is the vector dimension.

[0184] The first encoding is sampled using a normal distribution to obtain the noise encoding noise_vec, such as noise_vec = [[0.987, 0.269, ..., 0.387], ..., [0.012, 0.412, 0.062]], with the same dimensions [45, 768]. After multiple mapping transformations, the resulting vector has the same dimensions as the BERT output, and simultaneously generates the combined vector y' of the two generated statements. middle =([0.742,0.213,…,0.251],

[0185] [0.215, 0.618, ..., 0.512]). Calculate MSE_loss = (y middle -y′ middle ) 2 =6.79, where the loss value is an assumed value.

[0186] Simultaneously, the decoded vector gan_decoder, with the same dimensions as bert_encoder, is obtained as the second encoder. For example,

[0187] gan_decoder=[[0.219,-0.439,…,0.024],…,[0.697,0.117,-0.287]],

[0188] The same dimensions are [45, 768].

[0189] The third encoding, denoted as com_encoder, is obtained by selecting the character representation generated by gan_decoder with a 30% probability and the character representation generated by bert_encoder with a 70% probability. For example:

[0190] com_encoder=[[0.0568,-0.125,…,0.854],…,[0.697,0.117,-0.287]]

[0191] After the discriminator judges the third code, the probability of the word prediction result vocab_pro is obtained. For example, vocab_pro = [[0.161,0.745,…],…,[0.002,0.003,…]], where the dimension is [45,21118], and 21118 is the number of words.

[0192] Whether the prediction result is derived from real data is expressed as isreal_pro, for example, isreal_pro = [[0.890,0.110],…,[0.105,0.895]], with dimensions [45,2]; whether the prediction result is derived from two statements predicting the same phrase is expressed as is_identical, for example, is_identical = [[0.640,0.260],…,[0.780,0.220]], with dimensions [45,2]; the loss is calculated by comparing the results with the actual labels, and then the training model parameters are fed back.

[0193] Based on the embodiments described above, this disclosure proposes a pre-training method for semantic similarity of mechanical domain texts generated by GAN-based autoencoding of similar and dissimilar phrases. First, a large number of unstructured documents from the mechanical industry are collected, and "pseudo-similar question pairs" are constructed using a word2vec algorithm and ES. Second, based on the BERT pre-training concept, the data is constructed using a masking strategy of similar and dissimilar phrases to create similar question pairs. Third, the masked data is input into the BERT model to obtain corresponding outputs. Then, the autoencoding concept is used to obtain the comprehensive semantic vectors corresponding to the two sentences. GAN-based noise-similar BERT outputs with the same dimension are generated based on a normal distribution. Finally, a portion of the GAN noise output and a portion of the original BERT output are used as the final comprehensive model output. Finally, a discriminator is used to predict whether the word feature vectors originate from the original BERT input, predict the masked phrases, and predict whether the output position is the same phrase. This method improves the accuracy of text semantic similarity in vertical domains, while optimizing the previous pre-training methods that did not pay attention to the key "difference and similarity" phrase information in sentence pairs, and generates intermediate representations for each sentence pair; it solves the problems of difficult data construction and model training in vertical domains, and effectively reduces the work of manual annotation of industrial data.

[0194] Schematic diagrams of some embodiments of the question-and-answer service method disclosed herein are shown below. Figure 4 As shown.

[0195] In step 401, the statement to be analyzed is obtained. In some embodiments, the statement to be analyzed may be a question entered by a user or obtained through a predetermined channel. In some embodiments, the question relates to the field of mechanics.

[0196] In step 402, based on the statement to be analyzed, a predetermined first statement corresponding to the statement to be analyzed is determined using a statement analysis model. The statement analysis model is generated by training using any of the model pre-training methods mentioned above. In some embodiments, the predetermined first statement is a pre-stored question with a corresponding answer. In some embodiments, the text used when training the statement analysis model belongs to the same domain as the question obtained in step 401.

[0197] In step 403, the target statement is determined in the predetermined second statement according to the correspondence between the predetermined first statement and the predetermined second statement. In some embodiments, the predetermined second statement is the answer to the predetermined first statement.

[0198] In step 404, the target statement is fed back.

[0199] This approach enables the use of models obtained through the model pre-training method described above to provide services such as intelligent question answering, improving the efficiency of service deployment, facilitating the expansion of application scenarios, and enhancing the accuracy of responses.

[0200] Schematic diagrams of some embodiments of the model pre-training device 50 disclosed herein are shown below. Figure 5 As shown.

[0201] The statement determination unit 501 can obtain the matching statement corresponding to the initial statement in the text. In some embodiments, the statement determination unit 501 can perform the above-described step 101 or Figure 2A Any of the operations shown in the embodiments.

[0202] The training data acquisition unit 502 can acquire training data by masking phrases in the initial sentence and the matching sentence according to a predetermined phrase masking strategy. In some embodiments, the training data acquisition unit 502 can perform the above-described step 102 or Figure 2C Any one of the operations shown in the embodiments.

[0203] The training unit 503 can obtain a first encoding output by a preset BERT model based on training data and the BERT model, and obtain a first vector representation based on the first encoding and the convolutional encoder; obtain a noise encoding based on the first encoding and a normal distribution, and obtain a second vector representation based on the noise encoding and the convolutional encoder; decode the noise encoding based on the decoder to obtain a second encoding, and determine a first loss value based on the first and second vector representations; obtain a third encoding based on the first and second encodings, determine the word probability and prediction loss value for each element position in the third encoding, and obtain a second loss value; determine the type of encoding in the third encoding through a discriminator, and determine a third loss value; adjust the parameters of the target model based on the first, second, and third loss values ​​until training is complete, wherein the target model includes a BERT model, a convolutional encoder, a decoder, and a discriminator. In some embodiments, the training unit 503 can perform steps 103 to 108 above, or any one of the operations in the embodiment shown in Figure 3.

[0204] Such a device does not rely on labeled sample information for training, reducing the dependence on a large number of semantic annotations and expanding the scope of applications. During the model pre-training process, it obtains comprehensive sentence information through convolutional autoencoders and determines a loss value based on the comprehensive sentence information, which reduces the amount of sentence information loss and helps to improve the accuracy and efficiency of training.

[0205] Schematic diagrams of some embodiments of the question-and-answer service device 61 disclosed herein are shown below. Figure 6 As shown.

[0206] Input unit 611 can acquire the statement to be analyzed.

[0207] The statement analysis unit 612 can determine a predetermined first statement corresponding to the statement to be analyzed based on a statement analysis model. The statement analysis model is generated by training using any of the model pre-training methods mentioned above. In some embodiments, the predetermined first statement is a pre-stored question with a corresponding answer. In some embodiments, the text used when the statement analysis model is trained belongs to the same domain as the question obtained in step 401. Further, based on the correspondence between the predetermined first statement and the predetermined second statement, the target statement is determined in the predetermined second statement. In some embodiments, the predetermined second statement is the answer to the predetermined first statement.

[0208] Output unit 613 can provide feedback on the target statement.

[0209] Such a device can provide services such as intelligent question answering using models obtained based on the model pre-training method shown above, which improves the efficiency of service launch, facilitates the expansion of application scenarios, and helps improve the accuracy of responses.

[0210] A schematic diagram of the structure of an embodiment of the data processing device disclosed herein is shown below. Figure 7 As shown, the data processing device can be a model pre-training device or a question-answering service device. The data processing device includes a memory 701 and a processor 702. The memory 701 can be a disk, flash memory, or any other non-volatile storage medium. The memory is used to store instructions in the corresponding embodiments of the model pre-training method or question-answering service method described above. The processor 702 is coupled to the memory 701 and can be implemented as one or more integrated circuits, such as a microprocessor or microcontroller. The processor 702 is used to execute the instructions stored in the memory, which can improve the accuracy and efficiency of model training.

[0211] In one embodiment, it can also be as follows: Figure 8 As shown, the data processing device 800 includes a memory 801 and a processor 802. The processor 802 is coupled to the memory 801 via a BUS bus 803. The data processing device 800 can also be connected to an external storage device 805 via a storage interface 804 to access external data, and can also be connected to a network or another computer system (not shown) via a network interface 806. Further details are omitted here.

[0212] In this embodiment, storing data instructions in a memory and then processing those instructions with a processor can improve the accuracy and efficiency of model training.

[0213] In another embodiment, a computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the steps of the method in the corresponding embodiment of the model pre-training method or question-answering service method. Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0218] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit them; although this disclosure has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of this disclosure or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of this disclosure, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in this disclosure.

Claims

1. A model pre-training method, comprising: Obtaining the matching statement corresponding to the initial statement in the text includes: segmenting a predetermined text to obtain a first statement set, the first statement set containing multiple initial statements of the predetermined text; using a predetermined text search tool to obtain a predetermined first number of similar statements for each initial statement, the predetermined first number being greater than 1; and determining the matching statement among the similar statements based on semantic similarity, according to the initial statement and the corresponding similar statements. According to a predetermined phrase masking strategy, phrases are masked in the initial statement and the matching statement respectively to obtain training data, including: determining the same phrases and different phrases in the initial statement and the corresponding matching statement; masking a predetermined first proportion of the same phrases and masking a predetermined second proportion of different phrases in the initial statement and the matching statement respectively to obtain training data, wherein the masked same phrases in the initial statement and the matching statement are the same. Based on the training data and the BERT model, obtain the first encoding of the preset BERT model output, and obtain the first vector representation based on the first encoding and the convolutional encoder; Based on the first encoding and the normal distribution, a noise encoding is obtained, and based on the noise encoding and the convolutional encoder, a second vector representation is obtained; The noise code is decoded by the decoder to obtain the second code, and a first loss value is determined based on the first vector representation and the second vector representation. Based on the first encoding and the second encoding, a third encoding is obtained, and the word probability and prediction loss value of each element position in the third encoding are determined to obtain a second loss value. The discriminator determines the type of encoding in the third encoding and determines the third loss value; The parameters of the target model are adjusted based on the first loss value, the second loss value, and the third loss value until training is complete. The target model includes the BERT model, the convolutional encoder, the decoder, and the discriminator.

2. The method according to claim 1, wherein, The step of obtaining a predetermined first number of similar statements for each initial statement using a predetermined text search tool includes: Based on a predetermined search engine, a set of similar statements is obtained according to the initial statement; Based on the order of similarity rates of the statements in the set of similar statements from high to low, a predetermined first number of similar statements are obtained, and a second set is determined. The second set includes the initial statement, the similar statements, and the correspondence between each initial statement and the similar statement. The step of determining the matching statement among the similar statements based on semantic similarity, according to the initial statement and the corresponding similar statements, includes: Based on the second set and the word vector conversion algorithm, obtain the vectors of the sentences in the second set; The most similar statement is determined as the matching statement based on the cosine distance between the vector of the initial statement and the vector of the corresponding similar statement.

3. The method according to claim 1, wherein, The first encoding includes the encoding of the masked initial statement in the training data and the encoding of the masked matching statement in the training data.

4. The method according to claim 1, wherein, In the third encoding, the first encoding occupies a predetermined third proportion, the second encoding occupies a predetermined fourth proportion, and the sum of the predetermined third proportion and the predetermined fourth proportion is 1.

5. The method according to claim 1, wherein, Determining the word probability and prediction loss value for each element position in the third encoding, and obtaining the second loss value includes: The unmasked loss value is obtained based on the predicted loss value of the unmasked element position and the predetermined first weight; The masking loss value is obtained based on the predicted loss value of the masked element position and the predetermined second weight; The second loss value is obtained based on the non-masking loss value and the masking loss value.

6. The method according to claim 1, wherein, The determination of the third loss value includes: Determine the first sub-loss value of the discrimination result, wherein a true discrimination result corresponds to the code belonging to the first code, and a false discrimination result corresponds to the code belonging to the noise code; Determine the second sub-loss value to discriminate whether the predicted features are common words in the sentence pair; The third loss value is determined based on the first sub-loss value and the second sub-loss value.

7. The method according to claim 1, wherein, The method meets at least one of the following criteria: The step of obtaining the first vector representation based on the first encoding and the convolutional encoder includes: inputting the first encoding into the convolutional encoder, and obtaining the first vector representation through multiple convolutions and max pooling; or The step of obtaining the second vector representation based on the noise code and the convolutional encoder includes: inputting the noise code into the convolutional encoder, and obtaining the second vector representation through multiple convolutions and max pooling.

8. The method according to claim 1, wherein, The step of determining the first loss value based on the first vector representation and the second vector representation includes: The square of the difference between the first vector representation and the second vector representation is obtained as the first loss value.

9. The method according to claim 4, wherein, The predetermined first weight is less than the predetermined second weight.

10. The method according to claim 1, wherein, The predetermined text is unstructured text from a vertical domain.

11. A question-and-answer service method, comprising: Obtain the statement to be analyzed; Based on the statement to be analyzed, a predetermined first statement corresponding to the statement to be analyzed is determined based on the statement analysis model, wherein the statement analysis model is generated by training the model pre-training method according to any one of claims 1 to 10; Based on the correspondence between the predetermined first statement and the predetermined second statement, the target statement is determined in the predetermined second statement; The target statement is then fed back.

12. A model pre-training device, comprising: The statement determination unit is configured to obtain a matching statement corresponding to an initial statement in a text, including: segmenting a predetermined text to obtain a first statement set, the first statement set containing multiple initial statements of the predetermined text; obtaining a predetermined first number of similar statements for each initial statement according to a predetermined text search tool, the predetermined first number being greater than 1; and determining a matching statement among the similar statements based on semantic similarity according to the initial statement and the corresponding similar statements. The training data acquisition unit is configured to acquire training data by masking phrases in the initial statement and the matching statement according to a predetermined phrase masking strategy, including: determining the same phrases and different phrases in the initial statement and the corresponding matching statement; masking a predetermined first proportion of the same phrases and a predetermined second proportion of the different phrases in the initial statement and the matching statement, respectively, to acquire training data, wherein the masked same phrases in the initial statement and the matching statement are the same; The training unit is configured as follows: Based on the training data and the BERT model, obtain the first encoding of the preset BERT model output, and obtain the first vector representation based on the first encoding and the convolutional encoder; Based on the first encoding and the normal distribution, a noise encoding is obtained, and based on the noise encoding and the convolutional encoder, a second vector representation is obtained; The noise code is decoded by the decoder to obtain the second code, and a first loss value is determined based on the first vector representation and the second vector representation. Based on the first encoding and the second encoding, a third encoding is obtained, and the word probability and prediction loss value of each element position in the third encoding are determined to obtain a second loss value. The discriminator determines the type of encoding in the third encoding and determines the third loss value; The parameters of the target model are adjusted based on the first loss value, the second loss value, and the third loss value until training is complete. The target model includes the BERT model, the convolutional encoder, the decoder, and the discriminator.

13. A question-and-answer service device, comprising: The input unit is configured to acquire the statement to be analyzed. The statement analysis unit is configured as follows: Based on the statement to be analyzed, a predetermined first statement corresponding to the statement to be analyzed is determined based on the statement analysis model, wherein the statement analysis model is generated by training the model pre-training method according to any one of claims 1 to 10; Based on the correspondence between the predetermined first statement and the predetermined second statement, the target statement is determined in the predetermined second statement; The output unit is configured to provide feedback on the target statement.

14. A data processing apparatus, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 11 based on instructions stored in the memory.

15. A non-transitory computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images