A method, device and apparatus for obtaining semantic similarity based on sentences

Abstract

The present invention discloses a method, apparatus, and device for obtaining semantic similarity based on sentences. The semantic similarity obtaining method includes: obtaining a first semantic representation of a first sentence using a first encoder; obtaining a second semantic representation of a second sentence using the first encoder or the second encoder; calculating the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences; wherein the first encoder includes a first encoding layer, and the second encoder includes a second encoding layer and an interaction layer, wherein the model parameters of the first encoding layer are exactly the same as the model parameters of the second encoding layer, the output data of the second encoding layer serves as the input data of the interaction layer, and the output data of the interaction layer serves as the second semantic representation. The present application considers the interaction between sentences when converting sentences into semantic representations, which facilitates the accurate calculation of semantic similarity.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method, apparatus and device for obtaining semantic similarity based on sentences. Background Art

[0002] Sentence similarity refers to the semantic similarity between two sentences. It has a wide range of applications in natural language processing, such as dialogue systems, information retrieval, and semantic analysis. The performance of a semantic similarity model often determines the quality of the entire application system. A good semantic similarity model can provide faster information retrieval and more accurate results. Sentence similarity calculation is a key issue in multi-document automatic summarization technology. In search engines and search advertising scenarios, calculating semantic similarity is also one of the most important and challenging issues.

[0003] In the existing technology, obtaining semantic similarity is mainly divided into two directions: one is a method based on deep learning, and the other is a method based on statistics.

[0004] Deep learning-based methods mainly include Word2vec and the Deep Structured Semantic Model (DSSM). The Word2vec method converts each word in a sentence into a fixed vector representation through model training, averages the vector representations of all the words in the sentence to obtain the representation vector of the entire sentence, and finally calculates the similarity between the two sentences using cosine similarity. DSSM is a typical dual-tower structure that generally takes two sentences as input. The two input layers are each connected to an encoding layer, and two output vectors are obtained after encoding. Finally, the matching score of the two output vectors is calculated using cosine similarity. The encoding layer can use models such as RNN, CNN, Transformer, and BERT.

[0005] In deep learning-based methods, there is a lack of interaction in the process of converting each word into a fixed vector representation or encoding two sentences into vector representations independently, which has a certain impact on the similarity matching effect.

[0006] Statistical-based methods are mainly used in rough ranking or scenarios with low requirements for semantic accuracy, mainly including edit distance, BM25, TF-IDF, TextRank, etc. These methods mainly focus on the literal and word frequency changes between sentences. For example, the edit distance describes the minimum number of editing operations required to transform one sentence into another. If the number of editing operations is more, it means the similarity distance between sentences is farther and less similar. Editing operations include replacing one character with another, inserting a character, and deleting a character. Other methods such as BM25, TF-IDF, etc. are based on the edit distance and also consider the frequency of words appearing in the sentence and the importance of a word in all documents. Compared with the edit distance, there is a certain improvement, but the basic method is still based on statistical word frequency and other methods.

[0007] Statistical-based methods are superficial and difficult to understand the deep semantics of sentences. Taking the edit distance test of "I like you" as an example, if you want to change it to "I don't like you", you only need to insert a "don't" in the middle, that is, only one operation; while if you want to change it to "I really like you", you need to insert two characters "really" and "ah". At this time, the edit distance algorithm will think that "I like you" and "I don't like you" are more similar. Therefore, the accuracy of the statistical-based semantic representation method is relatively low. Summary of the Invention

[0008] In view of the above, the present invention aims to provide a method, device, and equipment for obtaining semantic similarity based on sentences, and correspondingly proposes a computer-readable storage medium, which considers the interaction between sentences when transforming sentences into semantic representations, making the semantic representations of sentences more accurate and contributing to the precise calculation of semantic similarity.

[0009] The technical solution adopted by the present invention is as follows:

[0010] In the first aspect, the present invention provides a method for obtaining semantic similarity based on sentences, including:

[0011] Obtaining a first semantic representation of a first sentence by using a first encoder;

[0012] Obtaining a second semantic representation of a second sentence by using the first encoder or a second encoder;

[0013] Calculating the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences;

[0014] The first encoder includes a first encoding layer, and the second encoder includes a second encoding layer and an interaction layer. The model parameters of the first encoding layer are exactly the same as the model parameters of the second encoding layer. The output data of the second encoding layer serves as the input data of the interaction layer, and the output data of the interaction layer serves as the second semantic representation.

[0015] In one possible implementation, the first encoder further includes an average pooling layer, the output data of the first encoding layer is the input data of the average pooling layer, and the output data of the average pooling layer is the first semantic representation.

[0016] In one possible implementation, the interaction layer includes a first sub-interaction layer and a second sub-interaction layer;

[0017] The first sub-interaction layer performs a first interaction operation on the output data of the second coding layer using multiple initialization vectors to obtain a first interaction result;

[0018] The second sub-interaction layer performs a second interaction operation on the first semantic representation using the first interaction result to obtain a second interaction result as the second semantic representation.

[0019] In one possible implementation, during the training phase, the first encoder and the second encoder form a dual-tower training model, and the dual-tower training model is optimized using the contrast loss function value between the interaction layer output data of the second encoder and the output data of the first encoder.

[0020] In one possible implementation, training the dual-tower training model includes:

[0021] Pre-training the dual-tower training model;

[0022] Generate incremental training datasets using seed data of the target application scenario;

[0023] The dual-tower training model is incrementally trained using the incremental training dataset.

[0024] In one possible implementation, seed data of the target application scenario is used to generate an incremental training dataset, specifically including:

[0025] The seed data is processed by a back-translation model to generate a first sentence pair as part of the incremental training dataset.

[0026] In one possible implementation, generating an incremental training dataset using seed data of a target application scenario also includes:

[0027] A second sentence pair is obtained through a syntactic analysis and reordering method as part of the incremental training dataset.

[0028] In one possible implementation, training the dual-tower training model further includes:

[0029] After the incremental training, the ground model is fine-tuned using a small sample of annotated sentences in the field to which the target application scenario belongs.

[0030] In one possible implementation manner, in the input data of the incremental training, the proportion of the second sentence pairs is within a first preset range.

[0031] In one possible implementation manner, the input data for the incremental training includes a third sentence pair, and two sentences in the third sentence pair are completely identical.

[0032] In one possible implementation manner, in the input data of the incremental training, the proportion of the third sentence pairs is within a second preset range.

[0033] In one possible implementation manner, the pre-training dataset includes a plurality of fourth sentence pairs, and the correlation between two sentences in the fourth sentence pairs is less than a threshold.

[0034] In a second aspect, the present invention provides a device for obtaining semantic similarity based on sentences, comprising a first semantic representation module, a second semantic representation module, and a similarity calculation module;

[0035] The first semantic representation module is used to obtain a first semantic representation of the first sentence using a first encoder;

[0036] The second semantic representation module is used to obtain a second semantic representation of the second sentence using the first encoder or the second encoder;

[0037] The similarity calculation module is used to calculate the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences;

[0038] The first encoder includes a first encoding layer, and the second encoder includes a second encoding layer and an interaction layer. The model parameters of the first encoding layer are exactly the same as the model parameters of the second encoding layer. The output data of the second encoding layer serves as the input data of the interaction layer, and the output data of the interaction layer serves as the second semantic representation.

[0039] In one possible implementation, during the training phase, the first encoder and the second encoder form a dual-tower training model;

[0040] The similarity obtaining device further includes a model training module, which includes a pre-training module, an incremental training data set obtaining module and an incremental training module;

[0041] The pre-training module is used to pre-train the dual-tower training model;

[0042] The incremental training data set acquisition module is used to generate an incremental training data set using seed data of the target application scenario;

[0043] The incremental training module is used to perform incremental training on the dual-tower training model using the incremental training data set.

[0044] In one possible implementation, the incremental training dataset acquisition module includes a first sentence pair acquisition module, which is used to process the seed data through a back-translation model to generate a first sentence pair as part of the incremental training dataset.

[0045] In one possible implementation, the incremental training dataset acquisition module further includes a second sentence pair acquisition module, and the second sentence pair acquisition module is used to obtain a second sentence pair as part of the incremental training dataset through a syntactic analysis and reordering method.

[0046] In a third aspect, the present invention provides a device for obtaining semantic similarity based on sentences, comprising:

[0047] One or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the sentence-based semantic similarity obtaining device, enable the sentence-based semantic similarity obtaining device to perform the sentence-based semantic similarity obtaining method as described above.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned sentence-based semantic similarity obtaining method.

[0049] The idea of the present invention is to consider the interaction between sentences when converting sentences into semantic representations, so that the semantic representation of the sentences is more accurate, which is helpful for the accurate calculation of semantic similarity. In addition, the present application utilizes a dual-tower training model, in which the first side has an interaction layer and the second side does not have an interaction layer. During the training process, the model parameters of the second side coding layer are corrected by the first side, so that the model parameters of the coding layers on both sides are completely consistent, and the model is optimized by using the contrast loss between the representation layers on both sides. The asymmetric structure is used to train both sides at the same time, solving the problem of lack of interaction. In addition, during the training process, the negative samples obtained by syntactic analysis and / or the positive samples obtained by the back-translation model are added to the batch input data of the training model, so that the model can learn richer semantic features while ensuring the efficiency of reasoning, thereby achieving better results. Finally, the training model of the present application uses weakly correlated data for pre-training. Compared with the collection of a large amount of supervised corpus, the collection of weakly correlated data greatly reduces the difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0051] Figure 1 A flowchart of a method for obtaining semantic similarity based on sentences provided by the present invention;

[0052] Figure 2 A structural diagram of the first encoder provided by the present invention;

[0053] Figure 3 A structural diagram of a second encoder provided by the present invention;

[0054] Figure 4 An example of the chat question and answer provided by the present invention;

[0055] Figure 5 A structural diagram of an embodiment of the dual-tower training model provided by the present invention;

[0056] Figure 6 A flowchart of an embodiment of the interactive operation provided by the present invention;

[0057] Figure 7 A flow chart of training the dual-tower training model provided by the present invention;

[0058] Figure 8 An example of the back-translation model provided by the present invention;

[0059] Figure 9 An example of the syntax analysis sequence provided by the present invention;

[0060] Figure 10 An example of a batch provided by the present invention;

[0061] Figure 11 A structural diagram of a device for obtaining semantic similarity based on sentences provided by the present invention;

[0062] Figure 12 This is a structural diagram of the incremental training data set acquisition module provided by the present invention;

[0063] Figure 13 This is a structural diagram of a device for obtaining semantic similarity based on sentences provided by the present invention. DETAILED DESCRIPTION

[0064] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0065] The idea of the present invention is to consider the interaction between sentences when converting sentences into semantic representations, so that the semantic representation of the sentences is more accurate, which is helpful for the accurate calculation of semantic similarity. In addition, the present application utilizes a dual-tower training model, in which the first side has an interaction layer and the second side does not have an interaction layer. During the training process, the model parameters of the second side coding layer are corrected by the first side, so that the model parameters of the coding layers on both sides are completely consistent, and the model is optimized by using the contrast loss between the representation layers on both sides. The asymmetric structure is used to train both sides at the same time, solving the problem of lack of interaction. In addition, during the training process, the negative samples obtained by syntactic analysis and / or the positive samples obtained by the back-translation model are added to the batch input data of the training model, so that the model can learn richer semantic features while ensuring the efficiency of reasoning, thereby achieving better results. Finally, the training model of the present application uses weakly correlated data for pre-training. Compared with the collection of a large amount of supervised corpus, the collection of weakly correlated data greatly reduces the difficulty.

[0066] In the DSSM model, the two encoding layers are independent of each other. Therefore, the two output vectors obtained after inputting a sentence pair into the model are independent of each other and cannot reflect the relationship between the two sentences in the sentence pair. Therefore, this semantic representation method cannot fully reflect the relationship between the two sentences.

[0067] In view of the aforementioned core concept, the present invention provides at least one embodiment of a method for obtaining semantic similarity based on sentences, such as Figure 1 As shown, the following steps may be included:

[0068] S110: Obtain a first semantic representation of a first sentence using a first encoder.

[0069] S120: Obtain a second semantic representation of the second sentence using the first encoder or the second encoder.

[0070] S130: Calculate the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences.

[0071] Specifically, in one possible implementation, the semantic representation model formed by the first encoder is as follows: Figure 2 As shown in Figure 1, the first encoder includes a first input layer, a first encoding layer, and an average pooling layer. The input data of the first input layer is the first sentence. After passing through the first encoding layer, its output data is input into the average pooling layer. The pooling result of the average pooling layer is the first representation layer, which serves as the output data of the first encoder, namely the first semantic representation.

[0072] In a possible implementation, the first encoding layer may use models such as RNN, CNN, Transformer, and Bert.

[0073] In another possible implementation, the structure of the first encoder may be other encoder structures in the prior art, for example, the output data of the first encoding layer is input into a fully connected layer, and the obtained output data is used as the output data of the encoder.

[0074] Based on the structure of the first encoder described above, in one possible implementation, the structure of the second encoder is exactly the same as that of the first encoder, and the model parameters of each layer are exactly the same, or the first encoder and the second encoder are the same model.

[0075] Application scenarios of this embodiment include similarity matching tasks (such as open question answering or casual question answering). Similarity matching tasks involve filtering out sentences in a candidate database (such as a question library) that are most similar to an input sentence (such as an input question), and then finding the answer corresponding to the input sentence based on the filtered sentences. Figure 4 An example of casual question-answering is shown, in which, through semantic similarity calculation, the similarity between the candidate set of the question library "I am in a terrible mood today" and the input question "I am in a really bad mood today" is high, and the two match each other.

[0076] In similarity matching, the candidate database is very large. To ensure efficiency and accuracy, a first editor is typically used to convert all sentences (e.g., all questions) in the candidate database into a first semantic representation. During task execution, after the user enters a sentence (e.g., a question), a second editor is used to obtain a second semantic representation. Finally, the similarity (e.g., cosine similarity) between the first and second semantic representations is calculated.

[0077] In another possible implementation, the semantic representation model formed by the second encoder is as follows: Figure 3 The second encoder includes a second input layer, a second encoding layer, and an interaction layer, where the model parameters of the first encoding layer are exactly the same as those of the second encoding layer. The input data of the second input layer is the second sentence, the output data of the second encoding layer is input into the interaction layer, and the output of the interaction layer is the second representation layer, which serves as the output data of the second encoder, i.e., the second semantic representation.

[0078] Specifically, if Figure 3 As shown, the interaction layer includes a first sub-interaction layer and a second sub-interaction layer. The first sub-interaction layer uses multiple initialization vectors to perform a first interaction operation on the output data of the second coding layer to obtain a first interaction result. The second sub-interaction layer uses the first interaction result to perform a second interaction operation on the first semantic representation to obtain a second interaction result as the second semantic representation.

[0079] It should be noted that in the implementation of the second encoder with an interactive layer, the two sentences are input into the first encoder and the second encoder at the same time, and the output data of the first encoder is used as the input data of the interactive layer of the second encoder. Therefore, the output data of the second encoder reflects the correlation between the two sentences, making the semantic similarity between the two more accurate.

[0080] This implementation is applicable to custom skills, which use a similarity threshold to determine whether two sentences are similar after calculating semantic similarity. Specifically, when the candidate library is small, the two sentences in the sentence pair are fed into the first encoder and the second encoder respectively to obtain the first and second semantic representations. The semantic similarity between the first and second semantic representations is then calculated, and the similarity threshold is used to determine whether the two sentences match.

[0081] In another possible implementation, the first encoder is further used to complete classification tasks such as small sample intent recognition, entity recognition, and text clustering. Such classification tasks are used to determine to which category a single input sentence belongs.

[0082] In this embodiment, the semantic representation model includes a first encoder, a fully connected layer, and a classifier. The output data of the first encoder is input into the fully connected layer, and the output data of the fully connected layer is input into the classifier for classification, which is the final output. It should be noted that when training this semantic representation model, the model parameters of the first encoder do not need to be adjusted, and only the parameters of the fully connected layer can be fine-tuned.

[0083] It should be noted that, during the training phase, the first encoder and the second encoder are trained simultaneously. Specifically, the first encoder and the second encoder form a dual-tower training model, and the dual-tower training model is optimized using the contrast loss function value between the interactive layer output data of the second encoder and the output data of the first encoder. Figure 5 As shown in the figure, the dual-tower training model is further explained by taking the encoder using the Bert model as an example.

[0084] It should be noted that the input data for the first and second input layers is imported in batches, and the input data is text. Specifically, after each word in a sentence passes through the Bert encoding layer, the output is an embedding vector (i.e., a 768-dimensional array). Assuming that the length of each sentence is limited to 32 and the number of batches is 64 (64 sentence pairs), the encoding layer output corresponding to each batch (the output of the first encoding layer and the output of the second encoding layer) is a 64x32x768-dimensional array.

[0085] Figure 5 In the example, the sentence length is 5, so the output data of the first encoding layer is out-11, out-12...out-15, and the output data of the second encoding layer is out-21, out-22...out-25.

[0086] In the first encoder, the output data of the first encoding layer is pooled by the average pooling layer to obtain the first representation layer.

[0087] In the second encoder, the output data of the second encoding layer is input into the interaction layer. In the first sub-interaction layer, a first interaction operation is performed on the output data of the second encoding layer using multiple initialization vectors (code-1, code-2, ..., code-m), obtaining a first interaction result. Here, m is the number of initialization vectors and is a hyperparameter of the model.

[0088] Figure 5 In

[15] , m=16, and Attention is a mathematical operation structure.

[0089] Specifically, the first interaction operation is to perform an Attention operation on the embedding vector corresponding to each word output by the second encoding layer and the 16 initialization vectors, respectively, to obtain 16 embedding vectors: Emb-1, Emb-2…Emb-16, that is, the output is a 5x16x768-dimensional array. These 16 embedding vectors serve as the first interaction result.

[0090] Specifically, in one possible embodiment, combined with Figure 6As shown, for each initialization vector code-n, the five output data of the second coding layer (out-21, out-22…out-25) are matrix-multiplied with the initialization vector code-n to obtain initial weights a-1, a-2…a-5. These are then normalized using softmax to obtain weights b-1, b-2…b-5, where b-1+b-2+…+b-5=1. The weights b-1, b-2…b-5 are then weighted and summed (mean) with the five output data of the second coding layer (out-21, out-22…out-25) to obtain the first interaction result Emb-n corresponding to the initialization vector.

[0091] In the second sub-interaction layer, the first interaction result (as the initialization vector) is used to perform a second interaction operation on the first representation layer (as input data) to obtain a second interaction result as the second representation layer.

[0092] In a possible implementation, the second interaction operation is the same as the first interaction operation. It is understandable that other existing interaction operations may also be used to implement the second interaction operation.

[0093] Therefore, after inputting a batch into the training model, the dimensions of the first representation layer and the second representation layer obtained are both 64x1x768.

[0094] Then, the contrast loss between the first representation layer and the second representation layer is calculated.

[0095] In one possible implementation, the contrastive loss function is as follows:

[0096]

[0097] Among them, χ represents all samples in the input batch, x k ∈χ, Indicates that x k The corresponding positive sample, x l ∈χ, Indicates that x l The corresponding positive sample, f(x k ) represents the output result of the k-th negative sample (e.g., the first representation layer), represents the output result of the kth positive sample (e.g., the second representation layer), |x| represents the number of sample pairs in a batch, and τ is a hyperparameter. Represents the number of all samples in a batch, which is twice |x|.

[0098] Based on the existing dual-tower structure model, an interaction layer is introduced to enable sentence representations to interact with semantic representations at the encoding layer. At the same time, contrastive loss is introduced to enable the model to learn negative samples between different sentence pairs in a batch. All positive pairs in the batch are used as negative samples, positive samples with similar distances are aggregated, and negative samples of different types are separated.

[0099] Based on the structure of the above dual-tower training model, such as Figure 7 As shown, the training process of the dual-tower training model includes the following steps:

[0100] S710: Pre-train the dual-tower training model.

[0101] It should be noted that traditional semantic similarity models rely on a large number of similar sentence pairs as supervised corpus pairs for training. However, obtaining such similar sentence pairs requires a large amount of business data support and manual annotation resources. This level of data acquisition is unrealistic in many downstream tasks.

[0102] Based on the above reasons, in this application, the pre-training data set includes multiple fourth sentence pairs, and the correlation between the two sentences in the fourth sentence pair is less than the threshold, that is, the fourth sentence pair is a weakly supervised correlation corpus. This expected acquisition is very simple. It only needs to ensure that there is a certain correlation between the sentence pairs, such as the sentence context of an article, QA pairs, etc. It is relatively easy to obtain a large amount of weakly correlated corpus in different downstream task scenarios. Therefore, compared with the acquisition of supervised corpus, the difficulty of obtaining weakly correlated corpus is greatly reduced.

[0103] S720: Generate an incremental training dataset using the seed data of the target application scenario.

[0104] Specifically, seed data are selected from different downstream task scenarios.

[0105] In one possible implementation, seed data of a target application scenario is used to generate an incremental training dataset, specifically including processing the seed data through a back-translation model to generate a first sentence pair as part of the incremental training dataset.

[0106] Specifically, the back-translation model translates seed data of the first language type (such as Chinese) into the second language type (such as English), and then translates the translation of the second language type into the first language type (such as Chinese), and obtains high-quality similar sentence pairs through a preset filtering method. Figure 8An example is shown in which an input sentence is translated into English by a first pair of encoders and decoders, and then translated into Chinese by a second pair of encoders and decoders to obtain a back-translation result. The input sentence and the back-translation result are expressed differently, but the meaning expressed is the same.

[0107] A good semantic similarity model should be able to understand the semantic differences brought about by slight changes in sentences. For example, "I like you" should be closer to "I like you" and farther away from "You like me".

[0108] Based on this idea, in another possible implementation, generating an incremental training dataset using seed data of a target application scenario further includes forging a second sentence pair as part of the incremental training dataset through a syntactic analysis and reordering method.

[0109] Specifically, the seed data is subjected to syntactic analysis to obtain the part of speech and sentence components of each word in the sentence, and words with the same part of speech are replaced according to the part-of-speech rules. This method can obtain negative sample sentences with a certain degree of similarity difference for use in incremental training. Figure 9 An example is shown in which a fake sentence can be obtained by replacing the positions of "you" and "he", "syntax" and "rule", and "analysis" and "replacement".

[0110] It is understandable that forged sentences can also be obtained through other methods (such as synonym replacement, antonym replacement, etc.) as positive samples or negative samples during training.

[0111] Based on the aforementioned incremental training dataset, in one possible implementation, in a batch, the proportion of the second sentence pairs is within a first preset range (eg, 18-12%).

[0112] In a preferred embodiment, the input data for incremental training also includes a third sentence pair, in which the two sentences are exactly the same (i.e., repeated samples). As can be seen from the structure of the above-mentioned dual-tower training model, due to the different conversion processes on both sides and the dropout setting of the entire network structure is 10%, the first representation layer and the second representation layer obtained after inputting the same sentence may be different. Such sentence pairs are used as positive examples in the input data to enhance the robustness of the model.

[0113] In a possible implementation, in the input data (a batch) of the incremental training, the proportion of the third sentence pairs is within a second preset range (eg, 3-6%).

[0114] Figure 10An example of a sentence pair in a batch is shown, where the upper dotted box is the first sentence pair, the middle dotted box is the second sentence pair, and the lower dotted box is the third sentence pair.

[0115] Such a batch enables the model to learn a more uniform vector representation and distinguish different negative samples.

[0116] The special batch construction method mentioned above is used in the incremental training process, so that the model can learn richer semantic features while ensuring inference efficiency, achieving better results.

[0117] S730: Perform incremental training on the dual-tower training model using the incremental training dataset.

[0118] Preferably, after the incremental training, the method further includes:

[0119] S740: Fine-tune the model using a small sample of labeled sentences within the domain of the target application scenario.

[0120] Weakly associated corpus is used for pre-training, and a single sentence is used as seed data. Data forgery is performed through methods such as syntactic analysis, and data enhancement methods such as back-translation models are used to generate enhanced training corpus. This makes the model no longer dependent on a large number of high-quality similar sentence pairs, saving the cost of data acquisition and manual labeling.

[0121] Corresponding to the above embodiments and preferred solutions, the present invention also provides an embodiment of a device for obtaining semantic similarity based on sentences, such as Figure 11 As shown, it may specifically include a first semantic representation module 1101 , a second semantic representation module 1102 and a similarity calculation module 1103 .

[0122] The first semantic representation module 1101 is configured to obtain a first semantic representation of a first sentence using a first encoder.

[0123] The second semantic representation module 1102 is configured to obtain a second semantic representation of the second sentence using the first encoder or the second encoder.

[0124] The similarity calculation module 1103 is configured to calculate the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences.

[0125] The first encoder includes a first encoding layer, and the second encoder includes a second encoding layer and an interaction layer. The model parameters of the first encoding layer are exactly the same as the model parameters of the second encoding layer. The output data of the second encoding layer serves as the input data of the interaction layer, and the output data of the interaction layer serves as the second semantic representation.

[0126] In one possible implementation, during the training phase, the first encoder and the second encoder form a dual-tower training model.

[0127] like Figure 11 As shown, the similarity obtaining device further includes a model training module 1104 , which includes a pre-training module 11041 , an incremental training data set obtaining module 11042 , an incremental training module 11043 and a fine-tuning module 11044 .

[0128] The pre-training module 11041 is used to pre-train the dual-tower training model.

[0129] The incremental training data set acquisition module 11042 is used to generate an incremental training data set using seed data of the target application scenario.

[0130] The incremental training module 11043 is used to perform incremental training on the dual-tower training model using the incremental training data set.

[0131] The fine-tuning module 11044 is used to fine-tune the model using a small sample of labeled sentence pairs within the domain of the target application scenario.

[0132] In one possible implementation, Figure 12 As shown, the incremental training dataset acquisition module 11042 includes a first sentence pair acquisition module 110421, which is used to process the seed data through a back-translation model to generate a first sentence pair as part of the incremental training dataset.

[0133] In one possible implementation, the incremental training dataset acquisition module 11042 further includes a second sentence pair acquisition module 110422, which is configured to obtain a second sentence pair as part of the incremental training dataset through a syntactic analysis and reordering method.

[0134] It should be understood that the above Figure 11-12The division of the various components of the apparatus for obtaining semantic similarity based on sentences shown is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these components can all be implemented in the form of software called through processing elements; they can also all be implemented in the form of hardware; some components can also be implemented in the form of software called through processing elements, and some components can be implemented in the form of hardware. For example, one of the above modules can be a separately established processing element, or it can be integrated into a chip of an electronic device. The implementation of other components is similar. In addition, all or part of these components can be integrated together, or they can be implemented independently. During the implementation process, each step of the above method or each of the above components can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0135] For example, the above components may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, these components may be integrated together to form a system-on-a-chip (SOC).

[0136] Based on the above embodiments and their preferred embodiments, those skilled in the art will appreciate that, in actual operation, the present invention is applicable to a variety of implementations. The present invention uses the following carriers as schematic illustrations:

[0137] (1) A device for obtaining semantic similarity based on sentences, which may include:

[0138] One or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the steps / functions of the aforementioned embodiment or equivalent implementation.

[0139] Figure 13This is a schematic diagram of the structure of an embodiment of the present invention for obtaining a device based on the semantic similarity of sentences, wherein the device can be an electronic device or a circuit device built into the above electronic device. The above electronic device can be a PC, a server, a smart terminal (mobile phone, tablet, watch, glasses, etc.), a smart TV, a stereo, a speaker, a set-top box, a remote control, a smart screen, an ATM, a robot, a drone, an ICV, a smart (car) car and a vehicle-mounted device, etc. This embodiment does not limit the specific form of the device for obtaining the semantic similarity of sentences.

[0140] Specific as Figure 13 As shown, the device 900 for obtaining semantic similarity based on sentences includes a processor 910 and a memory 930. The processor 910 and the memory 930 can communicate with each other through an internal connection path to transmit control and / or data signals. The memory 930 is used to store computer programs, and the processor 910 is used to call and run the computer program from the memory 930. The above-mentioned processor 910 and the memory 930 can be combined into a processing device, or more commonly, they are independent components. The processor 910 is used to execute the program code stored in the memory 930 to implement the above-mentioned functions. In specific implementation, the memory 930 can also be integrated into the processor 910, or be independent of the processor 910.

[0141] In addition, to further improve the functionality of the device 900 for obtaining semantic similarity based on sentences, the device 900 may further include one or more of an input unit 960, a display unit 970, an audio circuit 980, a camera 990, and a sensor 901. The audio circuit may further include a speaker 982, a microphone 984, etc. The display unit 970 may include a display screen.

[0142] Furthermore, the apparatus 900 for obtaining semantic similarity based on sentences may further include a power supply 950 for providing power to various devices or circuits in the apparatus 900 .

[0143] It should be understood that Figure 13 The illustrated apparatus 900 for obtaining semantic similarity based on sentences can implement the various processes of the methods provided in the aforementioned embodiments. The operations and / or functions of the various components of apparatus 900 can be used to implement the corresponding processes in the aforementioned method embodiments. For details, please refer to the descriptions of the aforementioned method, apparatus, and other embodiments. To avoid repetition, detailed descriptions are omitted here.

[0144] It should be understood that Figure 13The processor 910 in the sentence-based semantic similarity obtaining device 900 shown can be a system-on-chip SOC, and the processor 910 can include a central processing unit (CPU), and can further include other types of processors, such as a graphics processing unit (GPU), etc., which will be described in detail below.

[0145] In summary, the various processors or processing units within the processor 910 can work together to implement the previous method flow, and the corresponding software programs of the various processors or processing units can be stored in the memory 930.

[0146] (2) A readable storage medium having a computer program or the above-mentioned device stored thereon, which, when executed, causes a computer to execute the steps / functions of the aforementioned embodiment or an equivalent implementation.

[0147] In the several embodiments provided herein, any function, if implemented as a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, certain technical solutions of the present invention, or portions that contribute to the prior art, or portions of such solutions, may be embodied in the form of software products as described below.

[0148] (3) A computer program product (which may include the above-mentioned apparatus), which, when running on a terminal device, enables the terminal device to execute the sentence-based semantic similarity obtaining method of the aforementioned embodiment or an equivalent implementation.

[0149] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above implementation method can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the above-mentioned computer program product may include but is not limited to an APP; continuing from the above, the above-mentioned device / terminal may be a computer device (such as a mobile phone, a PC terminal, a cloud platform, a server, a server cluster, or a network communication device such as a media gateway, etc.). In addition, the hardware structure of the computer device may also specifically include: at least one processor, at least one communication interface, at least one memory and at least one communication bus; the processor, the communication interface, and the memory can all communicate with each other through the communication bus. Among them, the processor may be a central processing unit CPU, DSP, microcontroller or digital signal processor, and may also include GPU, embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include a specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present invention, etc. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium such as a memory; and the aforementioned memory / storage medium may include: non-volatile memory (non-volatile memory), such as a non-removable disk, USB flash drive, mobile hard disk, optical disk, etc., as well as read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), etc.

[0150] In the embodiment of the present invention, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0151] Those skilled in the art will appreciate that the various modules, units, and method steps described in the embodiments disclosed in this specification can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0152] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed to multiple places, such as nodes in a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying any creative work.

[0153] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred modes can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the scope of implementation of the present invention is not limited to what is shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the description and drawings, should be within the scope of protection of the present invention.

Claims

1. A method for obtaining semantic similarity based on sentences, characterized in that: include: Obtaining a first semantic representation of a first sentence using a first encoder; Obtaining a second semantic representation of the second sentence using a second encoder; Calculating the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences; The first encoder includes a first encoding layer, and the second encoder includes a second encoding layer and an interaction layer, wherein the model parameters of the first encoding layer are exactly the same as the model parameters of the second encoding layer, the output data of the second encoding layer serves as the input data of the interaction layer, and the output data of the interaction layer serves as the second semantic representation; The interaction layer includes a first sub-interaction layer and a second sub-interaction layer; The first sub-interaction layer performs a first interaction operation on the output data of the second coding layer using multiple initialization vectors to obtain a first interaction result; The second sub-interaction layer performs a second interaction operation on the first semantic representation using the first interaction result to obtain a second interaction result as the second semantic representation.

2. The method for obtaining semantic similarity based on sentences according to claim 1, characterized in that: The first encoder further includes an average pooling layer, the output data of the first encoding layer is the input data of the average pooling layer, and the output data of the average pooling layer is the first semantic representation.

3. The method for obtaining semantic similarity based on sentences according to claim 1, characterized in that: During the training phase, the first encoder and the second encoder form a dual-tower training model, and the dual-tower training model is optimized using a contrast loss function value between the interaction layer output data of the second encoder and the output data of the first encoder.

4. The method for obtaining semantic similarity based on sentences according to claim 3, characterized in that: Training the dual-tower training model includes: Pre-training the dual-tower training model; Generate incremental training datasets using seed data of the target application scenario; The dual-tower training model is incrementally trained using the incremental training dataset.

5. The method for obtaining semantic similarity based on sentences according to claim 4, characterized in that: Generate incremental training datasets using seed data of the target application scenario, including: The seed data is processed by a back-translation model to generate a first sentence pair as part of the incremental training dataset.

6. The method for obtaining semantic similarity based on sentences according to claim 5, characterized in that: Generate incremental training datasets using seed data of the target application scenario, including: A second sentence pair is obtained through a syntactic analysis and reordering method as part of the incremental training dataset.

7. The method for obtaining semantic similarity based on sentences according to claim 4, characterized in that: Training the dual-tower training model further includes: After the incremental training, the model is fine-tuned using a small sample of labeled sentences in the field to which the target application scenario belongs.

8. The method for obtaining semantic similarity based on sentences according to claim 6, characterized in that: In the input data of the incremental training, the proportion of the second sentence pairs is within a first preset range.

9. The method for obtaining semantic similarity based on sentences according to claim 4 or 8, characterized in that: The input data for the incremental training includes a third sentence pair, in which two sentences are completely identical.

10. The method for obtaining semantic similarity based on sentences according to claim 9, characterized in that: In the input data of the incremental training, the proportion of the third sentence pairs is within a second preset range.

11. The method for obtaining semantic similarity based on sentences according to claim 4, characterized in that: The pre-training data set includes a plurality of fourth sentence pairs, wherein the correlation between two sentences in the fourth sentence pairs is less than a threshold.

12. A device for obtaining semantic similarity based on sentences, characterized in that: It includes a first semantic representation module, a second semantic representation module and a similarity calculation module; The first semantic representation module is used to obtain a first semantic representation of the first sentence using a first encoder; The second semantic representation module is used to obtain a second semantic representation of the second sentence using a second encoder; The similarity calculation module is used to calculate the similarity between the first semantic representation and the second semantic representation as the semantic similarity between the two sentences; The first encoder includes a first encoding layer, and the second encoder includes a second encoding layer and an interaction layer, wherein the model parameters of the first encoding layer are exactly the same as the model parameters of the second encoding layer, the output data of the second encoding layer serves as the input data of the interaction layer, and the output data of the interaction layer serves as the second semantic representation; The interaction layer includes a first sub-interaction layer and a second sub-interaction layer; The first sub-interaction layer performs a first interaction operation on the output data of the second coding layer using multiple initialization vectors to obtain a first interaction result; The second sub-interaction layer performs a second interaction operation on the first semantic representation using the first interaction result to obtain a second interaction result as the second semantic representation.

13. The apparatus for obtaining semantic similarity based on sentences according to claim 12, characterized in that: During the training phase, the first encoder and the second encoder form a dual-tower training model; The similarity obtaining device further includes a model training module, which includes a pre-training module, an incremental training data set obtaining module and an incremental training module; The pre-training module is used to pre-train the dual-tower training model; The incremental training data set acquisition module is used to generate an incremental training data set using seed data of the target application scenario; The incremental training module is used to perform incremental training on the dual-tower training model using the incremental training data set.

14. The apparatus for obtaining semantic similarity based on sentences according to claim 13, characterized in that: The incremental training dataset acquisition module includes a first sentence pair acquisition module, which is used to process the seed data through a back-translation model to generate a first sentence pair as part of the incremental training dataset.

15. The apparatus for obtaining semantic similarity based on sentences according to claim 14, characterized in that: The incremental training data set acquisition module further includes a second sentence pair acquisition module, which is used to obtain a second sentence pair as part of the incremental training data set through a syntactic analysis and reordering method.

16. A device for obtaining semantic similarity based on sentences, characterized in that: include: One or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the sentence-based semantic similarity obtaining device, enable the sentence-based semantic similarity obtaining device to perform the sentence-based semantic similarity obtaining method according to any one of claims 1 to 11.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the sentence-based semantic similarity obtaining method according to any one of claims 1 to 11.

Images