Neural network training method, semantic similarity calculation method and semantic retrieval system

Abstract

The present invention discloses a neural network training method, a semantic similarity calculation method and a semantic retrieval system. The neural network training method comprises the following steps: S1: marking sentence similarity; S2: inputting two marked sentences into a neural network; S3: processing the features of the two sentences and calculating a loss function; S4: training the neural network according to the loss value of the loss function. The neural network training method used in the technical solution of the present invention has few neural network parameters, fast retrieval speed, can accurately calculate semantic similarity, and thus can obtain semantic information of the text, thereby improving the accuracy of retrieval, and is suitable for high concurrency and low latency demand scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a neural network training method, a semantic similarity calculation method and a semantic retrieval system. Background Art

[0002] Natural language processing (NLP) is an important research direction in computer science and artificial intelligence. It mainly studies various theories and methods that can achieve effective communication between humans and computers using natural language. It is a discipline that integrates linguistics, computer science, and mathematics.

[0003] In natural language processing, there are many scenarios that require semantic similarity matching between different texts. Therefore, semantic similarity calculation is one of the technical directions to solve such needs, and is the basic technology for applications such as text duplication detection and intelligent question answering. Semantic similarity is intended to measure the similarity between two given texts from a semantic perspective, and usually gives a semantic similarity score between 0 and 1, with higher scores representing greater similarity.

[0004] Among the existing technologies, some calculate semantic similarity based on static word vectors such as Word2Vec. However, the main technical defect of this static word vector is that it cannot consider the situation of polysemy of a word, word segmentation errors, etc., which greatly affects the accuracy of the calculation results.

[0005] In addition, since the BERT model came out, pre-trained language models represented by BERT have achieved amazing results in many NLP tasks. Therefore, text semantic similarity based on the BERT model has also been usefully explored. Although text similarity based on the BERT model has made significant progress, the huge number of BERT parameters leads to slow model prediction speed. In many scenarios, it cannot meet the high concurrency and low latency requirements of the Internet. Summary of the invention

[0006] In order to solve the technical problems existing in the prior art, the present invention provides a neural network training method, comprising the steps of:

[0007] S1: Mark the similarity of two sentences;

[0008] S2: Input the two annotated sentences into the neural network to obtain the features of each word in the sentence;

[0009] S3: Take the average of the features of each word to obtain the features S1 and S2 of the two sentences, and calculate the loss function;

[0010] S4: Train the neural network according to the loss value of the loss function.

[0011] Furthermore, step S1: annotate the sentence similarity as: sentenceA sentenceB Score.

[0012] Furthermore, calculating the loss function in step S3 includes:

[0013] Calculate the cosine similarity of S1 and S2, that is, sim = cos(S1, S2);

[0014] Divide the labels with scores of 1 to 5 by 5 respectively, normalize them to between 0 and 1, and get the normalized labels;

[0015] Calculate the loss function based on sim and label label.

[0016] Furthermore, the loss function formula is:

[0017] Loss = |sim-label|

[0018] Among them, sim=cos(S1,S2) is the cosine similarity.

[0019] Furthermore, the neural network structure includes a linear network unit, an embedding unit, a feature extraction unit and a compression unit, wherein:

[0020] The linear network unit is used to copy the input variable into three parts as input, and obtain the query, key and value of the sentence respectively;

[0021] The embedding unit is used to encode the input word, absolute position and paragraph number, and then process it to obtain the word vector;

[0022] The feature extraction unit is used to perform dimensionality processing on the word vector, extract features through the transformer module, and output the features of the word vector;

[0023] The compression unit is used to compress the features of word vectors.

[0024] Furthermore, the transformer is composed of multiple pairs of transformer groups connected in series, wherein each pair of transformer groups is composed of two layers of transformers, and the transformer parameters of each group are completely shared.

[0025] The present invention also provides a method for calculating semantic similarity, comprising the steps of:

[0026] Receive an input sentence;

[0027] The neural network searches for key sentences and related content based on the input sentences, extracts features and calculates semantic similarity;

[0028] Return the top N sentences with the highest semantic similarity scores.

[0029] The present invention also provides a semantic retrieval system, comprising a collection module, a processing module and an output module, wherein:

[0030] The acquisition module is used to receive input sentences;

[0031] The processing module is used to process the input sentences;

[0032] The output module is used to return the processing results of the processing module.

[0033] The present invention also provides a computer-readable storage medium, in which instructions or programs are stored. The instructions or programs are loaded and executed by a processor to implement the semantic similarity calculation method.

[0034] The present invention also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the semantic similarity calculation method.

[0035] In actual applications, the modules described in the method and system disclosed in the present invention can be deployed in multiple modules on one target server, or each module can be independently deployed on different target servers. In particular, in order to provide more powerful computing processing capabilities, the modules can also be deployed on cluster target servers as needed.

[0036] It can be seen that the technical solution adopted by the present invention uses a neural network training method that obtains fewer neural network parameters, has a fast retrieval speed, can accurately calculate the semantic similarity, and thus can obtain the semantic information of the text, thereby improving the accuracy of the retrieval, and is suitable for scenarios with high concurrency and low latency requirements.

[0037] In order to have a clearer and more comprehensive understanding of the present invention, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0039] Figure 1 Schematic diagram of the neural network structure of an embodiment of the present application. DETAILED DESCRIPTION

[0040] In view of the technical defects of the prior art, this application discloses a neural network training method, comprising the steps of:

[0041] S1: Mark the similarity of two sentences;

[0042] S2: Input the two annotated sentences into the neural network to obtain the features of each word in the sentence;

[0043] S3: Take the average of the features of each word to obtain the features S1 and S2 of the two sentences, and calculate the loss function;

[0044] S4: Train the neural network according to the loss value of the loss function.

[0045] The technical solution of the present application is further explained below in conjunction with various preferred implementations.

[0046] S1: Mark the similarity of two sentences;

[0047] The purpose of this step is to convert the input sentence into a sentence format that can be processed by training the neural network. The format of annotating two sentences is: sentenceA sentenceB Score, that is, the sentence and the similarity value. Enter two sentences and similarity labels. The labels are represented by 1, 2, 3, 4, and 5 points according to the similarity. The higher the score, the more similar. For example, the following sets of sentences are annotated with similarity label values ​​according to their meaning similarity:

[0048] Today is a very nice day. What a nice day today!

[0049] Today is a good day. Today is a sunny day.

[0050] Today the weather is good Today the weather is bad 1

[0051] S2: Input the two annotated sentences into the neural network to obtain the features of each word in the sentence;

[0052] After the sentences are labeled, the labeled data set can be input into the neural network for training.

[0053] Please refer to Figure 1 As a preferred implementation method of neural network training, the neural network structure constructed in this application is a neural network structure improved based on Albert, including a linear network unit, an embedding unit, a feature extraction unit and a compression unit, wherein:

[0054] The linear network unit is used to copy a set of input variables into three copies as input, and obtain the query, key and value of the sentence through the linear network respectively; the linear network can be composed of one linear layer or multiple linear layers.

[0055] In the embodiment of the present application, after extracting the query, key and value, a more preferred implementation is proposed, that is, adding an attention mechanism, including the following steps:

[0056] First, multiply the matrix consisting of query and key;

[0057] Divide by the number of groups for normalization;

[0058] Then remove the filling part of the border;

[0059] Then, after the softmax activation function, the attention coefficient of each word in the sentence is obtained;

[0060] The information attention coefficient of each word is processed by discarding some features, that is, dropout processing, which can reduce the amount of training calculations;

[0061] Then multiply it with the value matrix;

[0062] Finally, multiple groups are restored into one group, so that the relationship between each word can be better obtained.

[0063] The embedding unit is used to encode the input word, absolute position and paragraph number, and then process it to obtain a word vector. In this embodiment, absolute position coding is used, and the input word (word_id), the absolute position coding of the word and the sentence or paragraph to which it belongs are vectorized according to the sequence. This embodiment adopts 128 dimensions, and then the above information is superimposed, and then normalized and dropout are performed to obtain an embedding matrix and a mask vector after removing the boundaries.

[0064] The feature extraction unit is used to perform dimensionality processing on the word vector, extract features through the transformer module, and output the features of the word vector. In this embodiment, before extracting features, the word vector is first dimensionalized to 768. The transformer module of this embodiment is composed of 12 layers of transformers connected in series, which are connected in series into 6 groups. Each group contains 2 layers of transformers, and the transformer parameters of each pair are completely shared, which can further reduce the amount of training calculations.

[0065] The compression unit is used to compress the features of the word vector. This embodiment uses an average pooling solution to take the mean of each word feature vector as the feature of the entire sentence.

[0066] S3: Take the average of the features of each word to obtain the features S1 and S2 of the two sentences, and calculate the loss function;

[0067] The compression unit compresses the word features and takes the mean of each word feature vector as the feature of the entire sentence to obtain the features S1 and S2 of the two sentences, which can then be used to calculate the loss function. As a preferred implementation, the method for calculating the loss function includes:

[0068] The cosine similarity of the features S1 and S2 of the two sentences is calculated, and the formula is: sim = cos(S1, S2), where cos is the cosine similarity.

[0069] Then divide the labels with similarity scores from 1 to 5 by 5 respectively, normalize them to between 0 and 1, and get the normalized labels;

[0070] The loss function is calculated using sim and label label. The loss function formula in this embodiment is:

[0071] Loss = |sim-label|

[0072] label is the similarity between two manually annotated sentences sentenceA and sentenceB.

[0073] S4: According to the loss value of the loss function, the neural network is trained. The specific training or optimization method can be implemented by various existing algorithms.

[0074] Based on the above neural network training method, the obtained neural network can be used for prediction (retrieval). As an implementation method, a semantic similarity calculation method proposed in this application includes the steps of:

[0075] Receive an input sentence;

[0076] The neural network searches for key sentences and related content based on the input sentences, extracts features and calculates semantic similarity;

[0077] Return the top N sentences with the highest semantic similarity scores.

[0078] The neural network used is the neural network obtained by the above training method.

[0079] Based on the above semantic similarity calculation method, the present application also provides a semantic retrieval system, including a collection module, a processing module and an output module, wherein:

[0080] The acquisition module is used to receive input sentences;

[0081] The processing module is used to process the input sentences;

[0082] The output module is used to return the processing results of the processing module.

[0083] An embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores instructions or programs, and the instructions or programs are loaded by a processor and executed by any of the semantic similarity calculation methods described above.

[0084] An embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the semantic similarity calculation methods described above.

[0085] It should be noted that, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0086] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A neural network training method, The method is characterized in that it comprises the steps of: S1: Mark the similarity of two sentences; S2: Input the two annotated sentences into the neural network to obtain the features of each word in the sentence; S3: Take the average of the features of each word to obtain the features S1 and S2 of the two sentences, and calculate the loss function, including: calculating the cosine similarity of S1 and S2, that is, sim=cos(S1,S2); Divide the labels with scores of 1 to 5 by 5 respectively, normalize them to between 0 and 1, and get the normalized labels; Calculate the loss function based on sim and label label; S4: According to the loss value of the loss function, the neural network is trained, wherein the neural network structure includes a linear network unit, an embedding unit, a feature extraction unit, and a compression unit, wherein: The linear network unit is used to copy the input variable into three parts as input, and obtain the query, key and value of the sentence respectively; The embedding unit is used to encode the input word, absolute position and paragraph number, and then process it to obtain the word vector; The feature extraction unit is used to perform dimensionality processing on the word vector, extract features, and output the features of the word vector; The compression unit is used to compress the features of word vectors.

2. The neural network training method according to claim 1, characterized in that: Step S1: Annotate the sentence similarity as: sentenceA sentenceB Score.

3. The neural network training method according to claim 1, characterized in that: The loss function formula is: Loss=|sim-label| Among them, sim=cos(S1,S2), is the cosine similarity.

4. The neural network training method according to claim 1, wherein the feature extraction unit It is composed of multiple pairs of transformer groups connected in series, where: Each pair of transformer groups consists of two layers of transformers, and the transformer parameters of each group are completely shared.

5. A method for calculating semantic similarity, characterized in that it comprises the steps of: Receive an input sentence; The neural network searches for key sentences and related content based on the input sentence, extracts features respectively, and calculates semantic similarity, wherein the neural network is obtained by any method described in claims 1-4; Return the top N sentences with the highest semantic similarity scores.

6. A semantic retrieval system, characterized in that: It includes acquisition module, processing module and output module, among which: The acquisition module is used to receive input sentences; The processing module is used to process the input sentence, and the processing module is a neural network obtained by any method described in claims 1-4; The output module is used to return the processing results of the processing module.

7. A computer-readable storage medium, characterized in that: The storage medium stores instructions or programs, and the instructions or programs are loaded and executed by the processor to implement the semantic similarity calculation method as claimed in claim 5.

8. An electronic device, comprising: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the semantic similarity calculation method as described in claim 5.

Images