A Short Text Similarity Matching Method and Application Based on Sentence Representation

Abstract

The present invention relates to a short text similarity matching method and application based on sentence representation, constructs a model for text matching and representation learning, based on a siamese network, and calculates the loss by using a positive and negative sample enhancement method based on continuous batch data; the trained model is used for short text similarity matching; the method is applied to the question-answering system in the tax industry to match the standard questions in the tax industry for the user's questions. The present invention solves the defect of confidence calculation deviation for samples of the same length, strengthens the semantic expression of the samples themselves, increases the attention probability of semantic differences in different sentences, reduces the interference of sentence length on model judgment; makes full use of the existing feature vectors in the continuous batch samples of historical time steps to optimize the training objective of the model at the current time step, and does not increase the training duration while using more sample feature data; improves the model prediction accuracy and saves the labor cost of annotation; is particularly suitable for professional fields.

Description

Technical Field

[0001] The present invention relates to the technical field of electrical digital data processing, and particularly relates to a short text similarity matching method and application based on sentence representation. Background Art

[0002] With the continuous development of science and technology, in all walks of life, the demand for intelligent information consulting services is increasing day by day, and it has become common to realize intelligent consulting services through related technologies of deep learning and natural language processing. Under this premise, the core point of the service is that the intelligent consulting service can accurately recommend the answers to the questions consulted by users, which puts forward higher requirements for the performance of the question-and-answer system in the consulting service. Specifically, it puts forward higher requirements for the matching between the questions raised by users and the local questions in the question-and-answer system.

[0003] At present, most text similarity matching algorithms are implemented by encoding the concatenated sequences of two texts and outputting their correlation degree, that is, using the similarity of two texts as the basis for sorting the recommended questions in sequence. Although this method solves the problem of insufficient semantic feature extraction of traditional machine learning models to a certain extent, the increase in the number of model parameters leads to a significant increase in prediction time while improving the accuracy. Especially for the task of text similarity calculation, for each text to be sorted in the question-and-answer library, it is necessary to calculate the similarity with the question text of the user, which cannot meet the requirements in scenarios with high concurrency performance requirements.

[0004] The proposed siamese network architecture model alleviates the above problems to a great extent. However, although the siamese network architecture solves the problem of prediction speed to a great extent, it loses a certain amount of computational accuracy. This is because the model separately represents and calculates the two texts for prediction and then compares their similarities, which will cause the model to lose the interactive features between texts. Summary of the Invention

[0005] The present invention solves the problems existing in the prior art, provides a short text similarity matching method and application based on sentence representation, improves the matching accuracy between user questions and the questions stored in the local question-and-answer library, improves the user experience of the product, and fundamentally reduces the amount of computation.

[0006] The technical solution adopted by the present invention is a short text similarity matching method based on sentence representation, which constructs a model for text matching and representation learning. The model is based on a siamese network and calculates the loss using a positive and negative sample enhancement method based on continuous batch data;

[0007] The trained model is used for short text similarity matching.

[0008] Preferably, the positive and negative sample enhancement method includes positive sample data enhancement and combined negative sample construction; the training sets of the positive and negative samples are separate and unlabeled texts in the corresponding application fields.

[0009] Preferably, the positive sample data enhancement includes the following steps:

[0010] S1.1 For any positive sample, form a training sample pair with this positive sample and the positive sample itself;

[0011] S1.2 Check whether there is a professional term corresponding to the current application field in the current positive sample. If so, repeat and expand the positive sample itself in the training sample pair with the professional term corresponding to the current application field. The repeated professional term is repeated before or after the current professional term, and go to S1.4. Otherwise, go to the next step;

[0012] S1.3 Randomly and repeatedly select characters in the sentence of the positive sample for repeated expansion;

[0013] S1.4 Obtain the training sample pair x of the positive sample i and

[0014] Preferably, in S1.3, the number of repetitions is 2 to 6 times.

[0015] The solutions in the prior art use completely repeated corpora as a pair of positive samples, and rely entirely on the independent randomness brought by the dropout mechanism of the model itself to construct positive samples. This construction scheme will, to a great extent, result in all positive samples having the same length, thereby making the confidence calculated by the model for samples of the same length too large, and ultimately leading to poor prediction effects.

[0016] The positive sample data enhancement of the present invention is different from the positive sample construction scheme in the prior art; by repeating the application field keywords in the text or repeating random characters, it avoids the defect of semantic deviation that may be caused by traditional data enhancement schemes such as random insertion and deletion; by changing the sentence length difference between positive sample pairs through the data enhancement method of repeating random words and characters, it solves the defect of the confidence calculation deviation of the model for samples of the same length, and randomly repeating the professional terms of the current application field can strengthen the semantic expression of the sample itself, enabling the model to pay the most attention to the semantic differences in different sentences.

[0017] Preferably, the combined negative sample construction includes in-batch negative sample construction and consecutive batch negative sample construction.

[0018] Preferably, the in-batch negative sample construction is that the negative sample pairs within any batch are composed of each sample and another training sample except itself.

[0019] Preferably, let the batch size be N, and the learning objective function be

[0020]

[0021] where h i = f θ (x i , z i ), f θ (·) represents the encoder, z i and are two different random masks for dropout, i is the subscript of the current sample, j is the subscript of other samples within the batch, τ is the temperature hyperparameter, τ > 0, generally taken as 0.05, and sim() is the similarity calculation function; in this case, cosine similarity is used, satisfying

[0022] Preferably, a queue of size M is used to record the feature vectors of the training samples generated during the training process for each batch. For example, the size of M is 2 * batch_size; when the batch data stored in the queue is greater than the queue length, the earliest data entering the queue dequeues, and new batch data is collected; the negative samples for consecutive batches are constructed such that the negative sample pairs are composed of each sample and the other batch data to be enqueued and the batch data in the queue except itself.

[0023] In the present invention, since the weights of the model are updated after the data training for each batch is completed, the encoder for vectorizing the samples of each batch is the one generated during the training of the previous time step. This makes the feature vectors calculated at the current time step different from those calculated in the previous batch; therefore, a queue of size M is maintained during model training to record the feature vector results for each batch. When the encoded results stored in the queue are greater than the queue length M, the earliest data entering the queue dequeues and new encoded data results are collected. The encoded data in these queues are all constructed into negative samples with the current sample to be processed, optimizing the training objective of the model and enhancing the model's ability to obtain semantic information in the samples, enabling the model to learn better context representations.

[0024] Preferably, the training objective function is

[0025]

[0026] where M represents the queue length for storing consecutive batch data, represents the sentence vectors within the consecutive batches stored in the queue, indexed by m, h i = f θ (x i , z i ), f θ (·) represents an encoder, z i and are two different random masks for dropout, i is the subscript of the current sample, j is the subscript of other samples within the batch, τ is the temperature hyperparameter, τ > 0, and generally 0.05 is taken, and sim() is the similarity calculation function; cosine similarity is used this time, satisfying

[0027] An application of the described short text similarity matching method based on sentence representation, which is applied to the question - answering system in the tax industry. The question - answering system in the tax industry uses the short text similarity matching method based on sentence representation to match the standard questions in the tax industry for the user's questions.

[0028] The present invention relates to a short text similarity matching method and application based on sentence representation, constructs a model for text matching and representation learning. The model is based on a siamese network and calculates the loss using a positive and negative sample enhancement method based on continuous batch data; the trained model is used for short text similarity matching; the method is applied to the question - answering system in the tax industry. The question - answering system in the tax industry uses the short text similarity matching method based on sentence representation to match the standard questions in the tax industry for the user's questions.

[0029] The beneficial effects of the present invention are as follows:

[0030] (1) Through the data enhancement scheme, change the sentence length difference between positive sample pairs, solve the defect that the model's confidence calculation for samples of the same length is offset, strengthen the semantic expression of the samples themselves, increase the attention probability of semantic differences in different sentences, and reduce the interference of sentence length on the model's judgment;

[0031] (2) Complete the data training of each batch and continuous batches, make full use of the existing feature vectors in the continuous batch samples of historical time steps, optimize the training objective of the model at the current time step, and do not increase the training duration while using more sample feature data;

[0032] (3) Improve the model prediction accuracy, make up for the defect of insufficient feature extraction ability of the existing model, and at the same time support both unsupervised and supervised training methods. And in the unsupervised training mode, the accuracy index of the model is better than that of the existing unsupervised models;

[0033] (4) Do not require any manual annotation, meet the actual needs at the initial stage of the project, and save the annotation labor cost;

[0034] (5) It is especially suitable for professional fields, such as the application of the question - answering system in the tax industry. Description of the Drawings

[0035] Figure 1 The model calculation process of the present invention;

[0036] Figure 2 Schematic diagram of constructing negative samples within a batch in the present invention;

[0037] Figure 3 Schematic diagram of constructing negative samples for consecutive batches in the present invention. Detailed implementation manners

[0038] The following further describes the present invention in detail with reference to embodiments, but the protection scope of the present invention is not limited thereto.

[0039] The present invention relates to a short text similarity matching method based on sentence representation, constructs a model for text matching and representation learning, the model is based on a siamese network, and performs loss calculation with a positive and negative sample enhancement method based on consecutive batch data;

[0040] The trained model is used for short text similarity matching.

[0041] As Figure 1 shown, it is the model calculation process in the actual scenario processed by the present invention, that is, performing representation calculation on user text and Q&A library text, and then performing similarity comparison; in order not to lose the calculation progress, the present invention changes the training objective (loss function) to utilize more training samples and available features in the samples, that is, using the same model to vectorize the same sentence twice to obtain different feature vectors;

[0042] Specifically, the optimization objective of positive samples is reflected in the numerator term of the loss function, and the optimization objective of negative samples is reflected in the denominator term of the loss function. The training objective is to pull the feature vectors of samples with similar expressions (positive samples) closer in the vector space, and at the same time further distinguish the feature vectors of dissimilar samples (negative samples). The more the number of negative samples, the better the generated feature vectors; however, considering that the computational amount of their feature vectors will also increase when the number of negative samples increases, in addition to taking other samples in the batch except the positive samples themselves as negative samples, the feature vectors calculated in the historical batch are also stored and used in the calculation of the training objective at the current time step. While fully utilizing the data generated during the model training process, it also greatly reduces the computational amount and effectively improves the effect of the model in calculating sentence representation.

[0043] The following embodiments take a Q&A system applied to the tax industry as an example.

[0044] The positive and negative sample enhancement method includes positive sample data enhancement and combined negative sample construction; the training sets of positive samples and negative samples are separate and unlabeled texts in the corresponding application fields.

[0045] In the present invention, in the case of unsupervised training, the training set consists of separate and unlabeled texts in the tax industry, and the data is sourced from users consulting tax issues, as shown in Table 1.

[0046] Table 1 Examples of Partial Training Corpus

[0047] How to declare the employment security fund? The annual individual income tax settlement system prompts that the payment limit has been exceeded. What should I do? ...... Where can I consult about the invoicing system problems?

[0048] The data in Table 1 are partial example data.

[0049] The positive sample data augmentation includes the following steps:

[0050] S1.1 For any positive sample, form a training sample pair with this positive sample and the positive sample itself.

[0051] S1.2 Check whether there is a professional term corresponding to the current application field in the current positive sample. If so, repeat and expand the positive sample itself in the training sample pair with the professional term corresponding to the current application field, and the repeated professional term is repeated before or after the current professional term, then perform S1.4; otherwise, proceed to the next step.

[0052] S1.3 Randomly repeat and select several characters in the sentence of the positive sample for repeated expansion; in S1.3, the number of repetitions is 2 to 6 times.

[0053] S1.4 Obtain the training sample pair x of the positive sample i and

[0054] In this embodiment, the positive sample pairs for model training include the sample and the sample after reprocessing itself. The reprocessing means repeating and expanding the sample itself with the professional term of the current application field, such as a tax professional term. When there is no professional term in the current application field, a certain number of characters are randomly repeated and selected in the sentence for repeated expansion, and the number of repetitions is 2 to 6 times; based on Table 1, the reprocessed samples in the constructed positive sample pairs are shown in the second column of Table 2.

[0055] Table 2 Examples of Partial Constructed Positive Sample Training Corpus

[0056]

[0057] The first column and the second column in Table 2 correspond to positive sample pairs.

[0058] When there is no professional term in the current application field, for example, if the sample is "How to file the cultural undertakings construction tax return for general taxpayers", assuming there is no professional term in the current application field, the constructed positive sample can be "How to file the cultural undertakings construction tax return for general taxpayersssssss", and the two correspond to a positive sample pair.

[0059] In this embodiment, tax-related professional terms include, but are not limited to, post offices, withholding terminals, deduction terminals, import tickets, local tax bureaus, machine-printed tickets, etc.

[0060] In this embodiment, for the set of positive samples i is the index of the positive sample set, m is the set size, and χ i and is a positive sample pair. The feature vectors of the positive sample pair χ i and are calculated by using the method of two independent samplings. The method of two independent samplings means that the same encoder is used to calculate the vectors of positive samples. Since the dropout method is used in the calculation of the fully connected layer and the attention weights in the Transformer layer of BERT, different dropout masks z only need to be applied in the encoder to obtain the corresponding feature vectors h i and of χ i and

[0061] h i = f θ (x i , z i )

[0062]

[0063] where z i and are two random masks of dropout;

[0064] This will be used to calculate the loss function together with negative samples in subsequent training.

[0065] The construction of the combined negative samples includes the construction of negative samples within a batch and the construction of negative samples in consecutive batches.

[0066] The construction of negative samples within a batch is that for any batch, a negative sample pair is composed of each sample and another training sample except itself.

[0067] Let the batch size be N, and the learning objective function be

[0068]

[0069] where h i = f θ (x i , z i ), f θ (·) represents the encoder, and z i and Two different random masks for dropout, i is the subscript of the current sample, j is the subscript of other samples within the batch, τ is the temperature hyperparameter, τ > 0, generally taken as 0.05, and sim() is the similarity calculation function.

[0070] In this embodiment, as Figure 2 shown, for each mini-batch, the negative sample pairs are combinations of each sample and other training samples except itself.

[0071] Record the feature vectors of the training samples generated during the training process for each batch in a queue of size M. When the batch data stored in the queue is greater than the queue length, the data that entered the queue earliest dequeues, and new batch data is collected; the negative sample pairs for consecutive batches are constructed as negative sample pairs composed of each sample and other batch data to be enqueued and the batch data in the queue except itself; M is greater than the maximum value of the batch size for each batch.

[0072] The training objective function is

[0073]

[0074] Among them, M represents the queue length for storing consecutive batch data, represents the sentence vectors within the consecutive batches stored in the queue, indexed by m, h i = f θ (x i , z i ), f θ (·) represents the encoder, z i and are two different random masks for dropout, i is the subscript of the current sample, j is the subscript of other samples within the batch, τ is the temperature hyperparameter, τ > 0, generally taken as 0.05, and sim() is the similarity calculation function.

[0075] In this embodiment, as Figure 3 shown, maintain a queue of size M to record the encoding results for each batch. When the encoding results stored in the queue are greater than the queue length, the data that entered the queue earliest dequeues, and new encoding data results are collected; for consecutive batches, the negative sample pairs are each sample and other batch data to be enqueued and the batch data in the queue except itself.

[0076] The present invention also relates to an application of the above-mentioned short text similarity matching method based on sentence representation, which is applied to a question and answer system in the tax industry. The question and answer system in the tax industry uses the short text similarity matching method based on sentence representation to match the standard questions in the tax industry for the user's questions.

[0077] In the present invention, the trained model is applied in a question-and-answer system in the tax industry. By inputting the user's question through an input device, the question-and-answer system in the tax industry outputs the matching standard question and its answer or a direct answer.

[0078] To achieve the above, the present invention is implemented by a computer-readable storage medium and a computer device. The medium stores a short text similarity matching program based on sentence representation. When the program is executed by a processor, the above short text similarity matching method based on sentence representation is implemented. The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above short text similarity matching method based on sentence representation is implemented. The computer-readable storage medium and the computer device solve the problem in the prior art that it is impossible to balance prediction speed and calculation accuracy, and are particularly suitable for the field of tax consulting.

[0079] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0081] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps for implementing the functions specified in one block or a plurality of blocks.

[0083] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to cover the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0084] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A short text similarity matching method based on sentence representation, characterized in that: Construct a model for text matching and representation learning. The model is based on a siamese network and calculates the loss using a positive and negative sample enhancement method based on consecutive batch data. The positive and negative sample enhancement method includes positive sample data enhancement and combined negative sample construction. The training sets of the positive and negative samples are separate and unlabeled texts in the corresponding application fields. The positive sample data enhancement includes the following steps: S1.1 For any positive sample, form a training sample pair with this positive sample and the positive sample itself. S1.2 Check if there is a professional term corresponding to the current application field in the current positive sample. If so, repeat and expand the positive sample itself in the training sample pair with the professional term corresponding to the current application field. The repeated professional terms are repeated before or after the current professional term, and then proceed to S1.

4. Otherwise, proceed to the next step. S1.3 Randomly repeat and select characters in the sentences of the positive samples for repeated expansion; S1.4 Obtain the training sample pairs of positive samples and ; The combined negative sample construction includes in-batch negative sample construction and consecutive batch negative sample construction. The in-batch negative sample construction is that the negative sample pairs within any batch are composed of each sample and another training sample other than itself. Let the batch size be N, and the learning objective function is ， Among them, , , represents an encoder, and are two different random masks for dropout, i is the subscript of the current sample, j is the subscript of other samples within the batch, τ is the temperature hyperparameter, τ > 0, sim() is the similarity calculation function; a queue of size M is used to record the feature vectors of the training samples generated during the training process for each batch. When the batch data stored in the queue is greater than the queue length, the data that entered the queue earliest dequeues, and new batch data is collected; The consecutive batch negative sample construction is that the negative sample pairs are composed of each sample and the other batch data to be queued and the batch data in the queue other than itself. The training objective function is ， where M represents the queue length for storing consecutive batch data, represents the sentence vectors within consecutive batches stored in the queue, indexed by m, , , represents the encoder, and are two different random masks for dropout, i is the subscript of the current sample, j is the subscript of other samples within the batch, τ is the temperature hyperparameter, τ > 0, and sim() is the similarity calculation function; Use the trained model for short text similarity matching.

2. The short text similarity matching method based on sentence representation according to claim 1, wherein: In S1.3, the number of repetitions is 2 to 6 times.

3. Application of the short text similarity matching method based on sentence representation according to any one of claims 1 to 2, characterized in that: Applied to a question-and-answer system in the tax industry. The question-and-answer system in the tax industry matches the user's question with the standard question in the tax industry using a short text similarity matching method based on sentence representation.

Images