A text matching method integrating multi-head attention alignment

Abstract

The present invention discloses a text matching method that integrates multi-head attention alignment. The method comprises: constructing a text matching model, including an input layer, a BERT encoding layer, an interactive sentence representation layer, a fusion layer, and a fully connected layer; training the text matching model based on a set loss function; and estimating the degree of matching between target texts using the trained text matching model. The input layer takes two sentences, a first sentence and a second sentence, as input; the BERT encoding layer takes the superposition vector of the character-level vector, position vector, and segmentation vector of the corresponding sentence as input to extract a hidden layer vector with semantic information; the interactive sentence representation layer performs attention interaction between sentence vectors based on the hidden layer vector to extract the interactive features of the two sentences respectively; the fusion layer fuses the interactive features of the two sentences to obtain a feature fusion vector; and the fully connected layer obtains the matching results between the sentences based on the feature fusion vector. The present invention can improve the accuracy of text matching.

Description

Technical Field

[0001] The present invention relates to natural language processing, and more particularly, to a text matching method integrating multi-head attention alignment. Background Art

[0002] The text matching task is a basic task in the field of natural language processing, which aims to determine whether two sentences express the same semantic information.

[0003] Currently, the most popular pre-trained language model in the text matching field is Sentence-BERT (SBERT). The classic Siamese network SBERT model was proposed by Nils Reimers in 2019 to overcome the BERT model's inapplicability to unsupervised training tasks such as clustering and sentence regression. In his paper, Nils noted that directly using the final layer of BERT as sentence embeddings performed even worse than word embeddings, with [CLS] performing the worst. Therefore, it can be argued that the BERT model is only suitable for fine-tuning on specific tasks, and then using the fine-tuned BERT model for prediction to achieve optimal results. The original SBERT paper used a Siamese and Triplet network architecture to fine-tune the BERT pre-trained model, generating semantically informative sentence embeddings for similarity calculation. This architecture significantly improves computational efficiency by encoding each sentence individually. For example, in the task of detecting the two most similar texts from 10,000 texts, using BERT alone would incur a significant computational overhead, requiring the model to perform n*(n-1) / 2=49,995,000 operations (approximately 65 hours), which is extremely time-consuming. Under the same conditions, SBERT only needs to perform 10,000 operations (about 5 seconds) to obtain the sentence vector representation before performing subsequent operations, greatly improving efficiency.

[0004] In summary, the currently popular text matching model SBERT, as a representation-based model, can only extract representation vectors at the sentence level of text, but fails to consider the interactive information between text pairs at the word level, which affects the accuracy of text matching. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a text matching method integrating multi-head attention alignment, which comprises:

[0006] The language model SBERT is used to build a text matching model, which consists of an input layer, a BERT encoding layer, an interactive sentence representation layer, a fusion layer, and a fully connected layer.

[0007] Training the text matching model based on a set loss function;

[0008] estimating the degree of matching between target texts using the trained text matching model;

[0009] in:

[0010] The input layer takes the first and second sentences as input and uses character-level vectors, position vectors, and segment vectors to represent the sentences;

[0011] The BERT encoding layer takes the superposition vector of the character-level vector, position vector, and segment vector of the corresponding sentence as input and extracts the hidden layer vector with semantic information;

[0012] The interactive sentence representation layer performs attention interaction between sentence vectors based on the hidden layer vectors to extract interaction features of the two sentences respectively;

[0013] The fusion layer calculates the difference and product between the interactive feature vectors of the first sentence and the second sentence to obtain the feature fusion vector;

[0014] The fully connected layer obtains the matching results between sentences based on the feature fusion vector.

[0015] Compared with the existing technology, the advantage of the present invention is that it uses deep learning technology to improve the accuracy of text matching, and integrates the representation-based model and the interaction-based model by introducing a multi-head attention alignment mechanism. The improved model can better capture different levels of information in the original text and improve the accuracy of text matching.

[0016] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 is a flowchart of a text matching method integrating multi-head attention alignment according to an embodiment of the present invention;

[0019] Figure 2 is a structural diagram of a text matching model according to one embodiment of the present invention;

[0020] In the figure, Output refers to output; Fusion refers to fusion; Interactive Sentence Representation refers to interactive sentence representation; Mean-Pooling refers to average pooling; Last Hidden Vector refers to the last hidden vector; Prediction refers to prediction; Sentence A refers to sentence A; Sentence B refers to sentence B. Sub refers to subtraction; and Mul refers to multiplication. DETAILED DESCRIPTION

[0021] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0022] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0023] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0024] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0025] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0026] See also Figure 1 As shown, the provided text matching method integrating multi-head attention alignment includes the following steps:

[0027] Step S110 , using the language model SBERT to build a text matching model, which considers the interactive information between text pairs at the word level.

[0028] In one embodiment, an improved SBERT model integrating multiple attention alignment mechanism is proposed, or SBMAA (Improved SBert algorithm integrating Multiple Attention Alignment mechanism). Figure 2As shown in the figure, the SBMAA model consists of an input layer, a BERT encoding layer, an interactive sentence representation layer, a fusion layer, a fully connected layer, and an output layer. The BERT encoding layer and the interactive sentence representation layer form a twin network architecture.

[0029] In the following, we will focus on the implementation of the BERT encoding layer, interactive sentence representation layer, and fusion layer.

[0030] 1) Input layer and BERT encoding layer

[0031] For example, the input layer has two sentences A and B. According to the input characteristics of the BERT model, the input layer consists of character-level vectors (Token_Embeddings), position vectors (Position_Embeddings), and segment vectors (SegmentEmbeddings). These three vectors are added together and input into the encoding layer of BERT.

[0032] Character-level vectors (Token_Embeddings) are vectors corresponding to each character in the text. Position vectors (Position_Embeddings) contain the temporal information of each character in the text. Segment embeddings segment the text into sentences and locate the segments. The markers [CLS] and [SEP] are used to distinguish different sentences. [CLS] represents a special symbol for classification output, and [SEP] represents a special symbol for separating non-contiguous token sequences. The position information of each sentence is also stored. After stacking the three layers of vectors, they are introduced into the Transformer encoder, which uses a bidirectional encoding to represent each character. The Transformer encoder takes the stacked character-level vectors as input and passes them through the self-attention layer, residual layer, normalization layer, and feed-forward neural network layer in sequence to ultimately obtain a hidden layer vector with semantic information.

[0033] 2) Interactive Sentence Representation Layer

[0034] After obtaining the hidden layer vector, attention interaction is performed between the sentence vectors. For example, the similarity matrix is ​​first calculated to obtain the similarity between the two sentences, and then the attention mechanism is used to encode each sentence separately. The specific formula is as follows:

[0035] (1)

[0036] (2)

[0037] (3)

[0038] in, is the similarity matrix of the two hidden layer vectors, is the transposition of the hidden layer semantic vector of sentence A after BERT, is the hidden layer semantic vector of sentence B after BERT, so Indicates the similarity between the i-th word in sentence A and the j-th word in sentence B, Represents the similarity information between sentence A and sentence B extracted after attention interaction, Represents the similarity information between sentence B and sentence A extracted after attention interaction. Obtain important features for prediction and After that, average pooling is performed to obtain features and , Indicates the number of words contained in sentence A, Indicates the number of words contained in sentence B, Indicates the similarity between the i-th word in sentence A and the entire sentence B, Indicates the similarity between the jth word in sentence B and the entire sentence A.

[0039] In this step, the interactive sentence representation layer performs attention interaction between the hidden layer vectors to extract the interactive features of the two sentences. To account for the fusion between the two hidden layer vectors, a dot product is used to calculate the attention weights between them. Compared to other weight calculation methods, the present invention uses a dot product to calculate the interactive representation between the two hidden layers, which can improve computational efficiency. Due to the influence of sequence length, average pooling can be used to subsequently reduce the result to a fixed-length vector.

[0040] 3) Fusion layer, fully connected layer and output layer

[0041] The interactive sentence representation layer extracts interactive features in each sentence and The fusion layer enhances the local reasoning information by calculating the difference and product between the interactive feature vectors of the first sentence and the second sentence, that is, the fusion layer is used to fuse the interactive features of the two sentences to obtain a feature fusion vector.

[0042] Specifically, the fusion layer fuses the two interactive features according to the following formula (4):

[0043] (4)

[0044] in It is a subtraction operation between vectors, the purpose of which is to obtain difference features; It is a vector-matrix multiplication operation, the purpose of which is to obtain interactive features. Then, the results of the subtraction and multiplication operations are vector-concatenated to obtain the feature fusion vector Finally, Input into a fully connected network, adjust the weight of the features, and use a softmax function to predict the classification results. The calculation formula of the fully connected layer is as follows:

[0045] (5)

[0046] (6)

[0047] in Represents the weight, which is the dimension and The same matrix, represents the bias, which is the dimension of The same one-dimensional vector, Represents the predicted probability, and the predicted classification result is Get The maximum value in the range. represents the true label.

[0048] In the above steps, since it is necessary to integrate all the information and perform a global analysis, all the information is stored in a sequence, that is, the fusion feature vector is put into the fully connected layer classifier for classification. The activation function can use the tan function, and the result is sent to the softmax layer to finally obtain the matching result between sentences.

[0049] In summary, the embodiment of the present invention uses SBERT to achieve text vector representation, introduces multi-head attention alignment based on the twin network architecture, increases the interaction of sentence vectors, and then splices a fusion layer, so that the model itself has the ability to obtain interactive information.

[0050] Step S120: pre-training the text matching model using the set loss function.

[0051] The loss function can use the cross-entropy loss function or another type of loss function (such as the Mean Square Error Sequence (MSE) loss function). The training dataset can be the ATEC2018 NLP dataset or another dataset. After pre-training, the model's optimization parameters, such as weights and biases, are obtained.

[0052] Step S130: For the target text to be matched, obtain a matching result using the trained text matching model.

[0053] In this paper, text matching can include predicting the similarity between target texts, and is applied in various scenarios such as information retrieval, question answering systems, question retelling, dialogue systems, machine translation, etc.

[0054] In order to further verify the effect of the present invention, a comparative test was conducted. Based on the same dataset, five other classic text matching models were selected for experimental comparison: ESIM, SBERT, ConSERT, Bert-Whitening, and SimCSE. Among them, ESIM is a model based on BiLSTM and tree-LSTM, and is an enhanced version of LSTM designed specifically for natural language inference; ConSERT is a model that uses unsupervised and data augmentation methods to fine-tune BERT for contrastive learning; Bert-Whitening is a text matching model that converts the embedded vector into an isotropic text matching model through a simple whitening operation; the effect in the task is comparable to Bert-flow; SimCSE is a text matching model that uses contrastive learning to optimize the objective function to fine-tune the model to obtain text vector representation.

[0055] Table 1 below lists the experimental results of 6 different deep learning models on the ATEC2018 NLP dataset.

[0056] Table 1 Statistics of experimental results on the ATEC2018 dataset

[0057]

[0058] As shown in Table 1, the proposed SBMAA model achieves an F1 score of 84.7% and a recall rate of 92.6%, outperforming other models overall. Compared to the baseline SBERT model, the proposed model achieves an 18.6% improvement in F1. This is because the proposed model adds interaction between sentence vectors to the SBERT model, allowing it to capture more fine-grained semantic information than directly capturing similarity between sentences.

[0059] In summary, Sentence-BER, as a pre-trained language model, obtained the semantic representations of two sentences based on BERT for comparison and achieved good results. However, due to the different architecture of the model, it also belongs to the representation-based text matching algorithm. The main idea of ​​this type of algorithm is to extract the main semantics of the two sentences, encode them in the same vector space, then model the semantics of the two texts, and finally calculate the similarity. The disadvantage of this type of method is that it only extracts the representation vector at the text sentence level, and does not consider the interactive information between text pairs at the word level, thus losing the semantic focus, easily causing semantic deviations, and it is difficult to measure the meaning of a single word in the context. The SBMAA model proposed in the present invention aligns (interacts) the hidden layer vectors generated by the Multi-Head Attention part in the encoder module of the 12-layer transformer of SBERT to obtain interactive features for fusion, thereby strengthening the interaction between texts and effectively improving the effect of text information matching.

[0060] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0061] A computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure within a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0062] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0063] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present invention.

[0064] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0065] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0067] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0068] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A text matching method integrating multi-head attention alignment, comprising the following steps: The language model SBERT is used to build a text matching model, which consists of an input layer, a BERT encoding layer, an interactive sentence representation layer, a fusion layer, and a fully connected layer. Training the text matching model based on a set loss function; estimating the degree of matching between target texts using the trained text matching model; in: The input layer takes the first and second sentences as input and uses character-level vectors, position vectors, and segment vectors to represent the sentences; The BERT encoding layer takes the superposition vector of the character-level vector, position vector, and segment vector of the corresponding sentence as input and extracts the hidden layer vector with semantic information; The interactive sentence representation layer performs attention interaction between sentence vectors based on the hidden layer vectors to extract interaction features of the two sentences respectively; The fusion layer calculates the difference and product between the interactive feature vectors of the first sentence and the second sentence to obtain the feature fusion vector; The fully connected layer obtains the matching results between sentences based on the feature fusion vector; The interactive sentence representation layer first calculates the similarity between two sentences and uses the attention mechanism to encode each sentence separately, which is expressed as: in, is the transposition of the hidden layer semantic vector of the first sentence A after passing through the BERT encoding layer, is the hidden layer semantic vector of the second sentence B after passing through the BERT encoding layer, Represents the similarity between the i-th word in the first sentence A and the j-th word in the second sentence B, Represents the similarity information between the first sentence A and the second sentence B extracted after attention interaction, Represents the similarity information between the second sentence B and the first sentence A extracted after attention interaction. and Perform average pooling to obtain the interactive features of the two sentences. Indicates the number of words contained in the first sentence A, Indicates the number of words contained in sentence B, Indicates the similarity between the i-th word in the first sentence A and the entire sentence B, Indicates the similarity between the jth word in the second sentence B and the entire sentence A.

2. The method according to claim 1, characterized in that The fusion layer fuses the interactive features of the two sentences to obtain a feature fusion vector, which is expressed as: in, represents the interaction feature vector of the first sentence, represents the interaction feature vector of the second sentence, It is a subtraction operation between vectors, used to obtain difference features; It is a vector-matrix multiplication operation used to obtain interactive features. The results of the subtraction and multiplication operations are vector-concatenated to obtain the feature fusion vector. .

3. The method according to claim 2, characterized in that The calculation formula of the fully connected layer is expressed as: in, represents the weight, Is the dimension and The same matrix, Indicates bias, Is the dimension and The same one-dimensional vector, is the result of the predicted classification, represents the predicted probability, represents the true label.

4. The method according to claim 1, wherein The character-level vector is the vector corresponding to each word in the text, the position vector contains the timing information of each word in the text, and the segmentation vector segments the text into sentences and locates the text segments. Different sentences are distinguished by set marking symbols.

5. The method according to claim 1, wherein The BERT encoding layer sequentially includes a Transformer encoder, a self-attention layer, a residual layer, a normalization layer, and a feedforward neural network layer. The Transformer encoder takes character-level vector superposition as input and performs bidirectional encoding on each word.

6. The method according to claim 1, wherein The loss function is a cross entropy loss function.

7. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images