Sentence pair semantic matching method and system

Abstract

The application discloses a sentence pair semantic matching method and system, belongs to the natural language processing technical field and the computer artificial intelligence field, and aims to solve the technical problems of how to construct and utilize context information of a sentence pair to strengthen a sentence pair interaction process and how to realize direct interaction between the context information and the sentence pair, thereby improving the accuracy of sentence pair semantic matching. The method specifically comprises the following steps: obtaining a sentence pair semantic matching data set; downloading a publicly disclosed sentence pair semantic matching data set from a network; constructing a sentence pair semantic matching model based on a Bilinear Triple-Attention mechanism; and training the sentence pair semantic matching model on a sentence pair semantic matching training data set. The system comprises a data set obtaining unit, a model constructing unit and a model training unit.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and computer artificial intelligence, specifically to a sentence pair semantic matching method and system. Background Technology

[0002] In recent years, sentence pair semantic matching methods have gained increasing attention in the field of natural language processing (NLP). This is because many NLP tasks are based on sentence pair semantic matching techniques, and can be seen as extensions of these tasks. Therefore, sentence pair semantic matching plays a crucial role in NLP. However, measuring the degree of inherent semantic matching between sentence pairs is a very challenging task, and to date, existing methods have not substantially solved this problem.

[0003] Analysis and research reveal that existing methods can be broadly categorized into two types:

[0004] ① Representation-based methods: First, two sentences are encoded using an encoder. Then, the encoded representations of the two sentences are interacted to obtain the final sentence pair representation, and thus the matching degree of the sentence pair is obtained. The advantage of this type of method is that it is simple in concept and easy to implement. However, interacting only based on the deep features output by the encoder will result in insufficient interaction between sentence pairs.

[0005] ② Interaction-based methods can effectively avoid the above problems. Interaction-based methods directly interact with sentence pairs after obtaining their embedded representations; after obtaining the interaction results, further encoding processing is then performed. Many existing works have demonstrated that interaction-based methods often achieve better performance. However, these methods also have shortcomings: most sentence pair interaction mechanisms use attention mechanisms, but existing attention mechanisms can only achieve interaction between the two pairs and cannot incorporate overall contextual information into the calculation of attention weights. For sentence pair semantic matching tasks, without considering overall contextual information, the semantic interaction between sentence pairs may be insufficient, making it difficult to accurately calculate the semantic matching degree.

[0006] Therefore, how to construct and utilize the contextual information of sentence pairs to enhance the interaction process of sentence pairs, and how to realize the direct interaction between contextual information and sentence pairs, thereby improving the accuracy of semantic matching of sentence pairs, are currently urgent technical problems that need to be solved. Summary of the Invention

[0007] The technical objective of this invention is to provide a method and system for semantic matching of sentence pairs to solve the problem of semantic matching of sentence pairs.

[0008] The technical objective of this invention is achieved as follows: a sentence pair semantic matching method, the specific method of which is as follows:

[0009] Obtain sentence pair semantic matching datasets: Download publicly available sentence pair semantic matching datasets from the internet;

[0010] Constructing a sentence pair semantic matching model: Constructing a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism;

[0011] Training the sentence pair semantic matching model: Train the sentence pair semantic matching model on the sentence pair semantic matching training dataset.

[0012] As a preferred option, the Bilinear Triple-Attention mechanism is an attention mechanism based on the Bilinear ProductAttention algorithm. The specific implementation process is as follows:

[0013] (1) Calculate the relevance score between query, key, and context, as follows:

[0014] ① By introducing a third-order matrix ω to multiply the three input tensors query, key, and context, interactive computation among the three is achieved, as shown in the following formula:

[0015]

[0016] Where query represents the information of sentence 1 in the sentence pair; key represents the information of sentence 2 in the sentence pair; context represents the joint context of the sentence pair, that is, the concatenation of sentence 1 and sentence 2; and These represent the input tensors query, key, and context, respectively; d represents the input tensor. The element index in the tensor; D represents the input tensor. The number of elements in the input tensor; i and j represent the input tensor. and Element indices; i = 1, 2, ..., I kl j = 1, 2, ..., J cl ;I kl and J cl Representing the input tensors respectively and The number of elements in the array; ω is a third-order trainable weight tensor with shape D×D×D; ×1, ×2, and ×3 represent 1-modulo multiplication, 2-modulo multiplication, and 3-modulo multiplication, respectively. These are custom tensor operators defined for ease of writing formulas, and their operation process is as follows:

[0017] According to the relevant content in the literature T. Kolda and B. Bader, “Tensor decompositions and applications,” SIAM Review, vol. 51, no. 3, pp. 455–500, 2009, given two tensors and but express and Modular multiplication of n, where Similarly, X and column vectors The modular multiplication rule of n is defined as in,

[0018] ② Obtain the correlation score among the three through normalization, and denote it as α, as shown in the following formula:

[0019]

[0020] Among them, i=1, 2,...,I kl i' = 1, 2, ..., I kl j = 1, 2, ..., J cl j' = 1, 2, ..., J cl ;

[0021] (2) Feature aggregation operation: After mapping by the weight matrix, tensor multiplication is performed on the two input tensors, value and context, to complete the aggregation operation of the sentence to the joint context. The formula is as follows:

[0022]

[0023] Here, value represents the information of sentence 2 in the sentence pair, which is the same as the input tensor key; and These represent the input tensor value and context, respectively. This indicates that the weight matrix is ​​used to map pairs. and Perform aggregation operation; U' and H' are the weight matrices to be trained; i and j represent the input tensors. and Element indices; i = 1, 2, ..., I vl j = 1, 2, ..., J cl ;I vl and J cl Representing the input tensors respectively and The number of elements in;

[0024] (3) Dot product operation: Perform a dot product operation between the relevance score tensor obtained in step (1) and the feature aggregation representation obtained in step (2) to obtain the context-enhanced attention embedding representation, as shown in the following formula:

[0025]

[0026] Where i and j represent the element indices of α and v'; I and J represent the lengths of the two dimensions in the input tensors α and v';

[0027] Bilinear Triple-Attention is a stackable structure that dynamically adjusts the number of layers based on the specific task, denoted as... depth indicates the specific number of layers set for a particular task; This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. Its and same.

[0028] More specifically, the sentence-pair semantic matching model is constructed as follows:

[0029] Input data: For each data point in the dataset, firstly, two sentences are concatenated, with sentence 1 and sentence 2 separated by the separator "SEP". This concatenation is denoted as the context of the sentence pair. The sequence of sentence 1 is denoted as S1. All candidate sentence 2s are denoted as S2. The label of the data point is determined based on whether the semantics of the current sentence pair match. If it matches, it is denoted as 1; otherwise, it is denoted as 0. The three text sequences—the sentence pair context sequence, the sentence 1 sequence, and the candidate sentence 2 sequence—along with the label, constitute one input data point.

[0030] Pre-trained embedding representations: Pre-trained language models are used to perform embedding encoding operations on the input data, resulting in embedding representations of sentence pairs, joint context, sentence 1, and sentence 2 from the input data, denoted as follows: and Each sentence in the joint context sequence uses a separate encoding module, while sentence 1 and candidate sentence 2 sequences share the same encoding module. Since the pre-trained language model itself contains multiple layers of encoding networks, the output of different layers can be selected as its embedding representation based on different input objects; the formula is as follows:

[0031]

[0032] in, This indicates that the pre-trained language model BERT is used to embed and encode the joint context sequence of the input sentence; the subscript n indicates the number of network layers inside BERT used. This indicates the embedding of sentences into the context of a compound sentence;

[0033]

[0034] in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S1; the subscript t indicates the number of network layers inside BERT used. This indicates that sentence 1 is embedded.

[0035]

[0036] in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S2; the subscript t indicates the number of network layers inside BERT used. Sentence 2 embedding representation;

[0037] Bilinear Triple-Attention Interaction: The input consists of sentence pair joint context embeddings, sentence 1 embeddings, and sentence 2 embeddings. The first-layer interaction structure interacts with the sentence 1 and sentence 2 embeddings based on the sentence pair joint context embeddings to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, which are then passed to the second-layer interaction structure. Simultaneously, a fully connected network is used to map the sentence pair joint context embeddings to obtain the first-layer mapped representation of the sentence pair joint context, which is then passed to the second-layer interaction structure. The second-layer interaction structure interacts with the first-layer sentence 1 and sentence 2 interaction results based on the first-layer mapped representation of the sentence pair joint context to obtain the second-layer sentence 1 interaction result and the second-layer sentence 2 interaction result, which are then passed to the third-layer interaction structure. Simultaneously, a fully connected network is used to map the first-layer mapped representation of the sentence pair joint context to obtain the second-layer mapped representation of the sentence pair joint context, which is then passed to the third-layer interaction structure. This process is repeated multiple times to generate the depth-level sentence 1 interaction result, the depth-level sentence 2 interaction result, and the depth-level mapped representation of the sentence pair joint context.

[0038] Feature aggregation: Adaptive average pooling is used to filter the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the mapping representation of the sentence pair joint context at the depth layer. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, which is denoted as .

[0039] Label prediction: Based on the final aggregated feature representation, determine whether the sentence pairs semantically match; specifically as follows:

[0040] The final aggregated feature representation is processed by a fully connected network with dimension 1 and activation function sigmoid to obtain a matching degree value between [0,1], denoted as y_pred. The matching degree value is compared with a set threshold (0.5) to determine whether the semantics of the sentence pair match. That is, when y_pred≥0.5, the semantics of the sentence pair is predicted to match; otherwise, they do not match.

[0041] More preferably, the Bilinear Triple-Attention interaction is as follows:

[0042] The first-layer interaction structure is based on sentence pair joint context embedding representations. It interacts with sentence 1 embedding representations and sentence 2 embedding representations respectively to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, denoted as... and The sentence-to-joint context embedding representation is mapped through a fully connected network to obtain the first-level mapping representation of the sentence-to-joint context, denoted as . The formula is as follows:

[0043]

[0044] in, and These represent the sentence pair joint context embedding representation, sentence 1 embedding representation, and sentence 2 embedding representation obtained after processing by the pre-trained embedding representation module, respectively. Indicates use Interactive operations are performed on sentence 1 embedding representation and sentence 2 embedding representation based on sentence pair joint context embedding representation;

[0045]

[0046] in, Indicates use Interactive operations are performed on the sentence pair joint context embedding representation and the sentence 1 embedding representation based on sentence pair joint context embedding representation;

[0047]

[0048] in, Indicates the use of fully connected Dense 1 Perform mapping operations on the sentence-pair joint context embedding representation;

[0049] The interaction results of sentence 1 at the first level, the interaction results of sentence 2 at the first level, and the first-level mapping representation of the sentence pair in the joint context are represented as follows: and The result is passed to the second-level interaction structure. The second-level interaction structure, based on the first-level mapping representation of the sentence-to-context joint context, interacts with the first-level sentence 1 interaction result and the first-level sentence 2 interaction result to obtain the second-level sentence 1 interaction result and the second-level sentence 2 interaction result, denoted as... and The first-level mapping representation of the sentence to the joint context is then mapped through a fully connected network to obtain the second-level mapping representation of the sentence to the joint context, denoted as . The formula is as follows:

[0050]

[0051] in, Indicates use The first-level mapping representation based on sentence pairs in the joint context performs interactive operations on the interaction results of sentence 1 and sentence 2 at the first level.

[0052]

[0053] in, Indicates use Based on the first-level mapping representation of sentence pair joint context, interactive operations are performed on the interaction results of first-level sentence 2 and first-level sentence 1.

[0054]

[0055] in, Indicates the use of Dense 2 Perform mapping operations on the second-level mapping representation of the sentence to the joint context;

[0056] The interaction results of sentence 1 at the second level, the interaction results of sentence 2 at the second level, and the second-level mapping representation of sentence pairs in the joint context are represented as follows: and This is passed to the third-layer interaction structure; and so on, repeated interactions are performed to generate the sentence 1 interaction representation at the depth layer, the sentence 2 interaction representation at the depth layer, and the sentence-to-context mapping representation at the depth layer; for the depth layer, the formula is as follows:

[0057]

[0058] in, Indicates use Based on the sentence pair joint context, the depth-1 layer mapping representation performs interaction operations on the interaction results of sentence 1 and sentence 2 at the depth-1 layer.

[0059]

[0060] in, Indicates use Based on the sentence pair joint context, the depth-1 layer mapping representation performs interaction operations on the interaction results of sentence 2 and sentence 1 at the depth-1 layer.

[0061]

[0062] in, Indicates the use of Dense depth Perform mapping operations on the depth-1 level mapping representation of the sentence-to-joint context.

[0063] More specifically, the sentence pair semantic matching model is trained as follows:

[0064] Constructing the loss function: Cross-entropy is used as the loss function;

[0065] Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate, which was set to 2e-5, all other hyperparameters of AdamW were set to the default values ​​in PyTorch.

[0066] When the sentence pair semantic matching model has not yet been trained, further training is required to optimize the parameters of the sentence pair semantic matching model; when the sentence pair semantic matching model has been trained, it can predict whether the semantics of the target sentence pair match.

[0067] A sentence-pair semantic matching system, the system comprising,

[0068] The dataset acquisition unit is used to download publicly available sentence pair semantic matching datasets from the internet;

[0069] The model building unit is used to build a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism.

[0070] The model training unit is used to construct the loss function and optimization function to complete the semantic matching judgment.

[0071] Preferably, the model building unit includes,

[0072] The input module is used to preprocess the raw dataset to construct the input data;

[0073] The pre-trained embedding representation module is used to perform embedding encoding operations on the input data using a pre-trained language model, thereby obtaining the embedding representations of sentence pairs, joint context, sentence 1, and sentence 2 in the input data;

[0074] The Bilinear Triple-Attention interaction module is used to receive the sentence pair joint context embedding representation, sentence 1 embedding representation and sentence 2 embedding representation output by the pre-trained embedding representation module, and then repeatedly interact to generate the sentence 1 interaction result of the depth layer, the sentence 2 interaction result of the depth layer and the mapping result of the sentence pair joint context of the depth layer.

[0075] The feature aggregation module is used to perform feature filtering operations on the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the mapping representation of sentence pair joint context at the depth layer using adaptive average pooling operation, and then concatenates the three filtered feature representations to obtain the final aggregated feature representation.

[0076] The label prediction module is used to determine whether the semantics of the current sentence pair match based on the final aggregated feature representation.

[0077] More preferably, the model training unit includes,

[0078] The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function.

[0079] The optimization function building module is used to train and adjust the parameters during model training to reduce prediction error.

[0080] An electronic device includes: a memory and at least one processor;

[0081] The memory contains computer programs;

[0082] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the sentence pair semantic matching method as described above.

[0083] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the sentence pair semantic matching method as described above.

[0084] The sentence matching method and system of the present invention have the following advantages:

[0085] (i) This invention can construct and utilize the contextual information of sentence pairs to enhance the interaction process of sentence pairs and realize the direct interaction between contextual information and sentence pairs, thereby improving the accuracy of semantic matching of sentence pairs.

[0086] (ii) This invention can capture and utilize the semantic information contained in sentences through pre-trained embedding representation processing, making the captured semantic features richer and more accurate. At the same time, it can effectively extract and construct the overall contextual information of sentence pairs, making the subsequent interaction process more complete.

[0087] (III) This invention can incorporate contextual information into the sentence pair interaction process through Bilinear Triple-Attention interaction, thereby realizing direct interaction between contextual information and sentence pairs, making the semantic interaction between sentence pairs more complete; at the same time, it can take advantage of the flexibility of the Bilinear interaction calculation method to expand the application scenarios of the model.

[0088] (iv) This invention can aggregate and utilize multiple features through feature aggregation, thereby improving the accuracy of semantic matching of sentences;

[0089] (v) This invention combines the Bilinear Triple-Attention mechanism, which can effectively improve the accuracy of sentence pair matching at the semantic level. Attached Figure Description

[0090] The invention will be further described below with reference to the accompanying drawings.

[0091] Appendix Figure 1 This is a flowchart of the sentence pair semantic matching method;

[0092] Appendix Figure 2 A flowchart for constructing a sentence-pair semantic matching model;

[0093] Appendix Figure 3 A flowchart for training a sentence-pair semantic matching model;

[0094] Appendix Figure 4 This is a structural block diagram of a sentence-pair semantic matching system;

[0095] Appendix Figure 5 This is a schematic diagram of the Bilinear Triple-Attention interaction;

[0096] Appendix Figure 6 This is a schematic diagram of a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism. Detailed Implementation

[0097] The sentence pair semantic matching method and system of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0098] Example 1:

[0099] As attached Figure 1 As shown in the figure, this embodiment provides a sentence pair semantic matching method, which is as follows:

[0100] S1. Obtain sentence pair semantic matching datasets: Download publicly available sentence pair semantic matching datasets from the internet; for example, there are many publicly available sentence pair semantic matching datasets on the internet, such as the LCQMC dataset.

[0101] The data in this dataset is formatted as follows:

[0102]

[0103] The dataset contains a total of 260,068 sentence pairs, which are divided into three parts: training set (containing 238,766 sentence pairs), validation set (containing 8,802 sentence pairs), and test set (containing 12,500 sentence pairs). Positive examples (label: 1) indicate that the two sentences are semantically matched, while negative examples (label: 0) indicate that the two sentences are semantically mismatched.

[0104] S2. Constructing a sentence pair semantic matching model: Constructing a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism;

[0105] S3. Training the sentence pair semantic matching model: Train the sentence pair semantic matching model on the sentence pair semantic matching training dataset.

[0106] The Bilinear Triple-Attention mechanism in this embodiment is an attention mechanism based on the Bilinear ProductAttention algorithm. Bilinear ProductAttention first receives three input tensors: query, key, and value. Then, it calculates the attention weight between query and key. Finally, it calculates the dot product of the attention weight and value to obtain the final attention representation. The difference between Bilinear Triple-Attention and the Bilinear Attention algorithm is that the latter receives three inputs (query, key, and value) and can only calculate the attention weight between query and key, while the former can receive four inputs (query, key, value, and context) and calculate the attention weight between query, key, and context, where context represents the contextual information related to query and key. It should be noted that in both Bilinear Triple-Attention and BilinearAttention, the inputs value and key are the same.

[0107] The specific implementation process of the Bilinear Triple-Attention mechanism is as follows:

[0108] (1) Calculate the relevance score between query, key, and context, as follows:

[0109] ① By introducing a third-order matrix ω to multiply the three input tensors query, key, and context, interactive computation among the three is achieved, as shown in the following formula:

[0110]

[0111] Where query represents the information of sentence 1 in the sentence pair; key represents the information of sentence 2 in the sentence pair; context represents the joint context of the sentence pair, that is, the concatenation of sentence 1 and sentence 2; and These represent the input tensors query, key, and context, respectively; d represents the input tensor. The element index in the tensor; D represents the input tensor. The number of elements in the input tensor; i and j represent the input tensor. and Element indices; i = 1, 2, ..., I kl j = 1, 2, ..., J cl;I kl and J cl Representing the input tensors respectively and The number of elements in the tensor; ω is a third-order trainable weight tensor with shape D×D×D; ×1, ×2, and ×3 represent the operators for performing dot product calculations between tensors;

[0112] ② Obtain the correlation score among the three through normalization, and denote it as α, as shown in the following formula:

[0113]

[0114] Among them, i=1, 2,...,I kl i' = 1, 2, ..., I kl j = 1, 2, ..., J cl j' = 1, 2, ..., J cl ;

[0115] For example:

[0116] In PyTorch, the code implementation for the above operation is as follows:

[0117] # Receive three inputs, which are mapped to query, key, and context in the above formula through different fully connected networks.

[0118] s1_embed_layer=self.s1_embed(sen1)

[0119] key_layer = self.key(sen2)

[0120] context_layer=self.context(context_sens)

[0121] s1_embed_layer=self.gene_s1_embed(s1_embed_layer)

[0122] key_layer=self.gene_key(key_layer)

[0123] context_layer=self.gene_context(context_layer)

[0124] # Implement element-wise addition operations between three tensors: query, key, and context.

[0125] key_context=unsqueeze(key_layer,dim=2)* unsqueeze(context_layer,dim=3)

[0126] key_context=key_context.view(key_context.size(0), key_context.size(1),

[0127] key_context.size(2)*key_context.size(3), key_context.size(4))

[0128] #Normalization using softmax

[0129] attention_scores=torch.matmul(s1_embed_layer, key_context.transpose(-1,-2))

[0130] attention_probs=nn.Softmax(dim=-1)(attention_scores)

[0131] Where sen1 represents the formula sen2 represents the formula context_sens represents the formula attention_scores represents the value in the formula. attention_probs represents the relevance score in the formula, which is α.

[0132] (2) Feature aggregation operation: After mapping by the weight matrix, tensor multiplication is performed on the two input tensors, value and context, to complete the aggregation operation of the sentence to the joint context. The formula is as follows:

[0133]

[0134] Here, value represents the information of sentence 2 in the sentence pair, which is the same as the input tensor key;

[0135] and These represent the input tensor value and context, respectively. This indicates that the weight matrix is ​​used to map pairs. and Perform aggregation operation; U' and H' are the weight matrices to be trained; i and j represent the input tensors. and Element indices; i = 1, 2, ..., I vl j = 1, 2, ..., J cl ;I vl and J cl Representing the input tensors respectively and The number of elements in the array; for example:

[0136] In PyTorch, the code described above is implemented as follows:

[0137] # Perform aggregation operation on value and context by adding them element by element.

[0138] value_layer=self.gene_value(key_layer)

[0139] context_layer_value=self.gene_context_value(context_layer)

[0140] value_context=unsqueeze(value_layer,dim=2)*unsqueeze(context_layer_value,dim=3)

[0141] value_context=value_context.view(value_context.size(0), value_context.size(1),value_context.size(2)*value_context.size(3),value_context.size(4))

[0142] Here, value_context represents the aggregated result in the formula, which is v'.

[0143] (3) Dot product operation: Perform a dot product operation between the relevance score tensor obtained in step (1) and the feature aggregation representation obtained in step (2) to obtain the context-enhanced attention embedding representation, as shown in the following formula:

[0144]

[0145] Where i and j represent the element indices of α and v'; I and J represent the lengths of the two dimensions in the input tensors α and v';

[0146] For example, in PyTorch, the code implementation described above would be as follows:

[0147] # Perform a dot product operation on the attention weights and aggregate representations of the three input tensors.

[0148] query_new_layer=torch.matmul(attention_probs,value_context)

[0149] Where query_new_layer represents the interaction result in the formula, which is q new .

[0150] Bilinear Triple-Attention is a stackable structure that dynamically adjusts the number of layers according to the specific task, denoted as . depth indicates the specific number of layers set for a particular task; This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. Its and same.

[0151] In this invention, when applying the Bilinear Triple-Attention mechanism, query represents the information of sentence 1 in text matching, such as the sentence 1 embedding representation and the interaction result of sentence 1 at the i-th layer; key and value are the same, representing the information of sentence 2 in text matching, such as the sentence 2 embedding representation and the interaction result of sentence 2 at the i-th layer; context represents the joint context information of sentence pairs in text matching, such as the sentence pair joint context embedding representation and the i-th layer mapping result of the sentence pair joint context.

[0152] The specific steps for constructing the sentence pair semantic matching model in step S2 of this embodiment are as follows:

[0153] S201. Input Data: For each data point in the dataset, firstly, two sentences are concatenated, with sentence 1 and sentence 2 separated by the separator "SEP". This is taken as the joint context of the sentence pair, denoted as context. The sequence of sentence 1 is denoted as S1. All candidate sentence 2s are denoted as S2. The label of the data point is determined based on whether the semantics of the current sentence pair match. That is, if it matches, it is denoted as 1; otherwise, it is denoted as 0. The three text sequences—the joint context sequence of sentence pairs, the sentence 1 sequence, and the candidate sentence 2 sequence—along with the label, constitute one input data point.

[0154] For example: Using the data shown in step S1 as a sample, we can construct a single input data entry. The result is shown below:

[0155] (context: Which song has this lyric? SEP Which song has this lyric? S1: Which song has this lyric? S2: Which song has this lyric? 1)

[0156] S202, Pre-trained Embedding Representation: The pre-trained language model is used to perform embedding encoding on the input data, thereby obtaining the embedding representations of sentence pairs, joint context, sentence 1, and sentence 2 in the input data, denoted as follows: and Each sentence in the joint context sequence uses a separate encoding module, while sentence 1 and candidate sentence 2 sequences share the same encoding module. Since the pre-trained language model itself contains multiple layers of encoding networks, the output of different layers can be selected as its embedding representation based on different input objects; the formula is as follows:

[0157]

[0158] in, This indicates that the pre-trained language model BERT is used to embed and encode the joint context sequence of the input sentence; the subscript n indicates the number of network layers inside BERT used. This indicates the embedding of sentences into the context of a compound sentence;

[0159]

[0160] in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S1; the subscript t indicates the number of network layers inside BERT used. This indicates that sentence 1 is embedded.

[0161]

[0162] in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S2; the subscript t indicates the number of network layers inside BERT used. Sentence 2 embedding representation;

[0163] For example, when this invention is implemented on the RACE dataset, n is set to 12 and t is set to 1. In PyTorch, the code implementation described above is as follows:

[0164] # Encode the input using BERT

[0165] context_embed,_=BERT_n(context)

[0166] query_embed, _ = BERT_t(query)

[0167] response_embed,_=BERT_t(response)

[0168] Wherein, context_embed is the sentence pair joint context embedding representation, query_embed is the sentence 1 embedding representation, and response_embed is the sentence 2 embedding representation.

[0169] S203, Bilinear Triple-Attention Interaction: The input consists of sentence pair joint context embeddings, sentence 1 embeddings, and sentence 2 embeddings. The first-layer interaction structure interacts with the sentence 1 and sentence 2 embeddings based on the sentence pair joint context embeddings to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, which are then passed to the second-layer interaction structure. Simultaneously, a fully connected network is used to map the sentence pair joint context embeddings to obtain the first-layer mapped representation of the sentence pair joint context, which is then passed to the second-layer interaction structure. The second-layer interaction structure interacts with the first-layer sentence 1 and sentence 2 interaction results based on the first-layer mapped representation of the sentence pair joint context to obtain the second-layer sentence 1 interaction result and the second-layer sentence 2 interaction result, which are then passed to the third-layer interaction structure. Simultaneously, a fully connected network is used to map the first-layer mapped representation of the sentence pair joint context to obtain the second-layer mapped representation of the sentence pair joint context, which is then passed to the third-layer interaction structure. This process is repeated multiple times to generate the depth-level sentence 1 interaction result, the depth-level sentence 2 interaction result, and the depth-level mapped representation of the sentence pair joint context.

[0170] S204. Feature Aggregation: Adaptive average pooling is used to filter the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the mapping representation of the sentence pair joint context at the depth layer. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, which is denoted as S204.

[0171] For example, in PyTorch, the code implementation described above would be as follows:

[0172] query_out=torch.squeeze(self.pool_qo(torch.transpose(query_inter,dim0=1, dim1=2)),dim=-1)

[0173] response_out=torch.squeeze(self.pool_qo(torch.transpose(response_inter,dim0=1, dim1=2)),dim=-1)

[0174] context_out= torch.squeeze(self.pool_qo(torch.transpose(context_dense,dim0=1, dim1=2)),dim=-1)

[0175] #Feature aggregation via join operation

[0176] feature=torch.cat([query_out,response_out,context_out],dim=-1)

[0177] Here, query_out is the feature representation of sentence 1 after average pooling of the interaction result of sentence 1 in the depth layer, response_inter is the feature representation of sentence 2 after average pooling of the interaction result of sentence 2 in the depth layer, context_dense is the feature representation of the joint context of sentence pairs after average pooling of the mapping representation of sentence pairs in the depth layer, and feature is the final aggregated feature representation.

[0178] S205, Label Prediction: Based on the final aggregated feature representation, determine whether the sentence pairs semantically match; details are as follows:

[0179] The final aggregated feature representation is processed by a fully connected network with dimension 1 and activation function sigmoid to obtain a matching degree value between [0,1], denoted as y_pred. The matching degree value is compared with a set threshold (0.5) to determine whether the semantics of the sentence pair match. That is, when y_pred≥0.5, the semantics of the sentence pair is predicted to match; otherwise, they do not match.

[0180] As attached Figure 5 As shown, the Bilinear Triple-Attention interaction in step S204 of this embodiment is as follows:

[0181] S20401, the first-layer interaction structure is based on the sentence pair joint context embedding representation. It interacts with the sentence 1 embedding representation and the sentence 2 embedding representation respectively to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, denoted as... and The sentence-to-joint context embedding representation is mapped through a fully connected network to obtain the first-level mapping representation of the sentence-to-joint context, denoted as . The formula is as follows:

[0182]

[0183] in, and These represent the sentence pair joint context embedding representation, sentence 1 embedding representation, and sentence 2 embedding representation obtained after processing by the pre-trained embedding representation module, respectively. Indicates use Interactive operations are performed on sentence 1 embedding representation and sentence 2 embedding representation based on sentence pair joint context embedding representation;

[0184]

[0185] in, Indicates use Interactive operations are performed on the sentence pair joint context embedding representation and the sentence 1 embedding representation based on sentence pair joint context embedding representation;

[0186]

[0187] in, Indicates the use of fully connected Dense 1 Perform mapping operations on the sentence-pair joint context embedding representation;

[0188] S20402, The interaction results of sentence 1 at the first level, the interaction results of sentence 2 at the first level, and the first-level mapping representation of the sentence pair in the joint context are represented as follows: and The result is passed to the second-level interaction structure. The second-level interaction structure, based on the first-level mapping representation of the sentence-to-context joint context, interacts with the first-level sentence 1 interaction result and the first-level sentence 2 interaction result to obtain the second-level sentence 1 interaction result and the second-level sentence 2 interaction result, denoted as... and The first-level mapping representation of the sentence to the joint context is then mapped through a fully connected network to obtain the second-level mapping representation of the sentence to the joint context, denoted as . The formula is as follows:

[0189]

[0190] in, Indicates use The first-level mapping representation based on sentence pairs in the joint context performs interactive operations on the interaction results of sentence 1 and sentence 2 at the first level.

[0191]

[0192] in, Indicates use Based on the first-level mapping representation of sentence pair joint context, interactive operations are performed on the interaction results of first-level sentence 2 and first-level sentence 1.

[0193]

[0194] in, Indicates the use of Dense 2 Perform mapping operations on the second-level mapping representation of the sentence to the joint context;

[0195] S20403, The second-level mapping representation of the interaction results of sentence 1, sentence 2, and sentence pair in the joint context, i.e. and This is passed to the third-layer interaction structure; and so on, repeated interactions are performed to generate the sentence 1 interaction representation at the depth layer, the sentence 2 interaction representation at the depth layer, and the sentence-to-context mapping representation at the depth layer; for the depth layer, the formula is as follows:

[0196]

[0197] in, Indicates use Based on the sentence pair joint context, the depth-1 layer mapping representation performs interaction operations on the interaction results of sentence 1 and sentence 2 at the depth-1 layer.

[0198]

[0199] in, Indicates use Based on the sentence pair joint context, the depth-1 layer mapping representation performs interaction operations on the interaction results of sentence 2 and sentence 1 at the depth-1 layer.

[0200]

[0201] in, Indicates the use of Dense depth Perform mapping operations on the depth-1 level mapping representation of the sentence-to-joint context.

[0202] For example, in PyTorch, the code implementation described above would be as follows:

[0203]

[0204]

[0205] Where query_inter is the interaction result of sentence 1 at the i-th level, response_inter is the interaction result of sentence 2 at the i-th level, context_dense is the i-th level mapping representation of the sentence to the joint context, and depth is the number of layers when the structure achieves the best results on the dataset. In this invention, depth is set to 4.

[0206] As attached Figure 3 As shown, the specific details of the training sentence pair semantic matching model in step S3 of this embodiment are as follows:

[0207] S301. Constructing the loss function: Use cross-entropy as the loss function;

[0208] For example, in PyTorch, the code implementation described above would be as follows:

[0209] #Calculate the error between the predicted value and the label using the cross-entropy loss function.

[0210] loss_fct = CrossEntropyLoss()

[0211] loss=loss_fct(logits.view(-1,self.num_labels),labels.view(-1))

[0212] Here, labels are the true labels, and logits are the correct probabilities of the model output.

[0213] S302. Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate being set to 2e-5, all other hyperparameters of AdamW were set to the default values ​​in PyTorch.

[0214] For example, in PyTorch, the code implementation described above would be as follows:

[0215] #Optimize sentence pair semantic matching model parameters using the BertAdam optimizer

[0216] optimizer=AdamW(optimizer_grouped_parameters,lr=2e-5)

[0217] Here, optimizer_grouped_parameters are the parameters to be optimized, which by default are all the parameters in the sentence pair semantic matching model.

[0218] When the sentence pair semantic matching model has not yet been trained, step S3 needs to be executed to train it in order to optimize the parameters of the sentence pair semantic matching model; when the sentence pair semantic matching model has been trained, step S205 can predict whether the semantics of the target sentence pair match.

[0219] Example 2:

[0220] As attached Figure 4 As shown, this embodiment provides a sentence pair semantic matching system, which includes a dataset acquisition unit for downloading publicly available sentence pair semantic matching datasets from the network;

[0221] The model building unit is used to build a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism.

[0222] The model training unit is used to construct the loss function and optimization function to complete the semantic matching judgment.

[0223] The model building unit in this embodiment includes,

[0224] The input module is used to preprocess the raw dataset to construct the input data;

[0225] The pre-trained embedding representation module is used to perform embedding encoding operations on the input data using a pre-trained language model, thereby obtaining the embedding representations of sentence pairs, joint context, sentence 1, and sentence 2 in the input data;

[0226] The Bilinear Triple-Attention interaction module is used to receive the sentence pair joint context embedding representation, sentence 1 embedding representation and sentence 2 embedding representation output by the pre-trained embedding representation module, and then repeatedly interact to generate the sentence 1 interaction result of the depth layer, the sentence 2 interaction result of the depth layer and the mapping result of the sentence pair joint context of the depth layer.

[0227] The feature aggregation module is used to perform feature filtering operations on the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the mapping representation of sentence pair joint context at the depth layer using adaptive average pooling operation, and then concatenates the three filtered feature representations to obtain the final aggregated feature representation.

[0228] The label prediction module is used to determine whether the semantics of the current sentence pair match based on the final aggregated feature representation.

[0229] As attached Figure 6As shown, the pre-trained embedding representation module performs embedding encoding on the input sentence pair joint context sequence, sentence 1 sequence, and sentence 2 sequence to obtain their respective embedding encoding representations, which are then passed to the BilinearTriple-Attention interaction module. The BilinearTriple-Attention interaction module is a stackable module with controllable layer count. It receives the feature representation output from the pre-trained embedding representation module and performs interaction operations among the sentence pair joint context embedding representation, sentence 1 embedding representation, and sentence 2 embedding representation through the BilinearTriple-Attention mechanism, passing the results to the feature aggregation module. The feature aggregation module performs average pooling on the interacted sentence 1 and sentence 2 representations and concatenates them with the sentence pair joint context representation, which has also undergone average pooling, to obtain the final feature representation, which is then passed to the label prediction module. The label prediction module maps the matching tensor of the sentence pair to a floating-point value within a specified interval; it uses this as the matching degree and compares it with a preset threshold. Based on the comparison result, it determines whether the sentence pair semantically matches.

[0230] The model training unit in this embodiment includes,

[0231] The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function.

[0232] The optimization function building module is used to train and adjust the parameters during model training to reduce prediction error.

[0233] Example 3:

[0234] This embodiment also provides an electronic device, including: a memory and a processor;

[0235] The memory stores the instructions executed by the computer.

[0236] The processor executes computer execution instructions stored in the memory, causing the processor to perform the sentence pair semantic matching method in any embodiment of the present invention.

[0237] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0238] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0239] Example 4:

[0240] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the sentence pair semantic matching method in any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0241] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0242] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0243] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0244] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sentence pair semantic matching method, characterized in that, The method is as follows: Obtain sentence pair semantic matching datasets: Download publicly available sentence pair semantic matching datasets from the internet; Constructing a sentence pair semantic matching model: Constructing a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism; Training the sentence pair semantic matching model: Train the sentence pair semantic matching model on the sentence pair semantic matching training dataset; The Bilinear Triple-Attention mechanism is an attention mechanism based on the Bilinear Product Attention algorithm. Its specific implementation process is as follows: (1) Calculate the relevance score between query, key, and context, as follows: ① By introducing a third-order matrix The three input tensors, query, key, and context, are multiplied together to achieve interactive computation among them, as shown in the following formula: ; Where query represents the information of sentence 1 in the sentence pair; key represents the information of sentence 2 in the sentence pair; context represents the joint context of the sentence pair, that is, the concatenation of sentence 1 and sentence 2; , and These represent the input tensors query, key, and context, respectively; d represents the input tensor. The element index in the text; D represents the input tensor. The number of elements in the input tensor; i and j represent the input tensor. and Element indices; i = 1, 2, ..., I kl j=1,2,…,J cl ;I kl and J cl Representing the input tensors respectively and The number of elements in; It is a shape of D D The third-order trainable weight tensor of D; , and Operators that perform dot product calculations between tensors; ② Obtain the correlation score of the three through normalization operation, and denote it as The formula is as follows: ; where \(i = 1, 2, \ldots, I\) kl ; \(i' = 1, 2, \ldots, I\) kl ; \(j = 1, 2, \ldots, J\) cl ; \(j' = 1, 2, \ldots, J\) cl ; (2) Feature aggregation operation: After mapping by the weight matrix, tensor multiplication is performed on the two input tensors, value and context, to complete the aggregation operation of the sentence to the joint context. The formula is as follows: ； Here, value represents the information of sentence 2 in the sentence pair, which is the same as the input tensor key; and These represent the input tensor value and context, respectively. This indicates that the weight matrix is ​​used to map pairs. and Perform aggregation operations; and Here is the weight matrix to be trained; i and j represent the input tensors. and Element indices; i = 1, 2, ..., I vl j=1,2,…,J cl ;I vl and J cl Representing the input tensors respectively and The number of elements in; (3) Dot product operation: Perform a dot product operation between the relevance score tensor obtained in step (1) and the feature aggregation representation obtained in step (2) to obtain the context-enhanced attention embedding representation, as shown in the following formula: ； Where i and j represent and Element indices; I and J represent input tensors and The length of the two dimensions; Bilinear Triple-Attention is a stackable structure that dynamically adjusts the number of layers based on the specific task, denoted as... ;depth indicates the specific number of layers set for a particular task; , , This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. , and same; The specific steps for constructing the sentence-pair semantic matching model are as follows: Input data: For each data point in the dataset, firstly, two sentences are concatenated, with sentence 1 and sentence 2 separated by the separator "SEP". This is taken as the joint context of the sentence pair, denoted as context. The sequence of sentence 1 is denoted as S1. All candidate sentence 2s are denoted as S2. The label of the data point is determined based on whether the semantics of the current sentence pair match. If it matches, it is denoted as 1; otherwise, it is denoted as 0. The three text sequences—the joint context sequence of sentence pairs, the sentence 1 sequence, and the candidate sentence 2 sequence—along with the label, constitute one input data point. Pre-trained embedding representations: Pre-trained language models are used to perform embedding encoding operations on the input data, resulting in embedding representations of sentence pairs, joint context, sentence 1, and sentence 2 from the input data, denoted as follows: , and Sentence pairs using the joint context sequence use a separate encoding module, while sentence 1 sequence and candidate sentence 2 sequence share a single encoding module; different levels of output are selected as their embedding representations based on different input objects; the formula is as follows: ； in, This indicates that the pre-trained language model BERT is used to embed and encode the joint context sequence of the input sentence; the subscript n indicates the number of network layers inside BERT used. This indicates the embedding of sentences into the context of a compound sentence; ； in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S1; the subscript t indicates the number of network layers inside BERT used. This indicates that sentence 1 is embedded. ； in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S2; the subscript t indicates the number of network layers inside BERT used. Sentence 2 embedding representation; Bilinear Triple-Attention Interaction: The input consists of sentence pair joint context embeddings, sentence 1 embeddings, and sentence 2 embeddings. The first-layer interaction structure interacts with the sentence 1 and sentence 2 embeddings based on the sentence pair joint context embeddings to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, which are then passed to the second-layer interaction structure. Simultaneously, a fully connected network is used to map the sentence pair joint context embeddings to obtain the first-layer mapped representation of the sentence pair joint context, which is then passed to the second-layer interaction structure. The second-layer interaction structure interacts with the first-layer sentence 1 and sentence 2 interaction results based on the first-layer mapped representation of the sentence pair joint context to obtain the second-layer sentence 1 interaction result and the second-layer sentence 2 interaction result, which are then passed to the third-layer interaction structure. Simultaneously, a fully connected network is used to map the first-layer mapped representation of the sentence pair joint context to obtain the second-layer mapped representation of the sentence pair joint context, which is then passed to the third-layer interaction structure. This process is repeated multiple times to generate the depth-level sentence 1 interaction result, the depth-level sentence 2 interaction result, and the depth-level mapped representation of the sentence pair joint context. Feature aggregation: Adaptive average pooling is used to filter the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the joint context mapping representation of sentence pairs at the depth layer. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, which is denoted as . ; Label prediction: Based on the final aggregated feature representation, determine whether the sentence pairs semantically match; specifically as follows: The final aggregated feature representation is processed by a fully connected network with dimension 1 and activation function sigmoid to obtain a matching degree value between [0,1], denoted as y_pred. The matching degree value is compared with a set threshold (0.5) to determine whether the semantics of the sentence pair match. That is, when y_pred≥0.5, the semantics of the sentence pair are predicted to match; otherwise, they do not match.

2. The sentence pair semantic matching method according to claim 1, characterized in that, Bi The linear Triple-Attention interaction is as follows: The first-layer interactive structure, based on the sentence pair joint context embedding representation, interacts with the sentence 1 embedding representation and the sentence 2 embedding representation to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, denoted as... and The sentence-to-joint context embedding representation is mapped through a fully connected network to obtain the first-level mapping representation of the sentence-to-joint context, denoted as . The formula is as follows: ； in, , and These represent the sentence pair joint context embedding representation, sentence 1 embedding representation, and sentence 2 embedding representation obtained after processing by the pre-trained embedding representation module, respectively. Indicates use Interactive operations are performed on sentence 1 embedding representation and sentence 2 embedding representation based on sentence pair joint context embedding representation; ； in, Indicates use Interactive operations are performed on the sentence pair joint context embedding representation and the sentence 1 embedding representation based on sentence pair joint context embedding representation; ； in, Indicates the use of a fully connected layer. Perform mapping operations on the sentence-pair joint context embedding representation; The interaction results of sentence 1 at the first level, the interaction results of sentence 2 at the first level, and the first-level mapping representation of the sentence pair in the joint context are represented as follows: , and The result is passed to the second-level interaction structure. The second-level interaction structure, based on the first-level mapping representation of the sentence-to-context joint context, interacts with the first-level sentence 1 interaction result and the first-level sentence 2 interaction result to obtain the second-level sentence 1 interaction result and the second-level sentence 2 interaction result, denoted as... and The first-level mapping representation of the sentence to the joint context is then mapped through a fully connected network to obtain the second-level mapping representation of the sentence to the joint context, denoted as . The formula is as follows: ； in, Indicates use Based on the first-level mapping representation of sentence pair joint context, interactive operations are performed on the interaction results of first-level sentence 1 and first-level sentence 2. ； in, Indicates use The first-level mapping representation based on sentence pair joint context performs interactive operations on the interaction results of sentence 2 and sentence 1 at the first level. ； in, Indicates use Perform mapping operations on the second-level mapping representation of the sentence to the joint context; The second-level interaction results of sentence 1, sentence 2, and sentence-to-context second-level mapping representation are represented as follows: , and This is passed to the third-layer interaction structure; and so on, repeated interactions are performed to generate the sentence 1 interaction representation, the sentence 2 interaction representation, and the sentence-to-context mapping representation at the depth layer; for the depth layer, the formula is as follows: ； in, Indicates use Based on the sentence pair joint context, the depth-1 layer mapping representation performs interaction operations on the interaction results of sentence 1 and sentence 2 at the depth-1 layer. ； in, Indicates use Based on the sentence pair joint context, the depth-1 layer mapping representation performs interaction operations on the interaction results of sentence 2 and sentence 1 at the depth-1 layer. ； in, Indicates use Perform mapping operations on the depth-1 level mapping representation of the sentence-to-joint context.

3. The sentence pair semantic matching method according to claim 2, characterized in that, The specific steps for training the sentence-to-semantic matching model are as follows: Constructing the loss function: Cross-entropy is used as the loss function; Constructing the optimization function: The AdamW optimization function is used as the optimization function. Except for its learning rate, which is set to 2e-5, all other hyperparameters of AdamW are set to the default values ​​in PyTorch. When the sentence pair semantic matching model has not yet been trained, further training is required to optimize the parameters of the sentence pair semantic matching model; when the sentence pair semantic matching model has been trained, it can predict whether the semantics of the target sentence pair match.

4. A sentence-pair semantic matching system, characterized in that, The system includes, The dataset acquisition unit is used to download publicly available sentence pair semantic matching datasets from the internet; The model building unit is used to build a sentence pair semantic matching model based on the Bilinear Triple-Attention mechanism. The model training unit is used to construct the loss function and optimization function to complete the semantic matching judgment; The Bilinear Triple-Attention mechanism is an attention mechanism based on the Bilinear Product Attention algorithm. Its specific implementation process is as follows: (1) Calculate the relevance score between query, key, and context, as follows: ① By introducing a third-order matrix The three input tensors, query, key, and context, are multiplied together to achieve interactive computation among them, as shown in the following formula: ; Where query represents the information of sentence 1 in the sentence pair; key represents the information of sentence 2 in the sentence pair; context represents the joint context of the sentence pair, that is, the concatenation of sentence 1 and sentence 2; , and These represent the input tensors query, key, and context, respectively; d represents the input tensor. The element index in the text; D represents the input tensor. The number of elements in the input tensor; i and j represent the input tensor. and Element indices; i = 1, 2, ..., I kl j=1,2,…,J cl ;I kl and J cl Representing the input tensors respectively and The number of elements in; It is a shape of D D The third-order trainable weight tensor of D; , and Operators that perform dot product calculations between tensors; ② Obtain the correlation score of the three through normalization operation, and denote it as The formula is as follows: ; where \(i = 1, 2, \ldots, I\) kl ; \(i' = 1, 2, \ldots, I\) kl ; \(j = 1, 2, \ldots, J\) cl ; \(j' = 1, 2, \ldots, J\) cl ; (2) Feature aggregation operation: After mapping by the weight matrix, tensor multiplication is performed on the two input tensors, value and context, to complete the aggregation operation of the sentence to the joint context. The formula is as follows: ； Here, value represents the information of sentence 2 in the sentence pair, which is the same as the input tensor key; and These represent the input tensor value and context, respectively. This indicates that the weight matrix is ​​used to map pairs. and Perform aggregation operations; and Here is the weight matrix to be trained; i and j represent the input tensors. and Element indices; i = 1, 2, ..., I vl j=1,2,…,J cl ;I vl and J cl Representing the input tensors respectively and The number of elements in; (3) Dot product operation: Perform a dot product operation between the relevance score tensor obtained in step (1) and the feature aggregation representation obtained in step (2) to obtain the context-enhanced attention embedding representation, as shown in the following formula: ； Where i and j represent and Element indices; I and J represent input tensors and The length of the two dimensions; Bilinear Triple-Attention is a stackable structure that dynamically adjusts the number of layers based on the specific task, denoted as... ;depth indicates the specific number of layers set for a particular task; , , This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. , and same; The specific steps for constructing the sentence-pair semantic matching model are as follows: Input data: For each data point in the dataset, firstly, two sentences are concatenated, with sentence 1 and sentence 2 separated by the separator "SEP". This is taken as the joint context of the sentence pair, denoted as context. The sequence of sentence 1 is denoted as S1. All candidate sentence 2s are denoted as S2. The label of the data point is determined based on whether the semantics of the current sentence pair match. If it matches, it is denoted as 1; otherwise, it is denoted as 0. The three text sequences—the joint context sequence of sentence pairs, the sentence 1 sequence, and the candidate sentence 2 sequence—along with the label, constitute one input data point. Pre-trained embedding representations: Pre-trained language models are used to perform embedding encoding operations on the input data, resulting in embedding representations of sentence pairs, joint context, sentence 1, and sentence 2 from the input data, denoted as follows: , and Sentence pairs using the joint context sequence use a separate encoding module, while sentence 1 sequence and candidate sentence 2 sequence share a single encoding module; different levels of output are selected as their embedding representations based on different input objects; the formula is as follows: ； in, This indicates that the pre-trained language model BERT is used to embed and encode the joint context sequence of the input sentence; the subscript n indicates the number of network layers inside BERT used. This indicates the embedding of sentences into the context of a compound sentence; ； in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S1; the subscript t indicates the number of network layers inside BERT used. This indicates that sentence 1 is embedded. ； in, This indicates that the pre-trained language model BERT is used to embed and encode the input sentence sequence S2; the subscript t indicates the number of network layers inside BERT used. Sentence 2 embedding representation; Bilinear Triple-Attention Interaction: The input consists of sentence pair joint context embeddings, sentence 1 embeddings, and sentence 2 embeddings. The first-layer interaction structure interacts with the sentence 1 and sentence 2 embeddings based on the sentence pair joint context embeddings to obtain the first-layer sentence 1 interaction result and the first-layer sentence 2 interaction result, which are then passed to the second-layer interaction structure. Simultaneously, a fully connected network is used to map the sentence pair joint context embeddings to obtain the first-layer mapped representation of the sentence pair joint context, which is then passed to the second-layer interaction structure. The second-layer interaction structure interacts with the first-layer sentence 1 and sentence 2 interaction results based on the first-layer mapped representation of the sentence pair joint context to obtain the second-layer sentence 1 interaction result and the second-layer sentence 2 interaction result, which are then passed to the third-layer interaction structure. Simultaneously, a fully connected network is used to map the first-layer mapped representation of the sentence pair joint context to obtain the second-layer mapped representation of the sentence pair joint context, which is then passed to the third-layer interaction structure. This process is repeated multiple times to generate the depth-level sentence 1 interaction result, the depth-level sentence 2 interaction result, and the depth-level mapped representation of the sentence pair joint context. Feature aggregation: Adaptive average pooling is used to filter the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the joint context mapping representation of sentence pairs at the depth layer. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, which is denoted as . ; Label prediction: Based on the final aggregated feature representation, determine whether the sentence pairs semantically match; specifically as follows: The final aggregated feature representation is processed by a fully connected network with dimension 1 and activation function sigmoid to obtain a matching degree value between [0,1], denoted as y_pred. The matching degree value is compared with a set threshold (0.5) to determine whether the semantics of the sentence pair match. That is, when y_pred≥0.5, the semantics of the sentence pair are predicted to match; otherwise, they do not match.

5. The sentence pair semantic matching system according to claim 4, characterized in that, The model building unit includes, The input module is used to preprocess the raw dataset to construct the input data; The pre-trained embedding representation module is used to perform embedding encoding operations on the input data using a pre-trained language model, thereby obtaining the sentence pair joint context, sentence 1, and sentence 2 embedding representations in the input data. The Bilinear Triple-Attention interaction module is used to receive the sentence pair joint context embedding representation, sentence 1 embedding representation and sentence 2 embedding representation output by the pre-trained embedding representation module, and then repeatedly interact to generate the sentence 1 interaction result of the depth layer, the sentence 2 interaction result of the depth layer and the mapping result of the sentence pair joint context of the depth layer. The feature aggregation module is used to perform feature filtering operations on the interaction results of sentence 1 at the depth layer, the interaction results of sentence 2 at the depth layer, and the mapping representation of sentence pair joint context at the depth layer using adaptive average pooling operation, and then concatenates the three filtered feature representations to obtain the final aggregated feature representation. The label prediction module is used to determine whether the semantics of the current sentence pair match based on the final aggregated feature representation.

6. The sentence pair semantic matching system according to claim 4 or 5, characterized in that, The model training unit includes, The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function. The optimization function building module is used to train and adjust the parameters during model training to reduce prediction error.

7. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the sentence pair semantic matching method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the sentence pair semantic matching method as described in any one of claims 1 to 3.

Images