A scheme intelligent recommendation method based on semantic matching

Abstract

The application relates to the technical field of natural language processing and information retrieval, and discloses a scheme intelligent recommendation method based on semantic matching. A general text embedding model is constructed, a self-attention mechanism of a transformer architecture is used to fully mine the context relationship of scheme text, and the scheme semantic similarity is calculated. The general text embedding model is subjected to transfer training, so that the accuracy of the text embedding model in scheme recommendation in a specific field is improved. The recommendation effects of different models are evaluated by using public data sets, and indexes such as the average value of the average precision, the average value of the average reciprocal rank and the normalized loss cumulative gain are calculated. The scheme recommendation effect of the model adopted by the application is better than that of a traditional model, and significant improvement is realized in all evaluation indexes.

Description

Technical Field

[0001] This invention relates to the fields of Natural Language Processing (NLP) and Information Retrieval, specifically to an intelligent recommendation method for schemes based on semantic matching. Background Technology

[0002] This section provides only background information relevant to this disclosure and is not necessarily prior art.

[0003] Information Explosion and Personalized Needs: With the rapid development of the internet, information has exploded. Faced with this massive amount of information, how users can quickly and accurately find content that interests them has become a pressing problem. Fortunately, the emergence of semantic analysis technology offers a new approach to solving this challenge. Semantic analysis technology can deeply understand the meaning and content behind text, providing more accurate semantic information for solution recommendation systems by extracting key information and quantifying the semantic relationships between words. This technological development makes intelligent solution recommendation based on semantic matching possible.

[0004] Traditional keyword-based recommendation methods often fail to fully understand the semantics of text, resulting in low accuracy of recommendation results. Meanwhile, general-purpose text embedding models often perform poorly in specific domains, failing to extract sufficient semantic information and leading to distorted personalized recommendations. Summary of the Invention

[0005] The technical problem to be solved by this invention is to propose an intelligent recommendation method based on semantic matching, which addresses the shortcomings of the prior art.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] A scheme intelligent recommendation method based on semantic matching, such as Figure 1 As shown, it includes the following steps:

[0008] Step 1: Construct a text embedding model. The model is based on the transformer architecture and includes two layers: a self-attention mechanism and a fully connected layer (Feed Forward Neural Network). The model is trained using a general dataset to learn the semantic relationships in the text.

[0009] Step 2: Evaluate the model obtained in Step 1 using a publicly available dataset. Experiments show that the text embedding model used in this patent performs better than other models in terms of recommendation performance.

[0010] Step 3: Construct transfer training data and a query scheme for the specific domain based on the domain-specific dataset.

[0011] Step 4: Use the transfer training data obtained in Step 3 to perform transfer training on the text embedding model, adjust the model parameters, generate word vectors for each word, and establish a word vector table corresponding to the word table.

[0012] Step 5 involves segmenting, preprocessing, and encoding the corpus in the domain-specific dataset to transform the scheme into a word representation.

[0013] Step 6: Using the text embedding model trained by transfer learning obtained in Step 4, convert each word into a corresponding word vector, generate sentence vectors or segment vectors according to the pooling strategy, and construct the vector representation library of the corpus.

[0014] Step 7: When the user inputs a query, the query is segmented, preprocessed, encoded, and converted into sentence vectors and segment vectors. The specific processing method is the same as in steps 5 and 6.

[0015] Step 8: Calculate the cosine similarity between the sentence or segment vector of the query solution and all vectors in the query solution library, and return a specified number of solutions and their similarity scores based on the cosine similarity scores.

[0016] Furthermore, step 1, which involves constructing and training the text embedding model, includes:

[0017] Step S111: Load the general dataset and input the tokenized list after word segmentation into the token embedding layer, segment embedding layer, and position embedding layer respectively to obtain the corresponding embedding representations. Specifically, in the token embedding layer, tags such as [CLS] and [SEP] are added to the word segmentation results to generate an attention mask, and all tags are converted into embedding vectors. The segment embedding layer returns two types of embedding vectors, which can be used to distinguish between two sentences input simultaneously. In the position embedding layer, the order information of word occurrences in the sentence is encoded, and position vectors are output.

[0018] Step S112: Accumulate the two embedding vectors and the position vector obtained in step S111, and use them as the output vector e of the self-attention module. i .

[0019] Step S113, input vector e i With three weight matrices W Q W K W VThe multiplication, where the weight matrix is ​​randomly initialized and optimized during training via backpropagation, generates the query vector Q. i Key vector K i Sum vector V i .

[0020] Q i =e i ·W Q ,

[0021] K i =e i ·W K ,

[0022] V i =e i ·W V ;

[0023] Step S114: Calculate the attention score for each input vector. ij and on score ij To prevent the dot product from becoming too large and causing the softmax function to enter the saturation region, the following formula is applied:

[0024]

[0025] Among them, score ij Let d be the attention score between the i-th word and the j-th key. k For the key vector K i Dimensions.

[0026] Step S115: Convert the score into attention weights using the softmax activation function. ij ,

[0027]

[0028] AttentionWeight ij This represents the attention received by the word vector at position j when generating the output at position i.

[0029] Step S116: Weighted summation using attention weights. ij For value vector V i Perform a weighted summation to obtain the output vector of the i-th word. i ,

[0030] Output i =AttentionWeight ij ·V j,

[0031] The formulas for the above process are summarized as follows:

[0032]

[0033] In step S117, during the process of converting words into vectors, a multi-head attention mechanism is used to simultaneously focus on different positions in the input sequence. The input sequence is divided into multiple heads, and each head performs the above operation independently. Finally, the outputs of all heads are multiplied by their corresponding weights to obtain the output vector Z. i .

[0034] Step S118, the obtained word output vector Z i The signal is fed into a fully connected layer, which consists of two layers: the first layer uses a ReLU activation function, and the second layer uses a linear activation function, expressed as follows:

[0035] FFN(Z i ) = max(0, Z i W1+b1)W2+b2,

[0036] W1 and W2 are weight matrices, and b1 and b2 are bias terms (the parameters are randomly initialized and optimized through backpropagation during training).

[0037] The ReLU activation function is a non-linear function that learns the non-linear relationships in the input vector, capturing dependencies and patterns in the input sequence. The second layer is a linear layer without an activation function. The second-layer linear activation function further transforms the features after the non-linear transformation, generating the final word vector output Z′. i .

[0038] Step S119: Perform the Masked Language Modeling (MLM) pre-training task, replacing some words in the input sequence with special tags [MASK], and performing the word vector Z′ obtained in S118 on a linear layer. i Linear transformation:

[0039] O i =WZ′ i +b,

[0040] Among them O i Let O be the output vector, W be the weight matrix, and b be the bias term. The weight matrix W and bias term b are randomly initialized before training begins. During training, the values ​​of the weight matrix W and bias term b are adjusted according to the loss function. The output vector O of the linear layer... i Each element corresponds to a word in the vocabulary, and the value of the element represents the probability that the model believes that word appears in the masked position.

[0041] Step S120: Normalization is performed using the softmax activation function to generate a probability distribution, where each element represents the probability that the corresponding word in the vocabulary appears in the masked position. This probability distribution is the model's final prediction.

[0042]

[0043] Where p i Let be the probability that the i-th word in the vocabulary appears in the masked position.

[0044] Step S121: Using the cross-entropy loss function as the loss function, calculate the difference between the probability distribution obtained in S120 and the true value of the masked words. During model training, the goal is to minimize the loss function value, adjusting parameters such as word vectors, weight matrices, and bias terms to make the output probability distribution approximate the true probability distribution. The cross-entropy loss function is used, and the formula is as follows:

[0045]

[0046] Where y i Yi is a one-hot encoding vector that indicates whether a word is a real target word, i.e., the real value of the masked word. Yi is 1 at the position of the real target word, and 0 otherwise.

[0047] Furthermore, step 2, which involves evaluating the model obtained in step 1, includes:

[0048] Step S211: Use a portion of the publicly available dataset to evaluate the experiment. The portion of the dataset consists of the first 15,000 data points from the publicly available dataset.

[0049] Step S212: Evaluate the S-BERT model, the Text2vec model, and the model constructed in this patent, and calculate the metrics.

[0050] Step S213: Statistically analyze the values ​​of different indicators for different models in the two sets of data and create a table.

[0051] Furthermore, the metrics calculated in step S212 include map@1, map@10, mrr@1, mrr@10, ndcg@1, and ndcg@10 metrics, and the calculation formula for each metric is as follows:

[0052] (1) MAP@k (MeanAverage Precision at k) refers to the average AP (Average Precision) of each query in the first k returned recommendation results;

[0053] Average precision (AP) is an evaluation metric for a single query, taking into account the ranking position of the query results among relevant documents. The AP formula for each query is as follows:

[0054]

[0055] In this context, P(n) represents the precision of the first n results, specifically the number of relevant solutions in the first n results divided by n; rel(n) is a binary function indicating whether the nth result is relevant, with 1 indicating relevant and 0 indicating irrelevant; and Recall@k measures the proportion of truly relevant results in the first k returned results.

[0056]

[0057] MAP@k is the average precision (AP) of all queries.

[0058] (2) MRR@k (Mean Reciprocal Rank at k) refers to the average of the inverse values ​​of the rank of the first relevant result among the first k results. This metric measures how quickly the model finds the first relevant result.

[0059] For each query, the formula for calculating the reciprocal rank (RR) is:

[0060]

[0061] Here, rank is the position of the first relevant result, starting from 1.

[0062] MRR@k is the average of the reciprocals RR of all queries.

[0063] (3) NDCG@k (NormalizedDiscounted Cumulative Gain at k) refers to the normalized value of the cumulative gain considering location discounts in the first k results. It measures the quality of the list of results returned by the model, taking into account both relevance and location.

[0064] 1) For a single query, the formula for calculating the Discounted Cumulative Gain (DCG) is:

[0065]

[0066] Here, rel(i) is the relevance score of the i-th result. The relevance score is usually a value between 0 and a positive number, such as 0, 1, 2, 3, etc.

[0067] 2) Calculate the ideal DCG, i.e., IDCG, to normalize the DCG. The DCG is calculated as the IDCG after sorting the results by relevance score. k .

[0068] 3) Calculate the normalized discount cumulative gain NDCG:

[0069]

[0070] NDCG@k is the average of the NDCG values ​​for all queries.

[0071] Furthermore, step 3, which involves constructing transfer learning training data samples and a query solution library for a specific domain based on a domain-specific dataset, includes:

[0072] Step S311: Construct transfer training data samples based on specific terms that users are interested in from the specific domain dataset, use the data to perform transfer training on the model, and then generate feature vectors that are more in line with the semantics of the specific domain.

[0073] Step S312: Generate a database of query solutions for specific domains based on the domains of interest.

[0074] Furthermore, step 4, which involves transfer training of the text embedding model, includes:

[0075] Step S411: Select the loss function and calculate the loss.

[0076] Step S412: Start training, load transfer training data samples to enable the model to learn how to distinguish between positive and negative samples, generate a loss function and aim to minimize the loss function, so that the embedding vectors generated by the text embedding model are more in line with specific semantics.

[0077] Furthermore, the loss function in step S411 is the sentence triple negative softmax contrastive loss function, and the calculation process is as follows:

[0078] Step S4111, calculate the similarity matrix:

[0079]

[0080] Among them, s ij It is a similarity matrix. It is the embedding of the i-th specific term, E j It is the embedding of the j-th sample. The sample can be a positive sample or a negative sample. T is a temperature parameter. The scaling is performed by the temperature parameter T to balance the influence of positive and negative samples.

[0081] Step S4112, apply log-softmax:

[0082]

[0083] Where, p j This is the normalized similarity matrix, representing the probability distribution of each sample being a positive sample. Assume there is one positive example and M negative samples for each specific term.

[0084] Step S4113, calculate the loss L, where N is the number of positive examples in a batch:

[0085]

[0086] The goal of the loss function is to maximize the probability of positive samples and minimize the probability of negative samples, using the method of minimizing the negative log-likelihood function.

[0087] Step S4114, Exchange Loss: Calculate the loss function by exchanging the roles of specific terms with positive examples, and use this as the exchange loss. This helps the model learn a more symmetrical similarity measure.

[0088] Furthermore, step 5 involves word segmentation, data preprocessing, and encoding of the corpus in the specific domain dataset, transforming the scheme into a word representation, including...

[0089] Step S511: Initialize the vocabulary by segmenting the input scheme into characters, counting the frequency of each character, and obtaining a primary character-level vocabulary.

[0090] Step S512: Calculate the frequency of adjacent character pairs and count the frequency of all adjacent character pairs. The adjacent character pairs are two adjacent characters or characters that have been merged.

[0091] Step S513, merge operation: select the adjacent character pairs that appear most frequently in the vocabulary, merge them into a new character or vocabulary unit, and update the frequency statistics of the vocabulary and adjacent character pairs.

[0092] Step S514: Repeat the merging operation until the preset vocabulary size is reached or there are no adjacent character pairs that can be merged.

[0093] Step S515: Obtain the final vocabulary list. After all the above operations are completed, all the sub-words or words obtained constitute the vocabulary list, which is the word segmentation result of the input scheme.

[0094] Step S516: Data preprocessing, filling short schemes according to the preset length of the text embedding model, truncating long schemes; replacing special characters and unknown words; removing irrelevant characters in the schemes.

[0095] Step S517, encoding, is to transform the input sequence into a data form that the model can accept. The encoding result includes three parts: input_ids, token_type_ids, and attention_mask. Among them, input_ids represents the ID of each word in the word table, token_type_ids is used to distinguish which sentence each word belongs to when two sentences are input at the same time, and attention_mask is used to determine which words need to be paid attention to when generating word vectors.

[0096] Furthermore, step 6 uses the transfer-trained text embedding model to generate sentence vectors and segment vectors, constructing a vector representation library for the query scheme library, including...

[0097] Step S611: Based on the word segmentation results, find the ID of each word in the word table and generate an attention_mask. Use the attention_mask to determine the word that needs to be considered.

[0098] Step S612: Extract the word vector of each word from the text embedding model based on the ID of each word.

[0099] Step S613: Generate sentence vectors or segment vectors according to the pooling strategy.

[0100] Step S614: Use a two-dimensional heatmap to visually demonstrate the changes in text vectors before and after transfer training.

[0101] Furthermore, the pooling strategy in step S613 is to select an average pooling strategy, obtain all words that need attention based on the attention_mask, and take the average of each element in the word vectors of all words that need attention as the sentence or segment vector.

[0102] Furthermore, step 8 includes,

[0103] Step S811: Normalize the vector, which includes the embedding vector of the query scheme and the vector generated by the scheme in the query scheme library.

[0104] Step S812: Calculate the vector dot product, which is the cosine similarity, using the following formula:

[0105]

[0106] Where A represents the embedding vector of the query scheme, and B represents the vector generated by a certain scheme in the query scheme library.

[0107] Step S813: Sort the results and return a specified number of schemes and their similarity scores in descending order of cosine similarity score.

[0108] By adopting the above technical solution, the present invention can produce the following technical effects:

[0109] This invention proposes an intelligent recommendation method based on semantic matching. First, a general text embedding model is constructed, and the self-attention mechanism of the transformer architecture is used to fully mine the contextual relationships of the text and calculate the semantic similarity of the proposed solutions. Furthermore, the general text embedding model is transferred to improve its recommendation accuracy in specific domains.

[0110] The advantages of this method are as follows: This invention uses a semantic matching-based intelligent recommendation method. When calculating scheme similarity, it considers textual semantic information. Compared with keyword-based intelligent recommendation methods, this method can provide more accurate and personalized recommendation services. The text embedding model of this invention uses a self-attention mechanism to fully mine textual context information, solving the problem of insufficient semantic information mining in traditional neural networks. This method supports transfer training, allowing the model to be trained using domain-specific text based on the user's specific domain, solving the problem of poor performance of general text embedding models in specific domains. It effectively mines and accurately represents the specific semantics of a professional domain, calculates the similarity between heterogeneous texts, and improves the rationality and accuracy of scheme similarity calculation in specific domains. Attached Figure Description

[0111] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0112] Figure 1 This is a flowchart illustrating the intelligent recommendation method for schemes based on semantic matching according to the present invention.

[0113] Figure 2 This is a visual diagram of sentence vectors for the semantic matching-based intelligent recommendation method of the present invention.

[0114] Figure 3 This diagram illustrates the difference in feature vectors between a positive example and a previous example before transfer training in the semantic matching-based intelligent recommendation method of this invention.

[0115] Figure 4 This diagram illustrates the difference between the feature vectors of the semantic matching-based intelligent recommendation method of this invention after transfer training and those of a positive example.

[0116] Figure 5 This diagram illustrates the difference in feature vectors between the original and negative examples in the semantic matching-based intelligent recommendation method of this invention.

[0117] Figure 6 This diagram illustrates the difference in feature vectors between the transferred training and negative examples of the semantic matching-based intelligent recommendation method of this invention. Detailed Implementation

[0118] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0119] This invention designs an intelligent recommendation method for schemes based on semantic matching, comprising the following steps:

[0120] Step 1: Construct a text embedding model based on the transformer architecture. The model includes a self-attention mechanism and a fully connected layer (FeedForward Neural Network). It is trained using a general dataset to learn semantic relationships within the text. The specific processing steps are as follows:

[0121] Step S111: Load the general dataset and input the tokenized list after word segmentation into the token embedding layer, segment embedding layer, and position embedding layer respectively to obtain the corresponding embedding representations. Specifically, in the token embedding layer, tags such as [CLS] and [SEP] are added to the word segmentation results to generate an attention mask, and all tags are converted into embedding vectors. The segment embedding layer returns two types of embedding vectors, which can be used to distinguish between two sentences input simultaneously. In the position embedding layer, the order information of word occurrences in the sentence is encoded, and position vectors are output.

[0122] Step S112: Accumulate the above two embedding vectors and position vectors as the input vector e of the self-attention module. i .

[0123] Step S113, input vector e i With three weight matrices W Q W K W V Multiply the matrices (the weight matrix is ​​randomly initialized and optimized during training via backpropagation) to generate the query vector Q. i Key vector K i Sum vector V i .

[0124] Q i =e i ·W Q ,

[0125] K i =e i ·W K ,

[0126] V i =ei ·W V ;

[0127] Step S114: Calculate the attention score for each vector. ij (Attention score between the i-th word and the j-th key) and score ij Normalization, i.e., dividing by (d k K represents i (The dimension of the key vector) is used to prevent the dot product result from being too large, causing the softmax function to enter the saturation region. The formula is as follows:

[0128]

[0129] Step S115: Use the softmax activation function to convert the score into attention weights. ij .

[0130] These weights represent the attention given to the word vector at position j when generating the output at position i.

[0131]

[0132] Step S116: Weighted summation. The value vector is weighted and summed using attention weights to obtain the output vector of the i-th word.

[0133] Output i =AttentionWeight ij ·V j ,

[0134] The formulas for the above process are summarized as follows:

[0135]

[0136] Step S117: In the process of converting words into vectors, a multi-head attention mechanism is used to simultaneously focus on different positions in the input sequence. The input sequence is divided into multiple heads, and each head performs the above operation independently. Finally, the outputs of all heads are multiplied by their corresponding weights to obtain the output vector Z. i .

[0137] Step S118: The obtained word vectors are fed into the next module, namely the fully connected layer. This module has two layers: the first layer uses the ReLU activation function, and the second layer uses a linear activation function, which can be represented as:

[0138] FFN(Z i ) = max(0, Z iW1+b1)W2+b2,

[0139] In the first layer, the ReLU function is a non-linear function that learns the non-linear relationships in the input vectors, better capturing more complex dependencies and patterns in the input sequence. The second layer is a linear layer without an activation function. In this layer, the features after the non-linear transformation are further transformed to generate the final word vector output Z′. i .

[0140] Step S119: Perform the Masked Language Modeling (MLM) pre-training task, replacing some words in the input sequence with the special label [MASK], and then linearly transforming the word vectors obtained in step 1.8 on a linear layer, as shown in the following formula:

[0141] O i =WZ′ i +b,

[0142] O i Let O be the output vector, W be the weight matrix, and b be the bias term. The weight matrix and bias term are randomly initialized before training begins, but during training, their values ​​are adjusted according to the loss function to minimize the loss function. The output vector O of the linear layer... i Each element corresponds to a word in the vocabulary, and the value of that element represents the probability that the model considers that word to appear in a masked position.

[0143] Step S120: Normalize using the softmax activation function to produce a probability distribution where each element represents the probability that the corresponding word in the vocabulary appears in the masked position. This probability distribution is the model's final prediction.

[0144]

[0145] Where p i It is the probability that the i-th word in the vocabulary appears in the masked position.

[0146] Step S121: Using the cross-entropy loss function as the loss function, calculate the difference between the probability distribution obtained in step 1.10 and the true values ​​of the masked words. During model training, the goal is to minimize the loss function value, adjusting parameters such as word vectors, weight matrices, and bias terms to make the output probability distribution increasingly closer to the true probability distribution. The cross-entropy loss function is used. The formula is as follows:

[0147]

[0148] Among them, y iIt is a one-hot encoded vector indicating whether it is the true target word (the true value of the masked word). At the position of the true target word, y... i The value is 1 if it is 1, otherwise it is 0.

[0149] Step 2: Evaluate different text embedding models using the T2ranking dataset. Experiments show that the text embedding model used in this patent performs better than other models in recommendation, as shown in Table 1. The specific processing procedure is as follows:

[0150] Step S211: Use the first 15,000 data points of the T2ranking dataset for experimental evaluation.

[0151] Step S212: Evaluate the S-BERT model, the Text2vec model, and the model constructed in this patent respectively, and return the metrics map@1, map@10, mrr@1, mrr@10, ndcg@1, and ndcg@10. The calculation formula for each metric is as follows:

[0152] (1)MAP@k(MeanAverage Precision at k)

[0153] MAP@k refers to the average precision of each query among the top k returned recommendations. Average Precision (AP) is an evaluation metric for a single query, taking into account the ranking of the query results among relevant documents. The formula for AP for each query is as follows:

[0154]

[0155] In the formula, P(n) is the precision of the first n results, which is equal to the number of relevant solutions in the first n results divided by n; rel(n) is a binary function indicating whether the nth result is relevant (1 indicates relevant, 0 indicates irrelevant). Recall@k measures the proportion of truly relevant results in the first k returned results.

[0156] MAP@k is the average AP for all queries.

[0157] (2)MRR@k(Mean Reciprocal Rank at k)

[0158] MRR@k refers to the average of the reciprocals of the rank of the first relevant result among the first k results. This metric measures how quickly the model finds the first relevant result.

[0159] For each query, the formula for calculating the reciprocal rank (RR) is:

[0160]

[0161] Here, rank is the position of the first relevant result (ranking starts from 1).

[0162] MRR@k is the average value of all queries.

[0163] (3)NDCG@k(NormalizedDiscounted Cumulative Gain at k)

[0164] NDCG@k is the normalized value of the cumulative gain, taking into account location discounts, over the first k results. It measures the quality of the list of results returned by the model, taking into account both relevance and location.

[0165] 1) For a single query, the formula for calculating DCG (Discounted Cumulative Gain) is:

[0166]

[0167] Where rel(i) is the relevance score of the i-th result (usually a value between 0 and a positive number, such as 0, 1, 2, 3, etc.).

[0168] 2) Calculate the ideal DCG (IDCG) to normalize the DCG. Sort the results by relevance score, and then calculate the DCG as the IDCG. k .

[0169] 3) Calculate NDCG:

[0170]

[0171] NDCG@k is the average of the NDCG values ​​for all queries.

[0172] Step S213: Statistically analyze the values ​​of different indicators for different models in the two sets of data and create a table. The results are shown in Table 1.

[0173] Table 1. Recommendation performance of different models on the T2ranking dataset.

[0174]

[0175] Step 3: Construct transfer learning training data samples and a query solution library for the specific domain based on the domain-specific dataset. The specific processing procedure is as follows:

[0176] Step S311: Construct transfer training data samples based on specific terms that users are interested in from the specific domain dataset, use the data to perform transfer training on the model, and then generate feature vectors that are more in line with the semantics of the specific domain.

[0177] Step S312: Generate a database of query solutions for specific domains based on the domains of interest.

[0178] Step 4: Use transfer training data to perform transfer training on the text embedding model, adjust the relevant model parameters, and generate word vectors for each lexical unit, establishing a word vector table corresponding to the lexical unit table. The specific processing procedure is as follows:

[0179] Step S411: Select the loss function. This patent uses the sentence triple negative softmax contrastive loss function. The calculation process is as follows.

[0180] 1) Calculate the similarity matrix:

[0181]

[0182] Among them, s ij It is a similarity matrix. It is the embedding of the i-th specific term, E j It is the embedding of the j-th sample (which can be either a positive or negative sample). T is a temperature parameter, which is used for scaling to balance the influence of positive and negative samples.

[0183] 2) Applying log-softmax:

[0184]

[0185] Where, p j This is the normalized similarity matrix, representing the probability distribution of each sample being a positive sample. Assume there is one positive example and M negative samples for each specific term.

[0186] 3) Calculate the loss (N is the number of positive examples in a batch):

[0187]

[0188] The goal of the loss function is to maximize the probability of positive samples and minimize the probability of negative samples, using the method of minimizing the negative log-likelihood function.

[0189] 4) Exchange losses:

[0190] To improve model performance, a loss function can be calculated by swapping the roles of specific terms with those of positive examples, and this loss function can be used as the swap loss. This helps the model learn a more symmetrical similarity metric.

[0191] Step S412: Start training, load transfer training data samples, enable the model to learn how to distinguish between positive and negative samples, generate a loss function and aim to minimize the loss function, so that the embedding vectors generated by the text embedding model are more in line with the specific semantics.

[0192] Step 5: Perform word segmentation, data preprocessing, and encoding on the corpus in the specific domain dataset to transform the scheme into a word representation. The specific processing procedure is as follows:

[0193] Step S511: Initialize the vocabulary. First, the input scheme is divided into individual characters, and then the frequency of each character is counted to form a basic character-level vocabulary.

[0194] Step S512: Calculate the frequency of adjacent character pairs. Count the frequency of all adjacent character pairs. An adjacent character pair may be two adjacent characters or a pair of characters that have already been merged.

[0195] Step S513: Merge operation. Select the most frequent adjacent character pairs in the vocabulary, merge them into a new character or vocabulary unit, and then update the vocabulary and the frequency statistics of adjacent character pairs.

[0196] Step S514: Repeat the merging operation until the preset vocabulary size is reached or there are no adjacent character pairs that can be merged.

[0197] Step S515: Obtain the final vocabulary list. After all the above operations are completed, all the sub-words or words obtained constitute the vocabulary list, which is the word segmentation result of the input scheme.

[0198] Step S516: Data preprocessing mainly includes processes such as padding, truncation, character replacement, and text cleaning. Padding and truncation involve padding short schemes and truncating long schemes according to the preset length of the text embedding model. Character replacement includes replacing special characters and unknown words. Text cleaning refers to removing irrelevant characters from the scheme.

[0199] Step S517: Encoding transforms the input sequence into a data format acceptable to the model. The encoding result includes three parts: input_ids, token_type_ids, and attention_mask. Input_ids represents the ID of each word in the word table. Token_type_ids is used to distinguish which sentence each word belongs to when two sentences are input simultaneously. Attention_mask determines which words need attention when generating word vectors.

[0200] Step 6: Using the transferred learning-trained text embedding model, convert each word into a corresponding word vector, then select a pooling strategy to generate sentence or segment vectors, constructing a vector representation library for the query scheme database. The specific processing procedure is as follows:

[0201] Step S611: Based on the word segmentation results, look up the ID of each word in the word table and generate attention_mask (attention_mask is used to distinguish which word elements need to be considered and which word elements do not need to be considered).

[0202] Step S612: Extract the word vector of each word from the text embedding model based on the ID of each word.

[0203] Step S613: Select a pooling strategy and generate sentence or segment vectors. The text embedding model of this patent selects the average pooling strategy. First, it obtains all the words that need attention based on the attention_mask. Then, it averages the vectors of each element in the word vectors of all the words that need attention and uses the resulting vector as the sentence or segment vector.

[0204] Step S614: Use a two-dimensional heatmap to visually demonstrate the changes in text vectors before and after transfer training, such as... Figures 2-6 The 726-dimensional vector is reshaped into a two-dimensional matrix, with the horizontal and vertical axes representing the column and row indices after reshaping, respectively. Figure 2 This is an example of a two-dimensional heatmap generated from a sentence; Figure 3 and Figure 4 This is a visual representation of the difference between the embedding vectors generated by specific terms and those generated by positive examples before and after domain-adaptive training. The graph clearly shows that before domain-adaptive training, the vector difference heatmap is darker and each element has a larger value, indicating a significant difference between the vectors generated by specific terms and those generated by positive examples. However, after domain-adaptive training, the vector difference heatmap becomes lighter, each element's value decreases, and the difference approaches zero. Therefore, training makes specific terms more similar to positive examples. Figure 5 and Figure 6 This is a visual representation of the difference between the embedding vectors generated by a specific term ("panda") and its negative counterpart ("panda") before and after domain-adaptive training. As can be seen from the graph, before domain-adaptive training, the heatmap of the vector difference is lighter in color and has smaller element values, indicating a smaller difference between the vectors generated by the specific term and those generated by the negative examples. However, after domain-adaptive training, the heatmap of the vector difference becomes darker in color and has larger element values, indicating that training has made the specific term dissimilar to the positive examples.

[0205] Step 7: When the user inputs a query, the query is segmented, preprocessed, encoded, and converted into sentence and segment vectors. The specific processing method is the same as in steps 5 and 6.

[0206] Step 8: Calculate the cosine similarity between the sentence or segment vector of the query solution and all vectors in the query solution database. Sort the results and return a specified number of solutions and their similarity scores in descending order of cosine similarity score. The specific processing procedure is as follows:

[0207] Step S811: Normalize the vector, which includes the embedding vector of the query scheme and the vector generated by the scheme in the query scheme library.

[0208] Step S812: Calculate the vector dot product, which is the cosine similarity, using the following formula:

[0209]

[0210] Where A represents the embedding vector of the query scheme, and B represents the vector generated by a certain scheme in the query scheme library.

[0211] Step S813: Sort the results and return a specified number of schemes and their similarity scores in descending order of cosine similarity score.

[0212] This invention provides an idea and method for intelligent recommendation of solutions based on semantic matching. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for intelligent recommendation of solutions based on semantic matching, comprising the following steps: Step 1: Construct a text embedding model. The model is based on the transformer architecture and includes two layers: a self-attention mechanism and a fully connected layer. The model is trained using a general dataset to learn the semantic relationships in the text. Step 2: Evaluate the model obtained in Step 1 using a public dataset; Step 3: Construct transfer training data and a domain-specific query scheme based on the domain-specific dataset; Step 4: Use the transfer training data obtained in Step 3 to perform transfer training on the text embedding model, adjust the model parameters, generate word vectors for each word element, and establish a word vector table corresponding to the word element table. Step 4, which involves transfer training of the text embedding model, includes... Step S411: Select the loss function and calculate the loss; Step S412: Start training, load transfer training data samples, enable the model to learn how to distinguish between positive and negative samples, generate a loss function and aim to minimize the loss function, so that the embedding vectors generated by the text embedding model are more in line with the specific semantics. The loss function in step S411 is the sentence triple negative softmax contrastive loss function, and the calculation process is as follows. Step S4111, calculate the similarity matrix: in, It is the embedding of the i-th specific term. It is the embedding of the j-th sample, and T is the temperature parameter. Scaling is performed using the temperature parameter T to balance the influence of positive and negative samples. Step S4112, apply log-softmax for normalization: in, It is the normalized version , representing the probability distribution of each sample as a positive sample, assuming that there is one positive example and M negative samples for each specific term; Step S4113, calculate the loss N represents the number of positive examples in a batch. The goal of the loss function is to maximize the probability of positive samples and minimize the probability of negative samples, using the method of minimizing the negative log-likelihood function; Step S4114, Exchange Loss: Calculate the loss function by exchanging the roles of specific terms with positive examples, and use it as the exchange loss; Step 5: Perform word segmentation, data preprocessing, and encoding on the corpus in the domain-specific dataset to transform the scheme into word representation; Step 6: Using the text embedding model trained by transfer learning obtained in Step 4, convert each word into a corresponding word vector, generate sentence or segment vectors according to the pooling strategy, and construct the vector representation library of the corpus. Step 7: When the user inputs a query, the query is segmented, preprocessed, encoded, and converted into sentence vectors and segment vectors. Step 8: Calculate the cosine similarity between the sentence or segment vector of the query solution and all vectors in the query solution library, and return a specified number of solutions and their similarity scores based on the cosine similarity scores.

2. The intelligent recommendation method for schemes based on semantic matching according to claim 1, characterized in that, Step 1, which involves constructing and training a text embedding model, includes: Step S111: Load the general dataset, input the token list after word segmentation into the token embedding layer, fragment embedding layer and position embedding layer respectively to obtain the corresponding embedding representations; in the token embedding layer, tokens are added to the word segmentation results to generate attention_mask and all tokens are converted into embedding vectors; in the fragment embedding layer, two types of embedding vectors are returned; in the position embedding layer, the order information of word occurrence in the sentence is encoded and the position vector is output. Step S112: Accumulate the two embedding vectors and the position vector obtained in step S111, and use them as the output vector of the self-attention module. ; Step S113, input vector With three weight matrices , , Multiplication is performed using a weight matrix that is randomly initialized and optimized through backpropagation during training to generate a query vector. Key vector Sum value vector ; Step S114: Calculate the attention score for each input vector. The formula is as follows: in The attention score between the i-th word and the j-th key. Key vector The dimension; Step S115: The score is converted into attention weights using the softmax activation function. , in This represents the attention received by the word vector at position j when generating the output at position i. Step S116, weighted summation, using attention weights value vector We perform a weighted summation to obtain the output vector of the i-th word. ; In step S117, during the process of converting words into vectors, a multi-head attention mechanism is used to simultaneously focus on different positions in the input sequence. The input sequence is divided into multiple heads, and each head performs the above operation independently. Finally, the outputs of all heads are multiplied by their corresponding weights to obtain the output vector. ; Step S118, output the obtained word vector The signal is fed into a fully connected layer, which consists of two layers: the first layer uses a ReLU activation function, and the second layer uses a linear activation function, expressed as follows: in and This is the weight matrix. and As a bias term, the weight matrix and and bias terms and All are randomly initialized before training begins and optimized through backpropagation during training. The ReLU activation function is a non-linear function that learns the non-linear relationships in the input vector, capturing dependencies and patterns in the input sequence. The second-layer linear activation function further transforms the features after the non-linear transformation, generating the final word vector output. ; Step S119: Perform the Masked Language Modeling pre-training task, replacing some words in the input sequence with the special tag [MASK], and then performing the word vectors obtained in S118 on a linear layer. Linear transformation: in Let W be the output vector, W be the weight matrix, and b be the bias term. The weight matrix W and bias term b are randomly initialized before training begins. During training, the values ​​of the weight matrix W and bias term b are adjusted according to the loss function. The output vector of the linear layer... Each element corresponds to a word in the vocabulary, and the value of the element represents the probability that the model believes that the word appears in the masked position; Step 120: Normalize using the softmax activation function to generate a probability distribution, where each element represents the probability that a corresponding word in the vocabulary appears in the masked position. This probability distribution is the model's final prediction. in This represents the probability that the i-th word in the vocabulary appears in the masked position. Step S121: Using the cross-entropy loss function as the loss function, calculate the difference between the probability distribution obtained in S120 and the true value of the masked word. During model training, adjust the parameters with the goal of minimizing the loss function value so that the output probability distribution is close to the true probability distribution.

3. The intelligent recommendation method for schemes based on semantic matching according to claim 1, characterized in that, Step 2, which involves evaluating the model obtained in Step 1, includes: Step S211: Use a portion of the publicly available dataset to evaluate the experiment; Step S212: Evaluate the proposed model and the comparison model respectively, and calculate the indicators; Step S213: Statistically analyze the values ​​of different indicators for different models in the two sets of data and create a table.

4. The intelligent recommendation method for schemes based on semantic matching according to claim 3, characterized in that, The metrics calculated in step S212 include map@1, map@10, mrr@1, mrr@10, ndcg@1, and ndcg@10. The calculation formula for each metric is as follows: (1) MAP@k refers to the average precision AP of each query in the first k returned recommendation results; Average precision (AP) is an evaluation metric for a single query, taking into account the ranking position of the query results among relevant documents. The AP formula for each query is as follows: Among them, P The precision of the first n results is, specifically... The number of relevant solutions divided by n; Indicates whether the nth result is relevant, 1 indicates relevant, and 0 indicates irrelevant; where Recall It measures the proportion of truly relevant results among the first k returned results. MAP@k is the average precision AP of all queries; (2) MRR@k refers to the average of the inverse values ​​corresponding to the ranking of the first relevant result among the first k results; For each query, the formula for calculating the reciprocal rank (RR) is: Here, rank is the position of the first relevant result, starting from 1; MRR@k is the average of the reciprocals RR of all queries; (3) NDCG@k refers to the normalized value of the cumulative gain considering the location discount in the first k results; 1) For a single query, the formula for calculating the discount cumulative gain (DCG) is: in, It is the relevance score of the i-th result, which is usually a value between 0 and some positive number; 2) Calculate the ideal DCG, i.e., IDCG, to normalize the DCG. The DCG is calculated after sorting the results by correlation score. ; 3) Calculate the normalized discount cumulative gain NDCG: NDCG@k is the average of the NDCG values ​​for all queries.

5. The intelligent recommendation method for schemes based on semantic matching according to claim 1, characterized in that, Step 5 involves word segmentation, data preprocessing, and encoding of the corpus in the domain-specific dataset, transforming the scheme into a word representation, including... Step S511: Initialize the vocabulary by segmenting the input scheme into characters, counting the frequency of each character, and obtaining a primary character-level vocabulary. Step S512: Calculate the frequency of adjacent character pairs and count the frequency of all adjacent character pairs. The adjacent character pairs are two adjacent characters or characters that have been merged. Step S513, merge operation: select the adjacent character pairs that appear most frequently in the vocabulary, merge them into a new character or vocabulary unit, and update the frequency statistics of the vocabulary and adjacent character pairs. Step S514: Repeat the merging operation until the preset vocabulary size is reached or there are no adjacent character pairs that can be merged. Step S515: Obtain the final vocabulary list. After all the above operations are completed, all the sub-words or words obtained constitute the vocabulary list, and the sub-words or words are the word segmentation results of the input scheme. Step S516: Data preprocessing; fill short schemes according to the preset length of the text embedding model; truncate long schemes; replace special characters and unknown words; remove irrelevant characters from the schemes; Step S517, encoding, is to transform the input sequence into a data form that the model can accept. The encoding result includes three parts: input_ids, token_type_ids, and attention_mask. Among them, input_ids represents the ID of each word in the word table, token_type_ids is used to distinguish which sentence each word belongs to when two sentences are input at the same time, and attention_mask is used to determine which words need to be paid attention to when generating word vectors.

6. The intelligent recommendation method for schemes based on semantic matching according to claim 1, characterized in that, Step 6 uses the transfer-trained text embedding model to generate sentence vectors and segment vectors, constructing a vector representation library for the query scheme database, including: Step S611: Based on the word segmentation results, find the ID of each word in the word table and generate an attention_mask to determine the word that needs to be considered. Step S612: Extract the word vector of each word from the text embedding model based on the ID of each word; Step S613: Generate sentence or segment vectors according to the pooling strategy; Step S614: Use a two-dimensional heatmap to visually demonstrate the changes in text vectors before and after transfer training.

7. The intelligent recommendation method for schemes based on semantic matching according to claim 6, characterized in that, The pooling strategy in step S613 is to select the average pooling strategy. Based on the attention_mask, all words that need attention are obtained. The average of each element in the word vectors of all words that need attention is used as the sentence or segment vector.

8. The intelligent recommendation method for schemes based on semantic matching according to claim 1, characterized in that, Step 8 includes, Step S811: Normalize the vector, which includes the embedding vector of the query scheme and the vector generated by the scheme in the query scheme library; Step S812: Calculate the vector dot product, which is the cosine similarity, using the following formula: Where A represents the embedding vector of the query scheme, and B represents the vector generated by a certain scheme in the query scheme library; Step S813: Sort the results and return a specified number of schemes and their similarity scores in descending order of cosine similarity score.

Images