A method for constructing a scoring function for semantic retrieval and related devices

Abstract

Embodiments of the present application provide a scoring function construction method for semantic retrieval and related devices. The method comprises: obtaining a text library text and a query text; performing word segmentation on the text library text and the query text to obtain a first term set and a second term set, and obtaining a weight value of each term in the first term set; inputting the first term set and the second term set into a pre-trained BERT model to obtain a first term vector set and a second term vector set; determining a retrieval score according to a third term set, the weight value, the first term vector set and the second term vector set; and determining a scoring function according to a first compressed vector, a second compressed vector and the retrieval score. In this way, a model modeling and training process for semantic retrieval is formulated, so that the retrieval effect of a multi-document semantic retrieval task can be significantly improved, mainly in terms of retrieval efficiency and retrieval accuracy.

Description

Technical Field

[0001] This application relates to the fields of natural language processing and information retrieval, and in particular to a method for constructing a scoring function for semantic retrieval and related apparatus. Background Technology

[0002] Search engines are the most common way for people to obtain information. Users enter their search terms into the search engine, and then view and verify the search results one by one to determine whether they are the information they need.

[0003] Currently, mainstream search engines use information retrieval methods represented by BM25, which determine the relevance between a query and a document by calculating the lexical overlap between the user's query and the online text. However, this keyword-matching retrieval method often encounters situations where the query keywords fail to match the user's actual needs, thus requiring manual attempts with multiple keywords to increase the likelihood of directly retrieving the desired information. This process wastes a significant amount of the user's time. Faced with the vast and ever-increasing volume of internet data, the contradiction between accurately retrieving the desired information and retrieval efficiency is becoming increasingly prominent. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method and related apparatus for constructing a scoring function for semantic retrieval, aiming to realize the construction of a scoring function for semantic retrieval.

[0005] Firstly, a method for constructing a scoring function for semantic retrieval includes:

[0006] Retrieve text from the text library and query text;

[0007] The text in the text library and the query text are segmented to obtain a first word set and a second word set. The weight value of each word in the first word set is obtained. The first word set includes multiple text library words, and the second word set includes multiple query words.

[0008] The first word set and the second word set are input into a pre-trained BERT model to obtain a first word vector set and a second word vector set. The pre-trained BERT model is used to compress the semantic information of words in the text.

[0009] The retrieval score is determined based on the third word set, the weight value, the first word vector set, and the second word vector set, wherein the third word set is the set of texts in the first word set that have overlapping words with the second word set;

[0010] A scoring function is determined based on a first compression vector, a second compression vector, and the retrieval score. The first compression vector represents the overall semantics of the first word set, and the second compression vector represents the overall semantics of the second word set.

[0011] Optionally, constructing the first trained model includes the following steps:

[0012] Obtain the input sequence;

[0013] The semantic vector matrix is ​​obtained by concatenating the first CLS vector and the first character vector. The first CLS vector is the CLS vector of the last layer in the last eight layers of the pre-trained BERT model, and the first character vector is all the vectors of the last layer in the first four layers of the pre-trained BERT model that do not have CLS characters.

[0014] An encoding layer is added to the semantic vector matrix. The CLS compression vector is calculated based on the semantic vector matrix and the self-attention of the encoding layer to obtain the BERT pre-trained model. The CLS compression vector is a vector representing the semantics of the entire input sequence, obtained by interacting the semantics of the first CLS vector with the first character vector.

[0015] Large-scale pre-training was performed using a contrastive learning task and corpus until training was complete, and the trained model was used as the first training model.

[0016] Following the large-scale pre-training using the contrastive learning task and predictions, the method also includes:

[0017] Updating the parameters in the model using virtual adversarial training, wherein the virtual adversarial training includes:

[0018] The input sequence is trained using an MLM task to obtain a first probability distribution, which is the predicted probability distribution of the characters replaced by the mask in the vocabulary.

[0019] Obtain a first perturbation vector, add the first perturbation vector to the input sequence, and obtain a second input sequence;

[0020] The second input sequence is input again to train the MLM task and obtain the second probability distribution;

[0021] The first divergence Loss and the first gradient of the first perturbation vector are determined based on the first probability distribution and the second probability distribution.

[0022] The first perturbation vector is updated and regularized according to the first gradient to obtain the second perturbation vector;

[0023] The second perturbation vector is added to the second input sequence to obtain the third input sequence;

[0024] The third input sequence is input again to train the MLM task and obtain the third probability distribution;

[0025] Calculate the second divergence Loss of the actual output probability distribution and the third probability distribution, and add the cross-entropy Loss to the second divergence Loss. The cross-entropy Loss is the cross-entropy Loss of the actual output probability distribution and the one-hot label vector of the character replaced by the MASK.

[0026] The cross-entropy Loss and the second divergence Loss are weighted to obtain the overall Loss of the final model;

[0027] The gradient is calculated based on the overall loss, and the parameters in the BERT pre-trained model are updated.

[0028] Optionally, the large-scale pre-training using the contrastive learning task and corpus includes:

[0029] The aforementioned corpora are divided into positively correlated data and negatively correlated data;

[0030] The positive and negative correlation data are input into the BERT pre-trained model in batches for training.

[0031] Optionally, obtaining the input sequence includes:

[0032] High-frequency words are obtained and added to the pre-trained BERT model vocab.txt;

[0033] The input statement is segmented into words to obtain a segmentation result set, which includes several segmented words;

[0034] Iterate through the segmented words in the result set, and retain the words that match the word list.

[0035] In response to a mismatch between the segmented word and the vocabulary, the segmentation function is applied to segment the word, and the original segmented word in the segmentation result set is replaced and updated. The step of "traversing the segmented words in the segmentation result set" is then executed.

[0036] Several successfully matched word segmentation results are concatenated in an ordered manner to generate the input sequence.

[0037] Optionally, dividing the plurality of corpora into positively correlated data and negatively correlated data includes:

[0038] The text information of the first text segment is obtained as the first sample. The second text segment in the same article that does not intersect with the first text segment is obtained. The text information of the second text segment is labeled as positive correlation data of the first sample. The second text segment contains the same amount of text information as the first text segment.

[0039] Alternatively, obtain the similarity of the first sample, search the text library for a second sample whose similarity to the first sample is not less than a threshold according to the BM25 algorithm, and label the second sample as positively correlated data of the first sample.

[0040] Optionally, determining the retrieval score based on the third word set, the weight value, the first word vector set, and the second word vector set includes:

[0041] For each query word in the third word set, match it with all corresponding identical words in the text library;

[0042] Using the first word vector set and the second word vector set output by the pre-trained BERT model adapted to the retrieval task, the dot product similarity between each query word and its corresponding identical word is calculated respectively.

[0043] The highest similarity score is obtained as the retrieval score, which is the similarity score of the token with the highest similarity among all identical words.

[0044] Secondly, embodiments of this application provide a semantic retrieval scoring function construction apparatus, the apparatus comprising:

[0045] The text acquisition module is used to acquire text from the text library and query text.

[0046] The weight value acquisition module is used to segment the text in the text library and the query text to obtain a first word set and a second word set, and to acquire the weight value of each word in the first word set. The first word set includes multiple text library words, and the second word set includes multiple query words.

[0047] The word vector set acquisition module is used to input the first word set and the second word set into the pre-trained BERT model to obtain the first word vector set and the second word vector set. The pre-trained BERT model is used to compress the semantic information of words in the text.

[0048] The retrieval score determination module is used to determine the retrieval score based on the third word set, the weight value, the first word vector set, and the second word vector set, wherein the third word set is the set of texts in the first word set that have overlapping words with the second word set;

[0049] The scoring function determination module is used to determine a scoring function based on a first compression vector, a second compression vector, and the retrieval score. The first compression vector represents the overall semantics of the first word set, and the second compression vector represents the overall semantics of the second word set.

[0050] Thirdly, embodiments of this application provide a device including a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code to cause the device to perform the semantic retrieval scoring function construction method described in any of the first aspects above.

[0051] Fourthly, embodiments of this application provide a computer storage medium storing code, wherein when the code is executed, a device running the code implements the semantic retrieval scoring function construction method described in any of the first aspects above.

[0052] This application provides a method and related apparatus for constructing a scoring function for semantic retrieval. When executing the method, a text library and a query text are acquired; the text library and the query text are segmented into words to obtain a first word set and a second word set; the weight value of each word in the first word set is obtained; the first word set includes multiple text library words, and the second word set includes multiple query words; the first word set and the second word set are input into a pre-trained BERT model to obtain a first word vector set and a second word vector set; the pre-trained BERT model is used to compress the semantic information of words in the text; a retrieval score is determined based on a third word set, the weight values, the first word vector set, and the second word vector set; the third word set is the set of texts in the first word set that overlap with the second word set; and a scoring function is determined based on a first compression vector, the second compression vector, and the retrieval score. Therefore, the semantic information of words in the text is compressed into the [CLS] vector, thereby obtaining a better overall semantic representation. In addition, the use of virtual adversarial training enhances the robustness of the language model. The TF-IDF value of words is used as the weight of the word semantic vector, thereby reducing the role of common words in the semantic retrieval stage and highlighting the role of keywords, thus improving the quality of retrieval results. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of a method for constructing a scoring function for semantic retrieval provided in the embodiments of this application;

[0055] Figure 2 A flowchart of a method for constructing a scoring function for semantic retrieval provided in the embodiments of this application;

[0056] Figure 3 The BERT model adapted for retrieval tasks provided in the embodiments of this application;

[0057] Figure 4 A schematic diagram illustrating a large-scale pre-training method combining contrastive learning, provided as an embodiment of this application;

[0058] Figure 5 A flowchart illustrating a method provided in an embodiment of this application;

[0059] Figure 6 A detailed model framework diagram of a high-efficiency semantic retrieval model combining word weights provided in this application embodiment;

[0060] Figure 7 This is a schematic diagram of a semantic retrieval scoring function construction device provided in an embodiment of this application. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0062] As mentioned earlier, the mainstream search engines currently use information retrieval methods represented by BM25. However, the inventors have found through research that the main problems faced by keyword matching-based retrieval methods are as follows: (1) The words in the query and the words in the document have the same semantics, but the expressions are inconsistent, for example,

spending money

consuming

[0063] In existing technologies, BM25 keyword matching-based retrieval methods rely on lexical overlap between queries and documents to score their relevance. While BM25 keyword matching is currently the mainstream method in search engines, boasting high retrieval efficiency and good interpretability and controllability—allowing users to easily improve search results by adding keywords when the desired information is missing—it cannot address the issues of semantically consistent words with inconsistent expressions, or vice versa. Even with subsequent research using techniques like lexical normalization, N-gram matching, and query expansion to alleviate these problems, these improvements are still fundamentally based on the bag-of-words assumption and do not address the root causes of these issues. Therefore, these models have very limited capabilities for modeling natural language.

[0064] Deep Language Model-Based Retrieval: Deep language models (DLMs), represented by BERT, have had a significant impact on the development of information retrieval. While BERT-based ranking methods effectively address the lexical and semantic mismatch issues of traditional BM25 keyword matching methods through fully connected cross-attention, the computational cost of cross-attention is prohibitively high. Therefore, the method of concatenating the query and document and inputting it into BERT to obtain a matching score is generally only used in the final fine-grained ranking stage. Furthermore, while fully connected cross-attention-based retrieval models can effectively solve the lexical and semantic mismatch problems of BM25 keyword matching methods, the computational cost of cross-attention remains extremely high. If a query needs to be retrieved from a database of hundreds of millions of documents, the retrieval model needs to perform hundreds of millions of calculations in real time, and offline pre-calculation is not possible. While fast semantic retrieval models based on dense vectors can balance semantic retrieval and retrieval efficiency to some extent, this modeling approach assumes that the semantic vector of [CLS] represents the semantics of the entire query or document. However, pre-trained language models such as BERT do not perform special processing on [CLS] during the pre-training stage and directly use the semantic vector of [CLS] for retrieval. This is merely a conventional modeling approach and lacks rationality.

[0065] A semantic retrieval scheme based on SimCSE (Contrastive Learning) aims to learn effective feature representations by grouping semantically similar neighbors together and separating dissimilar samples. This aligns perfectly with the needs of semantic retrieval, thus enabling the development of a high-performing retrieval model. However, while contrastive learning can bring queries closer to relevant documents and distance them from irrelevant ones, during training, irrelevant negative samples are randomly selected, while relevant positive samples are pre-prepared or generated through data augmentation. This results in highly relevant positive samples and low-relevance negative samples, which is overly simplistic for model training and fails to produce a robust retrieval model. Furthermore, the prevalence of reposting and citations on the internet means that the number of query-related positive samples increases over time. During model training, it's impossible to exhaustively enumerate all positive samples. If a positive sample is treated as a negative sample during contrastive learning model training, it significantly impacts the model's learning performance, leading to false negatives.

[0066] Based on the detailed description of the shortcomings of the existing technology, it can be seen that the existing technology does not fully consider the contradiction between the efficiency and retrieval effect of semantic retrieval, and does not compress and concentrate the semantic modeling capability of the pre-trained language model. Directly using the language model vector output will impair the retrieval effect. To solve this problem, when implementing the method, this case proposes to use a pre-trained language model to perform semantic representation of the query and document, and obtain the vector representation of each word in the query and document. A word-granular pre-trained language model is pre-trained for the retrieval scenario, thereby obtaining the semantic representation vector of each word in the query and document. Using the semantic vector at the [CLS] position of the ordinary pre-trained BERT language model for modeling, this case improves the BERT pre-training structure by compressing the semantic vector of each word in the query or document to the [CLS] vector, improving the semantic richness of the [CLS] vector, and making it a semantic vector that can represent the entire text. Contrastive learning training method is also integrated into the pre-training process of the language model, laying a good language model foundation for training a better semantic retrieval model.

[0067] To improve the semantic retrieval performance of Chinese multi-document retrieval, this solution proposes a word-granular multi-document semantic retrieval method model based on the characteristics of the Chinese language and the advantages and disadvantages of traditional solutions. In the model pre-training stage, training strategies such as virtual adversarial training and contrastive learning pre-training tasks are added to more accurately model the semantic information of the text and establish a high-efficiency semantic retrieval model that combines word weights, thereby balancing retrieval accuracy and retrieval efficiency.

[0068] The following example illustrates the method for constructing the semantic retrieval scoring function provided in this application. Please refer to... Figure 1 , Figure 1 A flowchart of a method for constructing a scoring function for semantic retrieval provided in this application embodiment includes:

[0069] S101: Retrieve text from the text library and query text.

[0070] S102: Perform word segmentation on the text in the text library and the query text to obtain a first word set and a second word set.

[0071] The first word set includes multiple text database terms, and the second word set includes multiple query terms. All texts in the text database are segmented using a word-granular BERT model, and then the TF-IDF weight value of each word is calculated on the segmented text. Simultaneously, referencing a keyword matching method based on BM25, a subset of texts with overlapping words with the query is filtered from the massive text database documents based on the segmentation results. Subsequent searches are performed on this subset.

[0072] S103: Input the first word set and the second word set into the pre-trained BERT model to obtain the first word vector set and the second word vector set.

[0073] The pre-trained BERT model is used to compress the semantic information of words in the text. The segmented user query and the text database are independently input into the pre-trained BERT model adapted for the retrieval task, obtaining vector representations of their respective [CLS] characters and other normal tokens. The calculation formula is as follows:

[0074]

[0075]

[0076] Where LM represents a pre-trained BERT model adapted for the retrieval task, and W token It is a trainable weight matrix that maps and transforms the output of the language model. The vector representation of the i-th word in the query. This represents the vector representation of the j-th word in the document.

[0077] S104: Determine the retrieval score based on the third word set, the weight value, the first word vector set, and the second word vector set.

[0078] The third word set is a text set from the first word set that overlaps with the second word set. For each query word in the third word set, match it against all corresponding identical words in the text database. Using the first word vector set and the second word vector set output by the pre-trained BERT model adapted for the retrieval task, calculate the dot product similarity between each query word and its corresponding identical word. Obtain the highest similarity score as the retrieval score, which is the similarity score of the token with the highest similarity among all identical words.

[0079] The overlapping words q for query and document i ∈q∩d, generate a third word set, calculate the retrieval score of overlapping words. Specifically, first for each query word q i Matches all occurrences of the word "d" in the document. j =q i Then, using the word vectors output by the pre-trained BERT model adapted for the retrieval task, d is calculated respectively. j and q i The dot product similarity is used to generate the first word vector set and the second word vector set, and all d are extracted.j The similarity score of the token with the highest similarity score.

[0080] Since the query and document may contain multiple overlapping words with varying degrees of importance, this study combines the calculated TF-IDF weights of each word to perform a weighted sum of the semantic similarity of these overlapping words. The calculation formula is as follows:

[0081]

[0082] S105: Determine the scoring function based on the first compression vector, the second compression vector, and the retrieval score.

[0083] The first compressed vector is used to represent the overall semantics of the first word set, and the second compressed vector is used to represent the overall semantics of the second word set.

[0084] s token Similar to keyword matching methods based on BM25, this approach does not consider the similarity between different words, thus facing the problem of word mismatch. To address this issue, this paper utilizes [CLS] to integrate sentence-level representations:

[0085]

[0086]

[0087] and The similarity between them can provide high-level semantic matching information, alleviating the problem of word mismatch. Ultimately, the comprehensive scoring function for this case's retrieval is:

[0088]

[0089] As can be seen from the embodiments of this application, this application fully considers the relationship between retrieval efficiency and retrieval accuracy. Combining the advantages of the modified BERT pre-training method, virtual adversarial training, and deep semantic methods, the following innovations and improvements are made to the Chinese multi-document semantic retrieval task: A pre-trained language model integrating contrastive learning training objectives and virtual adversarial training strategies is proposed. This model is specifically designed for semantic retrieval scenarios, compressing the semantic information of words in the text into [CLS] vectors to obtain better overall semantic representation. In addition, the use of virtual adversarial training enhances the robustness of the language model; A word-granularity pre-trained language model for Chinese data in retrieval scenarios is proposed, which can obtain query and document... The semantic representation vectors of each word in t are used. Furthermore, due to the use of word-level information, the sequence length input into the language model is shortened, enabling the language model to output semantic vectors for queries and documents more quickly. Additionally, in high-efficiency retrieval scenarios, an inverted index can be built based on words, significantly improving retrieval efficiency. A semantic retrieval model combining word weights is proposed, where the TF-IDF value of a word is used as the weight of its semantic vector. This reduces the role of commonly used words in the semantic retrieval stage, highlighting the role of keywords and thus improving the quality of retrieval results. These modifications to the model training strategies are all beneficial for improving the training of the Chinese multi-document semantic retrieval model, significantly enhancing its retrieval efficiency and performance.

[0090] The following provides a detailed description of the method for constructing the scoring function for semantic retrieval provided in the embodiments of this application. See also... Figure 2 As shown, Figure 2 Another flowchart illustrating the method for constructing a scoring function for semantic retrieval provided in this application embodiment is shown below. The specific process is as follows:

[0091] S201: Obtain high-frequency words and add them to the pre-trained BERT model vocab.txt.

[0092] In practical applications, due to the infinite variety of word combinations, it's impossible to include all words in the BERT model's dictionary. Furthermore, some words often appear infrequently, which would significantly increase the number of parameters in the BERT model. Therefore, the retrieval pre-training scheme proposed in this paper only considers frequently occurring words in the pre-training corpus, while still using a character-based segmentation method for those less frequent words. This approach combines the advantages of avoiding out-of-vocabulary words with the high efficiency of the character-based method.

[0093] Because BERT's built-in word segmenter forcibly separates Chinese characters with spaces, even though high-frequency words are added to the dictionary, the built-in word segmenter still needs to be modified to adapt to the word-granularity retrieval BERT model proposed in this paper. This is achieved by adding a "pre-segmentation" operation, using existing Chinese word segmentation tools such as jieba and LTP to segment the input Chinese text first, and then performing BERT's segmentation operation.

[0094] S202: Perform word segmentation on the input statement to obtain a set of word segmentation results.

[0095] The word segmentation result set includes several segmented words. Input a sentence s, and use a Chinese word segmentation tool to segment it once, obtaining [w1, w2, ..., w...]. l ].

[0096] S203: Traverse the segmented words in the set of segmentation results.

[0097] If the segmented words match the word list, then retain them;

[0098] In response to a mismatch between the segmented word and the vocabulary, the segmentation function is applied to segment the word, and the original segmented word in the segmentation result set is replaced and updated. The step of "traversing the segmented words in the segmentation result set" is then executed.

[0099] Traverse each w i If W i If it is in the vocabulary, it is retained; otherwise, W is removed. i Use BERT's built-in word segmentation function to segment it again.

[0100] S204: Concatenate several successfully matched word segmentation results in an ordered manner to generate the input sequence.

[0101] Each W i The word segmentation results are concatenated in order and used as the final input sequence for the BERT model.

[0102] S205: Concatenate the first CLS vector with the first character vector to obtain the semantic vector matrix.

[0103] The first CLS vector is the CLS vector of the last layer in the last eight layers of the pre-trained BERT model, and the first character vector is all the vectors of the last layer in the first four layers of the pre-trained BERT model that do not have CLS characters.

[0104] BERT's primary pre-training task is Masked Language Model (MLM). This involves randomly replacing each token in the input sequence with the character [MASK] with a 15% probability, and then predicting the original word at the [MASK] position. Since MLM tasks tend to extract semantic vector representations at the token level and cannot directly obtain sentence-level representations, BERT uses the Next Sentence Prediction (NSP) task for pre-training to enable the model to understand the relationships between sentences. Simply put, this predicts whether two sentences are connected based on the semantic vector of the [CLS] character.

[0105] This paper proposes an improved BERT pre-training architecture that forces the semantic information of the entire sequence to be compressed towards the [CLS] position. The specific model framework is as follows: Figure 3 As shown, Figure 3 This application provides a BERT model adapted for retrieval tasks. In order to allow the [CLS] character vector to contain more sequence semantic information, the original BERT model architecture is split. The first four layers are considered as steps to learn the semantic representation of all tokens in the normal sequence, during which the [CLS] character is treated as a regular token. The last eight layers are then considered as a process of compressing the semantics of the entire input sequence into a [CLS] character vector. To achieve this semantic compression, the [CLS] character vector of the last layer in the last eight layers is concatenated with all token vectors except for the [CLS] character in the last layer of the first four layers.

[0106] S206: Add an encoding layer to the semantic vector matrix, and calculate and determine the CLS compression vector based on the semantic vector matrix and the self-attention of the encoding layer to obtain the BERT pre-trained model.

[0107] Add an encoding layer to the concatenated semantic vector matrix, so that the semantic interaction contained in the [CLS] character vector of the last layer of the last 8 layers is transmitted to all the token vectors of the last layer of the first 4 layers except for the [CLS] character.

[0108] S207: Perform large-scale pre-training using contrastive learning tasks and corpora until training is complete, and use the trained model as the first training model.

[0109] Based on this, large-scale pre-training is performed using the output of this added encoder layer.

[0110] After constructing the BERT pre-trained model framework adapted for the retrieval task, large-scale pre-training of the BERT model is required. The pre-training tasks used in this case are the Masked Language Model (MLM) task and the contrastive learning task. The MLM task is used in the same way as the usual BERT pre-training tasks; the main improvement lies in the contrastive learning pre-training task.

[0111] Comparative learning requires both positive and negative correlation data. Typically, the positive correlation data for a single sample needs to be manually labeled.

[0112] Therefore, this paper uses two methods to automatically generate a large number of positively correlated samples.

[0113] The first method involves obtaining the text information of a first text segment as the first sample, and then obtaining a second text segment from the same article that does not overlap with the first text segment. The text information of the second text segment is then labeled as positively correlated with the first sample. The second text segment contains the same amount of text information as the first text segment. In other words, two disjoint text segments obtained from the same article can be considered to contain related semantic information. For example, for a document A, if the first 128 characters of the document are sample A1, then characters 129-256 of the document are designated as sample A2, which is positively correlated with A1.

[0114] The second approach involves using the BM25 algorithm to obtain the similarity of the first sample. Then, based on the BM25 algorithm, a second sample with a similarity of at least a threshold to the first sample is searched from the text database. This second sample is then labeled as positively correlated with the first sample. Similarly, a sample A2 with a similarity exceeding the threshold to A1 is searched from the massive text database and is also considered positively correlated with A1.

[0115] During large-scale pre-training, massive amounts of data are input into the BERT model in batches. Therefore, within the same batch, all samples except sample A are considered negatively correlated. See [link to model pre-training architecture] for details. Figure 4 , Figure 4 This diagram illustrates a large-scale pre-training method combining contrastive learning, as provided in an embodiment of this application. It should be noted that, to maintain simplicity, only a contrastive learning example for sample A is shown in this specification. For all samples within the same batch, the contrastive loss with other samples needs to be calculated.

[0116] S208: Update the parameters in the model using virtual adversarial training.

[0117] Essentially, this involves adding random perturbations to the word vector layer of the pre-trained model, making the model's predictions before and after the perturbations more consistent, thereby increasing the model's robustness against interference.

[0118] This case proposes a virtual adversarial training strategy to mitigate the impairment caused by learning only the one-hot vector representation of the masked character in the MLM task. By changing the learning objective of the pre-trained model for the MLM task, it learns not only the one-hot label vector (hard-label) of the masked character, but also the probability distribution of each word in the vocabulary. The probability distribution is then used as the soft-label vector for each masked character to alleviate the impairment caused by simply learning the hard-label.

[0119] The detailed process steps include the following, which you can refer to. Figure 5 , Figure 5 This is a flowchart illustrating a method provided in an embodiment of this application.

[0120] S301: Perform MLM task training on the input sequence to obtain a first probability distribution;

[0121] The first probability distribution is the predicted probability distribution of the character replaced by the mask in the vocabulary; for each batch_size of data input into the model for pre-training, the normal MLM task is first performed to obtain the predicted probability distribution P (first probability distribution) of the masked character in the vocabulary.

[0122] S302: Obtain the first perturbation vector, add the first perturbation vector to the input sequence, and obtain the second input sequence.

[0123] Generate a perturbation vector δ, which follows a normal distribution with a mean of 0 and a variance of 1. Add this perturbation vector to the word vector layer of the BERT pre-trained model.

[0124] S303: Input the second input sequence again to train the MLM task and obtain the second probability distribution.

[0125] The perturbated vector is then input back into the model, and through the MLM task, a new predicted probability distribution P1 (second probability distribution) of the masked character on the vocabulary is obtained.

[0126] S304: Determine the first divergence Loss and the first gradient of the first perturbation vector based on the first probability distribution and the second probability distribution;

[0127] Calculate the DL divergence Loss (first divergence Loss) of the probability distribution P of the actual output and the probability distribution P1 with perturbation, and calculate the gradient of the perturbation vector δ. Update the gradient of the generated perturbation and regularize the perturbation.

[0128] S305: Update and regularize the first perturbation vector according to the first gradient to obtain the second perturbation vector.

[0129] S306: Add the second perturbation vector to the second input sequence to obtain the third input sequence.

[0130] S307: Input the third input sequence again to train the MLM task and obtain the third probability distribution.

[0131] The new perturbation vector is added to the word vector layer of the BERT pre-trained model to obtain the predicted probability distribution P2 of the masked character on the vocabulary (third input sequence).

[0132] S308: Calculate the second divergence Loss of the actual output probability distribution and the third probability distribution, and add the cross-entropy Loss to the second divergence Loss.

[0133] The cross-entropy Loss is the cross-entropy Loss between the actual output probability distribution and the one-hot label vector of the character replaced by the MASK; calculate the DL divergence Loss of the actual output probability distribution P and the perturbed probability distribution P2, and add the cross-entropy Loss between the actual output probability distribution P and the one-hot label vector of the character replaced by the MASK.

[0134] S309: Weight the cross-entropy Loss and the second divergence Loss to obtain the overall Loss of the final model. Weight the two Loss to obtain the overall Loss of the final model.

[0135] S310: Calculate the gradient based on the overall loss and update the parameters in the BERT pre-trained model. Based on the final loss, calculate the gradient and update the parameters in the pre-trained model.

[0136] The BM25 keyword matching-based retrieval method is highly efficient because the scoring process only relies on documents containing the query terms, requiring the retrieval of only a subset of a massive text database. Deep semantic retrieval achieves better results than keyword matching, but its efficiency is far lower. This is mainly because it cannot perform inverted indexing on the semantic vector of each document, making semantic retrieval methods unsuitable for retrieval needs involving hundreds of billions of documents. To combine the performance advantages of semantic retrieval with the efficiency advantages of keyword matching, this paper proposes a high-efficiency semantic retrieval model incorporating word weights. The specific model framework is as follows: Figure 6 As shown, Figure 6 This is a specific model framework diagram of a high-efficiency semantic retrieval model that combines word weights, provided for embodiments of this application. The method flow after step S208 is connected to the method in Embodiment 1, including word segmentation using a word-granular BERT model on all texts in the text library, etc., which will not be elaborated here.

[0137] The above describes some specific implementations of a semantic retrieval scoring function construction method provided in this application. Based on this, this application also provides a corresponding apparatus. The apparatus provided in this application will be described below from the perspective of functional modularity.

[0138] Please refer to Figure 7 , Figure 7 This is a schematic diagram of a semantic retrieval scoring function construction device provided in an embodiment of this application.

[0139] In this embodiment, the device may include:

[0140] Text acquisition module 701 is used to acquire text from the text library and query text.

[0141] The weight value acquisition module 702 is used to segment the text in the text library and the query text to obtain a first word set and a second word set, and to acquire the weight value of each word in the first word set. The first word set includes multiple text library words, and the second word set includes multiple query words.

[0142] The word vector set acquisition module 703 is used to input the first word set and the second word set into the pre-trained BERT model to obtain the first word vector set and the second word vector set. The pre-trained BERT model is used to compress the semantic information of words in the text.

[0143] The retrieval score determination module 704 is used to determine the retrieval score based on the third word set, the weight value, the first word vector set, and the second word vector set, wherein the third word set is the text set in the first word set that has overlapping words with the second word set;

[0144] The scoring function determination module 705 is used to determine a scoring function based on a first compression vector, a second compression vector, and the retrieval score, wherein the first compression vector represents the overall semantics of the first word set, and the second compression vector represents the overall semantics of the second word set.

[0145] This application provides an apparatus including a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code to cause the apparatus to perform the semantic retrieval scoring function construction method described in any of the first aspects above.

[0146] This application provides a computer storage medium storing code. When the code is executed, a device running the code implements the semantic retrieval scoring function construction method described in any of the first aspects above.

[0147] The foregoing has provided a detailed description of a method for constructing a scoring function for semantic retrieval and related apparatus provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0148] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0149] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0150] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0151] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for constructing a scoring function for semantic retrieval, characterized in that, include: Retrieve text from the text library and query text; The text in the text library and the query text are segmented to obtain a first word set and a second word set. The weight value of each word in the first word set is obtained. The first word set includes multiple text library words, and the second word set includes multiple query words. The first word set and the second word set are input into a pre-trained BERT model to obtain a first word vector set and a second word vector set. The pre-trained BERT model is used to compress the semantic information of words in the text. The retrieval score is determined based on the third word set, the weight value, the first word vector set, and the second word vector set, wherein the third word set is the set of texts in the first word set that have overlapping words with the second word set; A scoring function is determined based on a first compression vector, a second compression vector, and the retrieval score. The first compression vector represents the overall semantics of the first word set, and the second compression vector represents the overall semantics of the second word set. In the BERT model pre-training stage, virtual adversarial training and contrastive learning pre-training tasks are added. Large-scale pre-training is carried out using contrastive learning tasks and corpora until training is completed. The trained model is then used as the pre-trained BERT model. Updating model parameters using virtual adversarial training includes: The first probability distribution is obtained by training the input sequence using the MLM task. The first probability distribution is the predicted probability distribution of the characters replaced by the mask in the vocabulary. Obtain a first perturbation vector, add the first perturbation vector to the input sequence, and obtain a second input sequence; The second input sequence is input again to train the MLM task and obtain the second probability distribution; The first divergence Loss and the first gradient of the first perturbation vector are determined based on the first probability distribution and the second probability distribution. The first perturbation vector is updated and regularized according to the first gradient to obtain the second perturbation vector; The second perturbation vector is added to the second input sequence to obtain the third input sequence; The third input sequence is input again to train the MLM task and obtain the third probability distribution; Calculate the second divergence Loss of the actual output probability distribution and the third probability distribution, and add the cross-entropy Loss to the second divergence Loss. The cross-entropy Loss is the cross-entropy Loss of the actual output probability distribution and the one-hot label vector of the character replaced by the MASK. The cross-entropy Loss and the second divergence Loss are weighted to obtain the overall Loss of the final model; Calculate the gradient based on the overall loss and update the parameters in the BERT pre-trained model; The input sequence is generated in the following ways: High-frequency words are obtained and added to the pre-trained BERT model vocab.txt; The input statement is segmented into words to obtain a segmentation result set, which includes several segmented words; Iterate through the segmented words in the result set, and retain the words that match the word list. In response to a mismatch between the segmented word and the vocabulary, the segmentation function is applied to segment the word, and the original segmented word in the segmentation result set is replaced and updated. The step "traverse the segmented words in the segmentation result set" is then executed. Several successfully matched word segmentation results are concatenated in an ordered manner to generate the input sequence.

2. The method for constructing a scoring function for semantic retrieval according to claim 1, characterized in that, Constructing the pre-trained BERT model includes the following steps: Obtain the input sequence; The semantic vector matrix is ​​obtained by concatenating the first CLS vector and the first character vector. The first CLS vector is the CLS vector of the last layer in the last eight layers of the pre-trained BERT model, and the first character vector is all the vectors of the last layer in the first four layers of the pre-trained BERT model that do not have CLS characters. An encoding layer is added to the semantic vector matrix. The CLS compression vector is calculated based on the semantic vector matrix and the self-attention of the encoding layer to obtain the pre-trained BERT model. The CLS compression vector is a vector representing the semantics of the entire input sequence, obtained by interacting the semantics of the first CLS vector with the first character vector.

3. The method for constructing a scoring function for semantic retrieval according to claim 2, characterized in that, The large-scale pre-training using contrastive learning tasks and corpora includes: Several corpora were divided into positively correlated data and negatively correlated data; The positive and negative correlation data are input into the BERT pre-trained model in batches for training.

4. The method for constructing a scoring function for semantic retrieval according to claim 2, characterized in that, The acquisition of the input sequence includes: High-frequency words are obtained and added to the pre-trained BERT model vocab.txt; The input statement is segmented into words to obtain a segmentation result set, which includes several segmented words; Iterate through the segmented words in the result set, and retain the words that match the word list. In response to a mismatch between the segmented word and the vocabulary, the segmentation function is applied to segment the word, and the original segmented word in the segmentation result set is replaced and updated. The step "traverse the segmented words in the segmentation result set" is then executed. Several successfully matched word segmentation results are concatenated in an ordered manner to generate the input sequence.

5. The method for constructing a scoring function for semantic retrieval according to claim 3, characterized in that, The process of dividing several corpora into positively correlated data and negatively correlated data includes: The text information of the first text segment is obtained as the first sample. The second text segment in the same article that does not intersect with the first text segment is obtained. The text information of the second text segment is labeled as positive correlation data of the first sample. The second text segment contains the same amount of text information as the first text segment. Alternatively, obtain the similarity of the first sample, search the text library for a second sample whose similarity to the first sample is not less than a threshold according to the BM25 algorithm, and label the second sample as positively correlated data of the first sample.

6. The method for constructing a scoring function for semantic retrieval according to claim 1, characterized in that, The step of determining the retrieval score based on the third word set, the weight value, the first word vector set, and the second word vector set includes: For each query word in the third word set, match it with all corresponding identical words in the text library; Using the first word vector set and the second word vector set output by the pre-trained BERT model adapted to the retrieval task, the dot product similarity between each query word and its corresponding identical word is calculated respectively. The highest similarity score is obtained as the retrieval score, which is the similarity score of the token with the highest similarity among all identical words.

7. A computer device, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the semantic retrieval scoring function construction method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores code, and when the code is executed, the device running the code implements the semantic retrieval scoring function construction method as described in any one of claims 1-6.

Images