Sentence vector cycle-based retrieval relevance model and construction method thereof

Abstract

The embodiment of the application discloses a retrieval relevance model based on a sentence vector cycle and a construction method; the method comprises the following steps: obtaining a query and a corresponding document; generating a sequence text based on the query and the document; inputting the sequence text into a pre-training model to obtain a vectorized representation of the sequence text, and obtaining a text classification vector, a query vector, a title vector, a title end vector, a content vector and a content end vector; newly creating a vector with an output hidden layer dimension of an alignment pre-training model as a first vector of a retrieval task; performing cross attention mechanism interaction processing on the first vector of the retrieval task and the query vector to obtain a second vector of the retrieval task; performing cross attention mechanism interaction processing on the second vector of the retrieval task and a splicing vector of the title vector and the content vector to obtain a third vector of the retrieval task; splicing the third vector of the retrieval task and the text classification vector, multiplying by a task weighting weight, obtaining an unnormalized logarithmic probability of a calculation score, and mapping the unnormalized logarithmic probability by using a Sigmoid function to obtain a floating point score, wherein the floating point score is a relevance calculation score.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electric data processing, and particularly relates to a retrieval relevance model based on a sentence vector cycle and a construction method thereof. BACKGROUND

[0002] In a retrieval system, re-ranking is to re-rank a query and a document screened out by a coarse ranking, wherein relevance is the most important index for measuring the semantics of the query and the document, and a relevance model scores the semantic relevance of the query and different documents to determine which document corresponding to the current query better meets the user's demand.

[0003] With the rise of retrieval enhancement generation tasks, how to find a document with higher relevance to a user query in a large number of documents has become increasingly important, making re-ranking technology re-enter people's field of vision. The re-ranking task plays the role of an intelligent garbage text filter in retrieval enhancement generation, filtering out irrelevant documents and inputting the most relevant several documents as pre-prompt words into a large model.

[0004] The re-ranking technology mainly includes rule-based re-ranking, machine learning re-ranking model, linear model, tree model, and end-to-end deep learning neural network model; for example, based on a BERT model plus a classification layer, the classification layer mainly includes a TextMatching model based on CNN for feature extraction of adjacent words, and an Esim model for improving the accuracy of results by using mutual representation between a query and a document. Based on pre-training, the re-ranking technology, for example, performs secondary pre-training of word embedding semantic similarity on the basis of mask language model pre-training, such as UniLM, SimBERT, Sentence BERT, ERNIE-Gram, and BGE, can map a query and a document into a sentence vector, and then calculate the similarity score between the two by using the calculation method of text distance such as Euclidean distance, cosine similarity, and edit distance, and then perform re-ranking.

[0005] However, the existing technology has the following technical problems:

[0006] The interaction matching between the query and the document is mainly used, which does not match the training form of the original base model, and the basic semantic information obtained by pre-training of the original base model is easily damaged;

[0007] The relevance model based on the double-tower structure occupies more computing resources (GPU) during reasoning;

[0008] The existing retrieval pre-training model can only utilize and mine the semantic knowledge obtained by pre-training of the base model, and cannot perform training specific to the downstream task (for example, the relevance task);

[0009] Existing relevance models mostly perform interactive operations at the word vector level, lacking semantic calculations on the overall sentence vector and between sentence vectors. In the retrieval and re-ranking stage, queries and documents are mostly sorted according to their relevance scores, which easily causes the sorting to fall into local optimality. Summary of the Invention

[0010] In view of this, some embodiments disclose a method for constructing a retrieval relevance model based on sentence vector loop, including:

[0011] S1. Obtain the query and the corresponding document; the document includes the title of the web page and the content extracted from the web page;

[0012] S2. Mark the file position;

[0013] S3, generate sequence text based on query and document;

[0014] S4. Input the sequence text into the pre-trained model to obtain a vectorized representation of the sequence text, and obtain a text classification vector, a query vector, a title vector, a title end vector, a content vector, and a content end vector;

[0015] S5. Create a new vector of the hidden layer dimension of the aligned pre-trained model output as the first vector of the retrieval task;

[0016] S6. Perform a cross-attention interaction between the first retrieval task vector and the query vector: use the first retrieval task vector as the query of the cross-attention mechanism, and use the query vector as the key and value of the cross-attention mechanism to obtain the second retrieval task vector.

[0017] S7. Perform a cross-attention interaction between the second retrieval task vector and the document vector: concatenate the title vector and the content vector into a document vector, use the second retrieval task vector as the query of the cross-attention mechanism, and use the document vector as the key and value of the cross-attention mechanism to obtain the third retrieval task vector.

[0018] S8. Concatenate the third retrieval task vector and the text classification vector, multiply by the task weight, and obtain the unnormalized logarithmic probability of the calculation score. Use the Sigmoid function to map to obtain a floating-point score, which is the relevance calculation score. Specifically expressed as:

[0019] Score=Sigmoid(W final Concat([cls]′;V3))

[0020] Among them, W finalThe task weight is randomly initialized for a normal distribution, [cls] is a text classification vector, V3 is a third vector for a retrieval task, Score is a floating point score, and the value is between 0 and 1, and Concat is a vector splicing operation.

[0021] Further, some embodiments disclose a retrieval relevance model construction method based on a sentence vector cycle, in step S2, the query and the title and content of the document are labeled, divided into four grades of excellent, medium, poor and irrelevant, and the grade identification corresponding to the document is obtained, and the grade identification is a value between 0 and 1.

[0022] Some embodiments disclose a retrieval relevance model construction method based on a sentence vector cycle, in step S3, the generated sequence text is represented as: [cls] query [sep] title [sep] content [sep]; at the same time, different text type IDs are given to [cls], query [sep], title [sep] and content [sep] to distinguish different text types; wherein, [cls] is a sequence semantic symbol, and [sep] is a sequence end symbol.

[0023] Some embodiments disclose a retrieval relevance model construction method based on a sentence vector cycle, in step S4, the sequence text [cls] query [sep] title [sep] content [sep] is input into a retrieval pre-training model to obtain a vectorization representation of the sequence text, that is, a text classification vector [cls]', a query vector query', a title vector title', a title end vector [sep1]', a content vector snippet' and a content end vector [sep2]'.

[0024] Some embodiments disclose a retrieval relevance model construction method based on a sentence vector cycle, in step S6, the cross attention mechanism is a multi-head cross attention mechanism, including three parts: query sequence, key sequence and value sequence; step S6 specifically includes:

[0025] Initialize the weight W of the normal distribution q , W k , W v , W;

[0026] Q, K and V are calculated according to the following formula:

[0027] Q = W q ·V1

[0028] K = W k ·query'

[0029] V = W v ·query'

[0030] Wherein, Q is a query vector, K is a key vector, V is a value vector, V1 is a first vector of a retrieval task, W q is a weight of Q, W k is a weight of K, W v is a weight of V; query' is a query vector;

[0031] Q, K and V are divided into multiple heads, here n heads, and the single-head attention is calculated respectively; the calculation method includes:

[0032] The dot product of Q and K is scaled by the vector divided by the square root of the hidden dimension d, and the similarity score between the first vector of the retrieval task and each token of the query vector is calculated;

[0033] After softmax processing, the attention weight of the first vector of the retrieval task to each token of the query vector is obtained;

[0034] The attention weight corresponding to the token is multiplied by the value vector V and then added to obtain the final single-head vector expression Head i ; the single-head calculation process is represented as:

[0035]

[0036] All single-head vectors Head i are spliced and multiplied by the spatial weighting weight W to obtain the output vector V2, represented as:

[0037] V2=concat(Head1;…;Head i )·W

[0038] Wherein, V2 is a second vector of a retrieval task, and concat is a vector splicing operation.

[0039] Some embodiments disclose a retrieval relevance model construction method based on sentence vector cycle, and step S7 specifically includes:

[0040] The weights W q , W k , W v and W are initialized as normal distribution;

[0041] Q, K and V are calculated according to the following formula:

[0042] Q=W q ·V2

[0043] K=W k ·concat(title′;snippet′)

[0044] V=W v ·concat(title′;snippet′)

[0045] Wherein, Q is a query vector, K is a key vector, V is a value vector, V2 is a retrieval task second vector, W q is a weight of Q, W k is a weight of K, W v is a weight of V, title' is a title vector, snippet' is a content vector, and concat is a vector concatenation operation.

[0046] Q, K, and V are divided into multiple heads, here n heads, and single-head attention is calculated respectively, including:

[0047] The dot product of Q and K is scaled by the vector divided by the square root of the hidden dimension d, and the similarity score between each token of the retrieval task second vector and the document vector is calculated.

[0048] After softmax processing, the attention weight of each token of the retrieval task second vector with respect to the document vector is obtained.

[0049] The attention weight corresponding to the token is multiplied by the value vector and then added to obtain the final single-head vector expression Head i ; The calculation process is represented as:

[0050]

[0051] All Head i are concatenated and multiplied by a spatial weighting weight W to obtain an output vector V3, represented as:

[0052] V3 = concat (Head1; …; Head i )·W

[0053] Wherein, V3 is a retrieval task third vector, and concat is a vector concatenation operation.

[0054] Some embodiments of the retrieval relevance model construction method based on sentence vector cycle also include step S9, loss function calculation, specifically including: using a binary cross-entropy loss function BCEloss for probability calculation, and the calculation formula is:

[0055]

[0056] Wherein, L i is the gear identification of the document, taking a value between 0 and 1, S i is the floating point score Score output by the relevance model, and N is the sample quantity.

[0057] Some embodiments of the retrieval relevance model construction method based on sentence vector cycle, step S3 includes:

[0058] fixing the length of the query;

[0059] The query, the title and the abstract of the document are input into the encoder-only base model to obtain the sequence text.

[0060] In another aspect, some embodiments disclose a retrieval relevance model based on sentence vector cycle, which is obtained by the retrieval relevance model construction method based on sentence vector cycle disclosed by the embodiments of the present application.

[0061] The retrieval relevance model obtained by the retrieval relevance model construction method based on sentence vector cycle disclosed by the embodiments of the present application is a model based on a single tower structure. In the case of sufficient interaction between the query and the document, a new task-specific task embedding vector is introduced, the query is summarized, the query is summarized into a single word vector--query sentence vector, and then the query sentence vector is used to search the document, the relevance degree between the document and the query sentence vector is calculated, and the corresponding relevance score is given. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The retrieval relevance model construction method based on sentence vector cycle disclosed by some embodiments is a flowchart;

[0063] Figure 2 The relevant text vector model structure obtained by the text sequence input BGE pre-training model disclosed by some embodiments is shown in the figure;

[0064] Figure 3 The multi-head cross-attention mechanism model structure disclosed by some embodiments is shown in the figure. DETAILED DESCRIPTION

[0065] The special word "embodiment" in this paper is not necessarily interpreted as superior or better than other embodiments as "exemplary" described in any embodiment. Unless otherwise specified, the performance index test in the embodiments of the present application adopts the conventional test method in the art. It should be understood that the terms described in the embodiments of the present application are only for the description of the specific embodiments, and are not used to limit the content disclosed by the embodiments of the present application.

[0066] Unless otherwise specified, the technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the embodiments of the present application belong; as the experimental methods and technical means not specially noted in the embodiments of the present application are the experimental methods and technical means generally used by those skilled in the art.

[0067] As used herein, the terms "substantially" and "approximately" are used to describe small fluctuations. For example, they can refer to less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%. Numerical data expressed or presented in range format herein are used for convenience and brevity only and should therefore be interpreted flexibly to include not only the values ​​explicitly listed as the limits of the range, but also all independent values ​​or subranges contained within the range. For example, a numerical range of "1-5%" should be interpreted to include not only the explicitly listed values ​​of 1% to 5%, but also the independent values ​​and subranges within the indicated range. Thus, included in this numerical range are independent values ​​such as 2%, 3.5%, and 4%, and subranges such as 1% to 3%, 2% to 4%, and 3% to 5%, etc. This principle also applies to ranges that only list a single value. Furthermore, this interpretation applies regardless of the width of the range or the characteristics described.

[0068] Throughout this document, including in the claims, transitional terms such as "comprises," "includes," "with," "having," "contains," "involving," and "accommodating" are understood to be open-ended, meaning "including but not limited to." Only the transitional terms "consisting of" and "composed of" are closed transitional terms.

[0069] In order to better illustrate the present invention, numerous specific details are provided in the following specific examples. It should be understood by those skilled in the art that the present invention can be practiced without certain specific details. In the examples, some methods, means, instruments, and equipment well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0070] Under the premise of no conflict, the technical features disclosed in the embodiments of the present invention can be arbitrarily combined, and the resulting technical solutions belong to the contents disclosed in the embodiments of the present invention.

[0071] In some implementations, a method for constructing a retrieval relevance model based on a sentence vector loop includes:

[0072] S1. Obtain a query and a corresponding document; the document includes the title of a web page and content extracted from the web page. Typically, a query term and related web pages are obtained, and the relevant content of the web page is extracted based on the query term as a document content snippet. The title of the web page and the document content snippet are together used as a document doc corresponding to the query term. Typically, multiple related web pages are obtained based on the same query term, and multiple corresponding documents doc are obtained.

[0073] S2, mark the gear identification of the document; usually, the query word query and the title and content of the corresponding document can be marked and scored in an artificial manner or based on an artificial intelligence model such as a chatGPT, Qwen or the like large model; generally, the four gear identifications can be marked and scored, and the four gear identifications are further converted into identifications label between 0 and 1; for example, the four gears are excellent, medium, poor and irrelevant, wherein excellent indicates complete matching, and the document completely satisfies the main intention of the query; medium indicates moderate matching, the secondary demand of the query is greatly satisfied by the document, or the main demand of the query is partially satisfied by the document; poor indicates low matching, the query and the document have a low degree of correspondence, only a small number of contents are related; and irrelevant indicates that the query and the document have no relevance;

[0074] S3, generate a sequence text based on the query and the document; usually, the generated sequence text is represented as: [cls] query [sep] title [sep] content [sep]; at the same time, different text type IDs are assigned to [cls], query [sep], title [sep], content [sep] to distinguish different text types; wherein [cls] is a sequence semantic symbol, and [sep] is a sequence end symbol;

[0075] In some embodiments, the length of the query is fixed; the query, the title and the abstract of the document are input into a retrieval pre-training model, and the sequence text is obtained through interaction;

[0076] S4, input the sequence text into the pre-training model to obtain a vectorized representation of the sequence text, and obtain a text classification vector, a query vector, a title vector, a title end vector, a content vector and a content end vector; usually, a text embedding model can be used as a retrieval pre-training base model, such as a BGE model, a BERT model or the like;

[0077] In some embodiments, a BGE model is selected as the pre-training model, as shown in Figure 2 The sequence text [cls] query [sep] Title [sep] Snippet [sep] is input into the BGE model to obtain a vectorized representation of the sequence text, i.e. a text classification vector [cls]', a query vector query', a title vector title', a title end vector [sep1]', a content vector snippet' and a content end vector [sep2]';

[0078] The BGE model is a text embedding model proposed by the Beijing Wisdom Artificial Intelligence Research Institute, which aims to convert text into low-dimensional dense vectors for efficient computation and analysis. The BGE model structure is consistent with RoBert, which is a bert-like model. The sequence semantic symbol [cls] is used as the text vector to support search-related tasks such as retrieval, re-photography, clustering, and classification.

[0079] S5, a vector of the output hidden layer dimension of the newly created alignment pre-training model is created as the first vector for the retrieval task. Typically, a vector of one token size (1token size) is created for the output hidden layer dimension of the newly created alignment pre-training model. This vector is a task-specific embedding, which is used as the first vector for the retrieval task corresponding to the query word and the sequence text.

[0080] S6, the first vector for the retrieval task and the query vector are processed through cross-attention mechanism interaction: the first vector for the retrieval task is used as the query of the cross-attention mechanism, and the query vector is used as the key and value of the cross-attention mechanism to obtain a query-specific embedding vector based on the query as the second vector for the retrieval task. During the interaction process, the cross-attention mechanism calculates the similarity weight between each token of the first vector for the retrieval task and the query vector, so the second vector for the retrieval task summarizes the weight of each token of the query vector.

[0081] In some embodiments, the cross-attention mechanism is a multi-head cross-attention mechanism. Cross-attention is a natural language processing mechanism for handling dependencies between two different sequences, which includes three parts: query vector Q, key vector K, and value vector V. The query vector comes from one modality, while the key and value vectors come from another modality. This cross-attention mechanism allows the model to capture element-level dependencies between different modalities and obtain relevant information by calculating the similarity between the query and the key and value.

[0082] In some embodiments, as shown in Figure 3 the cross-attention mechanism interaction process between the first vector for the retrieval task and the query vector includes:

[0083] Initialize the weight W q , W k , W v , W

[0084] Q, K, and V are calculated according to the following formula:

[0085] Q = W q ·V1

[0086] K = W k ·query′

[0087] V = W v ·query′

[0088] wherein Q is a query vector, K is a key vector, V is a value vector, V1 is a first vector of a retrieval task, W q is a weight of Q, W k is a weight of K, W v is a weight of V; query' is a query vector;

[0089] Q, K, V are divided into multiple heads, here n heads, and the single-head attention is calculated respectively; including:

[0090] The dot product of Q and K is scaled by the vector divided by the square root of the hidden dimension d, and the similarity score between the first vector of the retrieval task and each token of the query vector is calculated;

[0091] After softmax processing, the attention weight of the first vector of the retrieval task on each token of the query vector is obtained;

[0092] The attention weight corresponding to the token is multiplied by the value vector v and then added to obtain the final single-head vector expression Head i ; the single-head calculation process is represented as:

[0093]

[0094] All single-head vectors Head i are spliced and multiplied by a spatial weighting weight W to obtain an output vector V2, represented as:

[0095] V2 = concat(Head1;... ; Head i )·W

[0096] wherein V2 is a second vector of a retrieval task task embedding2; concat is a vector splicing operation;

[0097] S7, cross-attention mechanism interaction processing between the second vector of the retrieval task and the document vector: the title vector and the content vector are spliced into a document vector, the second vector of the retrieval task is taken as the query of the cross-attention mechanism, and the document vector is taken as the key and value of the cross-attention mechanism, to obtain a third vector of the retrieval task;

[0098] In some embodiments, as shown in Figure 3 , the cross-attention mechanism interaction processing process between the second vector of the retrieval task and the document vector includes:

[0099] Initialize the weight W of the normal distributionq , W k , W v , W; usually, the weights W of the normal distribution are initialized q , W k , W v , W can adopt the same initialization weight in step S6;

[0100] Q, K, V are calculated according to the following formula:

[0101] Q = W q ·V2

[0102] K = W k ·concat(title'; snippet')

[0103] V = W v ·concat(title'; snippet')

[0104] Wherein, Q is a query vector, K is a key vector, V is a value vector, V2 is a retrieval task second vector, W q is the weight of Q, W k is the weight of K, W v is the weight of V; title' is a title vector, snippet' is a content vector; concat is a vector concatenation operation;

[0105] Q, K, V are divided into multiple heads, here n heads, and the single attention is calculated respectively, including:

[0106] The dot product of Q and K is scaled by the vector divided by the square root of the hidden dimension d, and the similarity score between each token of the retrieval task second vector and the document vector is calculated.

[0107] After softmax processing, the attention weight of the retrieval task second vector about each token of the document vector is obtained.

[0108] The attention weight corresponding to the token is multiplied by the value vector and added to obtain the final single head vector expression Head i ; the calculation process is represented as:

[0109]

[0110] All Head i are concatenated, multiplied by the spatial weighting weight W, and the output vector V3 is obtained, represented as:

[0111] V3 = concat(Head1; …; Head i )·W

[0112] Wherein, V3 is the third vector of the retrieval task, and concat is a vector concatenation operation.

[0113] S8, the third vector of the retrieval task is concatenated with the text classification vector, multiplied by the task weighting weight, and the unnormalized logarithmic probability of the calculation score is obtained. The floating point score is obtained by mapping using the Sigmoid function. The floating point score is the relevance calculation score. Specifically, it is represented as:

[0114] Score=Sigmoid(W final ·Concat([cls]′;V3))

[0115] Wherein, W final is a task weighting weight randomly initialized by normal distribution, [cls]′ is a text classification vector, V3 is the third vector of the retrieval task, Score is a floating point score, and the value is between 0 and 1. Concat is a vector concatenation operation.

[0116] S9, loss function calculation, specifically including: using the binary cross-entropy loss function BCEloss of probability calculation, the calculation formula is:

[0117]

[0118] Wherein, L i is the gear identification of the document, and the value is between 0 and 1. S i is the floating point score Score output by the relevance model, and N is the sample quantity.

[0119] Some embodiments disclose a retrieval relevance model based on sentence vector cycle, which is obtained by the retrieval relevance model construction method disclosed by the embodiments of the present application.

[0120] The retrieval relevance model based on the sentence vector cycle disclosed by the embodiment of the application is a single tower structure model. In the case that the query and the document are fully interacted through the pre-training model, a new task-specific task embedding is introduced, the query is summarized, the query is summarized into a single token vector, that is, a query-specific embedding, and then the query-specific embedding is used to search the document to calculate the relevance of the document and the query-specific embedding and give the corresponding relevance score. The overall design refers to the sequence input method of the recurrent neural network RNN, and the overall sentence vector is used as the input of each time step. The sentence vector calculation method at each cycle time step is the cross attention mechanism, which can calculate the weight of each token vector in the sentence and the relevance task, and then obtain the final vector expression.

[0121] The length of the query is fixed, the title Title and the abstract Snippet of the document doc are spliced into the encoder Encoder-only retrieval pre-training model, the query and the document doc pass through the self-attention coding module self-attention of each layer of the basic model at the same time, and can express each other. Under the same computing resources (such as a graphics processing unit GPU), the single tower model can better save the computing resources, and the query and the document are interacted in advance at the input end, the weight expression calculation of the self-attention between the two is performed in advance, and the pre-training semantic space of the retrieval pre-training model can be used to the greatest extent.

[0122] The classification layer refers to the sequence modeling method of the recurrent neural network RNN, and the elements are replaced by sentence vectors that can better express global information instead of single tokens. This method can generate subsequent elements according to the previous elements, discards the interaction matrix of the query vector and the document vector, reduces the space occupation of the GPU memory during the calculation, is more intelligent in capturing local information, and is also beneficial to the embedding of more NLP technologies, such as term weight information.

[0123] The new random initialization task-specific embedding is introduced as the first vector of the retrieval task, which is specifically used for query and document matching. Since the pre-training model does not match the task form of the downstream task, the interference of the pre-training model on the downstream task can be reduced, and the word embedding information given by the pre-training task can be used for relevance model relevance score calculation.

[0124] The cross-attention mechanism is used to calculate the intermediate hidden states of the classification layer. Compared with the convolutional neural network CNN, this method focuses more on global information, and the model can obtain the weight size of each token in the query query for the relevance task itself through data training. The use of cross-attention mechanism can enhance the expression ability of the model by calculating the different weights of each token in the query, capture the complex relationship between input data, and make the model have stronger generalization ability. It can learn to extract relevant information from different inputs to help the model better generalize to unseen data. Moreover, the cross-attention mechanism has better interpretability, which can intuitively show how the model focuses on the importance weight of each token in the query and document, facilitating better evaluation of the learning degree of the model when reasoning.

[0125] The cross-attention mechanism is used to calculate the intermediate hidden states of the classification layer. Compared with the convolutional neural network CNN, this method focuses more on global information, and the model can obtain the weight size of each token in the query query for the relevance task itself through data training. The use of cross-attention mechanism can enhance the expression ability of the model by calculating the different weights of each token in the query, capture the complex relationship between input data, and make the model have stronger generalization ability. It can learn to extract relevant information from different inputs to help the model better generalize to unseen data. Moreover, the cross-attention mechanism has better interpretability, which can intuitively show how the model focuses on the importance weight of each token in the query and document, facilitating better evaluation of the learning degree of the model when reasoning.

[0126] In terms of optimization, the BCE loss is used as a probability calculation loss to convert the relevance of the query query and the document doc into a probability value between 0 and 1. This can directly optimize the probability value output by the model, so that the output learned by the model is close to the probability representation in the true probability distribution. It is more sensitive to small changes in probability without losing calculation stability or imposing excessive penalties on extreme values, which helps the model to more accurately predict the probability of the current example.

[0127] The technical solutions disclosed in the embodiments of the present application and the technical details disclosed in the embodiments are only exemplary to illustrate the inventive concept of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application, and any conventional changes, substitutions or combinations of the technical details disclosed in the embodiments of the present application are within the protection scope of the claims of the present application.

Claims

1. A retrieval relevance model construction method based on sentence vector loop, characterized by: include: S1. Obtaining a query and a corresponding document; the document includes a title of a web page and content extracted from the web page; S2. Mark the file position; S3, generate sequence text based on query and document; S4. Input the sequence text into the pre-trained model to obtain a vectorized representation of the sequence text, and obtain a text classification vector, a query vector, a title vector, a title end vector, a content vector, and a content end vector; S5. Create a new vector of the hidden layer dimension of the aligned pre-trained model output as the first vector of the retrieval task; S6. Perform a cross-attention interaction between the first retrieval task vector and the query vector: use the first retrieval task vector as the query of the cross-attention mechanism, and use the query vector as the key and value of the cross-attention mechanism to obtain the second retrieval task vector. S7. Perform a cross-attention interaction between the second retrieval task vector and the document vector: concatenate the title vector and the content vector into a document vector, use the second retrieval task vector as the query of the cross-attention mechanism, and use the document vector as the key and value of the cross-attention mechanism to obtain the third retrieval task vector. S8. Concatenate the third retrieval task vector and the text classification vector, multiply by the task weight, and obtain the unnormalized logarithmic probability of the calculation score. Use the Sigmoid function to map to obtain a floating-point score, which is the relevance calculation score. Specifically expressed as: Score=Sigmoid(W final ·Concat([cls]′;V3)); Among them, W final is the task weight randomly initialized from the normal distribution, [cls]′ is the text classification vector, V3 is the third vector of the retrieval task, Score is a floating point score with a value between 0 and 1, and Concat is a vector concatenation operation.

2. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 1, characterized in that: In step S2, the query and the title and content of the document are marked and divided into four levels: excellent, medium, poor and irrelevant, and the level identifier corresponding to the document is obtained, and the level identifier is a value between 0 and 1.

3. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 2, characterized in that: In step S3, the generated sequence text is represented as: [cls] query [sep] title [sep] content [sep]; at the same time, different text type IDs are assigned to [cls], query [sep], title [sep], and content [sep] to distinguish different text types; Among them, [cls] is the sequence semantic symbol, and [sep] is the sequence end symbol.

4. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 3, characterized in that: In step S4, the sequence text [cls] query [sep] title [sep] content [sep] is input into the retrieval pre-training model to obtain the vectorized representation of the sequence text, namely the text classification vector [cls]', the query vector query', the title vector title', the title end vector [sep1]', the content vector snippet', and the content end vector [sep2]'.

5. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 4, characterized in that: In step S6, the cross attention mechanism is a multi-head cross attention mechanism, which includes three parts: query sequence, key sequence, and value sequence. Step S6 specifically includes: Initialize the weights W of the normal distribution q 、W k 、W v ,W; Calculate Q, K, and V according to the following formula: Q=W q ·V1 K=W k ·query′ V=W v ·query′ Among them, Q is the query vector, K is the key vector, V is the value vector, V1 is the first vector of the retrieval task, and W q is the weight of Q, W k is the weight of K, W v is the weight of V; query′ is the query vector; Divide Q, K, and V into multiple heads, here n heads, and calculate the attention of each head separately; the calculation method includes: The dot product of Q and K is divided by the square root of the hidden dimension d to calculate the similarity score between the first vector of the retrieval task and each word in the query vector; After softmax processing, the attention weight of the first vector of the retrieval task to each word in the query vector is obtained; Multiply the attention weight of the corresponding word segment by the value vector V and add them together to get the final single-head vector expression Head i ; The single head calculation process is expressed as: All single head vectors Head i Splicing, multiplying by the spatial weighted weight W, obtains the output vector V2, which is expressed as: V2=concat(Head1;…;Head i )·W Among them, V2 is the second vector of the retrieval task, and concat is the vector concatenation operation.

6. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 5, characterized in that: Step S7 specifically includes: Initialize the weights W of the normal distribution q 、W k 、W v ,W; Calculate Q, K, and V according to the following formula: Q=W q V2 K=W k ·concat(title′;snippet′) V=W v ·concat(title′;snippet′) Among them, Q is the query vector, K is the key vector, V is the value vector, V2 is the second vector of the retrieval task, and W q is the weight of Q, W k is the weight of K, W v is the weight of V; title' is the title vector, snippet' is the content vector; concat is the vector concatenation operation; Divide Q, K, and V into multiple heads, here n heads, and calculate the attention of each head separately, including: The dot product of Q and K is divided by the square root of the hidden dimension d to calculate the similarity score between the second vector of the retrieval task and each word in the document vector; After softmax processing, the attention weight of the second vector of the retrieval task on each word in the document vector is obtained; Multiply the attention weight of the corresponding word segmentation with the value vector and add them together to get the final single head vector expression Head i ; The calculation process is expressed as: All Head i Splicing, multiplying by the spatial weighted weight W, obtains the output vector V3, which is expressed as: V3=concat(Head1;…;Head i )·W Among them, V3 is the third vector of the retrieval task, and concat is the vector concatenation operation.

7. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 6, characterized in that: The method further includes step S9, loss function calculation, which specifically includes: using a binary cross entropy loss function BCEloss calculated by probability, and the calculation formula is: Among them, L i It is the file's gear mark, with a value between 0 and 1. i is the floating-point score Score output by the correlation model, and N is the number of samples.

8. The method for constructing a retrieval relevance model based on sentence vector loop according to claim 1, characterized in that: Step S3 includes: Fixed query length; The query, document title, and summary are fed into a base model with only an encoder to interactively generate sequence text.

Images