A long document retrieval method based on memory network semantic fusion

Abstract

The application relates to a long document retrieval method based on memory network semantic fusion, solves the technical problem of how to maximize the retention and fusion of semantic representations of different paragraphs in limited input in long document retrieval, and belongs to the technical field of information retrieval. The method uses the powerful context semantic modeling capability of a pre-trained language model to enrich the semantic information of user query vector representation and candidate document vector representation, and better realizes semantic matching. The method uses a memory network to store vector representations of each segment of a document as memory units, dynamically reads the memory units by using an attention mechanism for a user query, fuses semantic vectors between different paragraphs, can effectively extract relevant information from long documents, gets rid of the forgetting problem of a sequence model, and improves the user retrieval quality under a long document task.

Description

Technical Field

[0001] This invention relates to a long document retrieval method based on semantic fusion of memory networks, belonging to the field of information retrieval technology. Background Technology

[0002] Long document retrieval is a fundamental task in the field of information retrieval. Its characteristics include: candidate document texts have a relatively long average length, often containing multiple topics across different paragraphs, while the user's search intent may only be strongly relevant to a few paragraphs within the document. Given a user query, long document retrieval models need to accurately retrieve the most relevant target document from a large pool of candidate long documents. It is frequently applied in scenarios such as document retrieval and legal document retrieval.

[0003] Traditional search engines (such as BM25) are based on the Binary Independence Model (BIM), which assumes that text can be represented by the presence or absence of a particular word, and that the occurrence of different words in the text is independent of each other. When combined with word segmentation techniques and inverted indexes, this approach can achieve fast retrieval in large-scale corpora. However, this method ignores the contextual dependencies between different words within real-world text, overemphasizes an exact match between the user query and candidate documents, and lacks semantic modeling. Therefore, its retrieval performance is poor when the user query appears in natural language rather than keywords.

[0004] Some researchers have used sequence models such as RNNs and GRUs to fuse different paragraphs of long documents together based on their inherent logical order. However, sequence models like RNNs and GRUs suffer from forgetting problems when the sequence length is long, making it impossible to establish long-distance dependencies. Furthermore, compressing long texts into fixed-length vectors inevitably leads to information loss.

[0005] In recent years, the widespread adoption of pre-trained language models (PRMs) has led to rapid advancements in semantic matching. Their powerful contextual modeling capabilities and convenient transferability have helped retrieval models move beyond precise matching, enabling them to better assess the semantic similarity between user queries and candidate documents. However, in long document retrieval, the length of candidate documents far exceeds the input length limit of pre-trained language models, making it impossible to input the entire document into the model at once. Therefore, how to preserve and integrate the semantic representations of different paragraphs to the greatest extent possible within a limited input is a major problem and challenge facing long document retrieval. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies and to solve the technical problem of how to retain and integrate the semantic representations of different paragraphs to the greatest extent possible in long document retrieval with limited input, thereby improving the accuracy of user retrieval in long document retrieval tasks. This invention creatively proposes a long document retrieval method based on semantic fusion of memory networks.

[0007] The innovation of this invention lies in the following: a method for semantic fusion between different paragraphs is achieved by using a memory network. The memory network directly stores the semantic vector representations of different paragraphs, and the user query is iteratively updated through an attention mechanism. This effectively extracts content related to a given user query from long texts, thereby improving the accuracy of user retrieval in long document retrieval tasks.

[0008] To achieve the above objectives, the present invention adopts the following technical solution.

[0009] First, the data is preprocessed, and a recall algorithm is used to filter documents that may be relevant to the user query, thereby reducing the computational load of the model in subsequent stages. Then, candidate documents are segmented and encoded, with each segment serving as a unit in a memory network. The purpose is to divide long candidate documents. Next, a pre-trained language model is used to encode the user query, obtaining a semantic vector representation of the user query. In the semantic fusion stage based on the memory network, parts relevant to the user query in different storage units are selected, and the user query is updated and expanded. Then, iterative updates are performed to obtain accurate user query information. A multilayer perceptron model is used to score positive and negative samples in the candidate documents, and the model parameters are trained using a loss function. After model training is complete, for a given user query, positive and negative samples are no longer distinguished; each document in the candidate document set is predicted and scored, and the documents are sorted from highest to lowest score. The documents with the highest scores are returned as the final retrieval results.

[0010] Beneficial effects

[0011] The method of the present invention has the following advantages compared with the prior art:

[0012] 1. This method can leverage the powerful contextual semantic modeling capabilities of pre-trained language models to enrich the semantic information of user query vector representations and candidate document vector representations, thereby achieving better semantic matching.

[0013] 2. This method uses a memory network to store the vector representations of each segment of a document as memory units. For user queries, it dynamically reads memory units using an attention mechanism and fuses semantic vectors between different paragraphs. This can effectively extract relevant information from long documents, overcome the forgetting problem of sequence models, and improve the user retrieval quality for long document tasks. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention.

[0015] Figure 2 This is a flowchart illustrating the specific implementation of the document / query encoding method of the present invention. Detailed Implementation

[0016] The method of the present invention will be further described in detail below with reference to the accompanying drawings.

[0017] like Figure 1 As shown, a long document retrieval method based on memory network semantic fusion includes the following steps:

[0018] Step 1: Data Preprocessing. Recall algorithms can be used to filter out documents that may be relevant to the user's query, thereby reducing the computational load on the model in subsequent stages.

[0019] Specifically, step 1 includes the following steps:

[0020] Step 1.1: For a given set of user queries, filter out special characters in the queries and segment the candidate document set.

[0021] Regular expressions can be used to filter special characters in user queries.

[0022] Step 1.2: For each given user query, rank all candidate documents in descending order of similarity, and select the top F documents with the highest similarity (the number of F is determined according to the query requirements; for example, the top 100 documents with the highest similarity can be selected) as the candidate document set for the next stage.

[0023] For example, the BM25 algorithm can be used to calculate the similarity between a user query and each document in a candidate document set, and return the top 100 documents with the highest similarity scores.

[0024] Step 1.3: For each given user query, select the relevant and irrelevant documents from the candidate document set, and use them as positive and negative samples, respectively.

[0025] For example, given a user query, one document is randomly selected from the set of fact-related candidate documents as a positive sample, one document is randomly selected from the set of fact-irrelevant candidate documents as a negative sample, and one document is randomly selected from the set of documents returned by the BM25 algorithm as a hard negative sample. The three are concatenated together and used as input to the subsequent model in the form of a triple.

[0026] Step 2: Segment the candidate document. The purpose is to divide the candidate long document into segments. Each segment is used as a unit of the memory network.

[0027] like Figure 2 As shown, specifically, step 2 includes the following steps:

[0028] Step 2.1: Starting from the beginning of the document, select a fixed length of text. The selected text is divided into segments of equal length, with overlapping portions of equal length between each segment.

[0029] Step 2.2: Encode each segment using a pre-trained language model to obtain a vector representation of each segment containing contextual semantics. Each segment serves as a storage unit of a memory module in the memory network.

[0030] Step 3: Encode the user query using a pre-trained language model to obtain the semantic vector representation of the user query.

[0031] The pre-trained language model can be the BERT model. BERT includes embedding layers and multiple stacked transformer layers, each transformer layer including a multi-head attention part and a forward propagation network part. The encoding of the pre-trained language model refers to the weighted average of the vector representations of each character output by the last two transformer layers of BERT; the sum of the two layers yields the final vector representation of the input text.

[0032] Step 4: Semantic fusion stage based on memory network, filtering the parts of different storage units that are relevant to the user query.

[0033] Step 4 can be implemented using the following methods:

[0034] By utilizing an attention mechanism, the similarity between the user query and different paragraphs is calculated, and this similarity is used as weights to fuse the semantic vector representations of multiple paragraphs together, resulting in the semantic vector representation of the document.

[0035] α i =sim(q) k ,p i (1)

[0036]

[0037] Where, sim(q,p) i Let q be the similarity function, which measures the similarity of user queries in the k-th iteration. k With the i-th paragraph p of the text i The similarity between them is α. i ; m is the number of storage units, O k Let be the semantic vector representation of the document obtained in the k-th iteration.

[0038] The similarity function sim(q,p) is specifically: sim(q,p) = v T tanh(W[q;p]), where v and W are training parameters, tanh is the activation function, T is the transpose, and q and p are the user query vector and the candidate paragraph representation vector, respectively.

[0039] Step 5: Update the user query vector. Combine the text representation vector obtained in Step 4 to enrich the semantics of the user query vector.

[0040] Step 5 can be implemented using the following methods:

[0041] User query q k The semantic vector representation of the document obtained by the k-th iteration O k Combined together, the updated document vector is output:

[0042] O k+1 =W1q k +W2O k +b (3)

[0043] Among them, O k+1 W1 is the semantic vector representation of the document obtained in the (k+1)th iteration; W1 is the vector representation of q. k Linear terms mapped to the same semantic space, W2 represents the O k Linear terms mapped to the same semantic space, b is the result of q k O k Bias terms mapped to the same semantic space; W1, W2, and b are all learning parameters.

[0044] The obtained document semantic vector is represented as O k+1 As a new user query vector: q k+1 =O k+1 .

[0045] Step 6: Perform iterative updates to obtain accurate user query information.

[0046] Specifically, repeat steps 4 to 5 to obtain the final document vector representation O. n n represents the total number of iterations.

[0047] Step 7: Using a multilayer perceptron model, score the positive and negative samples in the candidate documents respectively, and train the model parameters using a loss function.

[0048] Specifically, step 7 can be performed using the following methods:

[0049] Step 7.1: For a given user query, the multilayer perceptron model scores the positive and negative samples respectively:

[0050] spos =MLP([q;O pos (4)

[0051] s neg =MLP([q;O neg (5)

[0052] Among them, s pos s neg These are the similarity scores between the user query vector and the positive and negative samples, O pos O neg These are vector representations of relevant and irrelevant documents, respectively.

[0053] MLP([q;O]) is a multilayer perceptron model responsible for calculating the score between the user query vector and the candidate document vector, where q represents the user query vector and O represents the candidate document vector.

[0054] Furthermore, MLP([q; O]) concatenates q and O together and then feeds them into the multilayer perceptron, as shown in the following equation:

[0055] MLP(q;O)=sigmoid(W2(RELU(W1[q;O]+b1)+b2) (6)

[0056] Here, sigmoid and ReLU are activation functions, and W1, W2, b1, and b2 are learning parameters.

[0057] Step 7.2: Train the model using the loss function.

[0058] The model requires the score difference between positive and negative samples to be as large as possible, and uses this as the loss function to update the parameters of each part of the model, as shown in the following formula:

[0059]

[0060] Where Loss is the difference between the model's predicted sample relevance score and the sample's actual relevance score, margin is the threshold of the loss function, and N is the total number of samples.

[0061] During the backpropagation of the loss, the Adam algorithm can be used to update the model parameters.

[0062] Step 8: After completing the model training, for a given user query, no longer distinguish between positive and negative samples, predict the score for each document in the candidate document set, sort them from high to low score, and return the top G documents with the highest scores (e.g., return the top 10 documents with the highest scores) as the final retrieval results.

Claims

1. A long document retrieval method based on memory network semantic fusion, characterized in that, Comprising the following steps: Step 1: screening out documents that may be related to user queries; Step 1.1: for a given set of user queries, filtering special characters in the queries and segmenting the set of candidate documents; Step 1.2: for each given user query, ranking all candidate documents in descending order of similarity, and selecting the top F documents as the candidate document set for the next stage; Step 1.3: for each given user query, selecting relevant and irrelevant documents in the candidate document set as positive and negative samples, respectively; Step 2: segment encoding of candidate documents, with each segment obtained as a unit of the memory network; Step 3: encoding user queries using a pre-trained language model to obtain semantic vector representations of user queries; Step 4: semantic fusion stage based on the memory network, screening out relevant parts of different storage units and user queries; Using attention mechanism, calculate the similarity between user query and different paragraphs, and use it as weight to fuse the semantic vector representations of multiple paragraphs together to get the semantic vector representation of the document: a i = sim(q k , p i ) (1) where sim(q, p i ) is a similarity function, which measures the similarity between the user query q k and the i-th paragraph p i of the text in the k-th iteration, and the result is α i ; m is the number of storage units, O k is the semantic vector representation of the document obtained in the k-th iteration; The similarity function sim(q, p) is: sim(q, p) = v T tanh(W[q; p]), where v, W are training parameters, tanh is an activation function, T is a transpose, q, p are user query vector and candidate paragraph representation vector respectively; Step 5: update user query vector; Enrich the semantics of the user query vector by combining the text representation vector obtained in step 4, as follows: User query q k The semantic vector representation of the document obtained by the k-th iteration O k Combined together, the updated document vector is output: O k+1 = W1q k + W2O k + b (3) wherein O k+1 is the document semantic vector representation obtained in the k+1th iteration; W1 is a linear term mapping q k to the same semantic space, and W2 is a linear term mapping O k to the same semantic space, and b is a bias term mapping q k , O k to the same semantic space; W1, W2, and b are all learning parameters. The resulting document semantic vector representation O k+1 as a new user query vector: q k+1 = O k+1 ; Step 6: repeat steps 4 to 5 to iteratively update and obtain accurate user query information; Step 7: use a multi-layer perception model to score positive and negative samples in the candidate document, and train the model parameters through a loss function; Step 7.1: for a given user query, the multi-layer perception model scores positive and negative samples: s pos = MLP([q; O pos ]) (4) s neg = MLP([q; O nge ]) (5) where s pos , s neg are the similarity scores between the user query vector and the positive and negative samples, respectively, O pos , O neg represent the vector representations of the relevant and irrelevant documents, respectively. MLP([q;O]) is a multi-layer perception model responsible for calculating the score between the user query vector and the candidate document vector, q represents the user query vector, and O represents the candidate document vector; MLP([q;O]) concatenates q and O and then puts them into a multi-layer perception, as shown in the following formula: MLP(q;O) = sigmoid(W2(RELU(W1[q;O] + b1) + b2) (6) where sigmoid and RELU are activation functions, and W1, W2, b1, and b2 are learning parameters; Step 7.2: train the model through a loss function; Specifically as follows: where Loss is the difference between the model's predicted sample relevance score and the sample's actual relevance score, margin is the loss function threshold, and N is the total number of samples; Step 8: after completing the model training, for a given user query, without distinguishing between positive and negative samples, predict the score of each document in the candidate document set, and sort them in descending order of score, returning the top G documents with the highest scores as the final retrieval results.

2. The long document retrieval method based on memory network semantic fusion according to claim 1, wherein, Step 2 includes the following steps: Step 2.1: from the beginning of the document, select a fixed length of text from the document, where the selected text is evenly segmented by length, and there is an overlap of the same length between each segment; Step 2.2: encode each segment using a pre-trained language model to obtain a vector representation of each segment containing contextual semantics, each segment as a storage unit in the memory module in the memory network.

3. The long document retrieval method based on memory network semantic fusion according to claim 1, characterized in that, In step 3, the pre-trained language model adopts a BERT model; Wherein, the BERT includes an embedding layer and a plurality of stacked transformer layers, and each transformer layer includes a multi-head attention part and a forward propagation network part. The encoding of the pre-trained language model refers to the weighted average of the vector representation of each character output by the last two transformer layers of the BERT, and the sum of the two layers is the final vector representation of the input text.

Images