Dense information retrieval method and system for long text

Abstract

In order to overcome the problems of long time-consuming and high cost of prior art retrieval, and the problem of insufficient correctness of the results after the existing retrieval scheme is divided into sentences, a dense information retrieval method and system for long text are provided. By filling the text to the same character amount and then converting it into a vector representation through BERT, and then compressing it, the features of the words are simplified while maintaining their semantic relationships, making the retrieval more focused on semantic relationships and reducing the size of the data. The retrieval system implemented through offline storage has a faster processing speed for retrieval requests than the prior art and is also more secure.

Description

Technical Field

[0001] This invention relates to the field of information retrieval methods, and in particular to a dense information retrieval method and system for long texts. Background Technology

[0002] In recent years, the emergence of large-scale pre-trained language models (such as ELMo, OpenAI GPT, and BERT) has continuously refreshed the evaluation results of various tasks in natural language processing. These pre-trained language models can estimate the relevance between texts well through fine-tuning. Among them, BERT is the most representative pre-trained language model. In less than a year since its emergence, many ranking models based on BERT have achieved state-of-the-art results on various retrieval benchmarks. This benefits from its ability to compute the semantic interaction of deep context between two text pairs to obtain semantic matching between texts, bridging the common lexical mismatch problem between documents and queries. The input for the BERT retrieval engine to perform interactive semantic matching is a connection between a question and each retrieved document, i.e., "[CLS]Query[SEP]Document[SEP]". The probability of relevance between each paragraph and the query is obtained from CLS. Because BERT adopts a transformer-based architecture, its memory and time consumption increase quadratically with the increase of input length. Therefore, the design of the BERT model limits the input text length to a maximum of 512 terms.

[0003] Current solutions for long-text information retrieval share some common characteristics: they all employ interactive semantic matching to retrieve long texts. For each ad-hoc query, the interaction with each document and even each paragraph must be calculated, resulting in long processing times and high costs. Typically, in real-world scenarios, a user waiting time exceeding 100ms for a query will negatively impact the user experience. For tasks involving a large number of documents, substantial information, and high repetition, such as plagiarism checks on large batches of papers, the time-consuming and costly nature of existing technologies becomes even more pronounced in terms of project timeline and cost.

[0004] Chinese patent publication CN202110815723.7, entitled "Long Text Classification Method, System, Electronic Device, and Computer-Readable Storage Medium," discloses a method for segmenting long texts according to textual patterns (paragraph types) to meet the input byte requirements of BERT. The problem with this method is that the text segmentation method itself introduces significant data bias. The data bias generated by such methods largely depends on the rationality of the chosen theory, which directly affects the model's judgment. Therefore, the results obtained by such methods in practical applications are difficult to accept. However, this method is highly practical for standardized documents. Summary of the Invention

[0005] To overcome the problems of long retrieval time and high cost in existing technologies, as well as the insufficient accuracy of results after sentence segmentation in existing retrieval schemes, a dense information retrieval method and system for long texts are provided. The dense information retrieval method for long texts is as follows:

[0006] S1: Train the model using text that meets the input length requirements of the BERT model, along with query-related training samples.

[0007] The text length that meets the input length requirements of the BERT model is less than or equal to 512 characters.

[0008] The query relevance training samples are in the form of triples:

[0009] <Q′,Passage + Passage - >

[0010] Passage + Passage represents text manually marked as relevant to the query. - This represents the top n articles that are not labeled as relevant. In this model, Passage + As a positive example, Passage - This is a negative example.

[0011] S2: The retrieved text is segmented into a segmented context encoding representation using the trained model. The segmented text is then cut into segments with a length less than 512. These segments are then input into the BERT model trained in step 1 to obtain the segmented encoding representations of the text.

[0012] S3: Take the query text and use the trained model to obtain the BERT encoded representation of the query.

[0013] S4: Interact with the encoded representation of the query and the encoded representation of the long document to obtain the query results.

[0014] As a preferred option, S2 also includes

[0015] S2.2 is performing vector compression.

[0016] The latent vectors of the last layer of BERT are compressed, concatenated, and then stored sequentially in the representation of each document according to their document numbers. The representation of document d is E. d The calculation method is as follows:

[0017]

[0018] Among them, E d The first dimension has a size of k×Ld , where represents the total length of the k paragraphs. E d The second dimension is dim, which represents the dimension of each word vector after compression (768 before compression). i p represents the i-th segment after segmentation. i ∈d, i∈{1, 2, .., k}. tok i，j The vector representation of the j-th word in the i-th paragraph of document d.

[0019] Similarly, when the length len(d) of a document does not exceed L d When using [mask], padding is applied up to a length of L. d The document representation process is as follows:

[0020]

[0021]

[0022] Among them, tok i This represents the vector representation of the i-th word in the document.

[0023] Correspondingly, S3 also includes the step of compressing the query statement through the same linear layer.

[0024] S3.1, the query statement vector is compressed through a linear layer to obtain the encoded representation of the query:

[0025] The output of BERT is then compressed through a linear layer to obtain the representation of query Q as E. Q The calculation method is as follows:

[0026]

[0027] Among them, E Q The size is L Q ×din. Where dim is the dimension of each word vector after compression (768 before compression).

[0028] The vector compression methods in S2 and S3 reduce the memory usage during computation. Since the required server memory size is a significant factor limiting the practical application of a product, this method saves costs, reduces server requirements, and provides economic benefits.

[0029] As a preferred option, S2 also includes

[0030] The encoded representation of this long text is stored offline.

[0031] Correspondingly, in S4, the process of interacting between the encoded representation of the query and the encoded representation of the long document also includes retrieving the encoded representation of the long text stored offline in S2 from the data stored offline.

[0032] Offline computation can reduce the runtime of the document encoding representation during querying, but existing techniques consume a lot of memory and are difficult to apply because the dimension of each word embedding in the representation of long text is quite large (e.g., 768 floating-point values ​​per token in BERT-base).

[0033] By combining the vector compression methods of S2 and S3, this scheme reduces the size of offline storage data, reduces hardware requirements, lowers the hardware cost of offline use of the model, reduces memory usage, improves the working time efficiency of the model, and effectively utilizes existing computing power and memory.

[0034] Preferably, in S2, when segmenting these long texts, for a document d, the document length is len(d). Let the segmentation size be L. d If the document length exceeds L d This will split the document. For documents with a length not exceeding L... d The text is then padded using a special marker [mask]. Assuming a document's length len(d) exceeds L... d The document will then be divided into a series of L... d A segment of length d is denoted as d = {p1, p2, ..., p...} k}. Where p i p represents the i-th segment after segmentation. i ∈d, i∈{1, 2, .., k}. tok i，j This represents the vector representation of the j-th word in the i-th paragraph of document d. The start of each paragraph input is marked with [D].

[0035] Length L d The text of the fragment is represented as:

[0036]

[0037] Length less than L d The text of the fragment is represented as:

[0038] [D], tok1, tok2, ..., tok len(d) ,[sep],[mask],...,[mask]

[0039] The text undergoes a standardization process, padding shorter texts with special markers [masks] to ensure all resulting segmented longer texts are of the same length. This approach adapts well to Transformer networks, yielding more accurate results.

[0040] The standardization process of filling in special markers [mask] makes the result of vector compression controllable. While reducing the dimension of each vector, the overall semantic structure of the paragraphs is preserved after compression because each paragraph has the same length and the word vectors have the same relationship. The details lost by this compression method have little impact on the query task; on the contrary, it makes the information needed for the query cleaner and reduces the risk of model overfitting.

[0041] Preferably, in S3.1, the BERT encoding representation of the query statement is obtained as follows:

[0042] First, the query terms in the query are decomposed and denoted as q1, q2, ..., q m The starting position of the input is marked with [Q]. Since queries are typically much shorter than documents, they may lack information, making matching between queries and documents difficult. However, query expansion helps find relevant documents better by supplementing the missing information in the query.

[94] This paper proposes a query expansion strategy: Let the maximum length of the query be L. Q For insufficient L Q The query uses a query expansion strategy, repeating the query terms once and marking the end with [Sep]. If less than L remains... Q The positions are filled with special markers [mask] until the length reaches L. Q For lengths exceeding L Q For this query, you will input the first L of the query. Q 10 terms. For more than L Q However, portions of the input that are less than the length of BERT will be padded with a special marker [mask].

[0043] As a preferred option, in S4

[0044] The process of interacting between the encoded representation of the query and the encoded representation of the long document includes...

[0045] S4.1: Obtain the word vector in each long document encoded representation segment that has the closest cosine similarity to each query word vector.

[0046] This fine-grained model obtained all words in the fragment that had the highest similarity to the query value.

[0047] S4.2: Calculate the representation of the fragment and calculate the match value between the query and the individual fragment representations.

[0048] S4.3: Calculate the total score of the document.

[0049] Furthermore, the specific calculation method for the segment representation in S4.2 is as follows:

[0050] The average of the S4.1 closest word vectors is used as the representation of the segment, and the average word embedding of the query term is used as the representation of the query. The semantic matching value between the query and a single segment is calculated as follows:

[0051]

[0052] in, It is a subset of the document's word vectors, representing the set of word vectors that are closest to the word vector of each query. q e represents the vector representation of query term q in Q. m For the word vector e in document d that matches the query q The vector representation of the term with the highest cosine similarity.

[0053] This method extracts sentences most likely to be similar to query objects from long documents and calculates matching values ​​by comparing the extracted encoded representations with the encoded representations of the queries. This method focuses more on extracting semantic structural relevance, resulting in higher accuracy and better adherence to language structure.

[0054] By combining the vector compression methods in S2 and S3, the drawback of losing semantic details of words in the compression method is avoided, highlighting the relevance of the extracted semantic structure, making the loss caused by compression have a smaller impact on the results, and the data processing results are more in line with the query requirements.

[0055] Furthermore, the specific calculation method for the total document score in S4.3 is as follows:

[0056] Calculate the final score for document d, and take the maximum score of the query and each segment of the document as the semantic relevance score of the document. The calculation method is as follows:

[0057]

[0058] Where d = {p1, p2, ..., p} k}, fragment p in d i ∈d, i∈{1, 2, .., k}.

[0059] The loss function used when training the model on short text data is as follows:

[0060]

[0061] Where y(Q, p) is the relevance label between the paragraph and the query. Score(Q, p) represents the final score of document d.

[0062] During model training, the learning speed is faster when the performance is poor and slower when the performance is good, which helps to find the global optimum more quickly. Furthermore, because this loss function employs an inter-class competition mechanism, it is better at learning information between different categories.

[0063] The advantages of this approach are that it eliminates the need for conventional methods that adjust the model structure (such as splitting it in certain ways) to accommodate the complex encoding operations of long texts, making it more suitable for modern retrieval needs that pursue high efficiency and more accurate semantic matching. Its data processing avoids the biases inherent in manual splitting, helping to ensure the validity of the final results, and it uses positive and negative correlation loss functions to improve the recognition ability of the fine-tuned model. Furthermore, data compression results in smaller storage and memory requirements and improves processing speed.

[0064] The present invention also provides a dense information retrieval system for long texts, the system including an interactive terminal, a server, and a storage medium.

[0065] The system deployment is as follows: the BERT semantic model is trained using S1, a dense information retrieval method for long texts; then, a large amount of long texts to be retrieved are converted using S1 and stored offline in a storage medium.

[0066] The retrieval process of this system is as follows: the query statement is input through the interactive terminal, and the selection range of the long text to be retrieved is limited. After the server receives the query, it transforms the query statement through S3 in the dense information retrieval method for long text. Then, the server retrieves the representation of the long text within the selected range from the storage medium in a sequential batch and obtains the query results through S4 in the dense information retrieval method for long text.

[0067] Preferably, the storage medium utilizes cloud storage, allowing the system to retrieve long text representations in batches from different cloud servers, making it suitable for large-scale deployment. This step, through compression optimization, reduces data size, resulting in lower transmission costs and making it more suitable for network deployment compared to existing technologies. Because the vector representation after linear compression is almost completely indistinguishable, the system also improves security, avoiding the risk of storing text data on third-party servers and reducing the possibility of hackers intercepting and parsing search content.

[0068] The beneficial effects of this solution are that it provides an efficient dense information retrieval system and utilizes highly available and scalable storage media, enabling the system to be rapidly deployed at various scales based on demand. Detailed Implementation

[0069] Example 1

[0070] The embodiments of the present invention include the following steps:

[0071] S1: Train the model using text that meets the input length requirements of the BERT model, along with query-related training samples.

[0072] The text length that meets the input length requirements of the BERT model is less than or equal to 512 characters.

[0073] The training dataset used in this embodiment is triplet data from MS MACRO.

[0074] The query relevance training samples are in the form of triples:

[0075] <Q′,Passage + Passage - >

[0076] Passage + Passage represents text manually marked as relevant to the query. - This represents the top n articles that are not labeled as relevant. In this model, Passage + As a positive example, Passage - For a negative example, in this embodiment, the value of n is between 50 and 500, while the sample Passage - The quantity should be controlled below 100.

[0077] This contrastive learning method provides the model with both negative and positive examples, improving the model's recognition ability and making the fine-tuned model more accurate.

[0078] S2: The retrieved text is processed by the trained model to obtain a segmented contextual encoding representation, and this encoding representation is stored, including:

[0079] S2.1 Segment these long texts, ensuring the length of the segmented text is less than 512. Input each segment into the BERT model trained in step 1 to obtain the encoded representation of the text segments. In this embodiment, some identifiers, such as [D] and [SEP], need to be input, which will occupy some length. Therefore, the actual length of the segmented text is approximately 500.

[0080] When segmenting these long texts, for a document d, the document length is len(d). Let the segment size be L. d If the document length exceeds L d This will split the document. For documents with a length not exceeding L...d The text is then padded using a special marker [mask]. Assuming a document's length len(d) exceeds L... d The document will then be divided into a series of L... d A segment of length d is denoted as d = {p1, p2, ..., p...} k}. Where p i p represents the i-th segment after segmentation. i ∈d, i∈{1, 2, .., k}. tok i，j This represents the vector representation of the j-th word in the i-th paragraph of document d. The start of each paragraph input is marked with [D].

[0081] Length L d The text of the fragment is represented as:

[0082]

[0083] Length less than L d The text of the fragment is represented as:

[0084] [D], tok1, tok2, ..., tok len(d) ,[sep],[mask],...,[mask]

[0085] The text undergoes a standardization process, padding shorter texts with special markers [masks] to ensure all resulting segmented longer texts are of the same length. This approach adapts well to Transformer networks, yielding more accurate results.

[0086] S2.2 After vector compression, the encoded representation is stored offline.

[0087] The latent vectors of the last layer of BERT are compressed, concatenated, and then stored sequentially in the representation of each document according to their document numbers. The representation of document d is E. d The calculation method is as follows:

[0088]

[0089] Among them, E d The first dimension has a size of k×L d , where represents the total length of the k paragraphs. E d The second dimension is dim, which represents the dimension of each word vector after compression (768 before compression). i p represents the i-th segment after segmentation. i ∈d, i∈{1, 2, .., k}. tok i，j The vector representation of the j-th word in the i-th paragraph of document d.

[0090] Similarly, when the length len(d) of a document does not exceed L d When using [mask], padding is applied up to a length of L. d The document representation process is as follows:

[0091]

[0092]

[0093] Among them, tok i This represents the vector representation of the i-th word in the document.

[0094] S3: Calculate the query text using the trained model to obtain the encoded representation of the query.

[0095] S3.1, obtain the BERT encoded representation of the query statement.

[0096] For the query Q: "Buy apples on iphone", the query sentence is first segmented into "buy", "apples", "on", and "iphone", and these query terms are denoted as q1, q2, ..., q m The starting position of the input is marked with [Q]. Since queries are typically much shorter than documents, they may lack information, making matching between queries and documents difficult. However, query expansion helps find relevant documents better by supplementing the missing information in the query.

[94] This paper proposes a query expansion strategy: Let the maximum length of the query be L. Q For insufficient L Q The query uses a query expansion strategy, repeating the query terms once and marking the end with [Sep]. If less than L remains... Q The positions are filled with special markers [mask] until the length reaches L. Q For lengths exceeding L Q For this query, you will input the first L of the query. Q 10 terms. For more than L Q However, portions of the input that are less than the length of BERT will be padded with a special marker [mask].

[0097] S3.2, the query statement vector is compressed through a linear layer to obtain the encoded representation of the query:

[0098] The output of BERT is then compressed through a linear layer to obtain the representation of query Q as E. Q The calculation method is as follows:

[0099]

[0100] Among them, E Q The size is L Q ×dim. Where dim is the dimension of each word vector after compression (768 before compression).

[0101] S4: Interact with the encoded representation of the query and the encoded representation of the long document to obtain the query results.

[0102] When a new query comes in, the dimensionality-reduced offline stored text representation obtained from S2 is extracted from the system and interactively calculated with the query text representation obtained from S3.

[0103] The process of interacting between the encoded representation of the query and the encoded representation of the long document is as follows:

[0104] S4.1: Obtain the word vector in each long document encoded representation segment that has the closest cosine similarity to each query word vector.

[0105] This fine-grained model obtained all words in the fragment that had the highest similarity to the query value.

[0106] S4.2: Take the average of these word vectors as the representation of the segment, and take the average word embedding representation of the query term as the representation of the query. The semantic matching value between the query and a single segment is calculated as follows:

[0107]

[0108] in, It is a subset of the document's word vectors, representing the set of word vectors that are closest to the word vector of each query. q e represents the vector representation of query term q in Q. m For the word vector e in document d that matches the query q The vector representation of the term with the highest cosine similarity.

[0109] S4.3: Calculate the final score of document d. Take the maximum score of the query and each segment of the document as the semantic relevance score of the document. The calculation method is as follows:

[0110]

[0111] Where d = {p1, p2, ..., p} k}, fragment p in d i ∈d, i∈{1, 2, .., k}.

[0112] The loss function used when training the model on short text data is as follows:

[0113]

[0114] Where y(Q, p) is the relevance label between the paragraph and the query. Score(Q, p) represents the final score of document d.

[0115] Example 2

[0116] The present invention also provides a dense information retrieval system for long texts, the system including an interactive terminal, a server, and a storage medium.

[0117] The system deployment is as follows: the BERT semantic model is trained using S1, a dense information retrieval method for long texts. Then, a large amount of long text to be retrieved is converted using S1 and stored offline on a storage medium. This offline storage content is placed in cloud storage.

[0118] The retrieval process of this system is as follows: the user inputs a query statement through the interactive terminal and limits the selection range of the long text to be retrieved. After receiving the query, the server transforms the query statement through S3 in the dense information retrieval method for long text. Then, the server retrieves the representation of the long text within the selected range from the cloud in batches and obtains the query results through S4 in the dense information retrieval method for long text.

Claims

1. A dense information retrieval method for long text, characterized in that, S1: training the model by using the text meeting the input requirement length of the BERT model and cooperating with the query correlation training sample; S2: obtaining the segmented context coding representation of the searched text by the trained model; S2.1: cutting the long text and inputting it into the BERT model trained in S1 respectively to obtain the segmented coding representation of the text; S2.2: compressing the hidden vector of the last layer of BERT, splicing and storing it in the representation of each document according to the document number, which is the coding representation of the long document; S3: obtaining the BERT coding representation of the query text by the trained model; S3.1: obtaining the BERT coding representation of the query sentence; S3.2: compressing the query sentence vector through a linear layer to obtain the coding representation of the query; S4: interacting the coding representation of the query with the coding representation of the long document to obtain the query result; S4.1: obtaining the word vector with the highest similarity to each query word vector in each long document coding representation segment; S4.2: calculating the representation of the segment and the matching value of the query and the single segment representation; S4.3: calculating the total score of the document. S2 further includes S2.2: vector compression; 2. The method for dense information retrieval for long text-oriented according to claim 1, characterized in that, wherein toki represents the vector representation of the i-th word in the document; The hidden vectors of the last layer of BERT are compressed and spliced, and then sequentially stored in the representation of each document according to the document number; the representation of the document d is E d The calculation method is as follows: wherein E d is the first dimension size of k * L d , representing the total length of k paragraphs; E d is the second dimension dim, representing the dimension number of each word vector after compression; p i represents the i-th paragraph after segmentation , represents the vector representation of the j-th word of the i-th paragraph of the document d; the start of each paragraph input is marked with [D], and the size of the segmentation is L d , if the length of the document exceeds L d , the document will be segmented; Similarly, when the length of a document len(d) does not exceed L d ; the document is padded to length L d using [mask]; the process of document representation is: Correspondingly, S3 further includes the step of compressing the query sentence through a linear layer, In S3.1, the BERT coding representation of the query sentence is obtained as follows: S3.2: compressing the query sentence vector through a linear layer to obtain the coding representation of the query; First, the query words in the query are decomposed, and these query words are respectively recorded as q1, q2, …, qm; wherein the start position of the input is marked with [Q]; let the maximum length of the query be L Q For queries less than L Q , use a query expansion strategy, repeat the input query terms once, and use [Sep] to represent the end position, fill the remaining positions less than L Q with special markers [mask] until the length reaches L Q ; for queries longer than L Q , the first L Q terms of the input query are used; for the part that exceeds L Q but is less than the input length of BERT, special markers [mask] will be used for padding; S2 further includes offline storage of the long text coding representation; For the output of BERT, a linear layer is applied for compression; the representation of the query Q is E Q which is computed as where E Q has size L Q *dim; where dim is the dimension of each compressed word vector.

3. The method for dense information retrieval for long text-oriented according to claim 1, characterized in that, Correspondingly, in S4, the process of interacting the coding representation of the query with the coding representation of the long document further includes retrieving the long text coding representation stored offline in S2 from the offline storage. The text is processed through a standardization process, and the length of the text is filled with a special mark [mask] to make the length of all cut long texts the same.

4. The method for dense information retrieval for long text-oriented according to claim 1, characterized in that, In S2, when cutting these long texts, for a document d, the document length is len(d); let the size of the cut be L d If the length of the document exceeds L d , the document will be cut; for texts whose length does not exceed L d , special markers [mask] are used for padding; assuming that the length of a document len(d) exceeds L d , the document will be cut into a series of paragraphs of length L d , denoted as ; where p; represents the i-th segment after segmentation , represents the vector representation of the j-th word of the i-th segment of the document d; the start of each segment input is marked with [D] Length L d The representation of the text of the segment is: Length less than L d The representation of the text of the segment is: The specific calculation method of the segment representation in S4.2 is as follows:

5. The method for dense information retrieval for long text-oriented according to claim 1, characterized in that, The average value of the most similar word vectors in S4.1 is taken as the representation of the segment, and the average word embedding representation of the query term is taken as the representation of the query, and the calculation method of the semantic matching value of the query and the single segment is as follows: The specific calculation method of the total score of the document in S4.3 is as follows: wherein, is a subset of the word vectors of the document, representing the set of word vectors that are most similar to the word vector of each query; e j represents the vector representation of the query word q in Q, e m is the vector representation of the word item in the document d that has the largest cosine similarity to the word vector e j of the query.

6. The method for dense information retrieval for long text-oriented according to claim 1, characterized in that, The final score of the document d is calculated, and the maximum value of the score of the query and each segment of the document is taken as the semantic relevance score of the document, and the calculation method is as follows: The loss function used by the model when training on short text data is as follows: wherein , fragments in d , ; wherein y(Q, p) is the relevance label of the paragraph and the query; Score(Q, p) represents the final score of the document d. ​ 7. A long text-oriented dense information retrieval system, the system is applied to a long text-oriented dense information retrieval method according to any one of claims 1 to 6, comprising an interactive terminal, a server, a storage medium, the interactive terminal and the storage medium are network connected with the server, characterized in that, The system deployment is specifically as follows: the BERT semantic model is trained through S1 in the dense information retrieval method for long text, and then a large amount of long text to be searched is converted through S1 in the dense information retrieval method for long text and stored offline in a storage medium; The retrieval process of the system is specifically as follows: a query statement is input through an interactive terminal, and the selection range of long text to be searched is limited; after receiving the query, the server converts the query statement through S3 in the dense information retrieval method for long text, and then the server sequentially and batchwise calls the representation of long text in the selected range from the storage medium to obtain a query result through S4 in the dense information retrieval method for long text.

8. A dense information retrieval system for long text oriented according to claim 7, characterized in that, The storage medium uses cloud storage.

Images