A Long Text Retrieval Method Based on Keyword Indexing

Abstract

This invention discloses a long text retrieval method based on keyword indexing, characterized by the following steps: designing a general method that preserves semantic integrity to segment long texts into short texts; extracting keywords using existing technologies; embedding text features of short texts and keywords using existing natural language processing models; fusing multiple short text feature embeddings using a recurrent neural network combined with an attention mechanism to obtain feature embeddings belonging to the long text; and designing a text retrieval method based on keyword indexing, which first calculates the relevance between query keywords and search text keywords, and when the relevance exceeds a threshold, calculates the matching degree between the feature embeddings of the query and the feature embeddings of the search text. This invention enables more efficient and accurate retrieval of search-related text in text retrieval tasks, improves the ability to extract text features and the processing efficiency of long texts, and saves time and memory.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and text retrieval, and in particular to a long text retrieval method based on keyword indexing. Background Technology

[0002] Text retrieval is an important task in the field of Natural Language Processing (NLP), aiming to retrieve documents or information relevant to a user's query from large-scale text datasets. Text retrieval has wide applications in real-world scenarios, such as search engines, information retrieval systems, and question-answering systems.

[0003] Text retrieval originated in the early field of information retrieval. With the development of the internet and the generation of massive amounts of text data, text retrieval has become increasingly important.

[0004] In recent years, with the development of deep learning and natural language processing technologies, neural network-based text retrieval methods have gradually emerged and achieved significant progress. For example, ELMo and BERT are both neural network-based text representation models that have been widely applied in text retrieval tasks. The ELMo model is an LSTM-based model that learns context-dependent word vector representations, enabling it to better capture semantic information in sentences. This model can be used in text retrieval tasks such as semantic matching and question answering systems. The BERT model is a Transformer-based pre-trained language model that learns rich semantic representations through large-scale unsupervised training. This model can be used for ranking and recommending relevant documents in text retrieval.

[0005] While these models have achieved significant results in text retrieval, some challenges and limitations remain. For example, some text data can be very long, such as news articles and blog posts, and existing large-scale natural language processing models still have length limitations when handling long texts. Furthermore, these models may suffer from efficiency and memory consumption issues when retrieving long texts. Therefore, there is an urgent need for a method that can efficiently retrieve long texts without length limitations. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of existing technologies and provide a long text retrieval method based on keyword indexing, which can extract rich semantic information from long texts without being limited by text length, and also makes text retrieval faster. It mainly includes the following steps:

[0007] Step 1: Perform text segmentation on the long text while preserving contextual semantics to obtain the short text;

[0008] Step 2: Extract keywords from the short text obtained in Step 1;

[0009] Step 3: Use a large natural language model to embed multi-text features into the segmented short text and keywords;

[0010] Step 4: Use a recurrent neural network combined with an attention mechanism to fuse the feature embeddings of multiple short texts to obtain the feature embeddings belonging to the long text;

[0011] Step 5: Perform text retrieval based on keyword index.

[0012] In step 1, the long text is segmented to preserve contextual semantics, resulting in a short text. Specifically, this includes:

[0013] Given a long text Q i It is divided into multiple clauses based on preset punctuation marks, namely: Q i =[q0,q1,q2,...,q n ], where q x It is a long text Q i The (x+1)th sentence in the text, 0 ≤ x ≤ n; the punctuation marks include periods, question marks, and exclamation marks;

[0014] The default length of each short text is K, and the length of each long text is defined as Q. i The segmented short text is:

[0015] Q i =[SQ0,SQ1,SQ2,...SQ m ], (1)

[0016] The first short text It is a text concatenation symbol. y must satisfy 0 ≤ y ≤ n, and the text length of SQ0 must be ≤ K. The largest integer value whose text length is greater than K. It is assumed that the next short text contains the last two clauses of the previous short text.

[0017] Step 2, keyword extraction from the short text, specifically includes:

[0018] Get long text Q i After segmenting the short text, we first use ChatGPT combined with prompt words to extract keywords from the short text:

[0019] KW x =ChatGPT(SQ x (2)

[0020] Extract 3 keywords from each short text, namely KW x It is the xth short text SQ x The list of keywords extracted from [the source]. Q for each long text i Divided into a list of short texts and a list of keywords:

[0021] Q i ={[SQ0,SQ1,SQ2,...SQ m ],[KW0,KW1,KW2,...KW m (3)

[0022] In step 3, the multi-text feature embedding specifically includes:

[0023] After obtaining the list of short texts and the list of keywords for each long text segment, the feature embedding method for the short texts is as follows:

[0024] e x =LLM(SQ x (4)

[0025] in, It is the xth short text SQ x The eigenvector representation, d LLM It is the feature embedding dimension of the large language model LLM(*); long text Q i The keyword feature embedding method is as follows:

[0026]

[0027] in, To concatenate the symbols, the weight matrix that needs to be learned is used. Scaling dimension, d KW It is the scaled dimension, which gives the long text Q. i Keyword feature embedding

[0028] In step 4, the fusion of multiple short text feature embeddings specifically includes:

[0029] After obtaining the feature embeddings for each short text, a recurrent neural network combined with an attention mechanism is used to fuse them, resulting in the final feature embeddings belonging to the long text:

[0030]

[0031] in, It is a long text Q i Semantic feature embedding. Long Short-Term Attention (LSTM) mechanisms are used to perceive contextual semantic relationships from sequence data. x (e x ) represents the xth long-short-term attention mechanism unit.

[0032] The attention mechanism is defined as follows:

[0033]

[0034] in It is the input vector. These represent the learnable query weight parameter matrix and the keyword weight parameter matrix, respectively. softmax(*) is the activation function.

[0035] In step 5, the text retrieval task based on keyword indexing is specifically as follows: the text retrieval task is a downstream task in natural language processing, used to find the correct text based on the query text, and to calculate the relevance between the keywords of the query text and the keywords of the retrieved text using the feature dot product method; when the relevance exceeds the threshold, the matching degree between the feature embedding of the query text and the feature embedding of the retrieved text is calculated using cosine similarity.

[0036] Step 5, specifically calculating the relevance between the keywords of the query text and the keywords of the search text using the feature dot product method, includes:

[0037] For a given query text R i and retrieve text P j Set the text embedding as follows: and Set the keyword feature embedding as follows: and The relevance value is obtained by multiplying the keyword features between the query text and the retrieved text:

[0038]

[0039] Where sigmoid is the activation function, and the threshold η is set to 0.3 by default. When the value is greater than the threshold η, the calculation of the matching degree using cosine similarity is specifically as follows:

[0040]

[0041] in, The query text R i and retrieve text P j The matching degree. τ is a temperature coefficient used to adjust the degree of attention given to difficult samples. The default setting is 0.01. The smaller the temperature coefficient, the more attention is paid to separating the current sample from the most similar difficult sample, so as to obtain a more uniform representation.

[0042] In step 5, the text pairs of binary tuples (R) are used. i ,P i Construct a training set D, in addition to retrieving text P i For query text R iThe exception is that n search texts are randomly selected from the remaining text pairs as query text R. i For negative examples, the loss function is defined as:

[0043]

[0044]

[0045] loss = loss1 + loss2, (11)

[0046] Where |D| represents the number of training sets D, loss1 is the first loss function used to train the relevance between keywords, loss2 is the second loss function used to train the matching degree between texts, and the training objective is to minimize the objective function loss, where all parameters in the objective function loss are updated through a combination of backpropagation and stochastic gradient descent.

[0047] The beneficial effects achieved by this invention are as follows:

[0048] This paper proposes an optimal configuration method for long text retrieval based on keyword indexing. It utilizes a recurrent neural network combined with an attention mechanism to fuse the feature embeddings of multiple short texts, ensuring the integrity and accuracy of semantic information and freeing text embedding from the limitation of text length. By using keyword similarity indexing and then performing text relevance matching, the paper significantly reduces the text retrieval time, saves memory usage, and makes text retrieval more efficient. Attached Figure Description

[0049] Figure 1 This is a simplified flowchart of a method according to an exemplary embodiment of the present invention;

[0050] Figure 2 This is a detailed schematic diagram of the x-th LSTM unit in an exemplary embodiment of the present invention;

[0051] Figure 3 In an exemplary embodiment of the present invention A schematic diagram illustrating the specific generation process. Detailed Implementation

[0052] This invention is a long text retrieval method based on keyword indexing, which mainly includes text segmentation that preserves contextual semantics, keyword extraction, multi-text feature embedding, semantic fusion, and text retrieval based on keyword indexing.

[0053] The text segmentation module that preserves contextual semantics designs a punctuation-based segmentation method that divides the text into multiple short texts of similar length. In addition, it takes into account the contextual relationships of the segmented texts to better learn semantic features.

[0054] Keyword extraction involves using existing technologies to extract keywords from segmented short texts for use in building indexes during subsequent text retrieval.

[0055] The semantic fusion module combines recurrent neural networks and attention mechanisms to fuse multiple short text feature embeddings. Each short text feature can perceive contextual semantics, while the attention mechanism can assign different weights to different short text features, thereby capturing global information of long texts.

[0056] The keyword-indexed text retrieval module first uses feature dot product to calculate the relevance between the query keywords and the searched text keywords. When the relevance exceeds a threshold, it then uses cosine similarity to calculate the matching degree between the feature embedding of the query and the feature embedding of the searched text, thereby achieving efficient and accurate long text retrieval.

[0057] To implement the above methods, the following will be combined with the appendix. Figure 1 The method flow of an exemplary embodiment of the present invention will be described as follows:

[0058] Step 1: Text segmentation that preserves contextual semantics. This invention designs a general text segmentation method that does not destroy contextual semantics to segment long texts into multiple short texts for subsequent text feature embedding.

[0059] Step 1, text segmentation that preserves contextual semantics:

[0060] Given a long text Q i This invention divides the clause into multiple subclauses based on punctuation marks (period, question mark, and exclamation mark), namely: Q i =[q0,q1,q2,...,q n ], where q x It is a long text Q i The (x+1)th sentence in the text states that 0 ≤ x ≤ n.

[0061] To accommodate the limitations of large models on text length and ensure semantic coherence of context, this invention specifies that the length of each short text is K, and defines the length of each long text as Q. i The segmented short text is:

[0062] Q i =[SQ0,SQ1,SQ2,...SQ m ], (1)

[0063] The first short text It is a text concatenation symbol. y must satisfy 0 ≤ y ≤ n, and the length of the SQ0 text must be ≤ K. The largest integer value whose text length is greater than K. To ensure semantic relevance between two adjacent short texts, this invention stipulates that the later short text will contain the last two clauses of the previous short text, that is, the last two clauses of the preceding short text will be added to the next short text. For example, Q i The first two clauses of the second short text SQ1 are q y-1 and q y Furthermore, the length rules for subsequent short texts are the same as those for SQ0, so they will not be elaborated upon. This text segmentation method ensures that each short text is semantically related to the preceding and following short texts, which is more conducive to the mining of text semantic features.

[0064] Step 2: Keyword Extraction. In the segmented short text, this invention utilizes existing technology to extract keywords for index building during subsequent text retrieval.

[0065] Step 2, keyword extraction, specifically includes:

[0066] After obtaining the long text Q i After segmenting the short text, as shown in formula (1), in order to speed up text retrieval, this invention requires keyword extraction from the short text first, that is:

[0067] KW x =ChatGPT(SQ x (2)

[0068] This invention utilizes ChatGPT combined with prompt words to extract three keywords from each short text, namely KW. x It is the xth short text SQ x The list of keywords extracted from [the source]. Thus, each long text Q i It will be split into a short text list and a keyword list:

[0069] Q i ={[SQ0,SQ1,SQ2,...SQ m ],[KW0,KW1,KW2,...KW m (3)

[0070] Step 3: Multi-text feature embedding. This invention utilizes existing large-scale natural language models to embed text features from the segmented short texts and keywords.

[0071] Step 3, multi-text feature embedding, specifically includes:

[0072] After obtaining the list of short texts and the list of keywords for each long text segment, this invention utilizes existing large language models for feature embedding. First, for short texts, the feature embedding method is as follows:

[0073] e x =LLM(SQ x (4)

[0074] in, It is the xth short text SQ x The eigenvector representation, d LLM It is the feature embedding dimension of the large language model LLM(*). The large language model LLM used in formula (4) can also be replaced by existing models, such as BERT, M3E, etc., so there is no need to train the relevant model independently, which reflects the compatibility of this method in practical applications.

[0075] In this way, a long text Q i All the short texts included can be embedded as feature vectors.

[0076] Then, the long text Q i The keyword feature embedding method is as follows:

[0077]

[0078] in, To construct the concatenation symbols, this invention connects all keywords, then uses a large language model to embed their features, and finally utilizes a weight matrix. Scaling dimension, d KW It is the scaled dimension, which gives the long text Q. i Keyword feature embedding After ChatGPT extracts keywords from a long text, the length of the resulting keywords is much smaller than the text length limit of all current large language models, so the keywords can be directly concatenated and then used for feature embedding.

[0079] Step 4: Semantic Fusion. This invention utilizes fully connected layers to fuse keyword semantic features, and employs recurrent neural networks combined with attention mechanisms to fuse multiple short text feature embeddings, resulting in feature embeddings and keyword embeddings belonging to the long text.

[0080] Step 4, semantic fusion, specifically includes:

[0081] After obtaining the feature embeddings of each short text, considering the order of the short texts and the different importance levels of different short texts, this invention proposes a method that uses a recurrent neural network combined with an attention mechanism to fuse them, obtaining the final feature embeddings belonging to the long text, namely:

[0082]

[0083] in, It is a long text Q iSemantic feature embedding. LSTM(*) is a long short-term attention mechanism that can perceive contextual semantic relationships from sequence data. For example... Figure 2 As shown, LSTM x (*) represents the x-th long short-term attention mechanism unit, which is defined as follows:

[0084]

[0085]

[0086]

[0087] Where ⊙ represents the Hadamard product. These are all parameters that need to be learned through backpropagation, and their initial values ​​are all random. x f x and o x These are the input gate, forget gate, and output gate of the x-th LSTM unit, respectively. σ is the activation function, which defaults to Sigmoid. It is based on the input of the x-th unit (i.e., e). x The output features obtained, These are the output states of the x-th LSTM unit, all of which are initialized to 0 (i.e.: ). Figure 2 Demonstrates LSTM x Details of the unit. Furthermore, this invention defines the attention mechanism att(*) as follows:

[0088]

[0089] in It is the input vector. These represent the learnable query weight parameter matrix and the keyword weight parameter matrix, respectively. softmax(*) is the activation function.

[0090] This fusion method ensures that semantic feature embeddings can be obtained regardless of the length of the text. The specific generation process is as follows Figure 3 As shown.

[0091] Step 5: Keyword-based text retrieval. This invention designs a long text retrieval method based on keyword indexing, which first uses feature dot product to calculate the relevance between query keywords and search text keywords; when the relevance exceeds a threshold, cosine similarity is then used to calculate the matching degree between the feature embedding of the query and the feature embedding of the search text.

[0092] Step 5, text retrieval based on keyword indexing, specifically includes:

[0093] Through the above steps, we can learn the feature representations of each query and retrieval text, as well as the keyword feature representations. Since a "query" is also a type of text, it is processed using our proposed long text vectorization method. Let's take query Q as an example. i and retrieve text P j For example, their text embeddings are respectively and Keyword feature embeddings are respectively and To accelerate text retrieval, this invention proposes a text retrieval method based on keyword indexing. First, this invention utilizes the keyword features between the query and retrieved texts to perform a dot product to obtain a relevance value:

[0094]

[0095] Where sigmoid is the activation function. It only applies when... When the value is greater than the threshold η (η is set to 0.3 by default), this invention uses cosine similarity for the next step of text matching, i.e.:

[0096]

[0097] in, Is it a query Q? i and query text P j The matching degree. τ is the temperature coefficient, which is set to 0.01 by default. The role of the temperature coefficient is to adjust the degree of attention given to difficult samples. The smaller the temperature coefficient, the more attention is paid to separating the current sample from the most similar difficult sample in order to obtain a more uniform representation.

[0098] In model training, this invention constructs a training set D in the form of text pairs of binary tuples, i.e., (Q i ,P i To improve parameter learning, this invention utilizes the concepts of cross-entropy and contrastive learning to design a loss function. Using text pairs (Q... i ,P i For example, besides retrieving text P i To query Q i With the exception of [other factors], this invention will randomly select n search texts from the remaining text pairs as the query text Q. i The negative examples. Therefore, the loss function is defined as:

[0099]

[0100]

[0101] loss = loss1 + loss2, (11)

[0102] Where |D| represents the size of the training set D. Loss1 is a loss function designed using the idea of ​​cross-entropy, used to train the relevance between keywords. Loss2 is a loss function designed using the idea of ​​contrastive learning, used to train the matching degree between texts. The training objective of this method is to minimize the objective function loss, where all parameters in the objective function loss are updated through a combination of backpropagation and stochastic gradient descent.

[0103] In practical text retrieval scenarios, this invention first uses the keyword features between the query text and the retrieved text to perform a dot product to obtain a relevance value. When the relevance is less than or equal to a threshold η, it indicates that the current retrieved text is unrelated to the query, and the text matching step for the current retrieved text is skipped, proceeding directly to the keyword matching of the next retrieved text. Since the keyword feature acquisition process is relatively simple and the keyword feature dimension is low, the text features can be stored in external storage during keyword matching. Only when the keyword relevance is greater than the threshold η will the next text matching step be performed. This method saves memory and significantly improves indexing speed. The higher the final text matching value, the more likely the corresponding retrieved text is to be provided to the user.

[0104] This invention presents a long text retrieval method based on keyword indexing using a recurrent neural network. First, a punctuation-based segmentation method is designed to divide the text into multiple short texts of similar length. Furthermore, the contextual relationships within the segmented texts are carefully considered to facilitate better semantic feature learning. Then, existing techniques are used to extract keywords from the segmented short texts for indexing during subsequent text retrieval. Next, a recurrent neural network and an attention mechanism are combined to fuse the feature embeddings of multiple short texts. Each short text feature can perceive contextual semantics, and the attention mechanism can assign different weights to different short text features, thereby capturing the global information of the long text. Finally, the keyword indexing-based text retrieval method first calculates the relevance between the query keywords and the searched text keywords using feature dot product. When the relevance exceeds a threshold, cosine similarity is used to calculate the matching degree between the feature embeddings of the query and the searched text, thus achieving efficient and accurate long text retrieval.

[0105] The beneficial effects achieved by this invention are as follows:

[0106] This paper proposes an optimal configuration method for long text retrieval based on keyword indexing. It utilizes a recurrent neural network combined with an attention mechanism to fuse the feature embeddings of multiple short texts, ensuring the integrity and accuracy of semantic information and freeing text embedding from the limitation of text length. By using keyword similarity indexing and then performing text relevance matching, the paper significantly reduces the text retrieval time, saves memory usage, and makes text retrieval more efficient.

[0107] With the development of the internet and the generation of massive amounts of text data, text retrieval has become increasingly important. However, some text data can be very long, such as news articles and blog posts. Processing long texts remains challenging because models need to capture important information and extract useful features. Existing large-scale natural language models still have length limitations when processing long texts. Furthermore, these models may suffer from efficiency and memory consumption issues when retrieving long texts. Therefore, there is an urgent need for a method that can efficiently retrieve long texts without length limitations. This invention proposes a long text retrieval method based on keyword indexing. This method segments long texts without compromising semantics, embeds short texts and keywords using a large-scale language model, and then fuses multiple short text feature embeddings through a recurrent neural network combined with an attention mechanism. This ensures the integrity and accuracy of semantic information, thereby achieving accurate long text retrieval without length limitations, providing users with a better experience and more accurate information matching and recommendation services.

[0108] The above embodiments do not limit the present invention in any way. Based on the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of the present invention. All other improvements and applications made to the above embodiments by equivalent transformation are within the protection scope of the present invention. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A long text retrieval method based on keyword indexing, characterized in that, Includes the following steps: Step 1: Perform text segmentation on the long text while preserving contextual semantics to obtain the short text; Step 2: Extract keywords from the short text obtained in Step 1; Step 3: Use a large natural language model to embed multi-text features into the segmented short text and keywords; The multi-text feature embedding specifically includes: After obtaining the list of short texts and the list of keywords for each long text segment, the feature embedding method for the short texts is as follows: ； in, It is the first A short text The eigenvector representation, It is a large language model Feature embedding dimension; long text The keyword feature embedding method is as follows: ； in, To concatenate the symbols, the weight matrix that needs to be learned is used. Scaling dimensions It is the scaled dimension, which is the result of long text. Keyword feature embedding ; Step 4: Use a recurrent neural network combined with an attention mechanism to fuse the feature embeddings of multiple short texts to obtain the feature embeddings belonging to the long text; Step 5: Perform text retrieval based on keyword index.

2. The long text retrieval method based on keyword indexing according to claim 1, characterized in that, In step 1, the long text is segmented to preserve contextual semantics, resulting in a short text. Specifically, this includes: Given a long text It is divided into multiple clauses based on preset punctuation marks, namely: ,in It is a long text The first in In short, 0 The punctuation marks mentioned include periods, question marks, and exclamation marks. The preset length of each short text is 1000. and define each long text The segmented short text is: ； The first short text , It is a text splicing symbol. Requires 0 , Text length ,and Text length The maximum integer value, presumably the next short text contains the last two clauses of the previous short text.

3. The long text retrieval method based on keyword indexing according to claim 2, characterized in that, Step 2, keyword extraction from the short text, specifically includes: Get long text After segmenting the short text, we first use ChatGPT combined with prompt words to extract keywords from the short text: Extract 3 keywords from each short text, i.e. It is the first A short text The list of keywords extracted from [the source]. ; for each long text Divided into a short text list and a keyword list: Q i ={[SQ0, SQ1, SQ2, ..., SQ m ],[KW0,KW1,KW2,...KW m ]}.

4. The long text retrieval method based on keyword indexing according to claim 3, characterized in that, In step 4, the fusion of multiple short text feature embeddings specifically includes: After obtaining the feature embeddings for each short text, a recurrent neural network combined with an attention mechanism is used to fuse them, resulting in the final feature embeddings belonging to the long text: ； in, It is a long text. Semantic feature embedding and long short-term attention mechanisms are used to perceive contextual semantic relationships from sequence data. Indicates the first A long-term and short-term attention mechanism unit, The attention mechanism is defined as follows: ； in It is the input vector. These represent the learnable query weight parameter matrix and the keyword weight parameter matrix, respectively. It is an activation function.

5. The long text retrieval method based on keyword indexing according to claim 4, characterized in that, In step 5, the text retrieval task based on keyword indexing is specifically as follows: the text retrieval task is a downstream task in natural language processing, used to find the correct text based on the query text, and to calculate the relevance between the keywords of the query text and the keywords of the retrieved text using the feature dot product method; when the relevance exceeds the threshold, the matching degree between the feature embedding of the query text and the feature embedding of the retrieved text is calculated using cosine similarity.

6. The long text retrieval method based on keyword indexing according to claim 5, characterized in that, Step 5, specifically calculating the relevance between the keywords of the query text and the keywords of the search text using the feature dot product method, includes: For a given query text and search text Set the text embedding as follows: and The keyword feature embeddings are set as follows: and The relevance value is obtained by multiplying the keyword features between the query text and the retrieved text. ； in, For activation function, threshold The default setting is 0.

3. Greater than the threshold In this case, the calculation of the matching degree using cosine similarity specifically involves: ； in, It is the query text and search text The degree of matching, This is the temperature coefficient, used to adjust the degree of attention given to difficult samples. The default setting is 0.

01. The smaller the temperature coefficient, the more attention is paid to separating the current sample from the most similar difficult samples, so as to obtain a more uniform representation.

7. The long text retrieval method based on keyword indexing according to claim 6, characterized in that, In step 5, the text pairs of binary tuples are used. Constructing the training set In addition to retrieving text For query text The positive exception is randomly selected from the remaining text pairs. The search text is used as the query text. For negative examples, the loss function is defined as: ； ； ； in, Indicates training set Quantity, It is the first loss function, used to train the relevance between keywords. This is the second loss function, used to train the matching degree between texts. The training objective is to minimize the objective function. The objective function All parameters are updated through a combination of backpropagation and stochastic gradient descent.

Images