A Two-Granularity Text Retrieval Method Based on Atomic Sentences
By segmenting documents into text blocks and atomic sentences, and utilizing cosine similarity calculation and weighted combination methods, the semantic inconsistency problem in the retrieval stage is solved, achieving higher retrieval accuracy and applicability.
Patent Information
- Application Number
- CN202510799592.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-16
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-06-16
AI Technical Summary
In existing technologies, there is a problem of inconsistency between the query and the external knowledge context in the semantic space during the retrieval stage, resulting in unsatisfactory retrieval results.
A dual-granularity text retrieval method based on atomic sentences is adopted, which divides documents into text blocks and atomic sentences, and improves retrieval accuracy through cosine similarity calculation and weighted combination.
It focuses on both the global semantics and local information of the text, improving the accuracy of retrieval and making it suitable for enterprise private domain data.
Smart Images

Figure CN120821825B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of text retrieval, and more specifically, to a two-granularity text retrieval method based on atomic sentences. Background Technology
[0002] With the rapid development of information technology, text-based tasks play a crucial role in fields such as intelligent question answering and customer service automation. Large language models, due to their powerful semantic understanding and generation capabilities, are widely used in these areas. However, despite their excellent generation capabilities, large language models still face challenges such as lagging data updates and generating false information. To address these issues, retrieval-enhanced generation technology has emerged. This technology retrieves the most relevant fragments to the user's query from a large-scale knowledge base and passes these fragments as context to the generative model, thereby producing more accurate answers. In a retrieval-enhanced generation system, the retrieval phase is the core foundation of the entire process. High-quality retrieval results directly determine the relevance and accuracy of the generated content. However, the current retrieval phase suffers from a bottleneck: the query and the external knowledge context are semantically inconsistent, leading to unsatisfactory retrieval results. Summary of the Invention
[0003] To overcome at least one deficiency in the prior art, this application provides a dual-granularity text retrieval method based on atomic sentences.
[0004] Firstly, a dual-granularity text retrieval method based on atomic sentences is provided, including:
[0005] Divide the document into multiple text blocks;
[0006] A large pre-trained language model is used to segment each text block into a set of atomic sentences;
[0007] Based on the embedding model, each user query, each text block, and each group of atomic sentences are transformed into vectors, resulting in query vectors, text block vectors, and atomic sentence group vectors.
[0008] For each user query, calculate the cosine similarity between the query vector corresponding to the user query and each text block vector. Sort all cosine similarities in descending order and select the text blocks corresponding to the first 4K cosine similarities as the first search result corresponding to the user query. The cosine similarity of each text block in the first search result is recorded as the text block similarity score.
[0009] For each text block in the first search result, calculate the cosine similarity between the query vector and all atomic sentences corresponding to the text block, and select the largest cosine similarity as the atomic sentence similarity score corresponding to the text block;
[0010] The text block similarity score and the atomic sentence similarity score corresponding to the text block are weighted and combined to obtain the final score of the text block;
[0011] Sort all text blocks in descending order of their final scores, and select the top K text blocks with the highest final scores as the final search results for the user query.
[0012] In one embodiment, the text block similarity score and the atomic sentence similarity score corresponding to the text block are weighted and combined to obtain the final score of the text block, using the following formula:
[0013]
[0014] in, Let α be the final score for the i-th user querying the j-th text block, and let α be the weight. This is the text block similarity score for the j-th text block queried by the i-th user. This is the atomic sentence similarity score for the j-th text block queried by the i-th user.
[0015] Secondly, a dual-granularity text retrieval device based on atomic sentences is provided, comprising:
[0016] The document segmentation module is used to split a document into multiple text blocks;
[0017] The text block segmentation module is used to segment each text block into a set of atomic sentences based on a pre-trained large language model;
[0018] The vector transformation module is used to transform each user query, each text block, and each group of atomic sentences into vectors based on the embedding model, resulting in query vectors, text block vectors, and atomic sentence group vectors.
[0019] The first retrieval module is used to calculate the cosine similarity between the query vector corresponding to the user query and each text block vector for each user query, sort all the cosine similarities in descending order, and select the text blocks corresponding to the first 4K cosine similarities as the first retrieval result corresponding to the user query; the cosine similarity of each text block in the first retrieval result is recorded as the text block similarity score.
[0020] The second retrieval module is used to calculate the cosine similarity between the query vector and all atomic sentences corresponding to the text block for each text block in the first retrieval result, and select the largest cosine similarity as the atomic sentence similarity score corresponding to the text block.
[0021] The combination module is used to weight and combine the text block similarity score and the atomic sentence similarity score corresponding to the text block to obtain the final score of the text block;
[0022] The final search result determination module is used to sort the final scores of all text blocks in descending order and select the text blocks corresponding to the top K final scores as the final search results for the user query.
[0023] In one embodiment, the text block similarity score and the atomic sentence similarity score corresponding to the text block are weighted and combined to obtain the final score of the text block, using the following formula:
[0024]
[0025] in, Let α be the final score for the i-th user querying the j-th text block, and let α be the weight. This is the text block similarity score for the j-th text block queried by the i-th user. This is the atomic sentence similarity score for the j-th text block queried by the i-th user.
[0026] Thirdly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the aforementioned dual-granularity text retrieval method based on atomic sentences.
[0027] Fourthly, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the aforementioned dual-granularity text retrieval method based on atomic sentences.
[0028] Compared with the prior art, this application has the following advantages: The dual-granularity text retrieval method based on atomic sentences in this application uses both original text block retrieval and atomic sentence retrieval, which not only ensures the global semantics of the text, but also pays more attention to the local information of the text, which can truly improve the accuracy of retrieval and is applicable to enterprise private domain data. Attached Figure Description
[0029] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings:
[0030] Figure 1 A flowchart of a two-granularity text retrieval method based on atomic sentences is shown;
[0031] Figure 2 A comparison chart of the recall rates of the method in this application and the prior art is shown;
[0032] Figure 3 A graph comparing the average reciprocal ranking of the method of this application with that of the prior art is shown;
[0033] Figure 4 A block diagram of a dual-granularity text retrieval device based on atomic sentences is shown. Detailed Implementation
[0034] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.
[0035] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.
[0036] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.
[0037] This application provides a dual-granularity text retrieval method based on atomic sentences. This method adopts a dual-granularity text retrieval algorithm that combines "query and original text block retrieval results" and "query and atomic sentence retrieval results". Figure 1 A flowchart of a two-granularity text retrieval method based on atomic sentences is shown. See [link / reference]. Figure 1 The method mainly includes the following steps:
[0038] Step S1: Divide the document into multiple text blocks.
[0039] The documents in the document library are merged, split, and have metadata settings applied to obtain multiple text blocks. These multiple text blocks form a text block set C = {c1, c2, ..., c...} j …c m}, where c j This represents the j-th text block, where m is the number of text blocks. During segmentation, the segmentation can be based on a specific number of tokens, a specific punctuation group, or a special document structure.
[0040] Step S2: Based on a pre-trained large language model, each text block is segmented into a set of atomic sentences.
[0041] The instructions are set up to segment each text block into a set of atomic sentences based on a pre-trained large-scale language model, and then associate the text block with the corresponding set of atomic sentences. For example, for text block c... jThe prompt is set to "Please break down the following paragraphs into independent atomic facts. Require each fact to be a semantically complete and context-independent sentence. Return these facts.{chunk}", and the text chunks are broken down using a large language model. j Divide into a group of atomic sentences in, For A j The t-th atom sentence in the middle, h is A j The number of atomic sentences is determined by this step, resulting in a set A = {A_{n}} of atomic sentence groups from the text block set C. 1 A 2 …A j …A m}
[0042] Step S3: Based on the embedding model, each user query, each text block, and each group of atomic sentences are transformed into vectors, and the text is mapped to a high-dimensional semantic space to obtain query vectors, text block vectors, and atomic sentence group vectors.
[0043] Here, the query set is Q = {q1, q2, ..., q}. i …q n}, where q i For the i-th user query, n is the number of user queries, and user query q i Transform into query vector vq i All query vectors constitute the query vector set VQ = {vq1, vq2, ..., vq} i …vq n};
[0044] For a set of file blocks C = {c1, c2, ..., c...} j …c m}, file block c j Convert to text block vector vc j All text block vectors constitute the file block vector set VC = {vc1, vc2, ..., vc...} j …vc m};
[0045] The set of atomic sentence groups A = {A 1 A 2 …A j …A m}, the h-th group of atomic sentences A jTransform into atomic sentence group vectors VA j ,
[0046] Step S4: For each user query, calculate the cosine similarity between the query vector corresponding to the user query and each text block vector. Sort all cosine similarities in descending order and select the text blocks corresponding to the first 4K cosine similarities as the first search result corresponding to the user query. The cosine similarity of each text block in the first search result is recorded as the text block similarity score.
[0047] The cosine value between vectors is used as the similarity between vectors. The range of the cosine value is [-1, 1]. Therefore, the closer the cosine value of two vectors is to 1, the more similar the two vectors are, and the higher the semantic similarity of the texts represented by the two vectors is.
[0048] For user query q i Calculate the corresponding query vector vq i With VC = {vc1, vc2, ... vc j …vc m The cosine similarity between each text block vector in the query is calculated, and all cosine similarities are sorted in descending order. The text blocks corresponding to the top 4K cosine similarities are selected as the query result for the user query q. i The corresponding first search result is Rqc i ;RqC i Let Rqc be a set of 4K elements, where each element belongs to a set of text blocks C; i This includes the j-th text block c. j c j The corresponding text block similarity score is denoted as
[0049] Step S5: For each text block in the first search result, calculate the cosine similarity between the query vector and all atomic sentences corresponding to the text block, and select the largest cosine similarity as the atomic sentence similarity score corresponding to the text block.
[0050] For user query q i Calculate the query vector vq i With text block c j The cosine similarity between all corresponding atomic sentences, with the highest cosine similarity denoted as . t represents A j Zhongyu VQ i The atomic sentences with the highest cosine similarity The subscript.
[0051] Step S6: The text block similarity score and the atomic sentence similarity score corresponding to the text block are weighted and combined to obtain the final score of the text block.
[0052] Specifically, the following formula is used:
[0053]
[0054] in, Let α be the final score for the i-th user querying the j-th text block, and let α be the weight. This is the text block similarity score for the j-th text block queried by the i-th user. This is the atomic sentence similarity score for the j-th text block queried by the i-th user.
[0055] Step S7: Sort the final scores of all text blocks in descending order, and select the text blocks corresponding to the top K final scores as the final search results for the user query.
[0056] The dual-granularity text retrieval method based on atomic sentences in this embodiment uses both original text block retrieval and atomic sentence retrieval. This not only ensures the global semantics of the text but also pays full attention to the local information of the text, which can truly improve the accuracy of retrieval and is applicable to enterprise private domain data.
[0057] To further verify the effectiveness of the method in this application, experimental analysis was conducted.
[0058] We selected a public text dataset, set up specific implementations of the method in this application and several retrieval algorithms in the prior art, and selected recall rate R@N (Recall at N) and mean reciprocal rank MRR@N (Mean Reciprocal Rank at N) as metrics for retrieval performance.
[0059] In the selected public dataset, each query has its corresponding most relevant text block. If the search results contain the most relevant text block, the search is successful, which is beneficial for subsequent generation tasks; otherwise, it is unsuccessful.
[0060] Here, recall (R@N) represents the ratio of the number of queries that retrieve the most relevant text blocks to the total number of queries, when only the top N text blocks in the search results are considered. R@N ranges from [0,1], with a higher value indicating a higher recall and better search performance. Mean Reciprocal Rank (MRR@N) represents the average of the reciprocals of the most relevant text blocks retrieved by all queries, when only the top N text blocks in the search results are considered. MRR@N also ranges from [0,1], with a higher value indicating that the most relevant text blocks rank higher in the search results, indicating better search performance.
[0061] The publicly available English text dataset SQuAD was selected for implementation of the method described in this application and several retrieval algorithms in the prior art. The SQuAD dataset contains 98,169 queries and 20,958 text blocks. In the specific comparative experiment, the weight 1-α corresponding to the "atomic sentence similarity score" was set to 0.2.
[0062] Table 1 shows the recall rates R@N obtained by multiple retrieval algorithms in this application and the prior art on the dataset. Figure 2 A comparison chart of the recall rates of the method in this application and the prior art is shown.
[0063] Table 1
[0064]
[0065] According to Table 1 and Figure 2 The retrieval recall rate of the method in this application is consistently higher than that of other existing retrieval algorithms. Among them, the HyDE retrieval algorithm and the Query rewriting retrieval algorithm have the lowest recall rate R@N. This is because these two retrieval algorithms require a large language model to fully understand the semantics of the query before they can transform or rewrite it. Therefore, without using a dataset to pre-train the large language model, the transformation and rewriting of the query is not effective, and the retrieval results cannot be truly improved, resulting in a relatively low recall rate. When the number of recall results to be focused on is small (i.e., N is small, such as N=1 to N=3), the introduction of sparse retrieval leads to problems such as polysemy and synonyms of keywords, which can result in the inability to accurately retrieve the text blocks most relevant to the query.
[0066] The average reciprocal ranking (MRR@N) table obtained by multiple retrieval algorithms in this application and the prior art on the dataset is shown in Table 2. Figure 3 A graph showing the average reciprocal ranking of the method of this application and the prior art is presented.
[0067] Table 2
[0068]
[0069] According to Table 2 and Figure 3 The MRR@N value obtained by the method in this application is consistently the highest, indicating that the retrieval effect is the best. The MRR@N values corresponding to the HyDE retrieval algorithm and the Query rewriting retrieval algorithm remain relatively low. The MRR@N value of the hybrid retrieval shows a significant change compared to its performance in R@N. When MRR@N is used as an indicator, the corresponding data for hybrid retrieval is particularly low. This is also because hybrid retrieval introduces the problem of synonyms and polysemous words for keywords, so it cannot accurately ensure that the most relevant text blocks are placed at the beginning.
[0070] Employing the same inventive concept as the atomic sentence-based dual-granularity text retrieval method, this embodiment also provides a corresponding atomic sentence-based dual-granularity text retrieval device. Figure 4 The diagram shows a structural block diagram of a dual-granularity text retrieval device based on atomic sentences. The device includes:
[0071] The document segmentation module is used to split a document into multiple text blocks;
[0072] The text block segmentation module is used to segment each text block into a set of atomic sentences based on a pre-trained large language model;
[0073] The vector transformation module is used to transform each user query, each text block, and each group of atomic sentences into vectors based on the embedding model, resulting in query vectors, text block vectors, and atomic sentence group vectors.
[0074] The first retrieval module is used to calculate the cosine similarity between the query vector corresponding to the user query and each text block vector for each user query, sort all the cosine similarities in descending order, and select the text blocks corresponding to the first 4K cosine similarities as the first retrieval result corresponding to the user query; the cosine similarity of each text block in the first retrieval result is recorded as the text block similarity score.
[0075] The second retrieval module is used to calculate the cosine similarity between the query vector and all atomic sentences corresponding to the text block for each text block in the first retrieval result, and select the largest cosine similarity as the atomic sentence similarity score corresponding to the text block.
[0076] The combination module is used to weight and combine the text block similarity score and the atomic sentence similarity score corresponding to the text block to obtain the final score of the text block;
[0077] The final search result determination module is used to sort the final scores of all text blocks in descending order and select the text blocks corresponding to the top K final scores as the final search results for the user query.
[0078] The atomic sentence-based dual-granularity text retrieval device of this embodiment has the same inventive concept as the atomic sentence-based dual-granularity text retrieval method described above. Therefore, the specific implementation of this device can be found in the embodiment section of the atomic sentence-based dual-granularity text retrieval method described above, and its technical effects correspond to the technical effects of the above method, so it will not be repeated here.
[0079] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described dual-granularity text retrieval method based on atomic sentences.
[0080] This application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described dual-granularity text retrieval method based on atomic sentences.
[0081] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A dual-granularity text retrieval method based on atomic sentences, characterized in that, include: Divide the document into multiple text blocks; A large pre-trained language model is used to segment each text block into a set of atomic sentences; Based on the embedding model, each user query, each text block, and each group of atomic sentences are transformed into vectors to obtain query vectors, text block vectors, and atomic sentence group vectors. For each user query, calculate the cosine similarity between the query vector corresponding to the user query and each text block vector, sort all cosine similarities in descending order, and select the text blocks corresponding to the first 4K cosine similarities as the first search result corresponding to the user query. The cosine similarity of each text block in the first search result is denoted as the text block similarity score. For each text block in the first retrieval result, calculate the cosine similarity between the query vector and all atomic sentences corresponding to the text block, and select the largest cosine similarity as the atomic sentence similarity score corresponding to the text block; The text block similarity score and the atomic sentence similarity score corresponding to the text block are weighted and combined to obtain the final score of the text block; Sort all text blocks in descending order of their final scores, and select the text blocks corresponding to the top K final scores as the final search results for the user query. The final score of the text block is obtained by weighting and combining the text block similarity score and the atomic sentence similarity score corresponding to the text block, using the following formula: in, For the first i The user query number The final score of each text block, As weight, For the first i The user query number Text block similarity score for each text block For the first i The user query number The atomic sentence similarity score corresponding to each text block.
2. A dual-granularity text retrieval device based on atomic sentences, characterized in that, include: The document segmentation module is used to split a document into multiple text blocks; The text block segmentation module is used to segment each text block into a set of atomic sentences based on a pre-trained large language model; The vector transformation module is used to transform each user query, each text block, and each group of atomic sentences into vectors based on the embedding model, resulting in query vectors, text block vectors, and atomic sentence group vectors. The first retrieval module is used to calculate the cosine similarity between the query vector corresponding to the user query and each text block vector for each user query, sort all the cosine similarities in descending order, and select the text blocks corresponding to the first 4K cosine similarities as the first retrieval result corresponding to the user query. The cosine similarity of each text block in the first search result is denoted as the text block similarity score. The second retrieval module is used to calculate the cosine similarity between the query vector and all atomic sentences corresponding to the text block for each text block in the first retrieval result, and select the largest cosine similarity as the atomic sentence similarity score corresponding to the text block; The combination module is used to perform a weighted combination of the text block similarity score and the atomic sentence similarity score corresponding to the text block to obtain the final score of the text block; The final search result determination module is used to sort the final scores of all text blocks in descending order and select the text blocks corresponding to the top K final scores as the final search results for the user query. The final score of the text block is obtained by weighting and combining the text block similarity score and the atomic sentence similarity score corresponding to the text block, using the following formula: in, For the first i The user query number The final score of each text block, As weight, For the first i The user query number Text block similarity score for each text block For the first i The user query number The atomic sentence similarity score corresponding to each text block.
3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the dual-granularity text retrieval method based on atomic sentences as described in claim 1.
4. A computer program product, characterized in that, Includes a computer program / instruction, which, when executed by a processor, implements the atomic sentence-based dual-granularity text retrieval method of claim 1.
