Retrieval method and device, processing equipment, program product and storage medium

Abstract

The embodiment of the invention provides a retrieval method and device, processing equipment, a program product and a storage medium, and is applied to the technical field of artificial intelligence. The method comprises the steps that a subject sentence corresponding to a target document and at least two text blocks corresponding to the target document are obtained, the subject sentence is used for representing global semantic features of the target document, and the text blocks are minimum sentence units which are segmented according to a semantic integrity principle and bear independent and complete semantics in the target document; determining a first similarity corresponding to each text block and a second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the subject sentence; determining a text field corresponding to each text block according to the first similarity corresponding to each text block and the second similarity between every two text blocks; and generating response information for the user query information according to the text fields corresponding to the text blocks. By the adoption of the method, the problem that an existing RAG retrieval method is low in retrieval precision and retrieval efficiency is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a retrieval method, apparatus, processing device, program product, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and natural language processing technologies, Retrieval-augmented Generation (RAG) technology has emerged.

[0003] Large Language Models (LLMs) excel in text generation, but they heavily rely on massive, dynamically updated knowledge bases and efficient retrieval mechanisms. In practical business applications such as intelligent customer service, intelligent writing, and intelligent question answering, RAG (Resource Aggregator) significantly improves the accuracy, relevance, and timeliness of content by accurately integrating external knowledge sources. This effectively mitigates the illusion of Artificial Intelligence (AI), accelerates knowledge base updates, and enhances the traceability of generated content.

[0004] In short, RAG is a highly efficient collaborative mechanism of "retrieval + generation"—it accurately extracts key information from massive external knowledge bases and combines it with the language capabilities of LLM to generate clear, natural, and credible responses. However, existing RAG retrieval methods suffer from low retrieval accuracy and efficiency. Summary of the Invention

[0005] The purpose of this application is to provide a retrieval method, apparatus, processing equipment, program product, and storage medium to solve the problems of low retrieval accuracy and efficiency of existing RAG retrieval methods.

[0006] One embodiment of this application provides a retrieval method, the method comprising:

[0007] Obtain the topic sentence corresponding to the target document and at least two text blocks corresponding to the target document, wherein the topic sentence is used to characterize the global semantic features of the target document, and the text blocks are the smallest sentence units in the target document that are segmented according to the principle of semantic integrity and carry independent and complete semantics;

[0008] Determine the first similarity corresponding to each of the text blocks and the second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the topic sentence;

[0009] Based on the first similarity corresponding to each text block and the second similarity between every two text blocks, the text field corresponding to each text block is determined, wherein the text field is a set of at least one text block;

[0010] Based on the text fields corresponding to each of the text blocks, generate response information for the user's query.

[0011] Optionally, determining the text field corresponding to each text block based on the first similarity corresponding to each text block and the second similarity between every two text blocks includes:

[0012] Based on the first similarity corresponding to each of the text blocks, a baseline deviation value is determined between every two text blocks, wherein the baseline deviation value is the difference between the first similarity values ​​corresponding to the two text blocks;

[0013] Based on the first threshold, a first target text field corresponding to each first text block is determined, wherein the reference deviation value between any text block in the first target text field and the first text block is less than the first threshold, and the first text block is any one of the at least two text blocks;

[0014] Based on the second threshold, a second target text domain corresponding to each first text block is determined, and the second similarity between any text block in the second target text domain and the first text block is greater than the second threshold.

[0015] The intersection between the first target text field and the second target text field is determined as the text field corresponding to the first text block.

[0016] Optionally, generating response information for the user's query based on the text fields corresponding to each of the text blocks includes:

[0017] Based on the text blocks in each text field, generate at least one keyword tag corresponding to each text field;

[0018] Generate text block vectors corresponding to each of the text blocks;

[0019] Based on the text blocks in each text field and the text block vectors corresponding to each text block, a text field vector corresponding to each text field is generated, and the text fields and the text field vectors correspond one-to-one.

[0020] Based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector, a response message for the user's query information is generated.

[0021] Optionally, generating response information for the user's query based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector includes:

[0022] Obtain at least one question keyword and a question vector corresponding to the user query information;

[0023] By keyword matching, at least one candidate text field is determined, wherein at least one keyword tag corresponding to the candidate text field matches at least one of the question keywords;

[0024] Based on the at least one candidate text field, the question vector, the text block vector, and the text field vector, a response message for the user's query information is generated.

[0025] Optionally, generating response information for the user's query based on the at least one candidate text field, the question vector, the text block vector, and the text field vector includes:

[0026] Based on the question vector, a third similarity is determined for each of the text domain vectors, wherein the third similarity is the similarity between the text domain vector and the question vector;

[0027] Based on the third threshold and the third similarity corresponding to each of the text field vectors, at least one candidate text field vector is determined, wherein the candidate text field vector is a text field vector whose third similarity is greater than the third threshold.

[0028] The intersection of the text field vectors corresponding to the at least one candidate text field and the at least one candidate text field vectors is determined as the target text field vector set;

[0029] Based on the question vector and the target text domain vector set, a response message is generated for the user's query information.

[0030] Optionally, generating response information for the user's query based on the question vector and the target text domain vector set includes:

[0031] Determine the fourth similarity between the question vector and each text domain vector in the target text domain vector set;

[0032] The text field vector with the highest fourth similarity in the target text field vector set is determined as the target text field vector corresponding to the question vector;

[0033] Based on the text field corresponding to the target text field vector, generate response information for the user's query.

[0034] One embodiment of this application also provides a retrieval device, the device comprising:

[0035] The first acquisition module is used to acquire the topic sentence corresponding to the target document and at least two text blocks corresponding to the target document, wherein the topic sentence is used to characterize the global semantic features of the target document, and the text blocks are the smallest sentence units in the target document that are divided according to the principle of semantic integrity and carry independent and complete semantics.

[0036] The first processing module is used to determine the first similarity corresponding to each of the text blocks and the second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the topic sentence;

[0037] The second processing module is used to determine the text field corresponding to each text block based on the first similarity corresponding to each text block and the second similarity between every two text blocks, wherein the text field is a set of at least one text block;

[0038] The response generation module is used to generate response information for user queries based on the text fields corresponding to each of the text blocks.

[0039] One embodiment of this application also provides a processing device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the retrieval method as described in any of the preceding claims.

[0040] One embodiment of this application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the retrieval method as described in any of the preceding claims.

[0041] One embodiment of this application also provides a readable storage medium, wherein a program is stored on the readable storage medium, and when the program is executed by a processor, it implements the steps of the retrieval method as described in any of the preceding claims.

[0042] At least one of the above-mentioned technical solutions of this application has the following beneficial effects:

[0043] In the retrieval method of this application embodiment, the target document is segmented into the smallest sentence units (text blocks) according to the principle of semantic integrity, and the first similarity between each text block and the topic sentence is calculated. This avoids the semantic fragmentation problem caused by traditional word segmentation retrieval, improves the accuracy of document structured semantic capture, and thus significantly improves retrieval accuracy. By defining a text domain as a set of text blocks, and dynamically aggregating related text blocks based on the first and second similarities, full document traversal can be avoided, which can improve retrieval efficiency, reduce system computational overhead, and reduce invalid and redundant information in user query responses. Thus, the retrieval method of this invention is a reasonable one that can solve the problems of low retrieval accuracy and efficiency of existing RAG retrieval methods. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the retrieval method according to one embodiment of this application;

[0045] Figure 2 This is a flowchart of an AI question-answering process according to one embodiment of this application;

[0046] Figure 3 This is a flowchart illustrating the technical solution of one embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the structure of a retrieval device according to one embodiment of this application. Detailed Implementation

[0048] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited; for example, the first object can be one or more.

[0049] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0051] It should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0052] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0053] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0054] Relevant technical specifications:

[0055] The existing RAG workflow generally includes the following steps:

[0056] (1) Text segmentation: The original long text is divided into independent segments (such as paragraphs or topic blocks) according to semantic units to avoid directly traversing the whole text and improve retrieval efficiency. In layman's terms, it is to break a large book into small pages. For example, if there is a book that takes a long time to search, it can be broken into small blocks page by page for easier searching, that is, text segmentation.

[0057] (2) Generate embeddings: Generate a vector representation (i.e., embedding vector) for each text block, which means labeling each page of text blocks after splitting. For example, after the blocks are divided, each text block needs to be labeled with a special label, a label that a computer can understand and process. The label can be a string of numbers, also known as an "embedding vector".

[0058] (3) Storing Vectors: Establishing a Text Repository. After labeling each page of text blocks after splitting, these text blocks will have unique digital fingerprints. These digital fingerprints need to be stored in a vector database. In this way, the vector database stores the fingerprints of the text blocks and their original content.

[0059] (4) Input query: Initiate a question. For example, receive user questions or instructions as input.

[0060] (5) Query vectorization: Label the question. Here, in order to find the answer in the vector library, the question needs to be converted into a digital fingerprint, that is, converted into a vector.

[0061] (6) Retrieve similar blocks: Query the most relevant text. That is, match the digital fingerprint of the question with the digital fingerprints stored in the vector database, and find the “most similar” text block from the database.

[0062] (7) Result reordering: In order to improve the accuracy and relevance of the answers, the system will score and sort the matched text blocks, just like picking out the most accurate ones from many answers, thereby improving the accuracy of text block matching.

[0063] (8) Generate final response: The sorted search results (text blocks) are integrated with the user's query and input into the LLM to generate a coherent, accurate and natural response based on real evidence.

[0064] However, existing methods for using RAG technology to retrieve and generate text have the following drawbacks:

[0065] (1) The quality of text segmentation is low, and key information is scattered in different blocks: Existing methods divide documents into different text blocks according to a fixed number of characters, words or tags. Although the operation is convenient and AI processing is convenient, the segmentation quality is not high. It may cut complete sentences, core arguments or key entities into multiple blocks, affecting the accuracy of subsequent retrieval matching and text generation.

[0066] (2) Low retrieval efficiency: Existing vector retrieval methods require matching and retrieving text vectors that are most similar to the input vector one by one in the vector database, resulting in low retrieval efficiency.

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0068] This application addresses the issues of low retrieval accuracy and efficiency in existing RAG retrieval methods by providing a retrieval method, apparatus, processing equipment, program product, and storage medium.

[0069] like Figure 1 As shown, one embodiment of this application provides a retrieval method, including:

[0070] Step 101: Obtain the topic sentence corresponding to the target document and at least two text blocks corresponding to the target document. The topic sentence is used to characterize the global semantic features of the target document, and the text block is the smallest sentence unit in the target document that carries independent and complete semantics and is segmented according to the principle of semantic integrity.

[0071] Here, the target document can specifically be a text document of type PDF, Word, TXT, etc.

[0072] It should be noted that, as an optional example, in step 101, the target document can be parsed (i.e., preprocessed) first, and then the parsed text can be imported into the large language model to obtain the main idea sentence of the whole text (i.e., the main idea sentence, or the baseline sentence).

[0073] It should also be noted that, as an optional example, the target document can be segmented into several text blocks with complete meanings, based on the semantics of the text, using a complete sentence in the target document (that is, the smallest sentence unit that carries independent and complete semantics) as the unit.

[0074] Step 102: Determine the first similarity corresponding to each of the text blocks and the second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the topic sentence.

[0075] Here, both the first and second similarities can be measured using the cosine similarity index.

[0076] Specifically, starting with the first text block, the similarity (i.e., the first similarity) between each text block and the topic sentence can be calculated one by one. The result (i.e., the first similarity, or the baseline similarity value) is recorded as O. i (i=1,2,3,4,...), where the maximum value of i is the total number of text blocks. Starting with the first text block, we can calculate the similarity (i.e., the second similarity) between that text block and all other text blocks in the entire text, then calculate the similarity between the second text block and other text blocks, and so on, to calculate the second similarity between every two text blocks.

[0077] Step 103: Determine the text field corresponding to each text block based on the first similarity of each text block and the second similarity between every two text blocks, wherein the text field is a set of at least one text block.

[0078] Step 104: Generate response information for the user's query information based on the text fields corresponding to each of the text blocks.

[0079] In this embodiment, by segmenting the target document into the smallest sentence units (text blocks) according to the principle of semantic integrity and calculating the first similarity between each text block and the topic sentence, the semantic fragmentation problem caused by traditional word segmentation retrieval is avoided, improving the accuracy of document structured semantic capture and thus significantly improving retrieval precision. By defining a text domain as a set of text blocks and dynamically aggregating related text blocks based on the first and second similarities, full document traversal can be avoided, improving retrieval efficiency, reducing system computational overhead, and reducing invalid and redundant information in user query responses. Thus, the present invention is a reasonable retrieval method that can solve the problems of low retrieval precision and efficiency of existing RAG retrieval methods.

[0080] In some embodiments, step 103, determining the text field corresponding to each text block based on the first similarity corresponding to each text block and the second similarity between every two text blocks, includes:

[0081] Step 1031: Determine the baseline deviation value between every two text blocks based on the first similarity corresponding to each text block. The baseline deviation value is the difference between the first similarity values ​​corresponding to the two text blocks.

[0082] Here, the baseline similarity difference (i.e., baseline deviation value) between different text blocks can be calculated one by one, and the baseline deviation value is denoted as D. ij (i=1,2,3,4,...;j=1,2,3,4,...;i≠j), D ij =O i -O j(i=1,2,3,4,...;j=1,2,3,4,...;i≠j), O i and O j This represents the first similarity between two different text blocks. For example, if text block W1 has a cosine similarity of 0.81 with the main idea sentence, and text block W2 has a cosine similarity of 0.53 with the main idea sentence, then the baseline deviation value D between text block W1 and text block W2 is... 12 The value is 0.81 - 0.53 = 0.28.

[0083] Step 1032: Based on the first threshold, determine the first target text field corresponding to each first text block, wherein the reference deviation value between any text block in the first target text field and the first text block is less than the first threshold, and the first text block is any one of the at least two text blocks.

[0084] It should be noted that the baseline deviation value D between the two text blocks i The smaller the value, the less the two text blocks deviate from the main idea sentence, and the higher the similarity between the two text blocks. In step 1032, text blocks with a baseline deviation value less than the first threshold can be selected according to a preset first threshold to form the first target text field corresponding to the first text block.

[0085] Step 1033: Based on the second threshold, determine the second target text domain corresponding to each of the first text blocks, wherein the second similarity between any text block in the second target text domain and the first text block is greater than the second threshold.

[0086] Here, the execution order of steps 1033 and steps 1031-1032 is not limited, and is only an optional example. For example, step 1033 can be executed first, followed by steps 1031-1032, or they can be executed simultaneously.

[0087] In an optional example, for each text block (e.g., the first text block), all text blocks with a second similarity greater than a second threshold to the first text block can be initially classified into the same text domain (i.e., the second target text domain corresponding to the first text block), and this second target text domain is denoted as A. i (i=1,2,3,4,...); then, all text blocks whose baseline deviation value from the first text block is less than the first threshold Ds are divided into the same domain (i.e., the first target text domain corresponding to the first text block), denoted as B.

[0088] Step 1034: Determine the intersection between the first target text field and the second target text field as the text field corresponding to the first text block.

[0089] Here, the second target text field A i The intersection with the first target text domain B (i.e., A) i ∩B) Finally determine the text field corresponding to the first text block. In this way, the double verification method can ensure that the content of text blocks in the same text field is highly related.

[0090] In this embodiment, when calculating the similarity between text blocks for text segmentation, a cross-validation approach is adopted, introducing a baseline deviation value as a metric. First, the cosine similarity between each text block and the main idea sentence is calculated as the baseline similarity value (i.e., the first similarity). Then, the baseline deviation value between different text blocks is calculated to measure the magnitude of the similarity deviation between different text blocks. Next, the cosine similarity between each text block in the entire text (i.e., the second similarity) is calculated one by one. In this way, from the perspective of the entire text, the baseline deviation value is considered during the process of dividing text domains, and text blocks with high similarity and small baseline deviation values ​​are divided into the same text domain. This makes the content of text blocks in each text domain highly correlated, which can effectively improve the quality of text segmentation while ensuring that the text block size is appropriate, and ensure that the text blocks in each text domain are highly correlated.

[0091] In some embodiments, generating response information for user queries based on the text fields corresponding to each of the text blocks includes:

[0092] Step (1): Generate at least one keyword tag corresponding to each text field based on the text blocks in each text field.

[0093] Here, the TextRank algorithm can be applied to extract the top N keywords (KW) from the content of each text field, and each text field can be labeled with a keyword. The value of N can be determined based on the total number of words in the target document. For example, when the total number of words W < 1000, N = W * 1%; when the total number of words ≥ 1000, N = 10.

[0094] Step (ii): Generate the text block vector corresponding to each of the text blocks.

[0095] It is understandable that the execution order of this step and the previous step (i.e., the step of generating keyword tags) is not limited to this; this is just an example.

[0096] In this step, each text block can be vectorized to generate a text block vector corresponding to each text block. For example, text block ① can be converted into the vector expression [X1,X2,X3,...] (that is, the text block vector corresponding to text block ①), text block ② can be converted into the vector expression [Y1,Y2,Y3,...], and text block ③ can be converted into the vector expression [Z1,Z2,Z3,...].

[0097] Step (3): Generate a text field vector corresponding to each text field based on the text blocks in each text field and the text block vector corresponding to each text block, wherein each text field corresponds one-to-one with the text field vector.

[0098] Here, each text field can be vectorized, meaning the vector representation of each text block within the text field is transformed into an element of the text field's vector representation. For example, a text field contains three text blocks: text block ①, text block ②, and text block ③. The vector expressions for these three text blocks are [X1,X2,X3,...], [Y1,Y2,Y3,...], and [Z1,Z2,Z3,...], respectively. The vector expression [X1,X2,X3,...] for text block ① is transformed into X... 11 X 11 That is, as the first element in the text field vector expression, the vector expression [Y1,Y2,Y3,...] of text block ② is transformed into Y. 11 Y 11 That is, as the second element in the text field vector expression, the vector expression [Z1,Z2,Z3,...] of text block ③ is transformed into Z. 11 Z 11 This is the third element in the vector expression for the text field. Therefore, the vector expression for this text field is [X]. 11 ,Y 11 Z 11 ], where X 11 Mapped to vectors [X1,X2,X3,...], Y 11 Mapped to vectors [Y1,Y2,Y3,...], Z 11 The mapping is to vector [Z1,Z2,Z3,...].

[0099] It should be noted that after generating text block vectors and text field vectors, they need to be stored in a vector database and indexed. This way, the vector database can be queried through the index.

[0100] Step (iv): Generate response information for the user's query information based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector.

[0101] In this embodiment, based on the hierarchical retrieval concept, a text domain vector is constructed based on the text block vector. Each numeric element in the text domain vector represents a text block vector. That is, the digital fingerprint (DF) of the text domain contains the digital fingerprint information of the text block. In this way, by further adopting the retrieval mode of "input question vector -> text domain vector -> text block vector", the efficiency of vector retrieval can be greatly improved.

[0102] In some embodiments, generating response information for user queries based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector includes:

[0103] (i) Obtain at least one question keyword and the question vector corresponding to the user query information.

[0104] Here, the question can be vectorized (that is, the question statement is vectorized, or the user query information is vectorized) to obtain the question vector (denoted as [v1, v2, v3, ..., v n After that, similar text domain vectors can be located and matched based on the question vector.

[0105] As an alternative example, the TextRank algorithm can be used to identify the question keywords corresponding to the user's query information.

[0106] (ii) By keyword matching, at least one candidate text field is determined, wherein at least one keyword tag corresponding to the candidate text field matches at least one of the question keywords.

[0107] Here, by using the question keywords, matching text field keyword tags can be queried in the vector database. Furthermore, the relevant text fields can be determined based on the matched keyword tags. For example, if the keyword tag corresponding to a text field in the vector database matches the question keywords, then that text field can be identified as a candidate text field.

[0108] (iii) Generate response information for the user's query information based on the at least one candidate text field, the question vector, the text block vector, and the text field vector.

[0109] In this embodiment, during retrieval and matching, a "double insurance" cross-validation matching mode of keyword matching and text domain vector matching is used to generate response information for user query information, which can greatly improve the accuracy of retrieval and matching.

[0110] In some embodiments, generating response information for user query information based on the at least one candidate text field, the question vector, the text block vector, and the text field vector includes:

[0111] Step 1: Based on the question vector, determine the third similarity corresponding to each of the text domain vectors, whereby the third similarity is the similarity between the text domain vector and the question vector.

[0112] Step 2: Based on the third threshold and the third similarity corresponding to each of the text domain vectors, determine at least one candidate text domain vector, wherein the candidate text domain vector is a text domain vector whose third similarity is greater than the third threshold.

[0113] In steps 1 and 2 above, the cosine similarity (i.e., the third similarity) between each text field vector and the question vector is calculated one by one. Then, based on the pre-set cosine similarity threshold P (i.e. the third threshold), text field vectors with a third similarity greater than the third threshold are selected as candidate text field vectors (i.e., similar text field vectors of the question vector).

[0114] Step 3: Determine the intersection of the text field vectors corresponding to the at least one candidate text field and the at least one candidate text field vectors as the target text field vector set.

[0115] Here, the candidate text domains are obtained by matching question keywords in the previous steps, and the candidate text domain vectors are obtained by matching question vectors. In this step 3, the intersection of the text domain vectors matched by the two methods is determined as the target text domain vector set.

[0116] Step 4: Generate response information for the user's query information based on the question vector and the target text domain vector set.

[0117] In this step, response information can be further generated based on the target text domain vector set determined in the previous step.

[0118] In some specific embodiments, generating response information for user query information based on the question vector and the target text domain vector set includes:

[0119] Determine the fourth similarity between the question vector and each text field vector in the target text field vector set; determine the text field vector with the highest fourth similarity in the target text field vector set as the target text field vector corresponding to the question vector; generate response information for the user's query information based on the text field corresponding to the target text field vector.

[0120] It should be noted that the most similar text found (i.e., the text in the text domain corresponding to the target text domain vector) can be used as the prompt word of the LLM large model, thereby generating a natural, fluent, easy-to-understand and accurate answer (i.e. response information).

[0121] In this embodiment, the target text field vector that best matches the question vector can be selected from the target text field vector set, and the final response information can be generated based on the text field corresponding to the target text field vector.

[0122] The following provides specific examples illustrating the retrieval method provided in the embodiments of the present invention.

[0123] like Figure 2 As shown, in an AI question-answering process, the user poses a question to the AI ​​(i.e., inputs a query), and the AI ​​searches the knowledge base for matches, inputting the top N most relevant results into the LLM (Large Language Model). Based on its powerful text generation capabilities, the LLM integrates and refines the input information into a natural and fluent answer (i.e., a response to the user's query) and returns it to the user. The AI ​​can apply the retrieval method provided in this embodiment of the invention during the retrieval and matching process. By integrating this retrieval method into question-answering agents in various question-answering scenarios, 24 / 7 question-answering services can be provided to customers, improving accuracy by 10% and efficiency by 60%.

[0124] like Figure 3 As shown, in a specific embodiment, the retrieval method provided by this invention mainly includes the following steps:

[0125] Step 1: Divide the text in the target document into complete sentences (that is, according to the principle of semantic integrity), with each sentence as a text block;

[0126] Step 2: Starting from the first sentence (i.e. the first text block), calculate the cosine similarity (i.e. the second similarity) between the first sentence and every other sentence (i.e., all other text blocks except the first text block).

[0127] Step 3: Sentences with a cosine similarity greater than the second threshold and the first sentence are initially classified into the same text domain (i.e., the second target text domain).

[0128] Step four: Import the text from the target document into the large language model to obtain the main idea sentence of the entire text;

[0129] Step 5: Calculate the relevance of each sentence to the main idea sentence using cosine similarity (i.e., the baseline relevance value, also known as the first similarity).

[0130] Step 6: Based on the first similarity calculated in Step 5, calculate the baseline deviation value between each pair of sentences one by one;

[0131] Step 7: Divide the first target text domain according to the preset first threshold D: If the baseline deviation between two sentences is less than the first threshold D, the two sentences are considered to be related and can be divided into the first target text domain; if the baseline deviation between two sentences is greater than the first threshold D, the two sentences are considered to be unrelated.

[0132] Step 8: Take the intersection of the results of Step 3 and Step 7 (that is, the intersection of the second target text field and the first target text field) to finally determine the text blocks in each text field;

[0133] Step 9: Repeat steps 2 to 8 to divide all sentences into text domains, obtaining the text domain corresponding to each sentence (text block);

[0134] Step 10: Use the TextRank algorithm to extract keywords for each text field, identify the core theme of each text field, and assign keyword tags to each text field;

[0135] Step 11: Construct text block vectors and text field vectors, where the text field vectors are a set of text block vectors, and each text field vector contains all the text block vectors under that field;

[0136] Step 12: Use keyword + vector retrieval to quickly locate and retrieve the most similar vector from the vector database. This involves precisely identifying text domain vectors by inputting keywords and vectors, and then matching the most similar text block vectors from these text domain vectors.

[0137] The aforementioned retrieval methods can automate the generation of large amounts of information, reducing the time cost of manual data searching and content creation, and improving enterprise operational efficiency. For example, in areas such as content creation and customer service, it can significantly improve work efficiency, enabling enterprises to handle more tasks in the same amount of time or complete the same workload with fewer people. It can also provide users with more accurate and personalized services and content, enhancing user satisfaction and loyalty. For instance, intelligent customer service can reduce user waiting time, effectively resolve user problems, and improve user experience by providing quick and accurate answers. Furthermore, it can provide more comprehensive and accurate information for enterprise decision-making, helping enterprises make more informed decisions. For example, in market research and competitive analysis, RAG can quickly retrieve and analyze large amounts of market data and industry dynamics, providing strong support for enterprises to formulate strategies, product plans, and marketing strategies, reducing decision-making risks, and improving enterprise competitiveness.

[0138] The retrieval method described in this application embodiment is based on partitioned digital fingerprints, which realizes efficient RAG retrieval. While effectively improving the quality of text segmentation, it significantly improves the efficiency of text retrieval and can generate the required content more quickly and accurately. It has broad application prospects in fields such as intelligent customer service, content generation, intelligent recommendation, and knowledge graphs.

[0139] like Figure 4 As shown, one embodiment of this application also provides a retrieval device, the device comprising:

[0140] The first acquisition module 410 is used to acquire the main idea sentence corresponding to the target document and at least two text blocks corresponding to the target document, wherein the main idea sentence is used to characterize the global semantic features of the target document, and the text blocks are the smallest sentence units in the target document that are divided according to the principle of semantic integrity and carry independent and complete semantics.

[0141] The first processing module 420 is used to determine the first similarity corresponding to each of the text blocks and the second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the topic sentence;

[0142] The second processing module 430 is used to determine the text field corresponding to each text block based on the first similarity corresponding to each text block and the second similarity between each two text blocks, wherein the text field is a set of at least one text block;

[0143] The response generation module 440 is used to generate response information for user query information based on the text fields corresponding to each of the text blocks.

[0144] In this embodiment, by segmenting the target document into the smallest sentence units (text blocks) according to the principle of semantic integrity and calculating the first similarity between each text block and the topic sentence, the semantic fragmentation problem caused by traditional word segmentation retrieval is avoided, improving the accuracy of document structured semantic capture and thus significantly improving retrieval precision. By defining a text domain as a set of text blocks and dynamically aggregating related text blocks based on the first and second similarities, full document traversal can be avoided, improving retrieval efficiency, reducing system computational overhead, and reducing invalid and redundant information in user query responses. Thus, the present invention is a reasonable retrieval method that can solve the problems of low retrieval precision and efficiency of existing RAG retrieval methods.

[0145] Optionally, the second processing module 430 includes:

[0146] The first processing submodule is used to determine the baseline deviation value between each two text blocks based on the first similarity corresponding to each text block, wherein the baseline deviation value is the difference between the first similarity values ​​corresponding to the two text blocks;

[0147] The second processing submodule is used to determine the first target text field corresponding to each first text block according to the first threshold, wherein the reference deviation value between any text block in the first target text field and the first text block is less than the first threshold, and the first text block is any one of the at least two text blocks;

[0148] The third processing submodule is used to determine the second target text domain corresponding to each first text block according to the second threshold, wherein the second similarity between any text block in the second target text domain and the first text block is greater than the second threshold.

[0149] The fourth processing submodule is used to determine the intersection between the first target text field and the second target text field as the text field corresponding to the first text block.

[0150] Optionally, the response generation module 440 includes:

[0151] The first generation submodule is used to generate at least one keyword tag corresponding to each text field based on the text blocks in each text field.

[0152] The second generation submodule is used to generate text block vectors corresponding to each of the text blocks;

[0153] The third generation submodule is used to generate a text field vector corresponding to each text field based on the text blocks in each text field and the text block vector corresponding to each text block, wherein the text field and the text field vector correspond one-to-one.

[0154] The response generation submodule is used to generate response information for user query information based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector.

[0155] Optionally, the response generation submodule includes:

[0156] A vector acquisition unit is used to acquire at least one question keyword corresponding to the user query information and a question vector corresponding to the user query information.

[0157] The first selection unit is used to determine at least one candidate text field by keyword matching, wherein at least one keyword tag corresponding to the candidate text field matches at least one of the question keywords;

[0158] The response generation unit is configured to generate response information for user query information based on the at least one candidate text field, the question vector, the text block vector, and the text field vector.

[0159] Optionally, the response generation unit includes:

[0160] The first processing subunit is configured to determine the third similarity corresponding to each of the text domain vectors based on the question vector, wherein the third similarity is the similarity between the text domain vector and the question vector;

[0161] The second processing subunit is used to determine at least one candidate text field vector based on the third threshold and the third similarity corresponding to each of the text field vectors, wherein the candidate text field vector is a text field vector whose third similarity is greater than the third threshold.

[0162] The third processing subunit is used to determine the intersection between the text field vector corresponding to the at least one candidate text field and the at least one candidate text field vector as the target text field vector set.

[0163] The fourth processing subunit is used to generate response information for the user's query information based on the question vector and the target text domain vector set.

[0164] Optionally, the fourth processing subunit includes:

[0165] The fifth processing subunit is used to determine the fourth similarity between the question vector and each text domain vector in the target text domain vector set, respectively;

[0166] The sixth processing subunit is used to determine the text field vector with the highest similarity in the target text field vector set as the target text field vector corresponding to the question vector;

[0167] The response generation subunit is used to generate response information for the user's query information based on the text field corresponding to the target text field vector.

[0168] The apparatus provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0169] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] One embodiment of this application also provides a processing device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the retrieval method as described in any of the preceding claims.

[0172] The specific implementation of the retrieval method by the program running on the processor of the processing device can be found in the detailed description of the retrieval method, and will not be repeated here.

[0173] In addition, specific embodiments of this application also provide a readable storage medium storing a program. When executed by a processor, this program implements the various processes of the above-described retrieval method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0174] This application also provides a computer program product, including computer instructions. When executed by a processor, the computer instructions implement the various processes of the above-described retrieval method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0175] Optionally, embodiments of this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] The computer program product described in this application includes computer instructions that, when executed by a processor, implement the various processes of the method embodiments shown above and achieve the same technical effects. To avoid repetition, these will not be repeated here.

[0177] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0178] It should be noted that many of the functional components described in this specification are referred to as modules / submodules in order to more specifically emphasize the independence of their implementation.

[0179] In this application embodiment, the module / submodule can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0180] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0181] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0182] The above describes the preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A retrieval method, characterized in that, The method includes: Obtain the topic sentence corresponding to the target document and at least two text blocks corresponding to the target document, wherein the topic sentence is used to characterize the global semantic features of the target document, and the text blocks are the smallest sentence units in the target document that are segmented according to the principle of semantic integrity and carry independent and complete semantics; Determine the first similarity corresponding to each of the text blocks and the second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the topic sentence; Based on the first similarity corresponding to each text block and the second similarity between every two text blocks, the text field corresponding to each text block is determined, wherein the text field is a set of at least one text block; Based on the text fields corresponding to each of the text blocks, generate response information for the user's query.

2. The retrieval method according to claim 1, characterized in that, The step of determining the text field corresponding to each text block based on the first similarity corresponding to each text block and the second similarity between every two text blocks includes: Based on the first similarity corresponding to each of the text blocks, a baseline deviation value is determined between every two text blocks, wherein the baseline deviation value is the difference between the first similarity values ​​corresponding to the two text blocks; Based on the first threshold, a first target text field corresponding to each first text block is determined, wherein the reference deviation value between any text block in the first target text field and the first text block is less than the first threshold, and the first text block is any one of the at least two text blocks; Based on the second threshold, a second target text domain corresponding to each first text block is determined, and the second similarity between any text block in the second target text domain and the first text block is greater than the second threshold. The intersection between the first target text field and the second target text field is determined as the text field corresponding to the first text block.

3. The retrieval method according to claim 1, characterized in that, The step of generating response information for user queries based on the text fields corresponding to each of the text blocks includes: Based on the text blocks in each text field, generate at least one keyword tag corresponding to each text field; Generate text block vectors corresponding to each of the text blocks; Based on the text blocks in each text field and the text block vectors corresponding to each text block, a text field vector corresponding to each text field is generated, and the text fields and the text field vectors correspond one-to-one. Based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector, a response message for the user's query information is generated.

4. The retrieval method according to claim 3, characterized in that, The step of generating response information for user queries based on at least one keyword tag corresponding to each of the text fields, the text block vector, and the text field vector includes: Obtain at least one question keyword and a question vector corresponding to the user query information; By keyword matching, at least one candidate text field is determined, wherein at least one keyword tag corresponding to the candidate text field matches at least one of the question keywords; Based on the at least one candidate text field, the question vector, the text block vector, and the text field vector, a response message for the user's query information is generated.

5. The retrieval method according to claim 4, characterized in that, The step of generating response information for user query information based on the at least one candidate text field, the question vector, the text block vector, and the text field vector includes: Based on the question vector, a third similarity is determined for each of the text domain vectors, wherein the third similarity is the similarity between the text domain vector and the question vector; Based on the third threshold and the third similarity corresponding to each of the text field vectors, at least one candidate text field vector is determined, wherein the candidate text field vector is a text field vector whose third similarity is greater than the third threshold. The intersection of the text field vectors corresponding to the at least one candidate text field and the at least one candidate text field vectors is determined as the target text field vector set; Based on the question vector and the target text domain vector set, a response message is generated for the user's query information.

6. The retrieval method according to claim 5, characterized in that, The step of generating response information for the user's query information based on the question vector and the target text domain vector set includes: Determine the fourth similarity between the question vector and each text domain vector in the target text domain vector set; The text field vector with the highest fourth similarity in the target text field vector set is determined as the target text field vector corresponding to the question vector; Based on the text field corresponding to the target text field vector, generate response information for the user's query.

7. A retrieval device, characterized in that, The device includes: The first acquisition module is used to acquire the topic sentence corresponding to the target document and at least two text blocks corresponding to the target document, wherein the topic sentence is used to characterize the global semantic features of the target document, and the text blocks are the smallest sentence units in the target document that are divided according to the principle of semantic integrity and carry independent and complete semantics. The first processing module is used to determine the first similarity corresponding to each of the text blocks and the second similarity between every two text blocks, wherein the first similarity is the similarity between the text block and the topic sentence; The second processing module is used to determine the text field corresponding to each text block based on the first similarity corresponding to each text block and the second similarity between every two text blocks, wherein the text field is a set of at least one text block; The response generation module is used to generate response information for user queries based on the text fields corresponding to each of the text blocks.

8. A processing device, characterized in that, It includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the retrieval method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the retrieval method as described in any one of claims 1 to 6.

10. A readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the steps of the retrieval method as described in any one of claims 1 to 6.

Images