An enhanced retrieval generation method based on parent-child segmentation and multi-source recall

Abstract

The application provides an enhanced retrieval generation method based on parent-child segmentation and multi-source recall, which is deployed in a system comprising a parent segment library, a child segment vector sub-library, a recaller and a generator. An original document set is cut into parent segments and child segments by a document segmenter, and child segment vectors are stored in the child segment vector sub-library. The method flow is as follows: obtaining a user query vector, judging whether a first recall condition is triggered, if yes, searching for matched child segment vectors in the child segment vector sub-library through a first recall channel, obtaining corresponding parent segment identifiers and locating parent segments, executing parent-child segment expansion to obtain an expanded child segment set, inputting the generator to generate an answer segment, and finally writing the answer segment into an answer buffer based on a fusion strategy. The application can improve the accuracy, integrity and semantic consistency of the retrieval generated answer.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval and natural language processing, and more particularly, to an enhanced retrieval generation method based on parent-child segmentation and multi-source recall. BACKGROUND

[0002] In today's information explosion era, users' demand for information retrieval is growing, especially in the massive document data, quickly and accurately finding the required information has become an important research direction. Traditional retrieval systems mainly rely on keyword matching technology, searching in documents through user input keywords, returning a list of documents related to the keywords. However, this method has obvious limitations. First, keyword retrieval cannot understand the true intention of user queries, which can easily lead to inaccurate and incomplete retrieval results. Second, traditional retrieval systems perform poorly when dealing with complex queries, and cannot effectively handle queries with ambiguous semantics or multiple concepts. In addition, traditional retrieval systems lack context awareness when generating answers, and cannot effectively integrate and optimize the retrieved information, resulting in generated answer fragments that may lack coherence and completeness.

[0003] With the development of natural language processing technology, semantic retrieval-based methods have emerged, which vectorize documents and queries, and use vector space models to calculate similarity to retrieve relevant information. Although this method improves the accuracy of retrieval to some extent, there are still some problems. For example, the vectorized document is usually a whole without fine-grained segmentation processing, which leads to inaccurate positioning of the key part in the document during retrieval. In addition, existing semantic retrieval systems often lack effective fusion mechanisms when dealing with multi-source data, and cannot fully utilize information from different sources to improve the quality of retrieval results.

[0004] In the implementation of the embodiments of the present application, there are at least the following problems or defects in the prior art: the traditional retrieval system cannot understand the true intention of user queries, resulting in inaccurate retrieval results; although the existing semantic retrieval method improves the accuracy of retrieval, it lacks fine-grained segmentation processing and effective multi-source data fusion mechanism, and cannot fully utilize the key information in the document and data from different sources to improve the quality of retrieval results. SUMMARY

[0005] The application provides an enhanced retrieval generation method based on parent-child segmentation and multi-source recall, which is deployed in an enhanced retrieval generation system, the enhanced retrieval generation system comprising a parent segment library storing an original document set, a node indexer storing a subsegment vector sublibrary, a recaller deploying a source recall channel, and a generator for generating an answer, the original document set being divided into multiple parent segments by a document segmenter, each parent segment being further divided into multiple subsegments, the node indexer performing vectorization on each subsegment and obtaining a subsegment vector, the enhanced retrieval generation method comprising:

[0006] obtaining a query vector input by a user, judging whether a first recall condition is triggered based on the query vector, if yes, activating a first recall channel;

[0007] retrieving at least one first subsegment vector matching the query vector in the subsegment vector sublibrary through the first recall channel;

[0008] obtaining a corresponding parent segment identifier based on the first subsegment vector;

[0009] locating a corresponding parent segment according to the parent segment identifier, and performing parent-child segmentation expansion based on the parent segment to obtain an expanded subsegment set;

[0010] inputting the expanded subsegment set into the generator and generating a first answer segment;

[0011] writing the first answer segment into an answer buffer based on a first fusion strategy.

[0012] Further, before retrieval through the first recall channel, further comprising:

[0013] calculating a first semantic density based on the query vector;

[0014] if the first semantic density is lower than a first density threshold, performing query expansion on the query vector based on a preset synonym mapping table and context expansion rule to obtain an expanded query vector according to a comparison result of the first semantic density and the first density threshold;

[0015] re-inputting the expanded query vector as a new query vector into the first recall channel, and recording an expansion mark for subsequent fusion stage weighting.

[0016] Further, after writing the first answer segment into the answer buffer based on the first fusion strategy, further comprising:

[0017] calling an entity coverage detector based on the answer segment in the answer buffer to obtain an uncovered entity set;

[0018] activating a second recall channel based on a comparison result of an element number of the uncovered entity set and a first number threshold, if the element number is greater than the first number threshold;

[0019] performing semantic complementary retrieval in the parent segment library through the second recall channel to obtain at least one second parent segment;

[0020] performing secondary segmentation based on the second parent segment to obtain a second sub-segment set through a sub-segment splitter;

[0021] inputting the second sub-segment set into a generator and generating a second answer segment;

[0022] merging the second answer segment into an answer buffer after paragraph-level alignment and redundancy elimination based on a second fusion strategy.

[0023] Further, after merging based on the second fusion strategy, it further includes:

[0024] calling a confidence evaluator based on the answer buffer to calculate a fusion confidence;

[0025] activating a third recall channel based on a comparison result of the fusion confidence and a first confidence threshold, if the fusion confidence is lower than the first confidence threshold;

[0026] calling an external knowledge graph interface through the third recall channel to obtain an external knowledge segment based on a core entity of a query vector;

[0027] calling a conflict detector based on the external knowledge segment to cross verify with the existing answer segment, and if a conflict is detected, generating a correction prompt based on a conflict position;

[0028] inputting the external knowledge segment and the correction prompt into the generator to generate a third answer segment;

[0029] inserting the third answer segment into a corresponding paragraph position of the answer buffer and updating a reference index based on a third fusion strategy.

[0030] Further, after updating based on the third fusion strategy, it further includes:

[0031] calling an assertion verifier based on a final answer of the answer buffer to extract all assertion triples;

[0032] performing evidence retrieval in the parent segment library and the external knowledge graph based on each assertion triple to obtain a support evidence list;

[0033] based on a consistency score of the support evidence list and the assertion triple, if there is an assertion with a score lower than a consistency threshold, positioning the assertion as a conflict sub-segment;

[0034] retrieve a modified parent segment with opposite semantics and a score higher than a consistency threshold from a parent segment library based on the conflict subsegment;

[0035] perform subsegment segmentation based on the modified parent segment to obtain a modified subsegment set;

[0036] input the modified subsegment set into a generator and generate a modified answer segment;

[0037] replace the answer segment corresponding to the conflict subsegment based on the modified answer segment, and synchronously update the paragraph index and reference chain of the answer buffer.

[0038] Further, the first recall channel retrieves at least one first subsegment vector matching the query vector from a subsegment vector library, including:

[0039] Calculate a first cosine similarity based on the query vector and each subsegment vector;

[0040] Generate a first ranking list based on all first cosine similarities;

[0041] Get the maximum first cosine similarity as a maximum value based on the first ranking list;

[0042] Based on the comparison result of the maximum value and a preset first similarity threshold, if it exceeds the first similarity threshold, generate a first subsegment vector based on the subsegment vector corresponding to the maximum value, and extract a parent segment identifier based on the subsegment vector.

[0043] Further, locate the corresponding parent segment according to the parent segment identifier and perform parent-child segmentation expansion, including:

[0044] Perform an exact match based on the parent segment identifier in the hash index table of the parent segment library to obtain a target parent segment;

[0045] Get the original text length based on the target parent segment;

[0046] Based on the comparison result of the original text length and a preset first length threshold, if it is greater than the first length threshold, call a sliding segmenter based on a preset window length and a preset step length to perform non-overlapping sliding segmentation on the target parent segment to obtain an expanded subsegment set, and generate a position offset label for each expanded subsegment;

[0047] If it is less than or equal to the first length threshold, encapsulate the target parent segment as a unique expanded subsegment based on the target parent segment as a whole, and generate a whole label to join the expanded subsegment set.

[0048] Further, judge whether the second recall condition is met based on the stored answer segment in the answer buffer, including:

[0049] Based on the stored answer fragment, call the entity recognizer to perform named entity recognition, and obtain the covered entity set;

[0050] Based on the user input query vector, call the same entity recognizer to perform named entity recognition, and obtain the query entity set;

[0051] Based on the query entity set and the covered entity set, perform a difference set operation to obtain the uncovered entity set;

[0052] Based on the uncovered entity set, call the counter to count the number of elements;

[0053] Based on the comparison result of the number of elements and the preset first number threshold, if greater than the first number threshold, generate a second recall trigger signal based on the comparison result and activate the second recall channel.

[0054] Further, based on the final answer, perform fact consistency verification, including:

[0055] Based on the final answer, call the triple extractor to perform syntax dependency analysis and extract all assertion triples;

[0056] Based on the subject and object of each assertion triple, perform keyword retrieval in the inverted index of the parent segment library to obtain a candidate parent segment list;

[0057] Based on the candidate parent segment list, perform sub-segment segmentation, and based on the semantic matcher, calculate the semantic similarity between each sub-segment and the predicate of the assertion triple to obtain a support evidence sub-segment list;

[0058] Based on the consistency score of the support evidence sub-segment list and the assertion triple, if there is an assertion with a score lower than the consistency threshold, generate a conflict sub-segment based on the assertion and output a conflict marker signal;

[0059] If all assertion scores are not lower than the consistency threshold, output a verification pass signal.

[0060] Further, based on the first fusion strategy, write the first answer fragment to the answer buffer, including:

[0061] Based on the first answer fragment, call the keyword extractor to perform TF-IDF weight calculation to obtain a first key information list;

[0062] Based on the text currently stored in the answer buffer, call the same keyword extractor to perform TF-IDF weight calculation to obtain a current key information list;

[0063] Based on the first key information list and the current key information list, perform an intersection operation to obtain a repeated keyword set;

[0064] calculate a repetition rate based on the repeated keyword set and the first key information list;

[0065] If the repetition rate is lower than the first repetition threshold, a buffer writer is invoked based on the comparison result to append the first answer fragment to the end of the answer buffer, and the buffer index is updated.

[0066] If the repetition rate is greater than or equal to the first repetition threshold, a buffer controller is invoked based on the comparison result to skip the write operation, and a skip log and a repetition mark are recorded.

[0067] The above embodiments of the present application have at least the following beneficial effects:

[0068] 1. The original document is finely granulated into parent segments and child segments by the parent-child segmentation technology, realizing multi-level structured processing of the document. In the retrieval process, the child segment matching the query vector can be accurately located, and more rich semantic information can be obtained based on the parent segment context expansion, so that more accurate and complete answer fragments can be generated, effectively solving the problem that the traditional retrieval system cannot accurately locate the key part of the document.

[0069] 2. A multi-source recall mechanism is adopted, and the first, second and third recall channels are combined to retrieve the child segment vector sub-library, the parent segment library and the external knowledge graph respectively. This method makes full use of information resources from different sources, not only can obtain relevant fragments from the internal document, but also can introduce authoritative information from the external knowledge graph for supplement and verification. Through the fusion of multi-source information, the generated answer is more comprehensive and has higher credibility, solving the problem of insufficient multi-source data fusion in the prior art.

[0070] 3. Various fusion strategies and verification mechanisms are introduced, such as the first, second and third fusion strategies, as well as assertion verification and fact consistency verification. These mechanisms can dynamically optimize and correct the generated answer fragments, ensuring that the generated answer is highly consistent with the query vector in semantics, while avoiding the appearance of redundant, conflicting and inaccurate information. In this way, the present application effectively solves the problems of poor coherence, lack of context awareness and insufficient fact consistency of the answer generated by the traditional retrieval system, significantly improving the quality of the generated answer. BRIEF DESCRIPTION OF DRAWINGS

[0071] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0072] Figure 1A flowchart of an enhanced retrieval generation method based on parent-child segmentation and multi-source recall provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0073] The technical solutions in the present application will be described in detail below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0074] As shown in Figure 1 The present application provides an enhanced retrieval generation method based on parent-child segmentation and multi-source recall. The method is deployed in an enhanced retrieval generation system, which includes a parent segment library storing an original document set, a node indexer storing a subsegment vector sublibrary, a recaller deploying a source recall channel, and a generator for generating an answer. The original document set is divided into multiple parent segments by a document segmenter, and each parent segment is further divided into multiple subsegments. The node indexer performs vectorization on each subsegment and obtains a subsegment vector. The enhanced retrieval generation method includes:

[0075] S1, obtaining a query vector input by a user, judging whether a first recall condition is triggered based on the query vector, and if so, activating a first recall channel;

[0076] S2, retrieving at least one first subsegment vector matching the query vector in the subsegment vector sublibrary through the first recall channel;

[0077] S3, obtaining a corresponding parent segment identifier based on the first subsegment vector;

[0078] S4, locating a corresponding parent segment according to the parent segment identifier, and performing parent-child segmentation expansion based on the parent segment to obtain an expanded subsegment set;

[0079] S5, inputting the expanded subsegment set into the generator and generating a first answer segment;

[0080] S6, write the first answer segment into an answer buffer based on a first fusion strategy.

[0081] The enhanced retrieval generation system of the present application includes a parent segment library, a node indexer, a retriever, and a generator. The original document set is cut into multiple parent segments by a document segmenter, and each parent segment is further cut into multiple child segments. The node indexer performs vectorization on each child segment to obtain a child segment vector.

[0082] The system first obtains the query vector input by the user, and judges whether the first recall condition is triggered. If the condition is triggered, the first recall channel is activated. The first recall channel retrieves at least one first child segment vector matching the query vector in the child segment vector sub-library. This retrieval based on child segment vectors can achieve more accurate semantic matching.

[0083] Next, the system obtains the corresponding parent segment identifier based on the first child segment vector. The parent segment identifier can be used to locate the complete parent segment text. The system locates the corresponding parent segment according to the parent segment identifier, and performs parent-child segment expansion based on the parent segment to obtain an expanded child segment set. This step can obtain more complete context information.

[0084] Then, the system inputs the expanded child segment set into the generator to generate the first answer segment. The generator can generate high-quality answers based on more abundant context information. Finally, the system writes the first answer segment into the answer buffer based on the first fusion strategy, completing a retrieval generation process.

[0085] This scheme ensures the accuracy of retrieval and guarantees the completeness of the context of the generation input through the parent-child segment structure and the multi-level recall mechanism, thereby improving the quality of retrieval generation.

[0086] Among them, the query vector refers to the conversion of the user input query text into a high-dimensional vector representation, which can be implemented by using the encoder of the pre-trained language model BERT or Sentence-BERT. Through vectorization processing, the deep semantic information of the query can be captured. The first recall channel refers to a retrieval module based on vector similarity matching, which can be implemented by using the approximate nearest neighbor algorithm FAISS or HNSW. Through vector space fast retrieval of related child segments, the recall efficiency is improved. The parent segment identifier refers to a position marker for uniquely identifying the parent segment in the original document, which can be implemented by using a hash value or a hierarchical coding structure. Through the identifier, the parent segment text is quickly located to reduce storage overhead. The parent-child segment expansion refers to a dynamic expansion process of generating child segments based on parent segments, which can be implemented by using a sliding window algorithm or a semantic boundary detector. Through the hierarchical segmentation strategy, the retrieval precision and context coverage range are balanced. The answer buffer refers to a temporary storage area for storing generated answer segments, which can be implemented by using a doubly linked list or a graph structure. Through buffer management of multi-source recall results, incremental answer fusion is achieved.

[0087] The core innovation of the present application is to construct a hierarchical retrieval architecture through parent-child segmentation and multi-source recall mechanism, to realize fine-grained semantic matching by using sub-section vectors, to dynamically supplement context information by combining parent section expansion strategy, and finally to improve the accuracy and integrity of the generated answer through multi-stage fusion strategy.

[0088] As a preferred embodiment, the scheme of the present application is implemented as follows:

[0089] The enhanced retrieval generation system is implemented by a server, including a storage module, an index module, a recall module and a generation module. The storage module includes a parent section library for storing the original document set. The index module includes a node indexer for storing a sub-section vector sub-library. The recall module includes multiple recall channels. The generation module includes a neural network model for generating answers.

[0090] Firstly, the system receives the natural language query input by the user, and encodes the query text into a query vector through a pre-trained language model. The system calculates the semantic density of the query vector. If the semantic density exceeds the preset threshold, the first recall condition is triggered, and the first recall channel is activated.

[0091] The first recall channel performs a nearest neighbor search in the sub-section vector sub-library, retrieves the K sub-section vectors with the highest cosine similarity with the query vector, and takes them as the first sub-section vector set. The system extracts the parent section identifier from each first sub-section vector.

[0092] Next, the system locuses the corresponding parent section text in the parent section library according to the parent section identifier. For each parent section text, the system re-segments it using the sliding window method to obtain an expanded sub-section set. Specifically, the system sets a fixed-size window, slides it over the parent section text and extracts text segments to form expanded sub-sections.

[0093] Then, the system inputs the expanded sub-section set into a pre-trained generation model. The generation model generates an initial answer text based on the input expanded sub-section set, which is taken as the first answer segment.

[0094] Finally, the system writes the first answer segment into the answer buffer. Specifically, the system first checks whether the answer buffer is empty. If it is empty, the first answer segment is written directly. If it is not empty, the semantic similarity between the first answer segment and the existing content in the buffer is calculated. If the similarity is lower than the threshold, the first answer segment is appended to the end of the buffer. If the similarity is higher than the threshold, the first answer segment is merged with the similar content and written.

[0095] The application further proposes that before retrieval through the first recall channel, a first semantic density is calculated based on a query vector, and according to a comparison result of the first semantic density and a first density threshold, if the first semantic density is lower than the first density threshold, query expansion is performed on the query vector based on a preset synonym mapping table and a context expansion rule to obtain an expanded query vector, the expanded query vector is input as a new query vector to the first recall channel again, and an expansion mark is recorded to facilitate weighted fusion in a subsequent stage.

[0096] The first semantic density is calculated by counting the vector distribution density of the query vector in the adjacent region in the vector space. Specifically, a spherical region within a radius threshold is defined with the query vector as the center, the number of sub-section vectors existing in the region is counted, and the number value is the first semantic density. The first density threshold is set according to the median of the semantic density distribution of the high-recall-rate query in the historical retrieval data. The synonym mapping table is generated by vector clustering of domain terms through a pre-trained language model, and each term is associated with multiple semantically similar synonyms. The context expansion rule is based on matching the query vector with the context mode frequently appearing in the parent section library to extract additional keywords with strong relevance. The expanded query vector is generated by using a weighted average method to fuse the original query vector and the vector of the expanded keyword according to a preset proportion. The expansion mark is recorded in the form of a binary identifier to record the triggering state of the expansion operation, and the answer fragments are dynamically weighted in the subsequent fusion stage through a weight adjustment coefficient.

[0097] Specifically, when the first semantic density is lower than the first density threshold, it indicates that the semantic coverage of the original query vector in the sub-section vector sub-library is insufficient, and expansion is needed. The synonym set of the query keyword is retrieved through the synonym mapping table, and additional keywords related to the query intent are extracted in combination with the context expansion rule. The original query keyword and the expanded keyword are jointly coded into an expanded query vector, which is re-retrieved through the first recall channel to improve the semantic relevance of the recalled sub-sections. The expansion mark is attached to the retrieval result, and in the answer generation stage, the generator weights the answer fragments from the expansion source according to the weight adjustment coefficient of the expansion mark to ensure that the expansion content obtains a reasonable priority in fusion. Thus, in the low semantic density scenario, the dynamic query expansion and weighted fusion mechanism effectively improves the coverage ability of the answer fragments to the user query intent.

[0098] As a preferred embodiment, the scheme of the application is implemented as follows:

[0099] Before retrieval through the first recall channel, a first semantic density is calculated based on a query vector. Specifically, the density of core semantic words in the query vector can be calculated by using a word frequency statistical method. For example, the query vector is subjected to word segmentation processing, the frequency of the core semantic words is counted, and the first semantic density is obtained by dividing the total number of words in the query vector.

[0100] Further, according to the comparison result of the first semantic density and the first density threshold, if lower than the first density threshold, performing query expansion on the query vector based on a preset synonym mapping table and context expansion rules to obtain an expanded query vector. The synonym mapping table can contain common words and their corresponding synonym list. The context expansion rules can define how to add relevant words based on the query context. For example, the keywords can be extracted from the query vector, and then the synonyms of these keywords are found in the synonym mapping table, and the found synonyms are added to the original query. In addition, the context of the query can also be analyzed, and relevant words can be added according to the predefined rules to obtain the expanded query vector.

[0101] Thus, the expanded query vector is re-input as a new query vector to the first recall channel, and an expansion mark is recorded for subsequent fusion stage weighting. Specifically, the expanded query vector can be input into the first recall channel for retrieval, while marking the query vector as an expanded query in the system. This expansion mark can be used to adjust the weight in the subsequent result fusion stage, for example, giving the results obtained by the expanded query a lower weight.

[0102] The application further proposes to call an entity coverage detector based on the stored answer fragments in the answer buffer to obtain an uncovered entity set; based on the comparison result of the number of elements in the uncovered entity set and the first number threshold, if greater than the first number threshold, activate the second recall channel; perform semantic complementary retrieval in the parent segment library through the second recall channel to obtain at least one second parent segment; call the sub-segment splitter based on the second parent segment to perform secondary segmentation to obtain a second sub-segment set; input the second sub-segment set into the generator and generate a second answer fragment; based on the second fusion strategy, the second answer fragment and the stored answer fragment are aligned and redundant content is eliminated at the paragraph level, and then merged into the answer buffer.

[0103] Wherein, the entity coverage detector extracts the entity set in the stored answer fragment through a named entity recognition algorithm, and performs a difference set operation with the entity set in the user query to determine the number of uncovered entities; the second recall channel uses a semantic retrieval algorithm based on the parent segment library, combined with the keywords in the uncovered entity set for semantic expansion, to retrieve parent segments complementary to the original query semantics; the sub-segment splitter dynamically adjusts the segmentation window size according to the parent segment length, and uses sliding window segmentation when the parent segment length exceeds the preset threshold, otherwise, the whole is segmented; the second fusion strategy calculates the semantic overlap and logical relevance between the answer fragments, uses a paragraph alignment algorithm to eliminate redundant content, and ensures the coherence of the merged answer.

[0104] Specifically, the entity coverage detector first performs named entity recognition on the stored answer segment, generates a covered entity list, and compares it with the entity list in the user query to determine the number of uncovered entities. When the number of uncovered entities exceeds the preset threshold, the second recall channel is triggered. The second recall channel performs semantic complementary retrieval in the parent segment library, expands the query semantics with the uncovered entity keywords, and retrieves the parent segment that is not covered by the first recall channel. The retrieved parent segment is divided into a more fine-grained sub-segment set after secondary segmentation, which is input into the generator to generate a supplementary answer segment. The second fusion strategy performs redundancy elimination and logical alignment between the supplementary answer and the stored answer through semantic similarity calculation and paragraph alignment algorithm, and finally merges them into the answer buffer. This process effectively solves the problem of insufficient entity coverage through supplementary retrieval and dynamic segmentation mechanism, and improves the completeness and accuracy of the answer.

[0105] As a preferred embodiment, the scheme of the application is implemented as follows:

[0106] Based on the stored answer segment in the answer buffer, an entity coverage detector is called to obtain an uncovered entity set. The entity coverage detector uses a named entity recognition model based on deep learning to extract entities from the stored answer segment and compare them with the entities in the query to generate an uncovered entity set.

[0107] Based on the comparison result of the number of elements in the uncovered entity set and the first number threshold, if it is greater than the first number threshold, the second recall channel is activated. The first number threshold can be set to 3, and when the number of uncovered entities exceeds 3, the second recall is triggered.

[0108] At least one second parent segment is obtained by performing semantic complementary retrieval in the parent segment library through the second recall channel. The semantic complementary retrieval uses a BERT-based semantic similarity calculation method to retrieve the parent segment with the highest relevance as the second parent segment in the parent segment library based on the uncovered entity as the query.

[0109] Based on the second parent segment, a sub-segment segmenter is called to perform secondary segmentation to obtain a second sub-segment set. The sub-segment segmenter uses a sliding window method with a window size of 128 tokens and a step size of 64 tokens to perform overlapping segmentation on the second parent segment to generate a second sub-segment set. The second sub-segment set is input into the generator to generate a second answer segment. The generator uses a text generation model based on a Text-to-Text Transfer Transformer to generate a supplementary explanatory second answer segment based on the second sub-segment set as the context.

[0110] The second answer fragment is merged into the answer buffer after paragraph-level alignment and redundancy elimination with the existing answer fragments based on a second fusion strategy. The second fusion strategy includes calculating the semantic similarity between the second answer fragment and the existing answer fragments, inserting the part with a similarity lower than 0.7 as supplementary content into the corresponding position of the existing answer, and performing redundancy deletion on the part with a similarity higher than 0.7. Finally, the paragraph index of the answer buffer is updated.

[0111] The application further provides a method, including: calling a confidence evaluator based on the answer buffer to calculate a fusion confidence; activating a third recall channel according to a comparison result of the fusion confidence and a first confidence threshold; calling an external knowledge graph interface through the third recall channel to obtain an external knowledge fragment based on the core entity of the query vector; calling a conflict detector based on the external knowledge fragment to perform fact cross verification with the existing answer fragment, and generating a correction prompt based on a conflict position if a conflict is detected; inputting the external knowledge fragment and the correction prompt into a generator to generate a third answer fragment; and inserting the third answer fragment into a corresponding paragraph position of the answer buffer based on a third fusion strategy and updating a reference index.

[0112] The confidence evaluator calculates the fusion confidence by weighted calculation of semantic coherence and entity coverage, for example, using linear weighting, with the semantic coherence weight set to 0.6 and the entity coverage weight set to 0.4. The third recall channel is configured to establish a connection with the external knowledge graph through an API interface, and the core entity is extracted from the query vector through an entity recognition model, for example, using a BiLSTM-CRF model to recognize entity types. The conflict detector uses a rule-based triple matching algorithm to compare the subject-predicate-object structure in the external knowledge fragment with the existing answer fragment sentence by sentence, and triggers a conflict mark when the predicate logic is inconsistent. The correction prompt is structured as structured data containing the conflict position and the correction instruction, for example, a JSON format containing the conflict paragraph number, the conflict entity pair, and the suggested correction operation. The third fusion strategy uses a paragraph-level insertion mechanism to determine the insertion point through position offset markers, and updates the reference index to associate the external knowledge source.

[0113] Specifically, when the confidence evaluator detects that the fusion confidence is lower than 0.7, the system automatically activates the third recall channel. At this time, the entity recognition model extracts core entities from the query vector, such as identifying the two entities of quantum computing and decoherence. Through the knowledge graph interface, retrieve external knowledge fragments related to the entity, such as obtaining an authoritative paper abstract containing quantum decoherence suppression methods. The conflict detector compares the topology coding error correction method in the external knowledge with the dynamic decoupling scheme in the existing answer, detects that the technical path description is inconsistent, and generates a correction prompt containing the paragraph number of the conflict. The generator receives the external knowledge fragment and the correction prompt, reorganizes the language to generate the third answer fragment, and clearly marks the differences in the application scenarios of the two methods. The third fusion strategy inserts the fragment at the end of the second paragraph in the answer buffer, and adds the DOI identifier of the knowledge graph in the reference index. In this way, the technical description in the answer buffer is supplemented by external authoritative knowledge, the factual conflict is explicitly marked, and the credibility of the overall answer is improved. As a preferred embodiment, the scheme of the present application is implemented as follows:

[0114] The confidence evaluator is called based on the answer buffer to calculate the fusion confidence. The confidence evaluator uses a deep learning-based multi-feature fusion model, which inputs features such as semantic vectors of answer fragments, keyword weights, and entity coverage, and outputs a confidence score between 0 and 1.

[0115] According to the comparison result of the fusion confidence and the first confidence threshold, if it is lower than the first confidence threshold, the third recall channel is activated. The first confidence threshold is set to 0.7. When the fusion confidence is lower than 0.7, the third recall channel is triggered.

[0116] Through the third recall channel, the external knowledge graph interface is called to obtain external knowledge fragments based on the core entities of the query vector. The external knowledge graph uses a large-scale general knowledge graph, which contains hundreds of millions of entities and relationships. Through entity linking, the core entities in the query vector are mapped to the nodes in the knowledge graph, and then multi-hop reasoning is performed to obtain related knowledge fragments.

[0117] Based on the external knowledge fragment, the conflict detector is called to perform factual cross-validation with the existing answer fragment, and if a conflict is detected, a correction prompt is generated based on the conflict position. The conflict detector uses a combination of rules and machine learning to compare the factual statements of external knowledge and existing answers to detect potential contradictions. For the detected conflicts, a prompt is generated to point out the conflict position and suggest a correction direction.

[0118] The external knowledge fragment and the correction prompt are jointly input into the generator to generate the third answer fragment. The generator uses a pre-trained language model to input external knowledge and correction prompts as additional context, guiding the model to generate more accurate answers.

[0119] The third answer fragment is inserted into the corresponding paragraph position of the answer buffer based on a third fusion strategy and the reference index is updated. The third fusion strategy adopts a semantic similarity-based insertion method to insert the newly generated answer fragment into the most semantically similar paragraph position and update the reference relationship between paragraphs.

[0120] The application further proposes calling an assertion verifier based on the final answer to extract all assertion triples, performing evidence retrieval in the parent segment library and external knowledge graph according to each assertion triple to obtain a support evidence list, positioning a conflict subsegment based on the consistency score of the support evidence list and the assertion triple, retrieving a semantic opposite correction parent segment to generate a correction answer fragment to replace the conflict subsegment, and synchronously updating the paragraph index and reference chain of the answer buffer.

[0121] The assertion verifier identifies the subject-predicate-object structure to form an assertion triple through syntax dependency analysis, and each triple contains three semantic units of subject, predicate and object. The evidence retrieval process adopts inverted index to accelerate keyword matching, and the candidate parent segment list is screened by combining multiple keywords through Boolean logic operators. The semantic matcher calculates the vector similarity by using a pre-trained language model, and the consistency score is integrated by using a weighted average algorithm. The correction parent segment retrieval process adopts a bidirectional attention mechanism to capture semantic opposite features, and the correction subsegment set is adaptively divided by using a dynamic window adjustment algorithm. The conflict subsegment replacement operation adopts a version control mechanism to retain historical modification records, and the reference chain updating process establishes the association between the new and old paragraphs by using a hash pointer.

[0122] Specifically, the triple extractor performs dependency syntax analysis on the final answer to identify the subject-predicate-object structure and generate structured assertion data. Each assertion triple performs Boolean retrieval in the inverted index to screen candidate parent segments containing the subject and object. After the candidate parent segments are divided by a sliding window, the semantic matcher calculates the cosine similarity between the subsegments and the assertion predicate, and retains the subsegments with a similarity higher than 0.8 as support evidence. The consistency scoring module calculates the evidence support degree for each assertion, and triggers the conflict detection mechanism when the score is lower than 0.6. The correction parent segment retrieval module generates a negative query vector according to the conflict assertion, and searches for a semantically opposite paragraph in the parent segment library. The secondary segmentation module adopts a variable window mechanism to automatically adjust the segmentation granularity according to the parent segment length to generate correction subsegments. After the correction answer is generated, the version controller creates a new version in the buffer, and the hash indexer establishes the position mapping relationship between the new and old paragraphs to ensure the integrity of the reference relationship.

[0123] As a preferred embodiment, the scheme of the application is implemented as follows:

[0124] The final answer calling assertion verifier based on the answer buffer extracts all assertion triplets. The assertion verifier adopts a dependency syntax analysis-based method to parse each sentence in the final answer, identify the subject, predicate and object, and thereby construct assertion triplets.

[0125] According to each assertion triplet, evidence retrieval is performed in the parent segment library and the external knowledge graph to obtain a support evidence list. During the evidence retrieval process, the inverted index of the parent segment library and the entity linking function of the external knowledge graph are used to quickly locate relevant paragraphs or knowledge entries containing the entities in the assertion triplets.

[0126] Based on the consistency score of the support evidence list and the assertion triplets, if there is an assertion with a score lower than the consistency threshold, the assertion is located as a conflict subsegment. The consistency score is measured by calculating the semantic similarity between the support evidence and the assertion triplets, a pre-trained language model is used for semantic representation, and cosine similarity is used as the scoring indicator.

[0127] Based on the conflict subsegment, a modified parent segment with opposite semantics and a score higher than the consistency threshold is retrieved in the parent segment library. During the retrieval process, the conflict subsegment is first inverted in semantics, and then a vector retrieval technique is used to find semantically similar paragraphs in the parent segment library.

[0128] Based on the modified parent segment, a modified subsegment set is obtained by performing subsegment segmentation. The subsegment segmentation uses a sliding window method, sets a fixed window size and step, and segments the modified parent segment.

[0129] The modified subsegment set is input into the generator to generate a modified answer segment. The generator uses a pre-trained language model to generate a modified answer segment that meets the grammar and semantics, with the modified subsegment set as the context.

[0130] Based on the modified answer segment, the corresponding answer segment of the conflict subsegment is replaced, and the paragraph index and reference chain of the answer buffer are updated synchronously. During the replacement process, the original answer structure is maintained, only the conflict part is replaced, and the corresponding paragraph index and reference relationship are updated.

[0131] The application further proposes a retrieval method, comprising: calculating a first cosine similarity based on a query vector and each subsegment vector; generating a first ranking list based on all first cosine similarities; obtaining a maximum first cosine similarity as a maximum value based on the first ranking list; based on the comparison result of the maximum value and a preset first similarity threshold, if the first similarity threshold is exceeded, a first subsegment vector is generated based on the subsegment vector corresponding to the maximum value, and a parent segment identifier is extracted based on the subsegment vector.

[0132] The first cosine similarity is calculated by using the cosine angle formula in the vector space model, and the query vector and the sub-segment vector are mapped to the same dimensional space for dot product operation. The first ranking list is arranged in descending order of similarity value, and the maximum value screening mechanism compares the similarity peak value with the preset threshold value to determine whether to trigger the subsequent operation. The parent segment identifier is extracted based on the hash matching of the metadata field corresponding to the sub-segment vector, to ensure the accuracy of the index positioning. The preset first similarity threshold is set to 0.85, which is used to filter low-confidence matching results. When the maximum value exceeds the threshold, only the sub-segment vector with the highest similarity is retained as the effective result, to avoid multiple candidate interference.

[0133] Specifically, in the vectorization retrieval process, the cosine similarity calculation performs a standardized inner product operation on the query vector and the sub-segment vector, to eliminate the influence of the vector length difference on the similarity evaluation. The first ranking list arranges the similarity results in descending order through a sorting algorithm, to quickly locate the most relevant sub-segment. The maximum value screening mechanism compares the highest similarity with the preset threshold value, to ensure that the parent segment positioning operation is triggered only when the matching quality meets the requirements. For example, when the highest similarity is 0.9 and the threshold value is 0.85, the sub-segment vector is determined as an effective match, and its corresponding parent segment identifier is quickly extracted through the metadata index table. This process uses a double verification mechanism to ensure the semantic relevance of the retrieval results and avoid parent segment mispositioning caused by low-quality matching, thereby improving the accuracy of subsequent parent-child segment expansion and the effectiveness of answer generation.

[0134] As a preferred embodiment, the scheme of the application is implemented as follows:

[0135] The cosine similarity is calculated based on the query vector and each sub-segment vector in the sub-segment vector library. Specifically, the cosine similarity value of the query vector and each sub-segment vector is calculated by using vector dot product operation.

[0136] A ranking list is generated based on all the calculated cosine similarity values. For example, the cosine similarity values are sorted from large to small to form an ordered list.

[0137] The maximum value of the cosine similarity is obtained from the ranking list. Further, the first element in the ranking list is selected as the maximum similarity value.

[0138] The obtained maximum similarity value is compared with a preset similarity threshold value. The similarity threshold value can be set to 0.8.

[0139] If the maximum similarity value exceeds the similarity threshold value, a first sub-segment vector is generated based on the corresponding sub-segment vector. In this way, the most similar sub-segment vector is selected as the retrieval result.

[0140] The corresponding parent segment identifier is extracted based on the generated first sub-segment vector. Specifically, the unique identifier of the parent segment is extracted from the metadata of the sub-segment vector.

[0141] The present application further proposes to perform an exact match based on the parent segment identifier in the hash index table of the parent segment library to obtain the target parent segment, obtain the original text length and compare it with the preset first length threshold value, if greater than the threshold value, perform non-overlapping sliding segmentation on the target parent segment by a sliding segmenter with a preset window length and step to obtain an expanded sub-segment set, and generate a position offset marker for each expanded sub-segment; if less than or equal to the threshold value, the target parent segment is encapsulated as a unique expanded sub-segment as a whole and a whole marker is generated to join the set.

[0142] The hash index table is used to quickly locate the parent segment, the preset window length and step control the granularity of sliding segmentation, the position offset marker records the starting and ending positions of the sub-segment in the parent segment, and the whole marker identifies the parent segment that is not segmented. The sliding segmenter adopts a non-overlapping manner to avoid information duplication, and the preset window length is dynamically adjusted according to the processing capacity of the generator. For example, when the parent segment text length exceeds 2000 characters, a segmentation method with a window length of 500 characters and a step of 500 characters is adopted.

[0143] Specifically, the exact match quickly obtains the target parent segment through the hash index table, avoiding traversal query to improve efficiency. The text length and the threshold value are compared to determine whether to segment, and when the length exceeds the threshold value, the sliding segmenter divides the parent segment into multiple fixed-length sub-segments, each of which carries a position offset marker to preserve the original context relationship. For parent segments that do not exceed the threshold value, the whole encapsulation preserves the complete semantics. The position offset marker enables the generator to backtrack the position information in the original parent segment when integrating the answer, ensuring the accuracy and coherence of the answer. Non-overlapping segmentation reduces redundant calculation, the preset window length matches the processing capacity of the generator, avoiding performance degradation due to excessively long input. The whole marker simplifies the processing flow, directly passing the complete content for shorter parent segments, avoiding unnecessary segmentation loss.

[0144] As a preferred embodiment, the scheme of the present application is implemented as follows:

[0145] An exact match is performed based on the parent segment identifier in the hash index table of the parent segment library to obtain the target parent segment. The original text length is obtained based on the target parent segment. The original text length is compared with the preset first length threshold value. When the original text length is greater than the first length threshold value, a sliding segmenter is called based on the preset window length and the preset step to perform non-overlapping sliding segmentation on the target parent segment to obtain an expanded sub-segment set, and a position offset marker is generated for each expanded sub-segment. When the original text length is less than or equal to the first length threshold value, the target parent segment is encapsulated as a unique expanded sub-segment as a whole, and a whole marker is generated to join the expanded sub-segment set.

[0146] Specifically, the parent segment library stores a plurality of parent segments after the original document set is segmented, and each parent segment has a unique parent segment identifier. A hash index table is used to quickly locate the parent segment. The first length threshold can be set to 500 characters. The preset window length can be set to 200 characters, and the preset step length can be set to 100 characters. The sliding segmenter segments the parent segment by sliding window, and the step length of each sliding is the preset step length, until the entire parent segment is covered. The position offset marker records the starting position of each expanded subsegment in the original parent segment. The whole marker is used to identify the complete parent segment that is not segmented.

[0147] The application further proposes a method for determining whether the second recall condition is met based on the stored answer fragments in the answer buffer, comprising: calling an entity recognizer to perform named entity recognition based on the stored answer fragments to obtain a covered entity set; calling the same entity recognizer to perform named entity recognition based on the user input query vector to obtain a query entity set; performing a difference set operation based on the query entity set and the covered entity set to obtain an uncovered entity set; calling a counter to count the number of elements based on the uncovered entity set; and based on the comparison result of the number of elements and the preset first number threshold, if greater than the first number threshold, generating a second recall trigger signal based on the comparison result and activating the second recall channel.

[0148] The entity recognizer is implemented using a bidirectional long short-term memory network model, which is pre-trained in the named entity recognition task and loaded into the memory. The difference set operation is implemented through a hash table structure, and the covered entity set is stored in the hash table and the query entity set is traversed for existence verification. The counter uses an ordered set structure based on a red-black tree to count elements, and automatically filters duplicate entity items during the counting process. The first number threshold is set as a dynamic variable, which is calculated by a linear interpolation formula according to the query vector length and the parent segment library size.

[0149] Specifically, when the answer buffer stores the initial answer fragments, the entity recognizer scans the buffer text sentence by sentence, extracts the names of people, places, and organizations, and stores them in the covered entity set. At the same time, the user's original query vector is input into the same entity recognizer, and the same processing procedure is used to obtain the query entity set. The differences between the two sets are quickly compared through the hash table, and the query entities that do not appear in the covered set are classified into the uncovered entity set. When the counter detects that the number of elements in the set exceeds the preset threshold, the system automatically triggers the activation signal of the second recall channel. This mechanism ensures that the second recall is only started when there is indeed a missing entity, avoiding redundant recall operations due to entity coverage misjudgment, and at the same time, the dynamic threshold adjustment adapts to the needs of different size data scenarios.

[0150] As a preferred embodiment, the scheme of the present application is implemented as follows: the stored answer fragments in the answer buffer are input into a well-trained named entity recognition model, which is constructed based on a bidirectional long short-term memory network and a conditional random field, to perform entity type labeling and boundary detection and output a covered entity set. The query vector input by the user is processed by the same named entity recognition model to perform entity extraction on the original text corresponding to the query vector and generate a query entity set. The set operation module performs a difference set calculation on the two entity sets to filter out the query entities that are not covered by the answers and form an uncovered entity set. The statistical module counts the uncovered entity set based on a hash table structure, and when the count value exceeds a preset threshold, a trigger signal is generated and the second recall channel based on semantic complementary retrieval is activated.

[0151] The present application further proposes that the named entity recognizer is called based on the stored answer fragments in the answer buffer to perform named entity recognition and obtain a covered entity set, the same named entity recognizer is called based on the query vector input by the user to perform named entity recognition and obtain a query entity set, a difference set operation is performed based on the query entity set and the covered entity set to obtain an uncovered entity set, and a counter is called based on the uncovered entity set to count the number of elements, and a second recall trigger signal is generated and the second recall channel is activated based on the comparison result of the number of elements and the first number threshold.

[0152] Among them, the entity recognizer adopts a model structure based on the combination of a bidirectional long short-term memory network and a conditional random field, which is used to extract named entities such as names, place names, and organization names from text; the difference set operation stores the entity names of the covered entity set in a hash table, iterates through each entity name of the query entity set and matches it with the hash table, and adds the unmatched entity to the uncovered entity set; the counter adopts a sliding window mechanism to count the total number of different entity types in the uncovered entity set; the first number threshold is dynamically adjusted according to the average number of user queries in historical data, for example, set to trigger the second recall channel when the number of uncovered entities exceeds 30% of the number of query entities.

[0153] Specifically, after the stored answer fragments in the answer buffer are extracted by the entity recognizer to obtain a covered entity set, the system performs the same entity recognition operation on the user's original query to ensure consistent entity recognition standards. The uncovered entity set is filtered out through difference set operation, which can accurately locate the entity information missing in the answer. The counter counts the number of uncovered entities and compares it with the preset threshold, and if it exceeds the threshold, the second recall channel is triggered to generate new answer fragments by supplementing the semantic complementary content in the parent section library. This process realizes the automatic judgment of the recall condition by quantifying the number of uncovered entities, avoids manual intervention, dynamically adjusts the threshold to adapt to the needs of different query scenarios, and effectively improves the entity coverage completeness and semantic coherence of the answer.

[0154] As a preferred embodiment, the scheme of the present application is implemented as follows: after generating the final answer, a triple extractor is called to perform syntactic dependency analysis on the answer text, and all assertion triples are extracted by identifying subject-predicate-object structures. For each assertion triple, a full-text search is performed in the inverted index of the parent segment library with the subject and object as keywords respectively, and a list of candidate parent segments containing at least one keyword is screened. The candidate parent segments are input into a sub-segment splitter, which is divided into multiple sub-segments according to the preset semantic boundary rules. The semantic similarity between each sub-segment and the predicate part of the assertion triple is calculated using a semantic matcher, and the sub-segments with a similarity higher than a preset threshold are retained to form a list of supporting evidence sub-segments. For each assertion triple, the overall consistency score with the list of supporting evidence sub-segments is calculated, and if there is an assertion with a score lower than a preset consistency threshold, the answer paragraph corresponding to the assertion is marked as a conflict sub-segment and a conflict marking signal is triggered; if all assertion scores meet the threshold requirement, a verification pass signal is sent to the system.

[0155] The present application further proposes that the first answer fragment calls a keyword extractor to perform TF-IDF weight calculation to obtain a first key information list; the same keyword extractor is called to perform TF-IDF weight calculation based on the text currently stored in the answer buffer to obtain a current key information list; an intersection operation is performed based on the first key information list and the current key information list to obtain a repeated keyword set; a repetition rate is calculated based on the repeated keyword set and the first key information list; based on the comparison result of the repetition rate and the first repetition threshold, if it is lower than the first repetition threshold, the first answer fragment is appended to the end of the answer buffer by calling the buffer writer, and the buffer index is updated; if it is greater than or equal to the first repetition threshold, the buffer controller skips the write operation and records the skip log and the repetition mark.

[0156] The keyword extractor uses the TF-IDF algorithm to calculate the weight of the words in the text, and the words with high weight are identified as key information. The intersection operation of the first key information list and the current key information list is realized by a hash table for fast matching, and the generation efficiency of the repeated keyword set is improved. The calculation method of the repetition rate is the ratio of the number of repeated keywords to the total number of words in the first key information list, which is used to quantify the degree of content repetition. The buffer writer uses a non-overwrite writing method when appending the answer fragment, which preserves the original text structure. When the buffer controller skips the write operation, it generates a log entry containing a timestamp and a repetition mark, which facilitates subsequent tracing.

[0157] Specifically, when the first answer fragment output by the generator needs to be written into the answer buffer, the system first extracts the TF-IDF key information list of the fragment. At the same time, the same keyword extraction operation is performed on the text currently stored in the buffer to obtain the existing key information list. By comparing the intersection of the two lists, the number of repeated keywords is determined. The repetition rate is calculated by the ratio of the number of repeated keywords to the total number of keywords in the first answer fragment. If the ratio is below the preset threshold, it indicates that the new content has a low degree of repetition with the existing content, allowing the answer fragment to be appended to the end of the buffer and the index to be updated to reflect the position of the new content. If the repetition rate exceeds the threshold, it is determined that the new content has a high degree of redundancy, and the system automatically skips the write operation to avoid wasting storage space. This process automates the writing decision by quantifying the degree of repetition, effectively reducing the accumulation of redundant information. For example, when the first repetition threshold is set to 30%, if the calculated repetition rate is 25%, the system performs the write operation; if the repetition rate reaches 35%, the skip mechanism is triggered. This technical solution balances the contradiction between answer integrity and storage efficiency through dynamic threshold control.

[0158] As a preferred embodiment, the scheme of the application is implemented as follows: when the system generates a first answer fragment containing the preparation process of graphene, a keyword extraction module based on the TF-IDF algorithm is called to calculate the inverse document frequency weight of each word in the fragment, and the chemical vapor deposition and copper foil substrate with a weight value higher than 0.85 are selected as the first key information list. At this time, the answer buffer has stored text content containing chemical vapor deposition and high-temperature annealing, and the same algorithm is used to extract the existing key information list. The two lists are input into the set operation module to perform intersection processing to obtain a repeated keyword set containing chemical vapor deposition. Based on the proportion of the number of repeated keywords to the total number of elements in the first key information list, the repetition rate is calculated to be 33.3%. The system compares this value with the preset 40% repetition threshold to determine that the current repetition rate does not meet the threshold condition, and then calls the buffer management module to append the first answer fragment to the end of the buffer, and records the starting position of the fragment as byte 512 and the ending position as byte 689 in the index table.

[0159] Through the above technical solution, the application effectively avoids the superimposed writing of repeated information in the answer buffer, dynamically controls the fusion operation of the answer fragment through the quantitative repetition rate determination mechanism, improves the information density and diversity of the generated content under the premise of ensuring the integrity of the answer, solves the problem of bloated answers caused by the accumulation of redundant information in traditional retrieval systems, and realizes the precise incremental update of the answer text.

[0160] The above merely provides an example of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An enhanced retrieval generation method based on parent-child segmentation and multi-source recall, the method is deployed in an enhanced retrieval generation system, the enhanced retrieval generation system comprises a parent segment library storing an original document set, a node indexer storing a sub-segment vector sub-library, a recaller deploying a source recall channel, and a generator for generating an answer, the original document set is cut into multiple parent segments by a document segmenter, each parent segment is further cut into multiple sub-segments, the node indexer performs vectorization on each sub-segment and obtains a sub-segment vector, characterized in that, The enhanced retrieval generation method comprises: obtaining a query vector input by a user, determining whether a first recall condition is triggered based on the query vector, if yes, activating a first recall channel; retrieving at least one first sub-segment vector matching the query vector in a sub-segment vector sub-library through the first recall channel; obtaining a corresponding parent segment identifier based on the first sub-segment vector; locating a corresponding parent segment according to the parent segment identifier, and performing parent-child segment expansion based on the parent segment to obtain an expanded sub-segment set; inputting the expanded sub-segment set into the generator and generating a first answer segment; writing the first answer segment into an answer buffer based on a first fusion strategy; after the first answer segment is written into the answer buffer based on the first fusion strategy, further comprising: calling an entity coverage detector based on the stored answer segment in the answer buffer to obtain an uncovered entity set; based on the comparison result of the number of elements in the uncovered entity set and the first quantity threshold, if greater than the first quantity threshold, activating a second recall channel; performing semantic complementary retrieval in a parent segment library through the second recall channel to obtain at least one second parent segment; calling a sub-segment splitter based on the second parent segment to perform secondary segmentation to obtain a second sub-segment set; inputting the second sub-segment set into the generator and generating a second answer segment; after the second answer segment is aligned and redundancy eliminated at the paragraph level with the stored answer segment based on the second fusion strategy, merging into the answer buffer; wherein the second recall channel adopts a semantic retrieval algorithm based on the parent segment library, combined with the keywords in the uncovered entity set for semantic expansion, to retrieve parent segments complementary to the original query semantics.

2. The method of claim 1, wherein, Before retrieving at least one first sub-segment vector matching the query vector in a sub-segment vector sub-library through the first recall channel, further comprising: calculating a first semantic density based on the query vector; based on the comparison result of the first semantic density and the first density threshold, if lower than the first density threshold, performing query expansion on the query vector based on a pre-set synonym mapping table and context expansion rules to obtain an expanded query vector; re-inputting the expanded query vector as a new query vector into the first recall channel, and recording an expansion mark for subsequent fusion stage weighting.

3. The method of claim 2, wherein, After merging the second answer segment into the answer buffer based on the second fusion strategy after aligning and eliminating redundancy at the paragraph level with the stored answer segment, further comprising: calling a confidence evaluator based on the answer buffer to calculate a fusion confidence; based on the comparison result of the fusion confidence and the first confidence threshold, if lower than the first confidence threshold, activating a third recall channel; calling an external knowledge graph interface through the third recall channel to obtain an external knowledge segment based on the core entity of the query vector; calling a conflict detector based on the external knowledge segment to cross verify with the stored answer segment, if a conflict is detected, generating a correction prompt based on the conflict position; inputting the external knowledge segment and the correction prompt into the generator to generate a third answer segment; inserting the third answer segment into a corresponding paragraph position of the answer buffer based on the third fusion strategy and updating the reference index.

4. The method of claim 3, wherein, after merging the third answer segment into the answer buffer based on the third fusion strategy, further comprising: calling an assertion verifier based on the final answer of the answer buffer to extract all assertion triples; performing evidence retrieval in the parent segment library and the external knowledge graph according to each assertion triple to obtain a support evidence list; based on the consistency score of the support evidence list and the assertion triple, if there is an assertion with a score lower than the consistency threshold, positioning the assertion as a conflict subsegment; based on the conflict subsegment, retrieving a modified parent segment in the parent segment library which is semantically opposite and has a score higher than the consistency threshold; based on the modified parent segment, performing subsegment segmentation to obtain a modified subsegment set; inputting the modified subsegment set into the generator and generating a modified answer segment; based on the modified answer segment, replacing the answer segment corresponding to the conflict subsegment, and synchronously updating the paragraph index and the reference chain of the answer buffer.

5. The method of claim 1, wherein, retrieving at least one first subsegment vector matching the query vector in the subsegment vector sublibrary through the first recall channel, comprising: calculating a first cosine similarity based on the query vector and each subsegment vector; generating a first ranking list based on all first cosine similarities; obtaining the first cosine similarity with the largest value as the maximum value based on the first ranking list; based on the comparison result of the maximum value and the preset first similarity threshold, if it exceeds the first similarity threshold, generating a first subsegment vector based on the subsegment vector corresponding to the maximum value, and extracting a parent segment identifier based on the subsegment vector.

6. The method of claim 5, wherein, locating the corresponding parent segment according to the parent segment identifier, and performing parent-child segment expansion based on the parent segment to obtain an expanded subsegment set, comprising: performing an exact match in the hash index table of the parent segment library based on the parent segment identifier to obtain a target parent segment; obtaining the original text length based on the target parent segment; based on the comparison result of the original text length and the preset first length threshold, if it is greater than the first length threshold, calling a sliding segmenter based on a preset window length and a preset step to perform non-overlapping sliding segmentation on the target parent segment to obtain an expanded subsegment set, and generating a position offset marker for each expanded subsegment; if it is less than or equal to the first length threshold, the target parent segment is encapsulated as a unique expanded subsegment, and an overall marker is generated and added to the expanded subsegment set.

7. The method of claim 1, wherein, determining whether the second recall condition is met based on the stored answer segment in the answer buffer, comprising: calling an entity recognizer to perform named entity recognition based on the stored answer segment to obtain a covered entity set; calling the same entity recognizer to perform named entity recognition based on the query vector input by the user to obtain a query entity set; performing a difference set operation based on the query entity set and the covered entity set to obtain an uncovered entity set; calling a counter to count the number of elements based on the uncovered entity set; based on the comparison result of the element number and the preset first number threshold, if it is greater than the first number threshold, generating a second recall trigger signal based on the comparison result and activating the second recall channel.

8. The method of claim 4, wherein, performing fact consistency verification on the final answer, comprising: Based on the final answer, call the triple extractor to perform syntactic dependency analysis and extract all assertion triples; Based on the subject and object of each assertion triple, perform keyword retrieval in the inverted index of the parent segment library to obtain a list of candidate parent segments; Based on the list of candidate parent segments, perform sub-segment segmentation, and based on the semantic matcher, calculate the semantic similarity between each sub-segment and the predicate of the assertion triple to obtain a list of supporting evidence sub-segments; Based on the consistency score of the list of supporting evidence sub-segments and the assertion triple, if there is an assertion with a score below the consistency threshold, generate a conflict sub-segment based on the assertion and output a conflict flag signal; If all assertions have scores not lower than the consistency threshold, output a pass signal.

9. The method of claim 1, wherein, Based on the first fusion strategy, write the first answer fragment to the answer buffer, including: Based on the first answer fragment, call the keyword extractor to perform TF-IDF weight calculation to obtain a list of first key information; Based on the text currently stored in the answer buffer, call the same keyword extractor to perform TF-IDF weight calculation to obtain a list of current key information; Based on the intersection operation of the first key information list and the current key information list, obtain a set of duplicate keywords; Based on the duplicate keyword set and the first key information list, calculate the duplication rate; Based on the comparison result of the duplication rate and the preset first duplication threshold, if it is lower than the first duplication threshold, call the buffer writer based on the comparison result to append the first answer fragment to the end of the answer buffer and update the buffer index; If it is greater than or equal to the first duplication threshold, call the buffer controller based on the comparison result to skip the write operation and record the skip log and the duplicate flag.

Images