Data processing method and apparatus based on search enhancement generated data

Abstract

The application provides a data processing method and device based on retrieval enhancement generation, and relates to the technical field of artificial intelligence. A data processing method based on retrieval enhancement generation comprises the following steps: acquiring a reasoning problem in a specified field and initializing a retrieval tracking process; based on the reasoning problem, performing multi-path parallel retrieval based on the specified field in a knowledge graph and / or a vector database to obtain retrieval results, wherein the knowledge graph and the vector database are processed by synonym normalization; scoring candidate entities, candidate relationships and candidate text blocks in the retrieval results, and deduplicating and screening the retrieval results according to the scoring results to obtain target entities, target relationships and target text blocks; assembling the target entities, the target relationships and the target text blocks into input data, inputting the input data into a preset data processing model to obtain a generation result, and outputting complete tracking information based on the retrieval tracking process, so that efficient and high-precision data processing is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a data processing method and apparatus based on retrieval enhancement generation. Background Technology

[0002] Currently, GraphRAG (Knowledge Graph-Based Retrieval Enhancement Generation) is an important technical approach for improving the quality of data processing model generation in recent years. Its core idea is to extract entities and relationships from documents using a data processing model to construct a knowledge graph. When a user queries, the graph retrieves structured context, which is then injected into the data processing model to generate the answer. Compared to traditional pure vector retrieval RAG, GraphRAG can capture complex relationships between entities (such as "System A calls the interface of System B" or "Data model X is stored in table Y"), thus providing more accurate and structured context.

[0003] However, the retrieval process of existing GraphRAG systems is opaque. The internal operations, intermediate results, and truncation of each stage are unobservable. When retrieval results are poor, developers cannot pinpoint the problem and can only blindly adjust parameters or rebuild the index, resulting in low efficiency. Furthermore, during the retrieval process, existing solutions typically employ single-path vector retrieval and simply truncate the results, making it difficult to balance precise entity matching and semantic relevance. In addition, the same concept often has multiple expressions in different documents (such as abbreviations, full names, Chinese and English versions, aliases, etc.). These expressions are extracted as independent entity nodes by the data processing model, potentially leading to the omission of a large amount of information during the retrieval process and low recall. Summary of the Invention

[0004] Based on this, this application provides a data processing method and apparatus based on retrieval enhancement generation, achieving high-efficiency and high-accuracy data processing.

[0005] According to one aspect of this application, a data processing method based on retrieval enhancement generation is proposed, comprising: acquiring a reasoning question in a specified domain input by a user and initializing a retrieval tracking process; performing multi-path parallel retrieval based on the reasoning question in a pre-constructed knowledge graph and / or vector database to obtain retrieval results, wherein the knowledge graph and vector database have undergone synonym normalization processing; scoring candidate entities, candidate relations, and candidate text blocks in the retrieval results, and deduplicating and filtering the retrieval results according to the scoring results to obtain target entities, target relations, and target text blocks; assembling the target entities, target relations, and target text blocks into input data, and inputting the input data into a preset data processing model to obtain generated results, and outputting complete tracking information based on the retrieval tracking process.

[0006] According to some embodiments, the knowledge graph and vector database are pre-constructed through the following steps: The document to be processed is labeled with a domain identifier based on its domain and stored in a document library; the labeled document to be processed is divided into blocks to obtain text blocks; multiple entities and relationships between multiple entities are extracted from the text blocks as extraction results; the entity names in the extraction results are normalized using synonyms, and the normalized extraction results are written into the historical knowledge graph and historical vector database as new knowledge graphs and vector databases.

[0007] According to some embodiments, the entity names in the extraction results are subjected to synonym normalization processing, including: normalizing the entity names in the extraction results using a pre-set synonym mapping table and its reverse mapping table; comparing the entities in the normalized extraction results with entities in the historical knowledge graph, and extracting entities with the same name as the first entity; obtaining the domain identifier of the first entity in the historical knowledge graph as the first domain identifier, and obtaining the domain identifier of the first entity in the extraction results as the second domain identifier; skipping the merging if the document to be processed has domain isolation enabled, the configured isolation domain includes the domain corresponding to the first entity, and the first domain identifier and the second domain identifier do not match; merging the first entity in the historical knowledge graph and the first entity in the extraction results if the document to be processed has not domain isolation enabled, the configured isolation domain does not include the domain corresponding to the first entity, and / or the first domain identifier and the second domain identifier match.

[0008] According to some embodiments, based on the problem to be reasoned, multi-path parallel retrieval based on a specified domain is performed in a pre-constructed knowledge graph and / or vector database to obtain retrieval results. This includes: extracting document metadata from a pre-defined document library for documents belonging to the specified domain to construct a domain document whitelist; identifying keywords for the problem to be reasoned based on a pre-defined data processing model to obtain multiple keywords; expanding the multiple keywords with synonyms to obtain an expanded keyword list; performing multi-path parallel retrieval in the knowledge graph and / or vector database based on the expanded keyword list to obtain a retrieval list, wherein the parallel retrieval paths include local paths, global paths, and naive paths, and the retrieval list includes candidate entities and their vector retrieval rankings, candidate relations and their vector retrieval rankings, and candidate text blocks; and filtering the retrieval list based on the domain document whitelist to obtain retrieval results.

[0009] According to some embodiments, candidate entities, candidate relations, and candidate text blocks in the search results are scored, and the search results are deduplicated and filtered based on the scoring results to obtain target entities, target relations, and target text blocks. This includes: calculating a comprehensive score for candidate entities as a first scoring result, and calculating a comprehensive score for candidate relations as a second scoring result; sorting candidate entities in descending order based on the first scoring result, and deduplicating based on the sorting result to obtain a deduplicated entity list; sorting candidate relations in descending order based on the second scoring result, and deduplicating based on the sorting result to obtain a deduplicated relation list; selecting the first first preset number of entities from the entity list as target entities, and selecting the first second preset number of relations from the relation list as target relations; merging the text blocks corresponding to the target entities, the text blocks corresponding to the target relations, and the candidate text blocks, and deduplicating based on the text block identifiers to obtain text blocks to be processed; calculating a comprehensive score for the text blocks to be processed as a third scoring result; sorting the text blocks to be processed in descending order based on the third scoring result, and selecting the first third preset number of text blocks from the sorting result as target text blocks.

[0010] According to some embodiments, calculating the comprehensive score of the candidate entity as the first scoring result includes: calculating the graph topology factor based on the node degree of the candidate entity; calculating the semantic position factor based on the vector retrieval ranking of the candidate entity; determining the information source weight based on the retrieval path corresponding to the candidate entity; and calculating the comprehensive score of the candidate entity based on the graph topology factor, the semantic position factor, and the information source weight as the first scoring result.

[0011] According to some embodiments, calculating a comprehensive score for candidate relations as a second scoring result includes: calculating a graph topology factor based on the degree of the edges of the candidate relations; calculating a semantic position factor based on the vector retrieval ranking of the candidate relations; calculating a relation strength factor based on the edge weights of the candidate relations; determining the information source weight based on the retrieval path corresponding to the candidate relations; and calculating a comprehensive score for the candidate relations based on the graph topology factor, semantic position factor, relation strength factor, and information source weight, as a second scoring result.

[0012] According to some embodiments, calculating a comprehensive score of the text block to be processed as a third scoring result includes: calculating a base score based on the vector similarity of the text block to be processed; determining the source path weight based on the source path corresponding to the text block to be processed, wherein the source path includes entity association path, relation association path and direct vector path; and calculating a comprehensive score of the text block to be processed based on the base score and the source path weight as a third scoring result.

[0013] According to some embodiments, the target entity, target relationship, and target text block are assembled into input data, and the input data is input into a preset data processing model to obtain the generated result. Based on the retrieval and tracking process, complete tracking information is output, including: S1: performing structured context assembly on the target entity, target relationship, and target text block to obtain input data; S2: if the number of lexical units in the input data is greater than a preset lexical unit number threshold, the input data is truncated to a safe range according to lexical units and used as new input data, then proceeding to step S4; S3: if the number of lexical units in the input data is not greater than the preset lexical unit number threshold, then proceeding to step S4; S4: inputting the input data into the preset data processing model to obtain the generated result; S5: outputting complete tracking information based on the retrieval and tracking process.

[0014] According to some embodiments, the retrieval tracking process is used to track and record key stages of the retrieval, including keyword identification, keyword synonym expansion, entity retrieval, relation retrieval, lexical truncation, text block merging, and / or context assembly.

[0015] According to one aspect of this application, a data processing apparatus based on retrieval enhancement generation includes: an initial input unit for acquiring a reasoning question in a specified domain input by a user and initializing a retrieval tracking process; a parallel retrieval unit for performing multi-path parallel retrieval based on the reasoning question in a pre-constructed knowledge graph and / or vector database to obtain retrieval results, wherein the knowledge graph and vector database have undergone synonym normalization processing; a scoring and deduplication unit for scoring candidate entities, candidate relations, and candidate text blocks in the retrieval results, and deduplicating and filtering the retrieval results according to the scoring results to obtain target entities, target relations, and target text blocks; and a result output unit for assembling target entities, target relations, and target text blocks into input data, inputting the input data into a preset data processing model to obtain generated results, and outputting complete tracking information based on the retrieval tracking process.

[0016] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0017] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0018] Through the embodiments provided in this application, in the retrieval stage, a retrieval tracking process is initialized to track the retrieval process; the pre-constructed knowledge graph and / or vector database has undergone synonym normalization processing to improve the retrieval recall rate; based on the user-inputted domain-specific reasoning question, multi-path parallel retrieval based on the domain corresponding to the reasoning question is performed in the pre-constructed knowledge graph and / or vector database to achieve domain-based end-to-end knowledge isolation, while taking into account entity precision matching and semantic relevance; the candidate entities, candidate relations, and candidate text blocks in the obtained retrieval results are scored, and the retrieval results are deduplicated and filtered according to the scoring results to obtain target entities, target relations, and target text blocks. The retrieval results are staged through scoring, which greatly preserves the key entity information; the target entities, target relations, and target text blocks are assembled into input data, passed to the preset data processing model to generate answers, and complete tracking information is output based on the retrieval tracking process, making the data at each stage of the retrieval observable and achieving high-efficiency and high-accuracy data processing. Attached Figure Description

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0021] Figure 1 A flowchart illustrating the data processing method based on retrieval enhancement provided in this application embodiment;

[0022] Figure 2 One of the flowcharts for synonym normalization processing provided in the embodiments of this application; Figure 3 The second flowchart of the synonym normalization process provided in the embodiments of this application; Figure 4 A flowchart illustrating entity merging and isolation based on domain identifiers is provided for embodiments of this application; Figure 5 A flowchart illustrating the multi-path parallel retrieval provided in this application embodiment; Figure 6 The flowchart provided in this application embodiment is used to score candidate entities, candidate relationships, and candidate text blocks in the search results, and to remove duplicates and filter the search results based on the scoring results to obtain target entities, target relationships, and target text blocks; Figure 7A flowchart for calculating the comprehensive score of candidate entities provided in an embodiment of this application; Figure 8 A flowchart for calculating the comprehensive score of candidate relationships provided in an embodiment of this application; Figure 9 The flowchart provided in this application embodiment assembles target entities, target relationships, and target text blocks into input data, inputs the input data into a preset data processing model, obtains the generated results, and outputs complete tracking information based on the retrieval and tracking process; Figure 10 A block diagram of a data processing apparatus based on retrieval enhancement provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0028] For specific implementation details, please refer to the following examples.

[0029] Figure 1 A flowchart illustrating the data processing method based on retrieval enhancement provided in this application embodiment. Figure 1 As shown, the method includes steps S110-S140.

[0030] In step S110, the reasoning question in the specified domain input by the user is obtained, and the retrieval and tracking process is initialized.

[0031] Receive user input queries, denoted as the problem to be reasoned.

[0032] It should be emphasized that the problem to be reasoned about must include the specified domain of the query association.

[0033] According to the example implementation, based on the pre-defined domain_ids, the query parameters support the domain_ids parameter, which can specify one or more domains.

[0034] The retrieval and tracking process is a pre-built tracking program used to track and record data information (such as thought chains) during data processing according to a preset data structure.

[0035] In step S120, based on the question to be reasoned, a multi-path parallel search based on a specified domain is performed in a pre-constructed knowledge graph and / or vector database to obtain search results. The knowledge graph and vector database have undergone synonym normalization processing.

[0036] During the retrieval phase, the reasoning question in the specified domain input by the user is obtained, and multi-path parallel retrieval based on domain filtering is performed in a pre-built knowledge graph and / or vector database to obtain the retrieval results.

[0037] According to the example implementation, multi-path parallel retrieval includes parallel retrieval of local paths, global paths, and naive paths (keyword paths).

[0038] The search results include candidate entities, candidate relations, and candidate text blocks.

[0039] It is important to emphasize that the pre-built knowledge graph and vector database have undergone synonym normalization. This synonym normalization process understandably includes merging entities with the same name and meaning.

[0040] In step S130, the candidate entities, candidate relations, and candidate text blocks in the search results are scored, and the search results are deduplicated and filtered according to the scoring results to obtain the target entity, target relation, and target text block.

[0041] This application proposes a factor fusion scoring mechanism to score candidate entities, candidate relations, and candidate text blocks in the retrieval results. Based on the scoring results, the retrieval results are truncated to solve the problem that the crude result truncation in the prior art does not consider the token budget allocation, which easily leads to the truncation of key entity information while redundant text blocks are retained. This results in the loss of important contextual information passed to the data processing model and the retention of irrelevant information.

[0042] In step S140, the target entity, target relationship, and target text block are assembled into input data, and the input data is input into a preset data processing model to obtain the generation result. Based on the retrieval and tracking process, complete tracking information is output.

[0043] The obtained target entities, target relationships, and target text blocks are assembled into a context, which is then used as input data for the data processing model to generate an answer. The generated answer is recorded as the generation result.

[0044] The data processing model can be selected according to the situation, and this application does not restrict the model or name of the data processing model.

[0045] The initialized retrieval tracking process tracks and records the retrieval process from the user query to the final context generation, and then outputs complete tracking information based on the retrieval tracking process.

[0046] In the retrieval phase, this application initializes a retrieval tracking process to monitor the retrieval process. A pre-constructed knowledge graph and / or vector database, having undergone synonym normalization, improves retrieval recall. Based on the user-inputted, domain-specific reasoning question, multi-path parallel retrieval based on the domain corresponding to the reasoning question is performed in the pre-constructed knowledge graph and / or vector database, achieving domain-based end-to-end knowledge isolation while simultaneously considering precise entity matching and semantic relevance. Candidate entities, candidate relations, and candidate text blocks in the obtained retrieval results are scored, and the retrieval results are deduplicated and filtered based on the scoring results to obtain target entities, target relations, and target text blocks. The scoring method stages the retrieval results, greatly preserving key entity information. The target entities, target relations, and target text blocks are assembled into input data, passed to a pre-defined data processing model to generate answers, and complete tracking information is output based on the retrieval tracking process, making data observable at each stage of the retrieval and achieving high-efficiency, high-accuracy data processing.

[0047] According to some embodiments, the knowledge graph and vector database are pre-built through steps S121-S124.

[0048] In step S121, the document to be processed is labeled with a domain identifier according to the domain to which it belongs, and then stored in the document library.

[0049] In the process of building a knowledge graph and vector database using documents to be processed, the domain identifier (domain_id) of the documents to be processed is marked according to the domain to which the documents belong and stored in the document library.

[0050] According to the example implementation, when a document is added to the database, the domain_id information (e.g., "tms", "wms") is carried through metadata.

[0051] Understandably, in existing technologies, especially in enterprise-level GraphRAG applications, the same knowledge base often needs to host documents from multiple domains, such as TMS, WMS, and OMS subsystem documents in a logistics system. Different domains may contain entities with the same name but different meanings (e.g., "scheduling module" refers to vehicle dispatching in TMS and picking dispatching in WMS). The original LightRAG merges these entities with the same name into a single node, leading to knowledge confusion—a single node contains descriptions and relationships from multiple domains. This application's embodiments employ domain-based entity merging and isolation during the construction phase to avoid erroneous merging of entities with the same name across domains, eliminate knowledge pollution, and ensure that the descriptions and relationships of each entity node maintain domain consistency.

[0052] In step S122, the annotated document to be processed is divided into blocks to obtain text blocks.

[0053] Documents marked with domain identifiers (domain_id) are chunked to obtain multiple text blocks.

[0054] In step S123, multiple entities and the relationships between them are extracted from the text block as the extraction result.

[0055] Entities and relationships are extracted from text blocks using a data processing model, and this result is denoted as the extraction result.

[0056] The data processing model can be selected according to the situation, and this application does not restrict the model or name of the data processing model.

[0057] In step S124, the entity names in the extraction results are normalized using synonyms, and the normalized extraction results are written into the historical knowledge graph and historical vector database as new knowledge graphs and vector databases.

[0058] The extracted entity names are normalized using the synonym normalization function. During this process, entities within the same domain and with the same meaning are merged to obtain the normalized extraction result.

[0059] The normalized extraction results are then written into the historical knowledge graph and historical vector database as new knowledge graphs and vector databases.

[0060] According to some embodiments, refer to Figure 2 In step S124, the entity names in the extraction results are subjected to synonym normalization processing, which can be specifically implemented through steps S210-S250.

[0061] In step S210, the entity names in the extraction results are normalized using a pre-set synonym mapping table and its reverse mapping table.

[0062] A normalization method is used to normalize the entity names in the extraction results using synonyms.

[0063] Specifically, this application maintains a synonym mapping table of {standard name → [alias list]} and automatically generates a reverse mapping table {alias → standard name} (including case variations). The synonym mapping table is pre-defined according to domain knowledge and includes the following common synonym types: 1) Synonyms in Chinese and English: such as "TMS" "Transportation Management System"; 2) Abbreviations and full names: such as "CDC" "Central Distribution Center" “Central Distribution Center”; 3) Typo / near synonyms: such as "Chain Gongfang" "Chain Workshop" (two common spellings); 4) Aliases in the field: such as "transfer" "Transfer order" "STO" "Inventory transfer".

[0064] In the embodiments of this application, based on the synonym mapping table and its reverse mapping table, the entity names in the extraction results are normalized for synonyms and uniformly modified to the standard names.

[0065] According to the example embodiments, as Figure 3 shown, the synonym normalization includes the following steps: 1) Obtain the extracted entity name entity_name.

[0066] 2) Query the entity name entity_name in the reverse mapping table; if a hit is found, return the standard name corresponding to the entity name entity_name based on the reverse mapping table; if no hit is found, convert the entity name entity_name to lowercase.

[0067] 3) Query the lowercase entity name entity_name in the reverse mapping table; if a hit is found, return the standard name corresponding to the entity name entity_name based on the reverse mapping table; if no hit is found, return the original entity name entity_name.

[0068] It should be emphasized that for each extracted relationship, the names of the source entity and the target entity need to be normalized to ensure that the relationship is connected to the standard node; the normalization occurs after the data processing model extraction and before writing to the graph, and does not affect the data processing model extraction process itself; entities that are already in the standard name will not be modified.

[0069] In existing technologies, extracting the same concept from different documents results in different naming formats. For example, for the same transportation management system, different documents may use four different expressions: "TMS", "tms", "transportation management system", and "Transportation Management System". Existing technologies create four separate entity nodes for each, leading to: descriptions that should be concentrated on one node being scattered across four nodes, resulting in incomplete information on each node; relationships related to the system (such as "TMS calls routing algorithm") only connecting to one node, making these relationships unretrievable on other nodes; and queries using the keyword "tms" only hitting the node named "tms", failing to retrieve knowledge from the "TMS" and "transportation management system" nodes. This application's embodiments employ a normalization method to normalize the entity names in the extraction results using synonyms, making the descriptions on each standard node more complete and the relationships richer, thus improving the recall and accuracy during the retrieval phase.

[0070] In step S220, the entities in the normalized extraction results are compared with the entities in the historical knowledge graph, and the entities with the same name are extracted as the first entity.

[0071] Based on the results obtained through normalization, entity merging and isolation are performed based on domain identifiers, referring to... Figure 4 Specifically, the entities in the normalized extraction results are first compared with the entities in the historical knowledge graph, and entities with the same name are extracted and denoted as the first entity.

[0072] In step S230, the domain identifier of the first entity in the historical knowledge graph is obtained as the first domain identifier, and the domain identifier of the first entity in the extraction result is obtained as the second domain identifier.

[0073] When merging entities, it is necessary to check the matching of domain identifiers (domain_id), extract the domain identifier of the first entity in the historical knowledge graph, and denote it as the first domain identifier. Extract the domain identifier of the first entity in the extraction result and denote it as the second domain identifier.

[0074] In step S240, if the document to be processed has domain isolation enabled, the configured isolation domain includes the domain corresponding to the first entity, and the first domain identifier and the second domain identifier do not match, the merging is skipped.

[0075] Before entity merging, the `domain_isolation_enabled_domains` configuration (domains with domain isolation enabled) is checked. A `domain_id` matching check is performed on the domains with isolation enabled. If the document to be processed has domain isolation enabled (`domain_isolation_enabled`) and the configured isolation domain includes the domain corresponding to the first entity, a merge check is triggered. Specifically, the domain identifier of the first entity in the historical knowledge graph (first domain identifier) ​​is further compared with the domain identifier of the newly extracted first entity (second domain identifier). If they do not match, the merge is skipped, allowing the entities with the same name in the two domains to exist independently.

[0076] In step S250, if the document to be processed does not have domain isolation enabled, the configured isolation domain does not include the domain corresponding to the first entity, and / or the first domain identifier and the second domain identifier match, the first entity in the historical knowledge graph and the first entity in the extraction result are merged.

[0077] Conversely, merge entities with the same name.

[0078] Furthermore, after step S220, the method further includes: if no entity with the same name exists, then create a new entity node.

[0079] According to some embodiments, refer to Figure 5 In step S120, based on the problem to be reasoned, a multi-path parallel retrieval based on a specified domain is performed in a pre-constructed knowledge graph and / or vector database to obtain retrieval results. This can be specifically achieved through steps S510-S550.

[0080] In step S510, documents whose document metadata belongs to a specified domain are extracted from a preset document library to construct a domain file whitelist.

[0081] To streamline the data, a domain whitelist is built during the retrieval phase to precisely filter the input data.

[0082] According to the example implementation, build_domain_fiter (domain filter) is called to build a domain whitelist.

[0083] In the specific implementation process, the DocStatusStorage is traversed to obtain all document metadata; documents whose metadata.domain_id (domain identifier field in document metadata) is in domain_ids (a list of all allowed domain identifiers corresponding to a specified domain) are filtered; and the allowed_file_paths set is constructed as a domain whitelist.

[0084] In step S520, based on the preset data processing model, keyword identification is performed on the reasoning problem to be solved, and multiple keywords are obtained.

[0085] The preset data processing model identifies keywords from the question to be reasoned, extracting multiple keywords, including high-level keywords and / or low-level keywords.

[0086] In step S530, multiple keywords are expanded using synonyms to obtain an expanded keyword list.

[0087] The process of expanding each keyword from multiple keywords using synonyms can be achieved using a pre-set synonym mapping table and its reverse mapping table. That is, each keyword is converted into a standard name, and all the corresponding aliases are used as expanded terms, resulting in a list of expanded keywords.

[0088] According to the example embodiment, the steps of synonym expansion include: 1) Convert multiple keywords into a keyword list [kwl,kw2,...]; 2) Initialize the expanded list expanded = []; 3) Extract the current keyword kw from the keyword list and add it to the expanded list; 4) Through normalization, the standard name of the current keyword kw, canonical, is obtained; 5) If the standard name canonical is the same as the current keyword kw, find the complete list of aliases for the standard name canonical and add the alias list to the expanded list; 6) If the standard name canonical is different from the current keyword kw, add the standard name canonical to the expanded list, find all aliases for the standard name canonical, and add the aliases to the expanded list. 7) Repeat steps 3)-6) until each keyword kw in the keyword list is traversed, resulting in the expanded list expanded; 8) Remove duplicates from the expanded list and output the deduplicated expanded list as the expanded keyword list.

[0089] The expanded set of keywords is used to retrieve entities and relationships, ensuring that standard entity nodes and all their associated knowledge are retrieved regardless of the user's expression.

[0090] It should be noted that the normalization method provided in this application is essentially a bidirectional normalization method, effective simultaneously in both the construction and query phases. Specifically, a pre-defined synonym mapping table and its reverse mapping table are used. In the construction phase, entity names extracted from documents are normalized using synonyms. In the query phase, keywords extracted from user-input queries are expanded using synonyms. Normalization in the construction phase reduces the number of graph nodes by approximately 30-40% (synonym fragments are merged), resulting in more complete descriptions and richer relationships on each standard node. Keyword expansion in the query phase improves retrieval recall by approximately 15-30%, particularly when users use abbreviations, aliases, or non-standard expressions in their queries.

[0091] In step S540, based on the extended keyword list, a multi-path parallel search is performed in the knowledge graph and / or vector database to obtain a search list. The parallel search paths include local paths, global paths, and naive paths. The search list includes candidate entities and their vector search rankings, candidate relations and their vector search rankings, and candidate text blocks.

[0092] Based on the expanded keyword list, multi-path parallel retrieval is performed in the knowledge graph and / or vector database to obtain the retrieval list. Multi-path parallel retrieval includes local paths, global paths, and naive paths (keyword paths). The retrieval list includes entity retrieval lists, relation retrieval lists, and text block retrieval lists. The entity retrieval list includes candidate entities and their vector search rankings; the relation retrieval list includes candidate relations and their vector search rankings; and the text block retrieval list includes candidate text blocks.

[0093] In step S550, the search list is filtered according to the domain file whitelist to obtain the search results.

[0094] Based on the domain document whitelist, the entity search list, relation search list, and text block search list in the search list are filtered to ensure that only knowledge in the specified domain is returned, and the returned content is recorded as the search results.

[0095] Understandably, in existing technologies, especially in enterprise-level GraphRAG applications, the same knowledge base often needs to carry documents from multiple domains, such as documents from subsystems like TMS, WMS, and OMS in a logistics system. Users often only care about knowledge in a specific domain (e.g., "What are the functions of the scheduling module in TMS?"), but the original solution returns relevant content from all domains, reducing retrieval accuracy and increasing model generation noise. This application's embodiment employs precise filtering based on a domain-specific document whitelist during the retrieval stage, reducing noise interference from irrelevant domains and making the retrieval results more focused on the user's relevant domain, improving retrieval accuracy by approximately 15-25%.

[0096] It should be noted that the entity merging and isolation method adopted in this application includes domain identifier labeling in the construction stage and precise filtering based on domain file whitelist in the retrieval stage. It belongs to application layer soft isolation, and the isolation granularity is at the level of a single document / entity. In the construction stage, selective isolation is performed within the same graph by domain_id (domain identifier), and multiple domain_ids can be dynamically specified for joint retrieval.

[0097] This method employs a domain-identifier-based end-to-end knowledge isolation approach. During the entity merging phase, it checks the `domain_id` matching to ensure that entities with the same name across different domains remain independent. Simultaneously, through bidirectional normalization using a pre-defined synonym mapping, different expressions of the same concept are merged into standard nodes. During the query phase, keywords are automatically expanded to cover all aliases. In the retrieval phase, precise domain filtering is achieved through the `domainids` parameter. This method avoids cross-domain knowledge pollution, reduces graph nodes by approximately 30-40%, and significantly improves both retrieval recall and precision.

[0098] According to some embodiments, refer to Figure 6 In step S130, the candidate entities, candidate relations and candidate text blocks in the search results are scored, and the search results are deduplicated and filtered according to the scoring results to obtain the target entity, target relation and target text block. This can be achieved through steps S610-S670.

[0099] In step S610, the comprehensive score of the candidate entities is calculated as the first score result, and the comprehensive score of the candidate relationships is calculated as the second score result.

[0100] This application proposes a pre-defined scoring mechanism that integrates the topological importance of graph structures with the semantic similarity of vector retrieval into a unified scoring framework. Candidate entities and relationships in the search results are scored according to this pre-defined mechanism.

[0101] The comprehensive score of the candidate entities is used as the first score result, and the comprehensive score of the candidate relationships is used as the second score result.

[0102] In step S620, the candidate entities are sorted in descending order according to the first scoring result, and duplicates are removed according to the sorting result to obtain a deduplicated entity list.

[0103] In step S630, the candidate relationships are sorted in descending order according to the second scoring result, and duplicates are removed according to the sorting result to obtain a list of duplicate relationships.

[0104] Unlike the "first-come, first-served" deduplication method in existing technologies, this application's embodiment uses a descending order of scores for deduplication—only the version with the highest score is retained for entities / relationships with the same name.

[0105] In other words, candidate entities are sorted in descending order according to the first score result and then deduplicated to obtain a deduplicated list of entities; candidate relations are sorted in descending order according to the second score result and then deduplicated to obtain a deduplicated list of relations.

[0106] The deduplication method in this application embodiment can ensure that when the same entity / relationship is hit by multiple search paths at the same time, the version with the higher overall score (more important + more relevant) is retained.

[0107] In step S640, the first first preset number of entities are selected from the entity list as target entities, and the second preset number of relationships are selected from the relationship list as target relationships.

[0108] Based on the obtained entity and relationship lists, the upper limit of the number is controlled independently by category. That is, for entities in the entity list, the top_top_k entities (denoted as the first preset number) are retained according to their scores as target entities; for relationships in the relationship list, the top_top_k relationships (denoted as the second preset number) are retained according to their scores as target relationships.

[0109] In step S650, the text block corresponding to the target entity, the text block corresponding to the target relationship, and the candidate text block are merged, and duplicates are removed according to the text block identifier to obtain the text block to be processed.

[0110] Collect the original text blocks associated with the target entities and target relationships that are retained after deduplication and quantity control, and merge them with the text blocks directly retrieved from the Naive path (i.e., candidate text blocks) for deduplication. The result of deduplication is recorded as the text block to be processed.

[0111] Specifically, deduplication can be performed based on the text block identifier (by chunk_id).

[0112] In step S660, a comprehensive score is calculated for the text block to be processed, which is used as the third score result.

[0113] The scoring mechanism for text blocks differs from that for entities and relationships. This application proposes a composite scoring strategy that emphasizes balanced fusion of multiple sources. The text blocks to be processed are scored according to the composite scoring strategy.

[0114] In step S670, the text blocks to be processed are sorted in descending order according to the third scoring result, and the top three preset number of text blocks in the sorting result are selected as the target text blocks.

[0115] Based on the obtained text blocks to be processed, the quantity limit is also controlled independently according to category. The embodiments of this application adopt separate control of the quotas of three types of information to ensure that entities, relations and text blocks each have independent quotas, and to avoid one type of information crowding out other types.

[0116] For each text block in the text block to be processed, the top 100 (referred to as the third preset number) chunks are retained according to the score and used as the target text blocks.

[0117] Furthermore, in some embodiments, instead of scoring the text blocks to be processed, the text blocks to be processed are directly sorted by vector similarity or weighted round-robin, and the top three preset number of text blocks in the sorting results are selected as the target text blocks.

[0118] This application employs a scoring-based deduplication method to eliminate duplicates in multi-path retrieval results, increasing the proportion of high-value entities in the top-k results by approximately 20-30%. It also uses category quantity control, independently setting top-k quotas for entities / relationships / text blocks to prevent certain types of information from crowding out others, resulting in a more balanced distribution of information types in the retrieval results. Finally, it employs text block merging deduplication: by fusing text blocks from knowledge graph paths and vector paths, it improves the coverage of contextual information without increasing token consumption.

[0119] According to some embodiments, refer to Figure 7 In step S610, the comprehensive score of the candidate entity is calculated as the first score result, which can be implemented through steps S710-S740.

[0120] In step S710, the graph topology factor is calculated based on the node degree of the candidate entities.

[0121] An entity's rank (graph topology factor) is its node degree.

[0122] According to the example implementation, node degree is defined as the number of edges connected to it. The higher the degree, the more central the entity is in the knowledge graph and the more documents / relationships reference it.

[0123] In step S720, the semantic position factor is calculated based on the vector retrieval ranking of the candidate entities.

[0124] position_score (semantic position factor) = 1 / (vector retrieval ranking position + 1).

[0125] The higher the ranking in vector retrieval, the higher the semantic similarity and the higher the score.

[0126] In step S730, the information source weight is determined based on the retrieval path corresponding to the candidate entity.

[0127] `source_weight` is used to differentiate the weights of different retrieval paths. In practice, different weights are assigned to different retrieval paths based on the characteristics of the query type.

[0128] According to the example embodiment, the information source weights for different retrieval paths are shown in Table 1.

[0129] Table 1 Information Source Weights for Local and Global Paths

[0130] In step S740, the comprehensive score of the candidate entity is calculated based on the graph topology factor, semantic location factor and information source weight, and is used as the first score result.

[0131] The comprehensive score of the candidate entities is calculated by weighting the graph topology factors, semantic location factors, and information source weights, and the result is recorded as the first score result.

[0132] According to the example embodiment, the comprehensive score of the candidate entity, entity_score, is calculated according to the first preset formula.

[0133] The first preset formula includes: entity_score=rank 0.7+position_score 0.3 source_weight.

[0134] According to some embodiments, refer to Figure 8 In step S610, a comprehensive score for the candidate relationship is calculated as the second score result, which can be implemented through steps S810-S850.

[0135] In step S810, the graph topology factor is calculated based on the degree of the edges of the candidate relations.

[0136] The rank (topological factor) of a relation is the degree of its edges.

[0137] According to the example implementation, the degree of an edge is defined as the sum of the degree of the source node and the degree of the target node. The higher the degree, the more central the two entities connected by the relationship in the knowledge graph, and the more important the relationship.

[0138] In step S820, the semantic position factor is calculated based on the vector retrieval ranking of the candidate relationship.

[0139] position_score (semantic position factor) = 1 / (vector retrieval ranking position + 1).

[0140] The higher the ranking in vector retrieval, the higher the semantic similarity and the higher the score.

[0141] In step S830, the relationship strength factor is calculated based on the edge weights of the candidate relationships.

[0142] The weight (relationship strength factor), also known as the edge weight, is used to reflect the semantic strength of a relationship.

[0143] According to the example implementation, the edge weights are assigned by the data processing model based on semantic strength when extracting relations.

[0144] In step S840, the information source weight is determined based on the retrieval path corresponding to the candidate relationship.

[0145] `source_weight` is used to differentiate the weights of different retrieval paths. In practice, different weights are assigned to different retrieval paths based on the characteristics of the query type.

[0146] According to the example embodiment, the information source weights for different retrieval paths are shown in Table 2.

[0147] Table 2 Information Source Weights for Local and Global Paths

[0148] In step S850, a comprehensive score for candidate relations is calculated based on graph topology factors, semantic location factors, relation strength factors, and information source weights, which serves as the second score result.

[0149] The comprehensive score of candidate relations is calculated by weighting the graph topology factors, semantic location factors, relation strength factors, and information source weights. The result is recorded as the second score result.

[0150] According to the example embodiment, the comprehensive score of the candidate relationship, relation_score, is calculated according to the second preset formula.

[0151] The second preset formula includes: relation_score=rank 0.4+ weight 0.3 + position_score 0.3 source_weight.

[0152] The merging of multi-path retrieval results in the existing technology has the following problems: 1) Ranking relies solely on the retrieval order, ignoring graph structure information: Vector retrieval results are sorted by cosine similarity, but do not utilize the topological importance information contained in the knowledge graph. For example, a core entity (height number node) referenced by multiple documents may be ranked lower and discarded because its similarity to the query text is slightly lower.

[0153] 2) Merging multi-path results into a simple concatenation: Existing technology concatenates the results of global path, local path, and vector path in a fixed order (global first), retaining the first version during deduplication. This results in: (a) global results always taking precedence over local results, even if local results are more relevant to some queries; (b) when the same entity is hit on different paths, only the first version is retained, potentially losing a more complete description.

[0154] 3) Differential processing without information type awareness: The same merging strategy is used for entity queries and relational queries, without distinguishing the characteristics of different query types (entity queries usually require exact matching, while relational queries focus more on global associations).

[0155] This application proposes a two-factor fusion scoring mechanism that integrates the topological importance of the graph structure with the semantic similarity of vector retrieval into a unified scoring framework. It also introduces differentiated weights for information sources and employs a "high-score-first" deduplication strategy instead of the upstream "first-come, first-served" approach. By incorporating graph topological information (node ​​degree) into the ranking, core entities (high-connectivity nodes) in the knowledge graph receive higher scores, even if their similarity to the query text is slightly lower. This solves the problem of "semantically similar but unimportant" results in pure vector retrieval, increasing the proportion of high-value entities in the search results by approximately 20-30%.

[0156] According to some embodiments, in step S660, a comprehensive score of the text block to be processed is calculated as a third score result, which can be specifically implemented through steps S661-S663.

[0157] In step S661, a basic score is calculated based on the vector similarity of the text blocks to be processed.

[0158] For the comprehensive scoring of the text blocks to be processed, this application embodiment adopts a composite scoring strategy of "multi-source weighting + vector similarity".

[0159] Specifically, a base score is calculated based on the vector similarity of the text blocks to be processed. The cosine similarity between the text block and the query is the fundamental ranking criterion for all text blocks. Regardless of the retrieval path, the vector similarity reflects the semantic matching degree between the text block and the user query.

[0160] In step S662, the source path weight is determined based on the source path corresponding to the text block to be processed. The source path includes entity association path, relation association path and direct vector path.

[0161] The same text block may be hit by multiple paths, and different weights are assigned to different paths.

[0162] Source paths include: Entity Chunks: These are text chunks that appear in the query, retrieved from the target entity. These chunks contain core entities that are highly relevant to the query and have a high degree of credibility. Relational Chunks: Text chunks retrieved from high-scoring relations. These chunks contain descriptions of relationships between entities and are suitable for answering questions like "What is the relationship between A and B?". Direct vector paths (vector_chunks): Pure semantic similarity retrieval, with broad coverage but relatively low accuracy.

[0163] The source path weights of different paths can be replicated according to the actual situation, and this application embodiment does not impose any restrictions on this.

[0164] Furthermore, a Round-Robin approach can be used to retrieve text blocks during the processing. Specifically, blocks are retrieved alternately from three sources to ensure that no single source monopolizes the final context. This is essentially an implicit source diversity guarantee mechanism—even if one source returns a large number of results, it will not crowd out the quotas of other paths.

[0165] In step S663, a comprehensive score for the text block to be processed is calculated based on the base score and the source path weight, which serves as the third scoring result.

[0166] Based on the base score and the source path weight, a weighted comprehensive score is calculated for the text block to be processed, and the result is recorded as the third score result.

[0167] The weight parameters for the base score and the source path weight can be set according to the situation, and this application does not impose any restrictions on them.

[0168] Furthermore, according to the example embodiment, in the process of merging text blocks to obtain the text block to be processed, a semantic reordering model can be used to perform a secondary relevance score on the text block and the query to filter out low-scoring blocks.

[0169] The text block scoring in this application embodiment employs a composite strategy: "vector similarity as the base score, diversity ensured through multi-source round-robin polling, and optional Reranking to improve relevance." Unlike the single-formula scoring of entities / relationships, text block scoring emphasizes a balanced fusion of multiple sources. It is understandable that the text block itself is the original carrier of entities and relationships, and its value is indirectly transmitted through entity / relationship scoring.

[0170] According to some embodiments, refer to Figure 9 In step S140, the target entity, target relationship and target text block are assembled into input data, and the input data is input into the preset data processing model to obtain the generation result. Based on the retrieval and tracking process, complete tracking information is output, which can be achieved through steps S1-S5.

[0171] In step S1, the target entity, target relationship, and target text block are assembled into a structured context to obtain the input data.

[0172] The obtained target entities, target relationships, and target text blocks are assembled into a context, which serves as the input data for the data processing model.

[0173] According to the example implementation, the obtained target entities, target relationships, and target text blocks are assembled into a structured context string, and structured metadata is generated, which includes information such as the source file path and matching score of each entity / relationship / text block for reference annotation.

[0174] In step S2, if the number of words in the input data exceeds a preset word count threshold, the input data is truncated to a safe range according to the words and used as new input data, then proceeds to step S4.

[0175] This application uses token truncation as a safety fallback, which is not triggered under normal circumstances and is only used to place excessively long descriptions. If the total number of tokens in the input data (e.g., entity description) exceeds a pre-set token count threshold (e.g., max_entity_tokens), the tokens in the input data are truncated to a safe range according to the token count threshold and used as new input data, then proceed to step S4.

[0176] It is important to emphasize that token truncation serves as a security mechanism to prevent excessively long descriptions from exceeding the model context window in extreme cases.

[0177] In step S3, if the number of tokens in the input data is not greater than the preset token number threshold, proceed to step S4.

[0178] If the total number of tokens in the input data is not greater than the pre-set threshold for the number of tokens, token truncation will not be triggered, and the next step will proceed.

[0179] In step S4, the input data is input into the preset data processing model to obtain the generated result.

[0180] The input data is passed to the data processing model to generate an answer, and the generated answer is recorded as the result.

[0181] In step S5, complete tracking information is output based on the retrieval and tracking process.

[0182] The retrieval tracking process tracks and records the entire retrieval process from the user's query to the final context generation, completes data generation, and outputs complete tracking information based on the retrieval tracking process.

[0183] According to some embodiments, the retrieval tracking process is used to track and record key stages of the retrieval, including keyword identification, keyword synonym expansion, entity retrieval, relation retrieval, lexical truncation, text block merging, and / or context assembly.

[0184] Table 3 shows the key stages of the tracking.

[0185] Table 3 Key Stages of Tracking

[0186] Based on the pre-set data structure, input, output and statistical information are recorded at each key stage of the retrieval process (keyword extraction, synonym expansion, entity retrieval, relation retrieval, token truncation, text block merging, and context assembly).

[0187] According to the example embodiment, the data structure is shown in Table 4.

[0188] Table 4 Data Structure of the Retrieval and Tracking Process

[0189] Furthermore, in step S5, during the process of retrieving the output of the tracing process, the retrieval results are returned through a QueryResult object, where the thinking_process field contains complete tracing information.

[0190] According to the example embodiment, the tracking information includes: { "content": "The answer generated by the data generation model...", "thinking_process": { "query": "Which interfaces are called by the TMS vehicle dispatch module?", "mode": "mix", "summary": "15 entities, 23 relationships, and 8 text blocks are retrieved. After truncation, 10 entities, 15 relationships, and 5 text blocks are used", "total_entities_found": 15, "final_entities_used": 10, "steps": [...] } }

[0191] The embodiments of this application achieve (a) full-link tracing - covering the complete retrieval path from user queries to the final context; (b) quantitative statistics - recording the recall volume and the final usage volume, and supporting truncation ratio analysis; (c) structured output - returning the tracing information in JSON format through the thinking_process field, which is convenient for programmatic analysis and visual display. This mechanism significantly improves the efficiency of fault location for retrieval effects and supports targeted parameter tuning.

[0192] It can be understood that in the existing GraphRAG system, the retrieval process from user queries to the final context is a "black box" - developers cannot know: which keywords are extracted by the model from the query? How many entities and relationships are recalled by vector retrieval? How many entities / relationships are truncated due to the quantity limit? What are the proportions of entities, relationships, and text blocks in the final context? Therefore, when the retrieval effect is not good, only parameter tuning (adjusting top_k, etc.) can be carried out blindly, and the root cause of the problem cannot be located. The embodiments of this application use the RetrievalThinkingProcess (retrieval tracing process) data structure and multi-stage tracing mechanism to completely record the input and output of each retrieval stage from keyword extraction to context assembly. Developers can clearly see: how many entities / relationships are recalled, how many are truncated due to the quantity limit, and the composition ratio of the final context. When the retrieval effect is not good, the problem link can be quickly located, and targeted tuning rather than blind parameter tuning can be achieved.

[0193] This application employs a data processing method based on retrieval enhancement. Compared to traditional RAG, this application's embodiments capture structured relationships between entities through knowledge graphs and integrate graph topology information into retrieval ranking, significantly improving the quality of answers to cross-document association questions. Compared to the native LightRAG (a lightweight GraphRAG framework), this application's embodiments systematically solve its problems of unreasonable retrieval ranking, graph fragmentation, cross-domain contamination, and retrieval black boxes, improving end-to-end accuracy by 20-30%. This solution uses on-demand retrieval instead of global summarization, resulting in indexing costs far lower than Microsoft GraphRAG (a GraphRAG framework based on community detection). Furthermore, this application's embodiments provide retrieval process interpretability, which is lacking in Microsoft GraphRAG, and solve the graph fragmentation problem, outperforming the coarse-grained retrieval of community detection in both accuracy and efficiency.

[0194] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0195] Figure 10 A block diagram of a data processing apparatus based on retrieval enhancement generation according to an exemplary embodiment is shown.

[0196] Figure 10 The apparatus shown can perform the data processing method based on retrieval enhancement generation according to the embodiments of this application.

[0197] like Figure 10 As shown, the data processing apparatus based on retrieval enhancement may include: See Figure 10 Referring to the preceding description, the initial input unit 1010 is used to obtain the reasoning question in the specified domain input by the user and to initialize the retrieval and tracking process.

[0198] The parallel retrieval unit 1020 is used to perform multi-path parallel retrieval based on a specified domain in a pre-built knowledge graph and / or vector database based on the question to be reasoned, and obtain retrieval results. The knowledge graph and vector database have been processed by synonym normalization.

[0199] The scoring and deduplication unit 1030 is used to score the candidate entities, candidate relations and candidate text blocks in the search results, and to deduplicate and filter the search results based on the scoring results to obtain the target entity, target relation and target text block.

[0200] The result output unit 1040 is used to assemble the target entity, target relationship and target text block into input data, input the input data into a preset data processing model, obtain the generated result, and output complete tracking information based on the retrieval and tracking process.

[0201] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0202] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0203] For example, refer to Figure 11 , Figure 11 The illustrated electronic device 1100 includes a processor 1101 and a memory 1103. The processor 1101 and the memory 1103 are connected, for example, via a bus 1102. Optionally, the electronic device 1100 may also include a transceiver 1104. It should be noted that in practical applications, the transceiver 1104 is not limited to one unit, and the structure of this electronic device 1100 does not constitute a limitation on the embodiments of this application.

[0204] Processor 1101 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this application. Processor 1101 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0205] Bus 1102 may include a pathway for transmitting information between the aforementioned components. Bus 1102 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1102 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0206] The memory 1103 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0207] The memory 1103 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 1101. The processor 1101 is used to execute the application code stored in the memory 1103 to implement the content shown in the foregoing method embodiments.

[0208] Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0209] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0210] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0211] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method based on retrieval enhancement generation, characterized in that, include: Obtain the reasoning question in the specified domain input by the user, and initialize the retrieval and tracking process; Based on the problem to be reasoned, a multi-path parallel search based on the specified domain is performed in a pre-constructed knowledge graph and / or vector database to obtain search results, wherein the knowledge graph and the vector database have undergone synonym normalization processing; The candidate entities, candidate relations, and candidate text blocks in the search results are scored, and the search results are deduplicated and filtered according to the scoring results to obtain the target entities, target relations, and target text blocks; The target entity, the target relationship, and the target text block are assembled into input data, and the input data is input into a preset data processing model to obtain the generated result. Based on the retrieval and tracking process, complete tracking information is output.

2. The method according to claim 1, characterized in that, The knowledge graph and the vector database are pre-constructed through the following steps: Based on the domain to which the document to be processed belongs, the document to be processed is labeled with a domain identifier and stored in the document library; The annotated document to be processed is divided into blocks to obtain text blocks; Extract multiple entities and the relationships between them from the text block as the extraction result; The entity names in the extraction results are normalized using synonyms, and the normalized extraction results are written into the historical knowledge graph and historical vector database as new knowledge graphs and vector databases.

3. The method according to claim 2, characterized in that, The entity names in the extracted results are subjected to synonym normalization processing, including: The entity names in the extraction results are normalized using a pre-set synonym mapping table and its reverse mapping table; The entities in the normalized extraction results are compared with the entities in the historical knowledge graph, and the entities with the same name are extracted as the first entity. Obtain the domain identifier of the first entity in the historical knowledge graph as the first domain identifier, and obtain the domain identifier of the first entity in the extraction result as the second domain identifier; If the document to be processed has enabled domain isolation, the configured isolation domain includes the domain corresponding to the first entity, and the first domain identifier and the second domain identifier do not match, then skip merging; If the document to be processed does not have domain isolation enabled, the configured isolation domain does not include the domain corresponding to the first entity, and / or the first domain identifier and the second domain identifier match, the first entity in the historical knowledge graph and the first entity in the extraction result are merged.

4. The method according to claim 1, characterized in that, Based on the problem to be reasoned, a multi-path parallel retrieval based on the specified domain is performed in a pre-constructed knowledge graph and / or vector database to obtain retrieval results, including: Extract the metadata of documents belonging to the specified domain from the preset document library to construct a domain file whitelist; Based on a preset data processing model, keyword identification is performed on the question to be reasoned, resulting in multiple keywords; The multiple keywords are expanded using synonyms to obtain an expanded keyword list; Based on the extended keyword list, a multi-path parallel search is performed in the knowledge graph and / or vector database to obtain a search list. The parallel search paths include local paths, global paths, and naive paths. The search list includes candidate entities and their vector search rankings, candidate relations and their vector search rankings, and candidate text blocks. The search list is filtered according to the domain document whitelist to obtain the search results.

5. The method according to claim 1, characterized in that, The candidate entities, candidate relations, and candidate text blocks in the search results are scored, and the search results are deduplicated and filtered based on the scoring results to obtain the target entities, target relations, and target text blocks, including: Calculate the comprehensive score of the candidate entities as the first score result, and calculate the comprehensive score of the candidate relationships as the second score result; The candidate entities are sorted in descending order according to the first scoring result, and duplicates are removed according to the sorting result to obtain a deduplicated entity list. The candidate relationships are sorted in descending order according to the second scoring result, and duplicates are removed according to the sorting result to obtain a list of duplicate relationships. Select the first preset number of entities from the entity list as target entities, and select the first second preset number of relationships from the relationship list as target relationships; The text block corresponding to the target entity, the text block corresponding to the target relationship, and the candidate text block are merged, and duplicates are removed according to the text block identifier to obtain the text block to be processed. Calculate the comprehensive score of the text block to be processed, and use it as the third scoring result; Based on the third scoring result, the text blocks to be processed are sorted in descending order, and the top three preset number of text blocks in the sorting result are selected as the target text blocks.

6. The method according to claim 5, characterized in that, Calculate the comprehensive score of the candidate entities as the first score result, including: Calculate the graph topology factor based on the node degree of the candidate entities; The semantic position factor is calculated based on the vector retrieval ranking of the candidate entities; The information source weight is determined based on the retrieval path corresponding to the candidate entity; The comprehensive score of the candidate entity is calculated based on the graph topology factor, the semantic location factor, and the information source weight, and is used as the first score result.

7. The method according to claim 5, characterized in that, Calculate the comprehensive score of the candidate relationships as the second scoring result, including: Calculate the graph topology factor based on the degree of the edges of the candidate relationships; The semantic position factor is calculated based on the vector retrieval ranking of the candidate relationships; Calculate the relationship strength factor based on the edge weights of the candidate relationships; The information source weights are determined based on the retrieval paths corresponding to the candidate relationships. The comprehensive score of the candidate relation is calculated based on the graph topology factor, the semantic location factor, the relation strength factor, and the information source weight, and is used as the second score result.

8. The method according to claim 5, characterized in that, Calculate the comprehensive score of the text block to be processed, as the third scoring result, including: A base score is calculated based on the vector similarity of the text blocks to be processed; Based on the source path corresponding to the text block to be processed, the weight of the source path is determined, wherein the source path includes entity association path, relation association path and direct vector path; Based on the base score and the source path weight, a comprehensive score is calculated for the text block to be processed, which serves as the third scoring result.

9. The method according to claim 1, characterized in that, The target entity, the target relationship, and the target text block are assembled into input data, which is then input into a preset data processing model to obtain the generated result. Based on the retrieval and tracking process, complete tracking information is output, including: S1: Perform structured context assembly on the target entity, the target relationship, and the target text block to obtain input data; S2: If the number of words in the input data is greater than the preset word count threshold, the input data is truncated to a safe range according to words and used as new input data. Then proceed to step S4. S3: If the number of lexical units in the input data is not greater than the preset lexical unit number threshold, proceed to step S4; S4: Input the input data into the preset data processing model to obtain the generated result; S5: Output complete tracking information based on the retrieval and tracking process.

10. The method according to claim 1, characterized in that, The retrieval tracking process is used to track and record key stages of the retrieval, including keyword identification, keyword synonym expansion, entity retrieval, relation retrieval, word segmentation, text block merging, and / or context assembly.

11. A data processing apparatus based on retrieval enhancement generation, characterized in that, include: The initial input unit is used to obtain the reasoning question in the specified domain input by the user and to initialize the retrieval and tracking process; The parallel retrieval unit is used to perform multi-way parallel retrieval based on the specified domain in a pre-constructed knowledge graph and / or vector database based on the question to be reasoned, and to obtain retrieval results, wherein the knowledge graph and the vector database have been processed by synonym normalization. The scoring and deduplication unit is used to score the candidate entities, candidate relations and candidate text blocks in the search results, and to deduplicatize and filter the search results based on the scoring results to obtain the target entities, target relations and target text blocks; The result output unit is used to assemble the target entity, the target relationship, and the target text block into input data, input the input data into a preset data processing model to obtain the generated result, and output complete tracking information based on the retrieval and tracking process.

12. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 10.

Images