Dual-path RAG evidence tracing method for multi-source heterogeneous data

Abstract

The invention discloses a dual-path RAG evidence tracing method for multi-source heterogeneous data, and relates to the field of artificial intelligence and knowledge engineering cross technologies, the method comprises the following specific steps: receiving the multi-source heterogeneous data, and the invention discloses a dual-path retrieval enhanced analysis and evidence tracing method for the multi-source heterogeneous data. For multi-source heterogeneous data such as a scanning copy, a PDF (Portable Document Format), voice and the like, a system in which a text retrieval enhancement path and a visual retrieval enhancement path are coordinated is constructed, so that the technical bottlenecks of an existing RAG system in the aspects of evidence integrity maintenance, cross-modal consistency verification and generation result attributability are solved; the method can be widely applied to audit, judicial evidence obtaining, financial risk control, medical archive management and other scenes with strict requirements for knowledge credibility, management, visualization and storage are carried out on RAG evidence traceability data and corresponding analysis results, RAG evidence traceability management can be achieved through off-line localized evidence management and cloud management and control, and the reliability of the system is improved. And the intelligent level of RAG evidence traceability management is improved.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and knowledge engineering, specifically a dual-path RAG evidence tracing method for multi-source heterogeneous data. Background Technology

[0002] Current mainstream search-enhanced generation (RAG) systems generally adopt a single-path architecture of "OCR + text RAG": after converting scanned images into plain text using OCR, a text vector library is built for semantic retrieval and generation. This approach has three fundamental drawbacks when dealing with multi-source heterogeneous data:

[0003] Key information is permanently lost: OCR has an extremely low recognition rate for blurred areas, handwritten annotations, and stamps, and the output is plain text, completely stripping away non-textual but evidentiary visual primitives such as layout, handwriting, color, and spatial relationships.

[0004] The search results are unreliable: the generated content lacks an explicit connection with the original evidence, the source of the facts cannot be verified, and it is difficult to meet the requirements of "verifiable conclusions and traceable processes" in high compliance scenarios.

[0005] Single modality and poor robustness: When OCR fails (e.g., low-quality scanned documents), the system completely loses its service capability and lacks a visual path fallback mechanism.

[0006] Although some studies have attempted to introduce multimodal RAG, most employ a unified encoder for end-to-end feature fusion (such as BLIP-2 and LLaVA), failing to design independent and verifiable processing paths to address the complementarity and conflict of heterogeneous evidence. This leads to:

[0007] 1. Visual noise interferes with textual reasoning;

[0008] 2. Unable to pinpoint the exact location of the evidence within the original document;

[0009] 3. The generated results are still a "black box" and lack an operable traceability interface.

[0010] Therefore, this invention requires the design of a dual-path RAG evidence tracing method for multi-source heterogeneous data to solve the aforementioned problems. Summary of the Invention:

[0011] The purpose of this invention is to provide a dual-path RAG evidence tracing method for multi-source heterogeneous data in order to solve the above-mentioned problems, thereby resolving the issues mentioned in the background art.

[0012] To address the above problems, the present invention provides a technical solution:

[0013] A dual-path RAG evidence tracing method for multi-source heterogeneous data includes the following specific steps:

[0014] S1, Receive heterogeneous data from multiple sources;

[0015] S2. Construct a dual-path retrieval enhancement system;

[0016] S3. Respond to user natural language queries and perform dynamic path selection and combined retrieval;

[0017] S4. Perform multimodal structured analysis on the retrieved evidence to generate standardized knowledge blocks;

[0018] S5. The large language model generates a natural language answer based on the knowledge block, and automatically embeds interactive evidence anchors in the answer; the evidence anchors allow users to directly jump to the corresponding position of the original evidence after clicking, realizing end-to-end verifiable traceability.

[0019] In a preferred embodiment of the present invention, the multi-source heterogeneous data in step S1 includes at least two types of data: scanned images, PDF documents, voice recordings, and structured data tables.

[0020] In a preferred embodiment of the present invention, the construction of the dual-path retrieval enhancement system in step S2 includes the following specific steps:

[0021] S201, Enhanced Text Retrieval Path: Semantically vectorize the text obtained by OCR recognition and the original electronic document text, construct a text vector index library, and associate each text block with its source file path, page number, and character-level spatial coordinates;

[0022] S202, Visual Retrieval Enhancement Path: Input the original image into the visual large model encoder, extract its global and local visual features, and construct an image feature vector index library; the visual features are used to characterize visual primitive information such as layout structure, seal area, handwritten annotation, and handwriting deformation in the image, and associate each feature vector with its corresponding source image file path, page number, and pixel-level bounding box coordinates;

[0023] The dual-path retrieval enhancement system in step S2 further includes a cross-modal evidence consistency verification mechanism:

[0024] When the text path and the visual path return conflicting evidence for the same fact, the system automatically marks it as "conflicting evidence" and outputs a comparison report, including text recognition results, visual region screenshots, confidence level comparisons, and location of the difference region, for manual tracing and verification.

[0025] In a preferred embodiment of the present invention, after completing the joint retrieval in step S3, it is also necessary to adaptively select any of the following modes for retrieval based on the semantic type of the query and the OCR confidence score of the multi-source heterogeneous data:

[0026] Enable only the text search enhancement path;

[0027] Enable only the visual retrieval enhancement path;

[0028] Simultaneously, text retrieval enhancement paths and visual retrieval enhancement paths are enabled, and the retrieval results from the two paths are weighted and fused according to the confidence level of the evidence to generate a cross-modal evidence set with unified ranking.

[0029] In a preferred embodiment of the present invention, each knowledge block in step S4 includes the following fields: modality type, content summary, source file path, page number, spatial coordinate range, and confidence score.

[0030] In a preferred embodiment of the present invention, the large visual model encoder is a visual basic model based on self-supervised pre-training of large-scale image data, selected from any of the following: DINOv2, CLIP image encoder, EVA02, Intern Image, MAE-Vi T; the large visual model encoder is fine-tuned with domain data including handwriting, stamps, and table wireframes.

[0031] In a preferred embodiment of the present invention, the decision logic for dynamic path selection in step S3 is as follows:

[0032] When the query contains "in the image", "handwritten", "annotation", "stamp", "blurred area" or similar visually sensitive semantics, or when the average confidence of the current document page to be inspected is lower than the preset threshold T, the visual retrieval enhancement path or the dual-path fusion mode will be activated first.

[0033] Where T∈[0.5,0.75], preferably 0.65.

[0034] In a preferred embodiment of the present invention, the multimodal structured analysis in step S4 includes:

[0035] For the image regions returned by the visual retrieval enhancement path, a lightweight visual parsing module is invoked to generate natural language descriptions as "visual semantic summaries". The descriptions include object categories, relative positional relationships, color attributes, and text content inferences, which are used by large language models for understanding and reasoning.

[0036] In a preferred embodiment of the present invention, the evidence anchor point in step S5 is embedded in the response text in a machine-readable structured format, the format of which is:

[0037] [Evidence ID][Modality=X][File=Y][Page=Z][Region=[x,y,w,h]],

[0038] Where X∈{text, visual, audio}, Y is a relative file path, Z∈{document page number, audio timestamp}, and [x,y,w,h] are the document normalized coordinates or the absolute pixel coordinates of the keyframe image;

[0039] After the front-end system parses the anchor point, it can highlight the corresponding area in the original document and support one-click jump, screenshot evidence collection, and operation log recording.

[0040] The method is implemented in a data-controlled environment; the data-controlled environment includes a local server, a private cloud, or an isolated network domain that meets the requirements of Level 3 Information Security Protection; the large language model, the large visual model encoder, the vector database, and the OCR engine are all deployed in the data-controlled environment to ensure that the original multi-source heterogeneous data does not leave the domain.

[0041] In a preferred embodiment of the present invention, the dual-path retrieval enhancement system includes:

[0042] A multi-source data access module is used to receive, parse, and preprocess the multi-source heterogeneous data, perform OCR recognition on scanned images and PDF documents, perform ASR speech recognition on voice recordings, and uniformly manage the metadata of all data.

[0043] The text retrieval enhancement subsystem is used to semantically vectorize the text obtained by OCR recognition and the original electronic document text, build and maintain a text vector index library, and associate each text block with its source file path, page number and character-level spatial coordinates to support semantic-based accurate retrieval.

[0044] A visual retrieval enhancement subsystem is used to input the original image into the visual large model encoder, extract its global and local visual features, and construct and maintain an image feature vector index library to support visual content-based retrieval.

[0045] The dynamic path routing and fusion retrieval module is used to respond to user queries, execute the dynamic path selection and joint retrieval logic, adaptively select single or dual path retrieval mode according to query semantics and OCR confidence, and perform confidence weighted fusion on the dual path results.

[0046] The multimodal structured parsing and evidence alignment module is used to perform structured parsing on the retrieved text and visual evidence, generate standardized knowledge blocks, perform cross-modal evidence consistency verification, and automatically generate a comparison report when conflicts are found.

[0047] A source-tracing answer generation module is used to drive a large language model to generate natural language answers based on the knowledge blocks, and automatically embed interactive evidence anchors in the answers;

[0048] The evidence visualization and interaction module is used to parse and render the evidence anchor points, support users to click to highlight the corresponding area of ​​the original document, and provide one-click jump, screenshot evidence collection and operation log recording functions.

[0049] The modules work together via API, and the overall system supports automatically linking at least one original evidence anchor point to each factual claim in the response.

[0050] The beneficial effects of this invention are as follows: This invention discloses a dual-path retrieval enhancement parsing and evidence tracing method for multi-source heterogeneous data. For multi-source heterogeneous data such as scanned documents, PDFs, and audio, it constructs a system that coordinates text retrieval enhancement paths and visual retrieval enhancement paths. The text path processes OCR recognition of text and original electronic documents, while the visual path directly encodes the original image to preserve key visual evidence such as handwritten annotations, seals, and layout structure. The system supports dynamically selecting the retrieval path based on query semantics and data quality, and performs confidence-weighted fusion of the dual-path results. The generated answer automatically embeds interactive evidence anchors containing modality type, file path, and spatial coordinates, allowing users to click and jump to the original evidence location, achieving end-to-end verifiable and traceable knowledge services. This invention solves the problems of information loss and uninterpretable results caused by OCR distortion in traditional retrieval enhancement generation. Addressing core issues such as untraceability of evidence, this invention significantly improves the reliability of knowledge services in high-credibility scenarios. Belonging to the interdisciplinary field of artificial intelligence and knowledge engineering, it specifically relates to a method for parsing and credibly tracing multi-source heterogeneous data based on dual-path retrieval enhancement. This method is particularly suitable for processing unstructured data with complex features such as blurred images, handwritten content, overlaid signatures, and format constraints. It solves the technical bottlenecks of existing RAG systems in maintaining evidence integrity, verifying cross-modal consistency, and attributing generated results. It can be widely applied in scenarios with stringent requirements for knowledge credibility, such as auditing, judicial evidence collection, financial risk control, and medical record management. It manages, visualizes, and stores RAG evidence tracing data and corresponding analysis results, facilitating RAG evidence tracing management through offline local evidence management and cloud-based control, thereby improving the intelligence level of RAG evidence tracing management. Attached image description:

[0051] For ease of explanation, the present invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0052] Figure 1 This is a flowchart illustrating the overall steps of a dual-path RAG evidence tracing method for multi-source heterogeneous data according to the present invention.

[0053] Figure 2 This is the overall system structure topology diagram of the dual-path RAG evidence tracing method for multi-source heterogeneous data according to the present invention. Detailed implementation method:

[0054] like Figure 1 and Figure 2 As shown, the specific implementation adopts the following technical solution:

[0055] A dual-path RAG evidence tracing method for multi-source heterogeneous data includes the following specific steps:

[0056] S1, Receive heterogeneous data from multiple sources;

[0057] Multi-source heterogeneous data includes at least two of the following: scanned images, PDF documents, voice recordings, and structured data tables;

[0058] S2. Construct a dual-path retrieval enhancement system;

[0059] The construction of a dual-path retrieval enhancement system includes the following specific steps:

[0060] S201, Enhanced Text Retrieval Path: Semantically vectorize the text obtained by OCR recognition and the original electronic document text, construct a text vector index library, and associate each text block with its source file path, page number, and character-level spatial coordinates;

[0061] S202, Visual Retrieval Enhancement Path: Input the original image into the visual large model encoder, extract its global and local visual features, and construct an image feature vector index library; the visual features are used to characterize visual primitive information such as layout structure, seal area, handwritten annotation, and handwriting deformation in the image, and associate each feature vector with its corresponding source image file path, page number, and pixel-level bounding box coordinates;

[0062] The dual-path retrieval enhancement system in step S2 further includes a cross-modal evidence consistency verification mechanism:

[0063] When the text path and the visual path return conflicting evidence for the same fact, the system automatically marks it as "conflicting evidence" and outputs a comparison report, including text recognition results, visual region screenshots, confidence comparisons, and location of the difference region, for manual tracing and verification.

[0064] S3. Respond to user natural language queries and perform dynamic path selection and combined retrieval;

[0065] After completing the joint retrieval, it is also necessary to adaptively select any of the following modes for retrieval based on the semantic type of the query and the OCR confidence score of the multi-source heterogeneous data:

[0066] Enable only the text search enhancement path;

[0067] Enable only the visual retrieval enhancement path;

[0068] Simultaneously, text retrieval enhancement paths and visual retrieval enhancement paths are enabled, and the retrieval results of the two paths are weighted and fused according to the confidence level of the evidence to generate a cross-modal evidence set with unified ranking.

[0069] The decision logic for dynamic path selection is as follows:

[0070] When the query contains "in the image", "handwritten", "annotation", "stamp", "blurred area" or similar visually sensitive semantics, or when the average confidence of the current document page to be inspected is lower than the preset threshold T, the visual retrieval enhancement path or the dual-path fusion mode will be activated first.

[0071] Where T∈[0.5,0.75], preferably 0.65;

[0072] S4. Perform multimodal structured analysis on the retrieved evidence to generate standardized knowledge blocks;

[0073] Each knowledge block contains the following fields: modality type, content summary, source file path, page number, spatial coordinate range, and confidence score;

[0074] The multimodal structured analysis includes:

[0075] For the image regions returned by the visual retrieval enhancement path, a lightweight visual parsing module is invoked to generate natural language descriptions as "visual semantic summaries". The descriptions include object categories, relative positional relationships, color attributes, and text content inferences, which are used by large language models for understanding and reasoning.

[0076] In a preferred embodiment of the present invention, the evidence anchor point in step S5 is embedded in the response text in a machine-readable structured format, the format of which is:

[0077] [Evidence ID][Modality=X][File=Y][Page=Z][Region=[x,y,w,h]],

[0078] Where X∈{text, visual, audio}, Y is a relative file path, Z∈{document page number, audio timestamp}, and [x,y,w,h] are the document normalized coordinates or the absolute pixel coordinates of the keyframe image;

[0079] After the front-end system parses the anchor point, it can highlight the corresponding area in the original document and support one-click jump, screenshot evidence collection, and operation log recording.

[0080] S5. The large language model generates a natural language answer based on the knowledge block, and automatically embeds interactive evidence anchors in the answer; the evidence anchors allow users to directly jump to the corresponding position of the original evidence after clicking, realizing end-to-end verifiable traceability.

[0081] The large visual model encoder is a visual basic model based on self-supervised pre-training of large-scale image data, selected from any of the following: DINOv2, CLIP image encoder, EVA02, Intern Image, MAE-Vi T; the large visual model encoder has been fine-tuned with domain data including handwriting, stamps, and table wireframes.

[0082] The method is implemented in a data-controlled environment; the data-controlled environment includes a local server, a private cloud, or an isolated network domain that meets the requirements of Level 3 Information Security Protection; the large language model, visual large model encoder, vector database, and OCR engine are all deployed in the data-controlled environment to ensure that the original multi-source heterogeneous data does not leave the domain. The dual-path retrieval enhancement system includes: a multi-source data access module, which is used to receive, parse, and preprocess the multi-source heterogeneous data, perform OCR recognition on scanned images and PDF documents, perform ASR speech recognition on voice recordings, and uniformly manage the metadata of all data;

[0083] The text retrieval enhancement subsystem is used to semantically vectorize the text obtained by OCR recognition and the original electronic document text, build and maintain a text vector index library, and associate each text block with its source file path, page number and character-level spatial coordinates to support semantic-based accurate retrieval.

[0084] A visual retrieval enhancement subsystem (including a large visual model encoder) is used to input the original image into the large visual model encoder, extract its global and local visual features, and construct and maintain an image feature vector index library to support visual content-based retrieval.

[0085] The dynamic path routing and fusion retrieval module is used to respond to user queries, execute the dynamic path selection and joint retrieval logic, adaptively select single or dual path retrieval mode according to query semantics and OCR confidence, and perform confidence weighted fusion on the dual path results.

[0086] The multimodal structured parsing and evidence alignment module is used to perform structured parsing on the retrieved text and visual evidence, generate standardized knowledge blocks, perform cross-modal evidence consistency verification, and automatically generate a comparison report when conflicts are found.

[0087] A source-tracing answer generation module is used to drive a large language model to generate natural language answers based on the knowledge blocks, and automatically embed interactive evidence anchors in the answers;

[0088] The evidence visualization and interaction module is used to parse and render the evidence anchor points, support users to click to highlight the corresponding area of ​​the original document, and provide one-click jump, screenshot evidence collection and operation log recording functions.

[0089] The modules work together via API, and the overall system supports automatically linking at least one original evidence anchor point to each factual claim in the response.

[0090] Existing RAG systems are mostly single-modal or simple parallel multimodal, failing to address the core issue of evidence conflict between modalities. This invention introduces for the first time an automated verification mechanism that can output structured comparison reports into the RAG architecture. This is not a simple combination, but a collaborative architecture created to solve the specific technical problem of information credibility, and it has irreplaceable value in practical applications (such as judicial and auditing).

[0091] The decision logic for dynamic path selection is implemented by a lightweight classifier (such as a text classification model based on BERT fine-tuning). This classifier is trained on corpora containing visually sensitive, text-sensitive, and mixed queries. Meanwhile, the path decision function P(Q,D) integrates the classification result of query Q and the average confidence C_ocr of document D's OCR, and is calculated through a predefined decision tree or weighted formula to finally output the path selection instruction.

[0092] The lightweight visual parsing module can be implemented using any of the following architectures:

[0093] Architecture A (End-to-End): Uses a miniaturized, finely tuned multimodal large model (such as QWen-VL) to directly receive image regions and output natural language descriptions;

[0094] Architecture B (Pipeline-style):

[0095] The following steps are called sequentially: 1) Use an object detection model (such as YOLO) to identify key visual elements (text blocks, seals, signature lines) within the region;

[0096] 2) Perform secondary OCR on the recognized text blocks;

[0097] 3) Based on a predefined template (such as "This area contains an [object], whose text content is [content], located at [location]"), the above information is synthesized into a natural language description;

[0098] Example:

[0099] Scenario: A financial institution is conducting a post-loan audit of a corporate loan and needs to verify the authenticity of the appraisal report for collateral A.

[0100] S1. Data Input:

[0101] Collateral Appraisal Report.pdf (Original electronic version, including standard forms, copyable text, and digital signature);

[0102] Handwritten on-site investigation record.jpg (scanned copy, some handwriting is blurry, but includes the investigator's signature and date annotation);

[0103] Internal approval meeting recording.mp3 (transcribed to text, mentioning concerns about a certain asset)

[0104] S2, User Inquiry:

[0105] "Please state the final appraised value of collateral A and confirm whether this value has been verified through on-site inspection?"

[0106] S3, System Execution Process:

[0107] Dynamic path selection: The query included on-site investigation confirmation, which triggered visually sensitive semantics. At the same time, the average confidence score of the OCR of the on-site investigation handwritten record.jpg was calculated to be 0.58 (lower than the preset threshold of 0.65). Therefore, the system simultaneously enabled both text and visual paths.

[0108] Joint retrieval and fusion:

[0109] Text path returned: "Appraised value: ¥5,000,000" from the table on page 5 of the collateral appraisal report.pdf, with high confidence level;

[0110] Visual path return: A handwritten annotation area from the on-site investigation handwritten record.jpg, whose features are related to confirmation and value, with moderate confidence;

[0111] S4. Multimodal analysis and evidence alignment:

[0112] Audio-to-text transcription + audio dual-dimensional knowledge blocks

[0113] The text evidence was parsed into a knowledge block: {Modality: Text, Content: "Assessed Value: ¥5,000,000", Source File: "Asset Appraisal Report.pdf", Page Number: 5, Coordinates: [x1,y1,x2,y2], Confidence Level: 0.95};

[0114] Audio-to-text knowledge block: {Modality: Audio, Content: "Participant: mentioned doubts about the value of collateral A, and after review, confirmed that the appraised value of collateral A was 5 million. Although the handwriting in the on-site inspection record was blurred, the core confirmed information was credible", Source file: "Internal approval meeting recording.mp3", Duration range: [89.2, 98.5]s, Transcription confidence: 0.96, Speaker: Audit reviewer B, Overall confidence: 0.90};

[0115] Visual evidence was analyzed into a knowledge block: {Modality: Visual-Image, Semantic Summary: "Handwritten text: 'Verified on-site, value is correct', with signature / date", Source file: "On-site verification form.jpg", Coordinates: [x1,y1,x2,y2], Confidence: 0.92};

[0116] Evidence consistency verification: The system determines that the two pieces of evidence support each other in terms of the fact that "value has been confirmed" and there is no conflict;

[0117] S5. Generating Responses and Evidence Anchors:

[0118] The large language model generated the following answer: According to the assessment report, the final assessed value of collateral A is RMB 5,000,000 (￥5,000,000), which has been confirmed by on-site inspection [Evidence 1][Evidence 2];

[0119] Embedded evidence anchors:

[0120] [Evidence 1][Modal = Text][File = Collateral Appraisal Report.pdf][Page = 5][Area = [0.25, 0.4, 0.15, 0.05]];

[0121] [Evidence 2][Modality = Visual][Document = Handwritten On-site Investigation Record.jpg][Page = 1][Area = [0.1, 0.7, 0.3, 0.15]];

[0122] User interaction: Clicking on [Evidence 2] in the answer immediately locates and highlights the corresponding handwritten annotation area in the on-site investigation handwritten record.jpg in the right panel, completing the end-to-end evidence tracing.

[0123] Specifically, in practical applications, multiple multi-source data access modules are used in conjunction with a text retrieval enhancement subsystem, a visual retrieval enhancement subsystem, a dynamic path routing and fusion retrieval module, a multimodal structured parsing and evidence alignment module, and a source-tracing answer generation module. These multiple multi-source data access modules are located in different geographical locations. This invention discloses a dual-path retrieval enhancement parsing and evidence source tracing method for multi-source heterogeneous data. For multi-source heterogeneous data such as scanned documents, PDFs, and audio, a system is constructed that coordinates text retrieval enhancement paths and visual retrieval enhancement paths. The text path processes OCR recognition of text and raw electronic documents, while the visual path directly encodes the original image to preserve handwritten annotations, seals, etc. The system provides key visual evidence such as layout and structure; it supports dynamic selection of retrieval paths based on query semantics and data quality, and performs confidence-weighted fusion of dual-path results; the generated answer automatically embeds interactive evidence anchors containing modality type, file path, and spatial coordinates, allowing users to click and jump to the original evidence location, realizing end-to-end verifiable and traceable knowledge services. This invention solves the core problems of information loss, unexplainable results, and untraceable evidence caused by OCR distortion in traditional retrieval enhancement generation, significantly improving the reliability of knowledge services in high-confidence scenarios. This invention belongs to the interdisciplinary field of artificial intelligence and knowledge engineering, specifically involving a method for multi-source heterogeneous data parsing and reliable tracing based on dual-path retrieval enhancement. It is particularly suitable for processing unstructured data with complex features such as blurred images, handwritten content, signature overlays, and layout constraints. It solves the technical bottlenecks of existing RAG systems in terms of evidence integrity maintenance, cross-modal consistency verification, and attributability of generated results. It can be widely used in scenarios with stringent requirements for knowledge credibility, such as auditing, judicial evidence collection, financial risk control, and medical record management. It manages, visualizes, and stores RAG evidence tracing data and corresponding analysis results, and helps to realize RAG evidence tracing management through offline local evidence management and cloud-based control, thereby improving the intelligence level of RAG evidence tracing management.

[0124] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, equipment, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0126] In the embodiments provided in this application, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or equipment, and may be electrical, mechanical, or other forms.

[0127] The modules described, including multi-source data access, text retrieval enhancement subsystem, visual retrieval enhancement subsystem, dynamic path routing and fusion retrieval, multimodal structured parsing and evidence alignment, and source-tracing answer generation, may or may not be physically separate. The components displayed as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0129] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0130] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dual-path RAG evidence provenance method for multi-source heterogeneous data, characterized in that, Comprise the following specific steps: S1, receiving multi-source heterogeneous data; S2, constructing a dual-path retrieval enhancement system; S3, responding to user natural language queries, performing dynamic path selection and joint retrieval; S4, multi-modal structured analysis of the evidence obtained by retrieval, generating standardized knowledge blocks; S5, generating natural language answers based on the knowledge blocks by a large language model, and automatically embedding interactive evidence anchor points in the answers; the evidence anchor points support direct jumping to the corresponding position of the original evidence after user clicks, realizing end-to-end verifiable traceability.

2. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, characterized in that: The multi-source heterogeneous data in step S1 includes at least two of scanned images, PDF documents, voice recordings, and structured data tables.

3. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, wherein: The dual-path retrieval enhancement system constructed in step S2 comprises the following specific steps: S201, text retrieval enhancement path: semantic vectorization of OCR-recognized text and native electronic document text, construction of a text vector index library, and association of each text block with its source file path, page number, and character-level spatial coordinates; S202, visual retrieval enhancement path: inputting the original image into a visual large model encoder to extract its global and local visual features, and constructing an image feature vector index library; the visual features are used to represent visual primitive information such as layout structure, seal area, handwritten annotations, and character deformation in the image, and each feature vector is associated with its corresponding source image file path, page number, and pixel-level bounding box coordinates; The dual-path retrieval enhancement system in step S2 further comprises a cross-modal evidence consistency checking mechanism: When the text path and the visual path return conflicting evidence for the same fact, the system automatically labels "evidence conflict" and outputs a comparison report, including text recognition results, visual area screenshots, confidence comparison, and difference area positioning, for manual traceability verification.

4. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, wherein: After completing joint retrieval in step S3, the following any mode is adaptively selected for retrieval according to the semantic type of the query and the OCR confidence score of the multi-source heterogeneous data: Only enable the text retrieval enhancement path; Only enable the visual retrieval enhancement path; Enable both the text retrieval enhancement path and the visual retrieval enhancement path, and perform weighted fusion of the dual-path retrieval results according to evidence confidence to generate a unified sorted cross-modal evidence set.

5. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, wherein: Each knowledge block in step S4 contains the following fields: modality type, content summary, source file path, page number, spatial coordinate range, and confidence score.

6. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 3, wherein: The visual large model encoder is a visual base model pre-trained based on large-scale image data in a self-supervised manner, and is selected from any one of the following: DINOv2, image encoder of CLIP, EVA02, Intern Image, and MAE-Vi T; the visual large model encoder is fine-tuned on domain data containing handwriting, seals, and table frames.

7. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, wherein: The decision logic of dynamic path selection in step S3 is: When the query contains "in image", "handwriting", "annotation", "seal", "blur area" or similar visual sensitive semantics, or the average confidence of the current document page to be checked by OCR is lower than the preset threshold T, the visual retrieval enhancement path is preferentially enabled or the dual-path fusion mode is triggered; Wherein, T∈[0.5, 0.75], preferably 0.

65.

8. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 7, characterized in that: The step S4 includes: For the image area returned by the visual retrieval enhancement path, a lightweight visual analysis module is called to generate a natural language description as a "visual semantic summary", which includes object categories, relative position relationships, color attributes and text content speculation, for the large language model to understand and infer.

9. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, wherein: In the step S5, the evidence anchor point is embedded in the answer text in a machine-readable structured format, which is: [evidence ID][modality=X][file=Y][page=Z][region=[x,y,w,h]], Wherein, X∈{text, visual, audio}, Y is the relative file path, Z∈{document page number, audio timestamp}, [x,y,w,h] is the document normalized coordinate or key frame image absolute pixel coordinate; After the front-end system analyzes the anchor point, it can highlight the corresponding area of the original document, and support one-key jump, screenshot evidence and operation log recording; The method is implemented in a data-controlled environment; the data-controlled environment includes a local server, a private cloud or an isolated network domain meeting the third level of network security protection requirements.

10. The dual-path RAG evidence provenance method for multi-source heterogeneous data according to claim 1, wherein: The dual-path retrieval enhancement system includes: A multi-source data access module for receiving, analyzing and preprocessing the multi-source heterogeneous data, and performing OCR identification on scanned images and PDF documents; A text retrieval enhancement subsystem for performing semantic vectorization on the text obtained by OCR identification and the text of native electronic documents, and constructing and maintaining a text vector index library; A visual retrieval enhancement subsystem for inputting the original image into the visual large model encoder to extract its global and local visual features; A dynamic path routing and fusion retrieval module for responding to user queries and executing the dynamic path selection and joint retrieval logic; A multi-modal structured analysis and evidence alignment module for structuring and analyzing the text and visual evidence obtained by retrieval to generate standardized knowledge blocks; A traceable answer generation module for driving a large language model to generate a natural language answer based on the knowledge blocks; An evidence visualization and interaction module for analyzing and rendering the evidence anchor point.

Images