RAG method and system for compliance analysis of multi-modal financial document

Abstract

The invention relates to the technical field of financial large models, and provides an RAG method and system for compliance analysis of a multi-modal financial document, and the method comprises the steps: carrying out the preprocessing of an input multi-modal financial document, generating a corresponding text block through the multi-modal data in the multi-modal financial document, and constructing a corresponding vector database; responding to the target query request; according to the target query statement, entering a multi-path retrieval process to generate a binding packet for responding to the sub-query statements; based on the binding packages generated by the multi-path retrieval module, the semantic alignment degree of each binding package and a sub-query statement is calculated, a corresponding time reward score is calculated in combination with a time reward mechanism to obtain a matching score, ranking optimization is conducted on the matching score through a direct preference optimization mechanism, and a preference text block set is obtained to respond and generate a corresponding answer; by integrating the preprocessing module, the retrieval module and the reordering module of the financial document, the defects of the traditional method in the aspects of processing heterogeneous financial data and improving the retrieval recall rate and correlation are effectively overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of financial big models, and in particular to a RAG method and system for compliance analysis of multimodal financial documents. Background Art

[0002] In the financial industry, as regulations continue to evolve, financial institutions face increasingly complex compliance requirements. To meet these challenges, more and more financial institutions are seeking question-answering (QA) systems that can efficiently retrieve and analyze compliance information. These systems often need to be combined with retrieval-augmented generation (RAG) technology, leveraging external knowledge bases or databases to enhance the capabilities of language models and thus improve the accuracy of decision support. Existing RAG systems often rely on a single text retrieval method, such as dense retrieval or sparse lexical matching, to analyze and extract text content from financial documents. However, financial documents often contain multimodal data, such as unstructured text, semi-structured tables, and images, and existing RAG methods have certain limitations when processing this multimodal information.

[0003] Currently, traditional RAG systems struggle with multimodal data processing, particularly when dealing with the complex structured and unstructured data found in financial documents. These systems are unable to effectively integrate different types of data, resulting in fragmentation or loss of contextual information. Furthermore, existing retrieval methods primarily rely on semantic similarity for matching, ignoring the implicit regulatory relationships and domain standards unique to the financial sector. This compromises the accuracy and compliance of retrieval results. Furthermore, traditional ranking methods fail to fully consider the importance of compliance and are prone to missing key legal or financial information, further impacting the reliability and effectiveness of retrieval generation.

[0004] In order to solve the above problems, there is an urgent need for a RAG method that can effectively process multimodal financial documents and accurately identify compliance requirements. Summary of the Invention

[0005] This application proposes a RAG method and system for compliance analysis of multimodal financial documents. This method overcomes the limitations of existing technologies by combining a multimodal preprocessing component, a flexible multi-path retrieval module, and a domain-specific document re-ranking module. This allows for comprehensive processing of multimodal information such as text, tables, and images in financial documents, improving the accuracy and efficiency of information retrieval. Furthermore, domain-specific re-ranking technology ensures high accuracy in compliance analysis, providing a scalable and practical compliance analysis solution for financial institutions. The present invention provides a RAG method for compliance analysis of multimodal financial documents, comprising: Step S1: Preprocess the input multi-modal financial document. Use the file preprocessing module to generate corresponding text blocks from the multi-modal data in the multi-modal financial document. Employ the BGE-M3 dense encoder to perform vector quantization encoding on all preprocessed text blocks and their metadata, and construct a corresponding vector database; Step S2: Respond to the target query request; Step S3: According to the target query statement in the target query request, enter the multi-path retrieval process; The multi-path retrieval stage process is to decompose the target query statement into at least one or more sub-query statements, and retrieve text blocks matching the sub-query statements from the vector database through the multi-path retrieval module, and dynamically bundle and expand and combine the text blocks according to the similarity threshold to generate a bundle for responding to the sub-query statement; Step S4: Based on the bundles generated by the multi-path retrieval module, use the domain-specific document re-ranking module to calculate the semantic alignment degree of each bundle with the sub-query statement, calculate the corresponding time reward score by combining the time reward mechanism to obtain the matching score, and optimize the ranking of the matching score through the direct preference optimization mechanism to obtain the corresponding preferred text block set to respond to the target query statement and generate the corresponding answer.

[0006] Further, in step S1, preprocess the input multi-modal financial document. Use the file preprocessing module to generate corresponding text blocks from the multi-modal data in the multi-modal financial document. Employ the BGE-M3 dense encoder to perform vector quantization encoding on all preprocessed text blocks and their metadata, and construct a corresponding vector database, including: Step S11: Obtain the financial document for processing the target query request; Step S12: Use the open-source PDF tool MinerU to decompose multi-modal blocks including text, images, and tables in the original layout order of the multi-modal data of the financial document; Step S13: Convert the multi-modal blocks into text blocks through a large language model, convert the image blocks into structured text summaries as the text blocks of the images through a vision-language model, and convert the table blocks into text descriptions with a consistent structure as the text blocks of the tables; Step S14: Process the text blocks based on semantic enhancement technology, and output a set of text blocks for constructing the vector database.

[0007] Further, in step S14, process the text blocks based on semantic enhancement technology, and output a set of text blocks for constructing the vector database, including: Calculate the cosine similarity between different text blocks through SBERT sentence embedding, and when the cosine similarity exceeds the set threshold, merge the text blocks to complete redundancy removal; Using a large language model for coreference resolution, we iteratively parse pronouns within a chapter, identify and replace them with clear entities based on contextual information, and group different references representing the same entity into an equivalent set, addressing the issue of ambiguous references within a text block. Leverage a large language model to add structured metadata to each text block, including chapter title, page location, and data type.

[0008] All preprocessed text blocks and their corresponding metadata are vectorized using the BGE-M3 dense encoder to construct a vector database for target query statements on the client side.

[0009] Furthermore, in step S3, according to the target query statement in the target query request, a multi-path search process is entered, including: Step S31: When the client initiates a target query request to query the database, the executor uses natural language processing technology through a large language model to break the target query statement into multiple independent sub-query statements, replaces the pronouns in the independent sub-query statements with clear entities through coreference resolution, and automatically associates context information; Step S32: The multi-path retrieval module calculates similarity scores between the sub-query statements and the document blocks respectively through multiple search engines, wherein the search engines include a BM25 sparse search engine, a FAISS dense search engine, a metadata search engine, and a HyDE search engine; Step S33: performing weighted fusion based on the similarity scores of each retriever and the preset weight coefficient of the retriever, sorting the text blocks according to the weighted fusion scores, and selecting the top K text blocks with the highest to lowest scores as candidate text blocks, where K is a preset natural number greater than 0; Step S34: dynamically bundle and expand the candidate text blocks based on the similarity threshold to generate a bundle package for responding to the sub-query statement.

[0010] Furthermore, in step S34, the candidate text blocks are dynamically bundled and extended based on the similarity threshold to generate a bundle package for responding to the sub-query statement, including: The candidate text blocks are initially retrieved as independent units, and the corresponding dense embeddings are calculated using a pre-selected and trained text embedding model, where the dense embeddings are vector representations of the candidate text blocks. Calculate the cosine similarity of adjacent candidate text blocks based on dense embedding to determine the content relevance of adjacent candidate text boxes Determine whether the cosine similarity of adjacent candidate text blocks reaches a similarity threshold; When the cosine similarity of adjacent candidate text blocks reaches the similarity threshold, the adjacent candidate text blocks are dynamically merged into a bundle; otherwise, no merging is performed and they remain independent.

[0011] Further, in step S4, based on the bundles generated by the multi-path retrieval module, the domain-specific document re-ranking module calculates the semantic alignment degree between each bundle and the sub-query statement, and combines the time reward mechanism to calculate the corresponding time reward score. The calculation formula for obtaining the matching score is: ; Among them, is the matching score between the sub-query statement and the candidate text blocks in the bundle; is the semantic alignment degree between the bundle and the sub-query statement calculated by the cross-encoder; is the Sigmoid function, which is used to map the result to the interval [0, 1]; is the transpose of the weight vector W, is the time reward score; is the bias term.

[0012] Further, in step S4, the matching score is optimized by the direct preference optimization mechanism to obtain the corresponding preferred text block set to respond to the target query and generate the corresponding answer, including: Preliminarily calculate the matching score between the text block and the query statement in the BAAI / bge-reranker-v2-Gemma model of the document re-ranking module; Construct positive and negative sample pairs, and use the direct preference optimization mechanism to adjust the weights of the BAAI / bge-reranker-v2-Gemma model by minimizing the cross-entropy loss function to optimize the ranking of text blocks, forming the corresponding preferred text block set; Generate the final answer to respond to the target query statement according to the preferred text block set.

[0013] Further, the calculation formula of the cross-entropy loss function is, ; Among them, is the matching score of the positive sample, is the matching score of the negative sample, is the expected value of the positive and negative samples.

[0014] Based on the same inventive concept, the present invention provides a RAG system for multi-modal financial document compliance analysis, which executes the RAG method for multi-modal financial document compliance analysis as described above, including: A document preprocessing module for preprocessing the input multi-modal financial documents, generating corresponding text blocks from the multi-modal data in the multi-modal financial documents, vectorizing and encoding all the preprocessed text blocks and their metadata using a BGE-M3 dense encoder, and constructing a corresponding vector database; A response module for responding to a target query request; A multi-path retrieval module for entering a multi-path retrieval process according to the target query statement in the target query request; the multi-path retrieval stage process is to decompose the target query statement into at least one or more sub-query statements, retrieve text blocks matching the sub-query statements from the vector database, and dynamically bundle and expand and combine the text blocks according to a similarity threshold to generate a bundle for responding to the sub-query statement; A document re-ranking module, based on the bundles generated by the multi-path retrieval module, calculates the semantic alignment degree of each bundle with the sub-query statement through a domain-specific document re-ranking module, obtains a matching score by combining a time reward mechanism to calculate the corresponding time reward score, and optimizes the ranking of the matching score through a direct preference optimization mechanism, for obtaining a corresponding set of preferred text blocks to respond to the target query statement and generating a corresponding answer Further, the calculation formula of the matching score is: ; Where, is the matching score between the sub-query statement and the candidate text blocks in the bundle; is the semantic alignment degree of the bundle and the sub-query statement calculated through a cross-encoder; is the Sigmoid function for mapping the result to the interval [0, 1]; is the transpose of the weight vector W, is the time reward score; is the bias term.

[0015] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) The present invention provides a RAG method and system for multi-modal financial document compliance analysis. By integrating the preprocessing, retrieval, and re-ranking modules of financial documents, it effectively solves the deficiencies of traditional methods in processing heterogeneous financial data, improving retrieval recall rate and relevance. The system uses a hybrid retrieval strategy and domain-specific DPO re-ranking technology, not only improving the accuracy of information extraction, but also being able to prioritize the identification of compliance-critical content, ensuring high-quality and high-compliance answer generation. After comprehensive experimental verification, it shows a significant performance improvement, especially exceeding the existing baseline methods in terms of accuracy and recall rate in financial compliance tasks.

[0016] (2) By integrating the preprocessing, retrieval, and re-ranking steps of financial documents, the present invention provides an efficient and accurate end-to-end financial compliance Q&A method that can process multi-modal financial data and ensure the accurate extraction of compliance information.

[0017] (3) Through the multi-modal file preprocessing module, the present invention can uniformly process heterogeneous data formats such as text, tables, and images, and generate a structured vector database, effectively making up for the deficiency that traditional methods cannot process complex financial document data.

[0018] (4) The present invention introduces a multi-path retrieval module, including a dense retriever, a sparse retriever, a metadata retriever, and a HyDE retriever, which improves the recall rate and relevance of complex questions, especially having significant advantages when dealing with cross-modal and cross-document complex financial data.

[0019] (5) The present invention fine-tunes the re-ranking model through a direct preference optimization mechanism, preferentially presenting key compliance-related information, suppressing irrelevant content, and ensuring that the generated answers are more in line with regulatory requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of the steps of the RAG method for multi-modal financial document compliance analysis of the present invention; Figure 2 It is a working principle diagram of the RAG method for multi-modal financial document compliance analysis of the present invention; Figure 3 It is a schematic diagram of the query process in the embodiment of the present invention; Figure 4 It is a working framework diagram of the file preprocessing component in the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present application belong to the scope of protection of the present application.

[0022] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0023] The first embodiment The application of existing Retrieval-Augmented Generation (RAG) methods in financial document compliance analysis faces several core challenges. First, financial documents often contain multiple types of data forms, including unstructured text (such as narrative disclosures), semi-structured data (such as tables, images), and structured data. Traditional text-based retrieval methods cannot effectively handle these heterogeneous data, resulting in information fragmentation and context loss, thus affecting the comprehensiveness and accuracy of compliance analysis. Second, existing RAG methods usually rely on dense retrieval or sparse matching techniques. Although these methods perform well in some general tasks, they lack the in-depth capture of implicit regulatory relationships and legal terms specific to the financial field. As a result, when faced with financial documents, the system is prone to overlooking key compliance information or making reasoning errors, thereby reducing the effectiveness and reliability of the system. Third, the retrieval ranking of existing technologies is usually based on semantic similarity rather than domain-specific compliance priorities. This leads to the failure to present key information in a timely and accurate manner, which may affect the compliance decisions of financial institutions and increase regulatory risks. Finally, due to the frequent updates of financial regulations, existing systems lack sufficient dynamic adaptability and cannot quickly respond to regulatory changes, resulting in insufficient stability for long-term use in the regulatory environment.

[0024] Specifically, existing retrieval methods mainly perform matching by calculating the semantic similarity between texts, that is, finding the document or paragraph that is closest in meaning to the query content. However, this method ignores the implicit regulatory relationships and domain standards specific to the financial field. There are many specific rules and requirements in the financial field (such as compliance requirements, legal terms, financial indicators, etc.). These contents are often not expressed through direct semantic similarity but involve specific regulatory provisions, regulatory interpretations, or industry practices. These "implicit" and non-explicit regulatory relationships and domain standards may not be captured solely through semantic similarity matching. Therefore, existing retrieval methods may not be able to fully identify and process these key, domain-specific contents, resulting in incomplete or inaccurate results. That is to say, existing methods only focus on surface semantic matching and ignore some implicit regulations and standards that must be followed in the financial field, which may affect the accuracy and reliability of compliance analysis.

[0025] Based on the above-mentioned problems, the inventors proposed the FinSage framework through thinking, aiming to fundamentally solve the key difficulties in the compliance analysis of financial documents. The FinSage framework in this application proposes a comprehensive and robust solution by innovatively combining multimodal preprocessing, domain-aware retrieval strategies, and compliance priority ranking mechanisms. Regarding the problem of multimodal data processing, FinSage converts heterogeneous data formats such as text, tables, and images into structured data through a file preprocessing component, overcoming the problem of traditional methods dealing with fragmented information and ensuring that different types of data can be efficiently processed in a unified framework. Secondly, through a multi-path retrieval module, sparse retrieval, dense retrieval, metadata-aware semantic search, and hypothesis expansion retrieval (HyDE) are combined to accurately capture implicit regulatory relationships and legal terms in financial documents, ensuring that key information is not missed. In terms of compliance priority ranking, a domain-specific re-ranking module based on direct preference optimization ensures that compliance-critical content is presented first, while suppressing irrelevant information, improving the processing efficiency of documents and the accuracy of compliance decisions. In summary, the retrieval-augmented generation (RAG) method proposed in this application has the ability of dynamic adaptation and real-time update, and can operate efficiently in an environment where financial regulatory standards are constantly changing, fully meeting the requirements of modern financial institutions for compliance, accuracy, and flexibility. The specific implementation is as follows: As Figure 1 , 2 shown, the present invention provides a RAG method for multimodal financial document compliance analysis, including: Step S1: Preprocess the input multimodal financial document, generate corresponding text blocks for the multimodal data in the multimodal financial document through a file preprocessing module, and perform vector quantization encoding on all preprocessed text blocks and their metadata using a BGE-M3 dense encoder to construct a corresponding vector database; Step S2: Respond to the target query request; Step S3: According to the target query statement in the target query request, enter the multi-path retrieval process; the multi-path retrieval stage process is to decompose the target query statement into at least one or more sub-query statements, and retrieve text blocks matching the sub-query statements from the vector database through a multi-path retrieval module, and dynamically bundle and expand the combination of text blocks according to a similarity threshold to generate a bundle for responding to the sub-query statement; Step S4: Based on the bundles generated by the multi-path retrieval module, calculate the semantic alignment degree of each bundle with the sub-query statement through a domain-specific document re-ranking module, calculate the corresponding time reward score by combining a time reward mechanism to obtain a matching score, and optimize the ranking of the matching score through a direct preference optimization mechanism to obtain a corresponding set of preferred text blocks to respond to the target query statement and generate a corresponding answer.

[0026] The specific process is as follows Figure 3 , 4 As shown, the document preprocessing module preprocesses the input financial document or financial file, including text encoding and semantic enhancement. Among them, text encoding mainly extracts multi-modal blocks through open-source PDF tools, including text, images, and tables, etc. The multi-modal blocks are converted into text representations through the large language model LLM. Semantic enhancement mainly includes three parts, namely (a) removing redundant blocks through similarity comparison; (b) resolving co-references within subheading chapters; (c) generating summaries based on subheading chapters as metadata. The enhanced blocks are embedded in the vector database for subsequent processing. When receiving the target query statement in the database query from the client, such as the user's query: How did Lotus Tech perform in 2024 and how does it plan to expand this year? When this target query statement enters the multi-retrieval path stage, For the target query statement of the user's query, the multi-path retrieval module first performs query paraphrasing, including query decomposition, splitting the target query statement into multiple independent sub-query statements, and performing co-reference resolution and context integration. For example, it is split into: What is the core marketing strategy for the growth of Lotus Technology Inc. in 2025? and What are the sales data of Technology Inc. in 2024? Based on the split sub-query statements, the multi-path retrieval module performs matching retrievals through the BM25 sparse retriever, FAISS dense retriever, metadata retriever, and HyDE retriever respectively, and dynamically bundles and combines the text blocks according to the similarity threshold to generate a bundle combination for responding to the sub-query statements. According to the generated bundle combination, it is re-optimized and sorted through the document re-ranking module to generate the final answer for responding to this target query statement.

[0027] Furthermore, in step S1, the input multi-modal financial document is preprocessed. The multi-modal data in the multi-modal financial document is generated into corresponding text blocks through the document preprocessing module, and the BGE-M3 dense encoder is used to vectorize and encode all the preprocessed text blocks and their metadata to construct a corresponding vector database, including: Step S11: Obtain the financial document for processing the target query request; Step S12: Use the open-source PDF tool MinerU to decompose the multi-modal data of the financial document into three types of multi-modal blocks: text, images, and tables in the original layout order; Step S13: Convert the multi-modal blocks into text blocks through the large language model, including: converting the image blocks into structured text summaries as the text blocks of the images through the vision language model, and converting the table blocks into text descriptions with consistent structures as the text blocks of the tables, and this text description highlights the data trends; Step S14: Process the text blocks based on semantic enhancement technology and output a set of text blocks for constructing a structured vector database.

[0028] Further, in step S14, processing the text blocks based on semantic enhancement technology and outputting a set of text blocks for constructing the vector database includes: Calculate the cosine similarity between different text blocks through SBERT sentence embedding, and when the cosine similarity exceeds a set threshold, merge the text blocks to complete redundancy removal; Use a large language model for anaphora resolution. By iteratively parsing pronouns in the same chapter and identifying and replacing them with explicit entities through context information, different references representing the same entity are divided into an equivalent set to solve the problem of unclear anaphora in text blocks; Use a large language model to add structured metadata to each text block. The metadata includes chapter titles, page locations, data structures, and time information.

[0029] Vectorize and encode all preprocessed text blocks and their corresponding metadata through a BGE-M3 dense encoder to construct a vector database for querying by the target query statement of the client.

[0030] It should be specifically noted that financial documents usually contain complex and diverse data formats, including unstructured text (such as narrative disclosures), semi-structured data (such as tables and images), and contextual metadata. These multimodal data are often interdependent and closely related and cannot be processed independently. For example, narrative text usually involves detailed company financial information and market analysis, while tables and images present data trends and financial metrics, and the combination of this information can truly reflect the company's financial situation. However, traditional retrieval systems usually focus on the processing of text data and have weak capabilities in understanding and indexing non-text information such as images and tables, resulting in fragmented information or lost context during retrieval.

[0031] Due to the heterogeneity of these data types, traditional keyword-based retrieval methods (such as sparse retrieval based on BM25) may not be able to comprehensively capture the semantic relationships of documents, especially when the user's query involves complex financial problems. For example, simple keyword matching may ignore numerical trends in images or key data in tables, thus affecting the accuracy of retrieval. On the other hand, modern multimodal methods need to process and understand different types of data simultaneously to ensure that all relevant information can be comprehensively considered and efficiently extracted during the retrieval process.

[0032] To address these challenges, the FinSage framework proposed in this application uses a multi-modal data file preprocessing component to uniformly convert information such as text, tables, and images into structured vector representations, ensuring that different types of data are consistently processed within the same framework. This processing method avoids the fragmentation of context information and enhances the integration ability of multi-modal information, enabling the retrieval system to more comprehensively and accurately respond to complex query requirements in the financial field. The specific implementation process is as follows: (1) Use the open-source PDF tool MinerU to parse the document content into three types of chunks in the natural reading order: text chunks, image chunks, and table chunks. These chunk sets are denoted as . Each chunk represents an independent part of the document, facilitating subsequent processing and analysis.

[0033] (2) Convert the multi-modal chunks, including: ① Use a vision-language model to convert the information in the image into a structured text summary. For example, use " Figure 1 Show a 15% year-on-year increase in revenue in 2023" as the description of the image chunk to provide clearer semantic information for subsequent text processing and analysis. ② Convert the data in the table into a text narrative with a consistent structure to highlight data trends. For example, convert "Title: Company operating costs. Content: Operating costs decreased by 8% compared to the previous year" into the text description of the table chunk to facilitate the extraction and understanding of the trends and changes behind the data.

[0034] (3) Semantic enhancement, including ① Calculate the cosine similarity between different text chunks by using the SBERT (Sentence-BERT) sentence embedding technique. When the similarity between two text chunks reaches or exceeds a preset threshold (e.g., 0.95), they are considered duplicates, and one of the chunks is merged to remove redundant content and optimize the storage and extraction of information. ② Add structured metadata to each text chunk , and the metadata includes information such as chapter title, page location, data type, and time (e.g., "Income Statement - Page 5 - Table - September 1, 2024"), which helps quickly locate and identify the source and relevant information of the chunk during subsequent retrieval. ③ Text normalization, including ① Expand the abbreviations in the text to their full forms. For example, expand "EBITDA" to "Earnings Before Interest, Taxes, Depreciation, and Amortization" to ensure the standardization of the text. ② Unify the numbers expressed in different formats. For example, convert "$1.2M" to "1.2 million US dollars" to ensure the standardization of numerical values. ③ For legal-related terms, perform standardized replacement, such as unifying "non-compliance" to "non-compliance" to improve the readability and consistency of legal documents.

[0035] (4) Output the set of processed text chunks , where each text block It consists of five components, including the enhanced , metadata , dense text embedding , BM25 sparse vector and metadata densely embedded , ensuring that the multi-dimensional information of the document can be represented in the structured vector database.

[0036] Furthermore, in step S3, according to the target query statement in the target query request, a multi-path search process is entered, including: Step S31: When the client initiates a target query request to query the database, the executor uses natural language processing technology through a large language model to break the target query statement into multiple independent sub-query statements, replaces the pronouns in the independent sub-query statements with clear entities through coreference resolution, and automatically associates context information; Step S32: The multi-path retrieval module calculates similarity scores between the sub-query statements and the document blocks respectively through multiple search engines, wherein the search engines include a BM25 sparse search engine, a FAISS dense search engine, a metadata search engine, and a HyDE search engine; Step S33: performing weighted fusion based on the similarity scores of each retriever and the preset weight coefficient of the retriever, sorting the text blocks according to the weighted fusion scores, and selecting the top K text blocks with the highest to lowest scores as candidate text blocks, where K is a preset natural number greater than 0; Step S34: dynamically bundle and expand the candidate text blocks based on the similarity threshold to generate a bundle package for responding to the sub-query statement.

[0037] It should be noted that existing RAG systems typically rely on dense search (such as semantic matching based on deep learning) or sparse lexical matching (such as the BM25 algorithm). Although these methods can provide good results in general scenarios, they have the following limitations in financial applications: ① Traditional RAG systems are often not adapted for specific fields. Therefore, when processing financial documents, they may not be able to effectively capture the terminology, regulatory requirements or legal relationships in specific fields, resulting in insufficient accuracy when answering complex compliance questions.

[0038] ② Financial documents often contain implicit or indirect regulatory requirements. Traditional retrieval methods based on semantic similarity or keyword matching may not be able to accurately capture these implicit relationships, thus affecting the answers to compliance questions.

[0039] In order to solve the above problems, this application introduces the following two innovative steps: Combine sparse retrieval (such as BM25) and dense retrieval (such as semantic matching based on the FAISS model), and enhance the ability to understand legal terms, policy regulations, and industry standards in financial documents through domain-specific fine-tuning. For example, adapt the retrieval model to the domain through a custom dataset to identify regulatory keywords such as "material defect" and "non-compliant behavior".

[0040] ②During the retrieval process, weight the query by combining the metadata of financial documents (such as chapter titles, table annotations, context summaries, etc.), so that the retrieval can better understand the structural information of the document and effectively capture the explicit and implicit regulatory relationships in the document.

[0041] ③Use a large language model to generate hypothetical document paragraphs to expand the semantic scope of the query. This process helps the retrieval system identify and capture more implicit relationships by transforming relevant questions into hypothetical scenarios. For example, for the query "liquidity risk factors", the system generates a hypothetical paragraph "A decline in the liquidity coverage ratio may affect the solvency" to guide the retrieval system to mine relevant regulatory document content.

[0042] ④By fusing the retrieval results of multiple paths, combining dense retrieval, sparse retrieval, and metadata retrieval, the recall rate is improved, and higher relevance retrieval for complex financial problems is also ensured. By weighting and fusing the scores of these paths, the retrieval accuracy of relevant documents can be effectively improved, and ultimately ensure that the document content meets the compliance requirements.

[0043] Through the above steps, the system of the present invention can effectively overcome the deficiencies of traditional RAG systems in the financial field, especially in capturing regulatory requirements and implicit legal relationships, and improve the performance of the system in financial compliance tasks. The specific implementation process is as follows: (1) After receiving the user's original target query statement, first perform query expansion through the HyDE (Hypothetical Document Expansion) method to generate a hypothetical document to expand the query semantics, so as to capture a wider range of relevant information. For example, for the query "financial risk factors", the extended query generated by HyDE may be "The query 'financial risk factors' may involve liquidity ratios or debt terms" to expand the semantic scope of the original query.

[0044] (2) On the basis of query expansion, the system uses the following retrievers to improve the recall rate and retrieval relevance: FAISS dense retriever, using the BGE-M3 model, calculates the cosine similarity between the query and text blocks based on text dense embeddings.

[0045] BM25 sparse retriever, which uses the BM25 algorithm to match keywords and simultaneously weights and considers metadata fields (such as chapter title weight x2).

[0046] A metadata retriever retrieves chapter summaries related to a query by embedding metadata, increasing the context information for retrieval. It concatenates the chapter titles and summaries of each text block into a unified metadata embedding to ensure that all blocks within the same semantic segment share the same metadata representation. When retrieving a single metadata instance, it automatically associates all blocks in the same chapter, enhancing the cross-block context relevance and reducing retrieval ambiguity in multi-document scenarios.

[0047] The above results are comprehensively scored through weighted score fusion (weights optimized by grid search). Finally, the top 50 candidate text blocks are selected. For the retrieved candidate text blocks, bundling expansion is performed by concatenating the candidate blocks with their adjacent blocks to form a contextually coherent bundle, solving the fragmentation problem.

[0048] In addition, the step introduction part of the multi-path retrieval method also includes a query translation process, which mainly performs triple optimization on the input query through a large language model, including: Query decomposition: decomposes complex queries into independent sub-queries. For example, "2023 revenue growth and cost control measures" is decomposed into "2023 revenue growth rate" and "2023 cost control strategies".

[0049] Coreference resolution: replaces pronouns with specific entities. For example, "its financial risk" is converted to "the financial risk of XX company".

[0050] Context integration: if the query involves a historical conversation, it automatically associates the context information.

[0051] Further, in step S34, based on the similarity threshold, the candidate text blocks are combined through the dynamic bundling expansion to generate the bundle for responding to the sub-query statement, including: The candidate text blocks are initially retrieved as independent units, and the corresponding dense embeddings are calculated through a pre-trained text embedding model, where the dense embedding is the vector representation of the candidate text block; Based on the dense embeddings, the cosine similarity of the adjacent text blocks of the candidate text blocks is calculated to judge the content relevance of the adjacent text blocks of the candidate text blocks Judge whether the cosine similarity of the adjacent text blocks reaches the similarity threshold; When the cosine similarity of the adjacent text blocks reaches the similarity threshold, the adjacent text blocks are added to the candidate block set, and at the same time, they are dynamically merged with the candidate text blocks into a bundle; otherwise, no merging is performed and they remain independent.

[0052] It should be specifically noted that to solve the problem of cross-block distribution of key information, a dynamic bundling strategy is designed, including: 1. Initial retrieval: At the beginning of the multi-path retrieval phase, the document is decomposed into multiple independent text blocks. Each text block is treated as a separate retrieval object and participates in the search. Each text block can be a paragraph, table, image, or other information unit from the document. The system will first evaluate the relevance of these text blocks to the query, and each retrieved text block will be assigned a matching score for subsequent processing.

[0053] 2. Adjacent expansion: Dynamically expand the candidate set by the similarity between adjacent text blocks of the candidate text blocks and the user query to solve the problem that key information is scattered in different blocks. For the retrieved text blocks , the system will check the similarity between its adjacent front and back text blocks (i.e., and ) and the user query. Mainly use dense embeddings to calculate the cosine similarity between them. When the cosine similarity between the adjacent text block and the user query is greater than a certain similarity threshold, for example, the set similarity threshold is 0.85, then it is considered that this adjacent text block has sufficient semantic relevance. The similarity threshold is usually tuned according to the document type and domain requirements to ensure that only text blocks highly relevant to the user query are merged. Through this expansion in this step, semantically related adjacent text blocks can be aggregated together to avoid the loss of fragmented information. The following is an example of bundling: During the retrieval of the user query "Company's net profit in 2023", the candidate text block "Revenue in 2023" is found, and its adjacent text block "Cost analysis" also has a high similarity with the user query (for example, the similarity reaches 0.86, exceeding the similarity threshold of 0.85). Combine these two text blocks into a bundle, named "Financial performance in 2023", to form a semantically coherent and complete context unit. The text blocks in this bundle are not necessarily simple concatenations, but are reasonably combined according to their semantic and context relationships. This bundling method helps to provide a more coherent and comprehensive answer when answering the target query statement, especially when dealing with long texts or fragmented information.

[0054] Through the above two steps, the problems of scattered and fragmented key information are effectively solved. First, text blocks are obtained through initial retrieval; then, semantically related blocks are dynamically judged and merged through adjacent expansion; finally, by bundling adjacent blocks, a coherent context is generated to ensure the integrity of information and the accuracy of answers.

[0055] Furthermore, the document re-ranking module calculates the semantic alignment degree between each bundle and the sub-query statement through a cross-encoder, and combines the time reward mechanism to calculate the corresponding time reward score. The formula for obtaining the matching score is: ; where, is the matching score between the subquery statement and the candidate text blocks in the bundle; is the semantic alignment degree between the bundle and the subquery statement calculated by the cross-encoder; is the Sigmoid function, which is used to map the result to the interval [0, 1]; is the transpose of the weight vector W, is the time reward score; is the bias term.

[0056] Furthermore, the matching score is optimized by the direct preference optimization mechanism to generate a corresponding set of preferred text blocks to respond to the target query statement and generate the corresponding answer, including: In the BAAI / bge-reranker-v2-Gemma model of the document re-ranking module, the matching score between the text block and the query statement is initially calculated; Construct positive and negative sample pairs, and use the direct preference optimization mechanism to adjust the ranking of the text blocks by minimizing the cross-entropy loss function to form a corresponding set of preferred text blocks; According to the set of preferred text blocks, generate the final answer to respond to the target query statement.

[0057] Furthermore, the calculation formula of the cross-entropy loss function is ; where is the matching score of the positive sample, is the matching score of the negative sample, is the expected value of the positive and negative samples.

[0058] Specifically, the document re-ranking module uses the BAAI / bge-reranker-v2-Gemma model as the basic model, which is a multi-language and high-performance re-ranking model. This model can process text data from different languages and domains and is suitable for cross-language and cross-domain document ranking tasks.

[0059] The cross-encoder CrossEncoder is used to calculate the context matching degree between the query q and the text block The purpose is to evaluate the semantic alignment degree between the query and the candidate text blocks to ensure that the returned text blocks are highly relevant to the query.

[0060] The time reward mechanism introduced in this application adjusts the score based on the publication time of the text block. This function Dynamically adjust the score according to the release time. Text blocks released recently will receive higher rewards (for example, the reward coefficient of text blocks released within 1 year is multiplied by 1.2). This mechanism can ensure that the system favors more recent and relevant content when sorting.

[0061] In this application, the direct preference optimization (DPO) generates positive and negative sample pairs from multi-path retrieval results. The positive sample is a text block containing compliance keywords, such as "major litigation", and the negative sample is a text block that is semantically related but not related to compliance, such as "corporate social responsibility". Through manual annotation and screening, it is ensured that the system can learn how to correctly distinguish compliant and non-compliant content.

[0062] The iterative optimization process of the DPO in this application: Retrieval and annotation: Use a vector search engine such as FAISS to obtain candidate documents. Then, manually annotate these candidate documents to generate a preference dataset containing positive and negative sample pairs. In this way, high-quality training data can be provided for the model, enabling it to accurately distinguish compliant and non-compliant content in subsequent learning.

[0063] Model evaluation: Use evaluation metrics such as NDCG (Normalized Discounted Cumulative Gain) and MRR (Mean Reciprocal Rank) to measure the performance of the model in the sorting task. NDCG evaluates the relevance of the model's document sorting, while MRR evaluates the efficiency of the model when retrieving the correct document.

[0064] Dynamic adjustment: When the performance of the model on the new test set is not ideal, re-annotate the data and fine-tune the model's parameters. In this way, the model can adapt to new data and scenarios and continuously optimize its performance.

[0065] Second Embodiment Based on the same inventive concept, the present invention also provides a RAG system for multi-modal financial document compliance analysis, which executes the above-mentioned RAG method for multi-modal financial document compliance analysis, including: Based on the same inventive concept, the present invention provides a RAG system for multi-modal financial document compliance analysis, which executes the RAG method for multi-modal financial document compliance analysis as described above, including: A file preprocessing module for preprocessing the input multi-modal financial documents, generating corresponding text blocks from the multi-modal data in the multi-modal financial documents, vectorizing and encoding all preprocessed text blocks and their metadata using a BGE-M3 dense encoder, and constructing a corresponding vector database; A response module for responding to a target query request; A multi-path retrieval module is used to enter a multi-path retrieval process according to the target query statement in the target query request; the multi-path retrieval phase process is to decompose the target query statement into at least one or more sub-query statements, retrieve text blocks matching the sub-query statements from the vector database, and dynamically bundle and expand and combine the text blocks according to a similarity threshold to generate a bundle for responding to the sub-query statement. A document re-ranking module, based on the bundles generated by the multi-path retrieval module, calculates the semantic alignment degree of each bundle with the sub-query statement through a domain-specific document re-ranking module, combines a time reward mechanism to calculate the corresponding time reward score to obtain a matching score, and optimizes the ranking of the matching score through a direct preference optimization mechanism, so as to obtain a corresponding set of preferred text blocks to respond to the target query statement and generate a corresponding answer. Further, the calculation formula of the matching score is: ; where is the matching score between the sub-query statement and the candidate text block in the bundle; is the semantic alignment degree of the bundle and the sub-query statement calculated by the cross-encoder; is the Sigmoid function, which is used to map the result to the interval [0, 1]; is the transpose of the weight vector W, is the time reward score; is the bias term.

[0066] The above has described the embodiments of the present disclosure. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A RAG method for multimodal financial document compliance analysis, characterized in that, Including: Step S1: Preprocess the input multi-modal financial document. Generate corresponding text blocks from the multi-modal data in the multi-modal financial document through a file preprocessing module. Use a BGE-M3 dense encoder to perform vector quantization encoding on all preprocessed text blocks and their metadata, and construct a corresponding vector database; Step S2: Respond to the target query request; Step S3: According to the target query statement in the target query request, enter a multi-path retrieval process; the multi-path retrieval stage process is to decompose the target query statement into at least one or more sub-query statements, and retrieve the text blocks matching the sub-query statements from the vector database through a multi-path retrieval module, and perform dynamic bundling expansion and combination on the text blocks according to a similarity threshold to generate a bundle for responding to the sub-query statement; Step S4: Based on the bundle generated by the multi-path retrieval module, calculate the semantic alignment degree of each bundle and the sub-query statement through a domain-specific document re-ranking module, calculate the corresponding time reward score by combining a time reward mechanism to obtain a matching score, and optimize the ranking of the matching score through a direct preference optimization mechanism to obtain a corresponding set of preferred text blocks to respond to the target query statement and generate a corresponding answer.

2. The RAG method for multi-modal financial document compliance analysis according to claim 1, wherein, In step S1, preprocess the input multi-modal financial document. Generate corresponding text blocks from the multi-modal data in the multi-modal financial document through a file preprocessing module. Use a BGE-M3 dense encoder to perform vector quantization encoding on all preprocessed text blocks and their metadata, and construct a corresponding vector database, including: Step S11: Obtain the financial document for processing the target query request; Step S12: Use the open-source PDF tool MinerU to decompose multi-modal blocks including text, images, and tables in the original layout order of the multi-modal data of the financial document; Step S13: Convert the multi-modal blocks into text blocks through a large language model, convert image blocks into structured text summaries as text blocks of the images through a vision-language model, and convert table blocks into text descriptions with a consistent structure as text blocks of the tables; Step S14: Process the text blocks based on semantic enhancement technology, and output a set of text blocks for constructing the vector database.

3. The RAG method for multi-modal financial document compliance analysis according to claim 2, wherein, In step S14, process the text blocks based on semantic enhancement technology, and output a set of text blocks for constructing the vector database, including: Calculate the cosine similarity between different text blocks through SBERT sentence embedding, and when the cosine similarity exceeds a set threshold, merge the text blocks to complete redundancy removal; Use the large language model for anaphora resolution. Iteratively parse pronouns in the same chapter, identify and replace them with explicit entities through context information, and divide different references representing the same entity into an equivalent set to solve the problem of unclear anaphora in the text blocks; Adding structured metadata to each of the text blocks using the large language model, the metadata including chapter title, page location, and data type; All the pre-processed text blocks and the metadata corresponding to the text blocks are subjected to text vectorization encoding through the BGE-M3 dense encoder and a vector database is constructed for querying the target query statement of the client.

4. The RAG method for multimodal financial document compliance analysis according to claim 3, wherein In step S3, according to the target query statement in the target query request, a multi-path search process is entered, including: Step S31: When the client initiates a target query request to query the database, the executor uses the large language model and natural language processing technology to decompose the target query statement into multiple independent sub-query statements, replaces the pronouns in the independent sub-query statements with the explicit entities through coreference resolution, and automatically associates the context information; Step S32: the multi-path retrieval module calculates the similarity scores between the sub-query statement and the document block respectively through multiple search engines, wherein the search engines include a BM25 sparse search engine, a FAISS dense search engine, a metadata search engine, and a HyDE search engine; Step S33: performing weighted fusion based on the similarity scores of each of the retrievers and the preset weight coefficients of the retrievers, sorting the text blocks according to the weighted fusion scores, and selecting the top K text blocks with the highest to lowest scores as candidate text blocks, where K is a preset natural number greater than 0; Step S34: dynamically bundling and expanding the candidate text blocks based on the similarity threshold to generate the bundle package for responding to the sub-query statement.

5. The RAG method for multi-modal financial document compliance analysis according to claim 4, characterized in that, In step S34, the candidate text blocks are dynamically bundled and extended based on the similarity threshold to generate the bundle package for responding to the sub-query statement, including: Performing preliminary retrieval on the candidate text blocks as independent units, and calculating corresponding dense embeddings using a pre-selected and trained text embedding model, wherein the dense embeddings are vector representations of the candidate text blocks; Calculate the cosine similarity of adjacent candidate text blocks based on the dense embedding to determine the content relevance of the adjacent candidate text boxes Determining whether the cosine similarity of the adjacent candidate text blocks reaches the similarity threshold; When the cosine similarity of the adjacent candidate text blocks reaches the similarity threshold, the adjacent candidate text blocks are dynamically merged into a bundle; otherwise, they are not merged and remain independent.

6. The RAG method for multimodal financial document compliance analysis according to claim 5, wherein In step S4, based on the bundles generated by the multi-path retrieval module, the domain-specific document re-ranking module calculates the semantic alignment between each bundle and the sub-query statement, and the time reward mechanism is combined to calculate the corresponding time reward points to obtain the matching score. The calculation formula is: ； wherein, is the matching score of the sub-query statement and the candidate text block in the bundle; is the semantic alignment degree of the bundle and the sub-query statement calculated by the cross-encoder; the is the Sigmoid function, which is used to map the result to the interval [0, 1]; is the transpose of the weight vector W, is the time reward score; is the bias term.

7. The RAG method for multi-modal financial document compliance analysis according to claim 6, characterized in that, In step S4, the matching scores are ranked and optimized by a direct preference optimization mechanism to obtain a corresponding set of preferred text blocks to respond to the target query and generate a corresponding answer, including: The matching score between the text block and the query statement is initially calculated in the BAAI / bge-reranker-v2-Gemma model of the document reordering module; Positive and negative sample pairs are constructed, and the direct preference optimization mechanism is used to adjust the weights of the BAAI / bge-reranker-v2-Gemma model by minimizing the cross-entropy loss function to optimize the ranking of the text blocks, constituting a corresponding set of preferred text blocks; According to the set of preferred text blocks, the final answer to the target query statement is generated.

8. The RAG method for multi-modal financial document compliance analysis according to claim 7, characterized in that, The calculation formula of the cross-entropy loss function is ； Among them, is the matching score of the positive sample, is the matching score of the negative sample, is the expected value of the positive and negative samples.

9. A RAG system for multi-modal financial document compliance analysis, which executes the RAG method for multi-modal financial document compliance analysis according to any one of claims 1 to 8, characterized in that including: A file preprocessing module for preprocessing the input multi-modal financial document, generating corresponding text blocks from the multi-modal data in the multi-modal financial document, vectorizing and encoding all preprocessed text blocks and their metadata using a BGE-M3 dense encoder, and constructing a corresponding vector database; A response module for responding to a target query request; A multi-path retrieval module for entering a multi-path retrieval process according to the target query statement in the target query request; The multi-path retrieval stage process is to decompose the target query statement into at least one or more sub-query statements, retrieve the text blocks matching the sub-query statements from the vector database, and dynamically bundle and expand and combine the text blocks according to a similarity threshold to generate a bundle for responding to the sub-query statement; A document reordering module, based on the bundle generated by the multi-path retrieval module, calculates the semantic alignment degree between each bundle and the sub-query statement through a domain-specific document reordering module, combines a time reward mechanism to calculate the corresponding time reward score to obtain a matching score, and optimizes the ranking of the matching score through a direct preference optimization mechanism to obtain a corresponding set of preferred text blocks to respond to the target query statement and generate a corresponding answer.

10. The RAG system for multimodal financial document compliance analysis according to claim 9, wherein The calculation formula of the matching score is: ； wherein, is the matching score of the sub-query statement and the candidate text block in the bundle; is the semantic alignment degree of the bundle and the sub-query statement calculated by the cross-encoder; the is the Sigmoid function, which is used to map the result to the interval [0, 1]; is the transpose of the weight vector W, is the time reward score; is the bias term.

Images