Research report intelligent analysis method, system and device based on multi-modal and adaptive RAG

Abstract

The application provides a research report intelligent analysis method, system and device based on multi-modal and adaptive RAG, the method comprises the following steps: obtaining a research report and a query intention, the query intention corresponds to the complexity category of a query question; target data is obtained by multi-modal analysis based on the research report; adaptive retrieval is performed based on the query intention and the target data, different retrieval strategies are matched for retrieval to obtain retrieval results; the relevance of the retrieval results is evaluated and reordered to return information segments, a preset large model is used to generate a structured answer through a self-defined prompt engineering based on the query intention and the information segments; and the structured answer is supported to obtain a query result and visual expression. The application systematically solves the multi-modal understanding, intelligent retrieval, answer credibility and visual display of financial research report analysis, has strong technical innovation and practicality, and is especially suitable for financial scenarios with high requirements for data accuracy and analysis depth.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method, system and device for intelligent analysis of research reports based on multimodal and adaptive RAG. Background Technology

[0002] In the financial sector, research reports serve as crucial information carriers supporting industry research, market assessment, and investment decisions. Financial research reports are characterized by their complex structure and information density, containing not only extensive textual descriptions but also numerous charts and graphs such as financial data tables, performance trend charts, and valuation graphs. These charts and data contain a wealth of information and are of significant value to financial analysis and decision-making. The key challenge for industry practitioners is how to quickly and accurately extract and understand core information from these massive amounts of unstructured research report documents.

[0003] Currently, the existing technologies for analyzing financial research reports still have the following problems:

[0004] 1. Limitations of single-modal processing: Most existing research report analyses only process text data and lack effective means to analyze multimodal information such as tables and charts in research reports. They cannot understand the structured relationships in tables and the data mapping relationships in charts, which leads to the omission or misinterpretation of a large number of key financial indicators, year-on-year / month-on-month data, and forecast data. Furthermore, traditional analysis methods cannot establish the relationship between different modal contents, while a complete viewpoint in a research report often requires multi-dimensional relationships such as text description, data support, and chart display to be described.

[0005] 2. Limited effectiveness of retrieval strategies: Existing research report analyses often employ a single retrieval strategy to handle user queries, failing to adaptively select a retrieval strategy based on the complexity and intent category of the user query. This affects the accuracy and depth of the analysis. Research report analyses built on basic RAG (Retrieval-Augmented Generation) typically use simple vector similarity searches, and the returned results may contain low-relevance or redundant information, leading to inaccurate answers.

[0006] 3. Limited presentation of analysis results: Current research reports mostly present results in plain text, lacking rich data visualization. This single text presentation fails to fully realize the value of the data and cannot meet the high requirements of financial analysis for data presentation. However, financial data analysis results often reveal trends, comparisons, and distributions more intuitively through charts and graphs, resulting in users being unable to intuitively understand the information and having a poor experience. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a research report intelligent analysis method, system and device based on multimodal and adaptive RAG, to solve the problems of single-modal limitation in research report analysis and the single retrieval strategy and result presentation in the prior art.

[0008] In a first aspect, the present invention provides a research report intelligent analysis method based on multimodal and adaptive RAG, the method comprising:

[0009] Obtain research reports and query intents, wherein the query intents correspond to the complexity category of the query question;

[0010] The target data is obtained by performing multimodal analysis based on the research report, wherein the data obtained by analyzing the research report includes at least one of the following: text data, chart data, table data, and formula data.

[0011] Adaptive retrieval is performed based on the query intent and the target data, wherein different retrieval strategies are matched based on the complexity category to obtain retrieval results;

[0012] The search results are re-ranked based on relevance evaluation to return search fragments. A structured answer is generated based on the query intent and the search fragments using a pre-defined large model and through customized prompt engineering.

[0013] The structured answer is subjected to a supportive evaluation to obtain the query results, and the query results are extracted and visualized for output to the user.

[0014] In some embodiments of the first aspect of this application, the search results are reordered based on relevance assessment to return search fragments, and the structured answer is supported to obtain query result output, specifically including:

[0015] Each search segment is scored and labeled; search segments with irrelevant labels are filtered out, and search segments with relevant labels are retained and reordered.

[0016] Calculate the proportion of irrelevant fragments. When the proportion of irrelevant fragments is higher than a preset value, a self-checking mechanism is triggered. The self-checking mechanism includes reconstruction issues and retrieval optimization.

[0017] Verify each structured answer and define a label; filter out structured answers that do not support the label and retain structured answers that support the label.

[0018] A visual chart is generated based on the query results, and the visual chart and the search results are synchronously output to the user.

[0019] In some embodiments of the first aspect of this application, the complexity category includes a first category, and the retrieval based on the first category specifically includes:

[0020] The first category of query questions corresponds to factual and / or numerical queries;

[0021] A hybrid retrieval strategy using keywords and vectors is employed. This strategy includes a keyword-first, vector-later strategy, a vector-first, keyword-later strategy, and a weighting strategy. Specifically, keyword-based matching queries and / or vector-based semantic similarity searches are used.

[0022] In some embodiments of the first aspect of this application, the complexity category includes a second category, and the retrieval based on the second category specifically includes:

[0023] The second category of query questions corresponds to queries that summarize viewpoints and / or make trend judgments;

[0024] A vector retrieval strategy is adopted for retrieval, in which the query question is converted into a vector, semantic similarity is searched in a vector database, and multi-source information is integrated.

[0025] In some embodiments of the first aspect of this application, the complexity category includes a third category, and the retrieval based on the third category specifically includes:

[0026] The third category of query questions corresponds to deep reasoning and / or relational queries;

[0027] A hybrid retrieval strategy combining keyword and graph retrieval is employed. This strategy includes multi-path recall, question decomposition, parallel retrieval, and result fusion. Specifically, queries are performed in graph databases, and / or semantic similarity searches are conducted in vector databases, while the question is decomposed into a logically related sequence of sub-questions for further querying.

[0028] In some embodiments of the first aspect of this application, multimodal analysis is performed based on the research report to obtain target data, specifically including:

[0029] The layout analysis model is used to perform area detection and layout analysis on the research report to identify target elements, which include at least one of text, images, tables, and formulas;

[0030] Based on the bounding box of the target element, different types of elements are extracted from the research report to obtain text block images, table images, chart images and formula images;

[0031] Text blocks are obtained based on the text block image; table descriptions are obtained based on the table image; chart descriptions are obtained based on the chart image; and formula descriptions are obtained based on the formula image.

[0032] Extract the text block, the table description, the chart description, and the formula description to generate a semantic vector, and store the semantic vector and its corresponding original text / description in the vector database of the target data.

[0033] In some embodiments of the first aspect of this application, multimodal analysis is performed based on the research report to obtain target data, specifically including:

[0034] The layout analysis model is used to perform area detection and layout analysis on the research report to identify target elements, which include at least one of text, images, tables, and formulas;

[0035] Based on the bounding box of the target element, different types of elements are extracted from the research report to obtain text block images, table images, chart images and formula images;

[0036] Text blocks are obtained based on the text block image; table descriptions are obtained based on the table image; chart descriptions are obtained based on the chart image; and formula descriptions are obtained based on the formula image.

[0037] Semantic content is obtained based on the text block, the table description, the chart description, and the formula description. Entity extraction, relation extraction, and knowledge fusion are performed based on the semantic content to obtain a knowledge graph, and the knowledge graph is stored in the graph database of the target data.

[0038] To achieve the above and other related objectives, a second aspect of this application provides a research report intelligent analysis system based on multimodal and adaptive RAG, the system comprising:

[0039] The acquisition module is used to acquire research reports and query intentions, wherein the query intentions correspond to the complexity category of the query question;

[0040] The parsing module is used to perform multimodal parsing based on the research report to obtain target data, wherein the data obtained by parsing the research report includes at least one of text data, chart data, table data, and formula data;

[0041] The retrieval module is used to perform adaptive retrieval based on the query intent and the target data, wherein different retrieval strategies are matched based on the complexity category to obtain retrieval results;

[0042] The reordering generation module is used to reorder the search results based on relevance evaluation to return search fragments. It uses a preset large model to generate structured answers based on the query intent and the search fragments through customized prompt engineering.

[0043] The output module is used to perform a supportive evaluation on the structured answer to obtain the query results, and extract the query results for visualization and output to the user.

[0044] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0045] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0046] As described above, the intelligent analysis method, system, and device for research reports based on multimodal and adaptive RAG of the present invention have the following beneficial effects:

[0047] 1. It achieves deep understanding and integration of multimodal information in financial research reports, solving the problems of information omission and fragmentation. Through multimodal parsing and extraction, it can accurately identify and deeply understand multimodal information such as text, tables, and charts in research reports, extracting key data, trends, and structured relationships. This fundamentally solves the problem of key information omission caused by neglecting non-textual content in traditional methods. By constructing a cross-modal knowledge graph, it establishes semantic connections between different modalities, breaking down information silos and forming a complete and unified knowledge representation. This provides a solid data foundation for in-depth analysis, achieving unified parsing and semantic association of text, charts, and tables, and avoiding information fragmentation.

[0048] 2. It enhances the processing capabilities for complex financial queries, significantly improving the accuracy and reliability of question answering. It can intelligently identify user intent and query complexity, and dynamically select the optimal retrieval path. It can efficiently handle simple fact queries as well as complex analysis tasks such as multi-hop inference, causal attribution, and multi-entity comparison, breaking through the limitations of traditional single retrieval strategies. Through retrieval quality assessment and factual verification of generated answers, a quality control closed loop is constructed, which can effectively filter noise information and support complex tasks from simple queries to multi-hop inference, resulting in more accurate retrieval and more reliable answers.

[0049] 3. Enhanced the self-optimization and adaptive capabilities of research report analysis, improved overall performance and user experience. By constructing a self-correction and closed-loop optimization mechanism, it can automatically diagnose problems, rewrite queries and re-retrieve when the retrieval or generation results are not good, significantly improving robustness. It has the ability to continuously learn, ensuring long-term performance improvement. The closed-loop optimization mechanism of retrieval evaluation, query rewriting and model fine-tuning continuously improves processing performance.

[0050] 4. It provides interactive and visualized intelligent analysis results, which greatly improves decision-making efficiency and user experience. It can automatically generate various professional charts based on the analysis conclusions and present them in a visually appealing way, transforming dry data into intuitive insights and improving the data presentation effect and user analysis and decision-making efficiency. Attached Figure Description

[0051] Figure 1 The diagram shows a step-by-step illustration of the intelligent analysis method for research reports based on multimodal and adaptive RAG in one embodiment of the present invention.

[0052] Figure 2 The diagram shown is a schematic representation of a research report analysis method based on multimodal and adaptive RAG in one embodiment of the present invention.

[0053] Figure 3 The diagram shows a knowledge graph of an embodiment of the intelligent analysis method for research reports based on multimodal and adaptive RAG of the present invention.

[0054] Figure 4 The diagram shows a query strategy in one embodiment of the intelligent analysis method for research reports based on multimodal and adaptive RAG of the present invention.

[0055] Figure 5 The diagram shows a correlation assessment of the intelligent analysis method for research reports based on multimodal and adaptive RAG in one embodiment of the present invention.

[0056] Figure 6 This diagram illustrates a support evaluation of the research report intelligent analysis method based on multimodal and adaptive RAG in one embodiment of the present invention.

[0057] Figure 7 The diagram shown is a structural schematic of an embodiment of the intelligent analysis system for research reports based on multimodal and adaptive RAG of the present invention.

[0058] Figure 8 The diagram shown is a structural schematic of an embodiment of the electronic device of the present invention.

[0059] Component designation explanation

[0060] S102~S110 step 70 Intelligent Research Report Analysis System Based on Multimodal and Adaptive RAG 71 Get Module 72 Parsing module 73 Search module 74 Rearrangement generation module 75 Output module 801 processor 802 memory 8021 operating system 8022 app 803 Network interface 804 bus system 805 User Interface Detailed Implementation

[0061] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0062] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0063] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0064] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0065] This application proposes a research report intelligent analysis method, system, and device based on multimodal and adaptive RAG, applicable to various financial research reports. It enables in-depth information extraction, intelligent question answering, and visual insights from massive, multimodal research reports, improving the efficiency and accuracy of investment research and decision-making. Suitable for diverse user needs, it provides analysts and investors with powerful and effective intelligent research assistance. The technical solutions in the embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, in one embodiment of the invention, the intelligent analysis method for research reports based on multimodal and adaptive RAG of the present invention includes the following steps:

[0067] Step S102: Obtain the research report and query intent, wherein the query intent corresponds to the complexity category of the query question;

[0068] Step S104: Perform multimodal analysis based on the research report to obtain target data, wherein the data obtained by analyzing the research report includes at least one of the following: text data, chart data, table data, and formula data;

[0069] Step S106: Based on the query intent and the target data, perform adaptive retrieval, wherein different retrieval strategies are matched based on the complexity category to obtain retrieval results;

[0070] Step S108: The search results are re-ranked based on relevance evaluation to return the search fragments. A structured answer is generated based on the query intent and the search fragments using a preset large model and through a custom prompting process.

[0071] Step S110: Perform a support evaluation on the structured answer to obtain the query results, and extract the query results for visualization and output to the user.

[0072] It should be noted that, in this embodiment, the solution is applied to different preset large models, including multiple models such as semantic embedding models, visual large models, table recognition models, formula recognition models, and large language models. When performing research report analysis, it is necessary to obtain the research report and perform multimodal parsing to obtain different databases, such as vector databases and graph databases in the target data. After obtaining the query intent, adaptive retrieval is performed based on the query question in the query intent and the target data to obtain different retrieval results. In this embodiment, the order of obtaining the research report and the query intent is not limited. If the research report is obtained first, multimodal parsing can be performed first, and then retrieval can be performed during the query. If the query intent is obtained first, the target data can be generated based on the research report before retrieval can be performed to ensure the accuracy and stability of data query.

[0073] Furthermore, in one embodiment of the invention, the target data includes a vector database and a graph database. Therefore, obtaining the target data through multimodal analysis based on the research report specifically includes:

[0074] The layout analysis model is used to perform area detection and layout analysis on the research report to identify target elements, which include at least one of text, images, tables, and formulas;

[0075] Based on the bounding box of the target element, different types of elements are extracted from the research report to obtain text block images, table images, chart images and formula images;

[0076] Text blocks are obtained based on the text block image; table descriptions are obtained based on the table image; chart descriptions are obtained based on the chart image; and formula descriptions are obtained based on the formula image.

[0077] Extract the text block, table description, chart description, and formula description to generate a semantic vector, and store the semantic vector and its corresponding original text / description in a vector database within the target data; and / or

[0078] Semantic content is obtained based on the text block, the table description, the chart description, and the formula description. Entity extraction, relation extraction, and knowledge fusion are performed based on the semantic content to obtain a knowledge graph, and the knowledge graph is stored in the graph database of the target data.

[0079] It should be noted that, in this embodiment, the research reports described, such as financial research reports (PDF, Word, etc.), are specifically uploaded by users. When using a large model for analysis, different models are used depending on different needs. First, based on the layout analysis model, the research report is used to perform area detection and layout analysis to identify target elements. These target elements include at least text, images, tables, and formulas. Different target elements are extracted to obtain different parsed data. Specifically, an adaptive content decomposition mechanism is used to extract four modalities of content from the research report: text, tables, charts, and formulas. This converts different content in the research report document into a structured descriptive data list, preserving the relationship between the structured descriptive data list and content elements, and enhancing its semantic relevance.

[0080] Furthermore, in this embodiment, as Figure 2 The diagram shown illustrates the analysis of a research report. Different content modalities—text, tables, charts, and formulas—are processed using the following methods:

[0081] (1) Text extraction, the steps include: extracting text content, dividing the text content into blocks and converting the obtained text blocks into semantic vectors for storage. Specifically, extracting the text content in the text blocks, inputting the text content into the semantic block model, splitting the text content into multiple sub-blocks with complete semantic units according to the text semantic logic and document structure, and outputting text blocks with different themes such as financial performance and industry analysis; using the semantic embedding model BGE-M3 to convert the text blocks into semantic vectors, inputting the split text blocks, encoding each text block into a 1024-dimensional semantic vector, and at the same time constructing a vector index to store the text and its semantic vectors in the vector database.

[0082] For example, for the research report text block "In Q1 2024, the company's semiconductor business revenue reached RMB 1.82 billion, a year-on-year increase of 23.5% and a quarter-on-quarter increase of 8.7%; the core driver of growth was the landing of orders from overseas automotive electronics customers, and the revenue in this field accounted for 42% of the total revenue of the semiconductor business from 35% in Q4 2023...", the auxiliary input text block corresponds to the research report chapter information "Chapter 3 Analysis of the Company's Core Business". The semantic segmentation model divides the text into text blocks such as "financial performance". The BGE-M3 model encodes the text blocks and converts them into 1024-dimensional vectors. The generated semantic vectors are bound to the original text of the text block and the chapter information and stored in the Milvus vector database.

[0083] (2) Chart extraction, the steps include: content recognition of text, images and tables in the chart image, generating a structured description of the recognized content to obtain chart data. Specifically, context enhancement extraction technology is used to extract text context from the title, annotation and body text around the chart to help understand the semantics of the chart. The visual big model Qwen2.5-VL is used to analyze the image and recognize the content to generate context-aware descriptive titles and detailed descriptions. For financial charts such as revenue bar charts and PE valuation trend charts, prompts are designed to extract key values, trend features and annotation information in the chart to generate a structured description of the chart to obtain chart data.

[0084] Input the chart image from the research report, titled "Company Revenue Bar Chart 2022-2024", and input the surrounding context as supplementary information, including the title "Chart A-1 2022-2024 XX Company Revenue Changes", the note "Unit: RMB 100 Million", the data source "Company Annual Financial Report and Analyst Forecasts", and the relevant sentence in the text "2023 Revenue declined by 5% year-on-year due to weak domestic market demand, and is expected to recover in 2024", etc. to enhance context processing.

[0085] The chart image and semantic background information are input into Qwen2.5-VL. The visual layer identifies the chart type (bar chart), coordinate axes (X-axis: year 2022 / 2023 / 2024, Y-axis: revenue (100 million yuan)), and the numerical values ​​corresponding to the height of the data bars. The model extracts the core information of the chart by prompting the engineer: "Please identify the chart type, the meaning of the coordinate axes, the specific values ​​of each data point (distinguish between actual values ​​and predicted values), year-on-year / month-on-month growth rate, trend characteristics, and explain the reasons for the trend in the context, and output a structured description." The model outputs the structured parsing results of the chart based on the prompts, in formats including JSON and Markdown.

[0086] (3) Table extraction, the steps include: content recognition of the text and table in the table image, generating a structured description of the recognized content to obtain table data. Specifically, context enhancement extraction technology is used to enhance understanding by using the title and explanatory text around the table, and a table recognition model (such as StructEqTable) is used to recognize the table structure, title, cell content and row and column relationships. For rotated tables, coordinate transformation and correction are performed. Furthermore, a large model is used to identify the semantic relationships and dependencies between multiple tables, such as inter-table related fields and data derivation relationships. Finally, the table data is obtained by performing a structured description based on the table content, row and column relationships and inter-table related fields obtained from the extracted table blocks.

[0087] Enter a table from the research report, such as "Comparison of Gross Profit Margin of Each Product Line in 2023", and at the same time, enter the surrounding context as supplementary information, including the title "Table 4-2 Gross Profit Margin of XX Company's Product Lines in 2023", the explanation "Gross profit margin calculation method: (Product line revenue - Product line cost) / Product line revenue × 100%", and the following related sentence "Compared with 2022, the gross profit margin of the consumer electronics product line decreased by 2 percentage points", etc. to enhance the context processing.

[0088] The StructEqTable model is used to process the input table image. Through contour detection and cell segmentation, it identifies the number of rows and columns (e.g., 3 rows and 4 columns), the location of merged cells (e.g., "Row 1, Columns 1-4: Table Title"), and the table header information (e.g., "Product Line Name, Revenue (RMB 100 Million), Cost (RMB 100 Million), Gross Profit Margin (%)"). If the table image is rotated (e.g., tilted by 15°), a coordinate transformation algorithm is used to correct the table to a horizontal state, ensuring accurate recognition of cell content. The text content of each cell is identified (e.g., "Consumer Electronics, 28.5, 22.1, 22.5"), and combined with contextual information, it is linked to the "Product Line Name" field of the 2022 table to confirm the relationships between tables.

[0089] (4) Formula extraction: Understand the semantic content of the formula image and convert the formula in image format into an understandable and searchable semantic representation. In the previous steps, the formula region has been parsed from the research report to obtain the formula image. The formula recognition model (such as UniMERNet model) is used to recognize and convert the formula to obtain the structured markup language code of the formula. The structured markup language code is extracted and described in natural language to obtain the formula language vector as the formula data for storage.

[0090] Input the formula image from the research report, such as "Net Profit Calculation Formula on Page 5", and simultaneously input the text around the formula, "Formula 1-1 Net Profit Calculation Logic: Used to measure the company's core profitability level, period expenses include sales, management, and financial expenses" to enhance context processing.

[0091] The UniMERNet model performs pixel-by-pixel analysis on the formula image, identifying symbols (such as "=" and "-"), text (such as "operating revenue"), and structures (such as linear equations) within the formula, and converts them into the structured markup language LaTeX format. Combined with supplementary information, it generates a natural language explanation of the formula (such as "Net profit is calculated by deducting operating costs, period expenses, and corporate income tax from operating revenue"). This natural language description is then input into the semantic embedding model BGE-M3, generating a semantic vector for the formula and storing it in a vector database.

[0092] It should be noted that the extraction of the above four types of content applies different models. For example, text extraction applies the semantic segmentation module and the semantic embedding module, chart extraction applies the visual large model, table extraction applies the table recognition model, and formula extraction applies the optical formula recognition model. Since the above existing models are only used as applications in this embodiment, the specific application process will not be described in detail.

[0093] Furthermore, in this embodiment, a multimodal large model is used to work collaboratively to transform the multimodal information data such as text, charts, and tables in the research report into a unified structured description. The structured descriptions of the four modalities are then vectorized to obtain semantic vectors corresponding to different modalities, which are then stored in a vector database.

[0094] Furthermore, in order to help achieve deep semantic understanding and structured information extraction of multimodal elements, specifically, to achieve deep fusion and semantic association of multimodal information, different large models are used for entity extraction and relation extraction, entities are constructed from different modal content, and the relationships between entities are mined to construct a cross-modal knowledge graph, establish a unified graph representation and semantic vector, and realize the semantic association of multimodal information.

[0095] The construction of the knowledge graph specifically includes: (1) Entity extraction: using large models and prompting engineering to extract key entities from the text, including different entity types such as companies, people, industries, and financial indicators. At the same time, the extracted charts and tables containing semantic descriptions are converted into structured knowledge entities, each entity containing attribute labels; (2) Relationship construction: using a relationship extraction model to extract semantic relationships between entities from the text, including relationship types such as belonging, influence, and support. At the same time, chapter-level relationships and attribution relationships are constructed according to the inherent structure of the research report. The original document organization structure is maintained through the "attribution" relationship chain. A deduplication strategy based on information entropy and a graph convolutional network (GCN) are introduced. (3) Graph storage and vectorization: Use graph database to store structured information of entities and relationships to realize graph storage. At the same time, use embedding model to generate vector representations of entity nodes and attributes and store them in vector database to form a hybrid storage mode of "graph database + vector database", which supports semantic similarity search and graph structure query; (4) Graph update: When users upload new research report data, use incremental update algorithm to dynamically expand the entities and relationships in the graph, automatically identify and merge the same entities and relationships from different text fragments, and improve data processing efficiency.

[0096] The following example research report excerpt will be used to explain the map construction process in detail:

[0097] Paragraph 1: "ABC Technology (stock code: ABCT) reported in its 2023 financial report that its cloud computing business revenue increased by 50% year-on-year, reaching 12 billion yuan. CEO Zhang Wei said that the growth was mainly due to strong demand in the Asia-Pacific market."

[0098] Paragraph 2: "As shown in Figure A, cloud computing has become the main source of revenue among the company's various business lines. At the same time, its gross profit margin remains at a high level of 40%, as detailed in Table B."

[0099] Chart 1: A bar chart titled "ABC Technology's Revenue by Business Segment 2021-2023".

[0100] Table 1: A table titled "Gross Profit Margin by Business Segment in 2023".

[0101] Large models such as Deepseek and prompts such as "Extract company, person, business, and financial indicator entities from the following text and classify them" are used to achieve text entity extraction. Entities are extracted from the structured descriptions after charts and tables are extracted to form chart entities or table entities.

[0102] The following is a list of entities extracted from the research report excerpt:

[0103] (1) Company: ABC Technology (Stock Code: ABCT)

[0104] (2) Person: Zhang Wei (Position: CEO)

[0105] (3) Business: Cloud Computing

[0106] (4) Financial Metric: Revenue, Gross Margin

[0107] (5) Chart: Figure A (Bar chart, title: ABC Technology's Revenue by Business Unit from 2021 to 2023)

[0108] (6) Table: Table B (Title: Gross Profit Margin by Business Segment in 2023)

[0109] (7) DataValue: 120 (Unit: RMB 100 million, Year: 2023, Business: Cloud Computing), 50 (Unit: percent, Type: growth_rate, Business: Cloud Computing), 40 (Unit: percent, Type: gross_margin, Business: Cloud Computing).

[0110] Based on the extracted entities, relation extraction models or cueing engineering are used to extract relationships between entities from the text, establish semantic connections between entities, and form a knowledge network. This includes semantic relationships extracted from the text, attribution relationships based on document structure, and complex associations obtained through reasoning. Deduplication and alignment are then performed: duplicate data within the knowledge graph itself is removed, and entities and relationships from different modalities that are related are aligned (i.e., linked together). Figure 3 As shown, the relationships are constructed as follows:

[0111] (1) Zhang Wei -> ABC Technology: BELONGS_TO (Zhang Wei is the CEO of ABC Technology)

[0112] (2) Cloud Computing -> ABC Technology: PART_OF (Cloud computing business belongs to ABC Technology)

[0113] (3) Cloud Computing -> Revenue: HAS_METRIC (Cloud computing business has revenue metrics)

[0114] (4) Revenue -> 120: HAS_VALUE (The revenue value is 12 billion yuan (2023))

[0115] (5) Revenue -> 50: HAS_VALUE (Revenue growth rate is 50%)

[0116] (6) Gross Profit Margin -> 40: HAS_VALUE (The gross profit margin value is 40%)

[0117] (7) Figure A -> Cloud Computing: DESCRIBES (Figure A describes cloud computing services)

[0118] (8) Table B -> Gross Profit Margin: DESCRIBES (Table B describes the gross profit margin metric)

[0119] (9) 120 -> 40: CORRELATED_WITH (High revenue is associated with high gross margin)

[0120] Furthermore, it should be noted that the above entities and relationships are stored using graph databases such as Neo4j. The query language Cypher is applied to efficiently traverse the graph structure, concatenating the name, type, and attributes of each entity into a text segment (after secondary concatenation via the knowledge graph). An embedding model such as BGE-m3 is used to generate vector representations for each entity and its key attributes, which are then stored in the Milvus vector database for semantic similarity search. The purpose of similarity retrieval is to also store the textual information of the knowledge graph in the vector database.

[0121] Furthermore, when new research report data is uploaded, the above extraction and construction process is repeated. The incremental update algorithm will compare entity names, attributes and vector similarity to determine whether the newly extracted "ABC Technology" entity is the same as the existing "ABC Technology" entity in the graph. If so, attribute merging (e.g., adding new business lines) and relationship updates will be performed instead of creating duplicate nodes, thereby ensuring the continuous evolution and consistency of the graph.

[0122] Furthermore, in this embodiment, after obtaining the knowledge graph, query applications can be performed, i.e., adaptive retrieval based on query intent. In this application, in response to the needs of multimodal data analysis of research reports, a multimodal retrieval engine is constructed, and an adaptive routing mechanism is combined to realize dynamic matching of retrieval resources and strategies. The core of the described strategy includes four links: intelligent question classification, adaptive retrieval routing, retrieval quality assessment, and self-optimization closed loop. It can dynamically adapt to various query scenarios such as factual, inference, and multi-intent types, and is particularly accurate in handling complex queries such as multi-hop inference, multi-entity comparison, and causal attribution commonly found in research report analysis.

[0123] Specifically, in one embodiment of the invention, adaptive retrieval is performed based on the query intent and the knowledge graph, wherein different retrieval strategies are matched based on the complexity category to obtain retrieval results.

[0124] It should be noted that, in this embodiment, a large model (such as Deepseek) is used as the core classifier, and a financial domain knowledge base containing financial terminology dictionaries, company information, policy and regulation information, financial business tags, etc. is connected to the system. Combined with the query question defined by prompt words, the system performs multi-level intent parsing of user queries. The intent classification tag system covers the needs of all scenarios of research report analysis. The obtained query intents include, but are not limited to, information extraction, opinion summarization, comparative analysis, attribution analysis, indicator query, trend judgment, valuation prediction, etc.

[0125] While identifying user intent, the complexity of the query is assessed, categorized into three levels: simple, medium, and complex. Simple queries involve a single entity or concept, where the required information can be directly obtained from a knowledge base or a single document, requiring no retrieval or only minimal reasoning. Examples include queries for basic financial indicators such as a company's price-to-earnings ratio or specific policy names, which can usually be answered directly using simple search strategies. Medium queries involve simple relationships between multiple entities or concepts, requiring the integration of a few facts or preliminary analysis. Examples include comparing the revenue growth rates of two companies or summarizing the core arguments of a research report, typically requiring multi-step retrieval or simple cross-modal retrieval, but without deep reasoning. Complex queries involve complex relationships between multiple entities and concepts, requiring multi-step reasoning, causal attribution, cross-modal information fusion, or deep analysis. Examples include analyzing the impact of macroeconomic policies on industry trends or performing multi-hop reasoning to predict stock valuations.

[0126] Different retrieval strategies will be adopted for different query intentions and problems of different complexities. The specific processing flow is as follows: the user queries a large model (such as Deepseek), the model extracts entities / concepts based on prompts, combines the identified concepts / entities, queries the financial knowledge base, recalls relevant domain knowledge, and further integrates the user query, identified entities, and knowledge base returned information for comprehensive analysis, and returns the results of intent recognition and complexity assessment.

[0127] Specifically, in this embodiment, the complexity category includes a first category, and the retrieval based on the first category specifically includes:

[0128] The first category of query questions corresponds to factual and / or numerical queries;

[0129] A hybrid retrieval strategy using keywords and vectors is employed. This hybrid strategy includes a keyword-first, vector-later strategy and a vector-first, keyword-later strategy. Specifically, keyword-based matching queries and / or vector-based semantic similarity searches are performed.

[0130] Specifically, in this embodiment, the complexity category includes a second category, and the retrieval based on the second category specifically includes:

[0131] The second category of query questions corresponds to queries that summarize viewpoints and / or make trend judgments;

[0132] A vector retrieval strategy is adopted for retrieval, in which the query question is converted into a vector, semantic similarity is searched in a vector database, and multi-source information is integrated.

[0133] Specifically, in this embodiment, the complexity category includes a third category, and the retrieval based on the third category specifically includes:

[0134] The third category of query questions corresponds to deep reasoning and / or relational queries;

[0135] A hybrid retrieval strategy combining keyword and graph retrieval is employed. This strategy includes multi-path recall, question decomposition, parallel retrieval, and result fusion. Specifically, queries are performed in graph databases, and / or semantic similarity searches are conducted in vector databases, while the question is decomposed into a logically related sequence of sub-questions for further querying.

[0136] It should be noted that, in this embodiment, based on the identified intent classification labels and complexity levels, the routing decision engine dynamically selects the retrieval source and retrieval strategy to achieve accurate retrieval of multimodal knowledge from research reports, such as... Figure 4 The diagram shown illustrates the query strategy, with the specific strategies as follows:

[0137] ① Factual / numerical queries:

[0138] For simple query intents such as information extraction and indicator lookup, the corresponding complexity category is the first category. The retrieval source is structured data extracted from research reports. A hybrid retrieval strategy of keywords plus vectors is adopted. The hybrid retrieval strategy includes a keyword-first and vector-later strategy, a vector-first and keyword-later strategy, and a weighting strategy. Among them, keyword precision matching based on the BM25 algorithm is given priority. If no matching results are found for keyword retrieval, the system automatically routes to the vector database. Through semantic similarity retrieval, relevant textual viewpoints are supplemented, and information from multiple sources is integrated. The results are reordered and merged to ensure that the answer is both comprehensive and highlights the key points.

[0139] ② Complex semantic / opinion-based queries:

[0140] For queries targeting viewpoint summaries and trend judgments, the retrieval source is a vector database, and the corresponding complexity category is the second category. It obtains relevant text paragraphs and chart descriptions through semantic similarity search and multimodal fusion sorting, and integrates and returns information from multiple sources. Vector retrieval corresponds to semantic similarity retrieval in the vector database.

[0141] ③ Deep reasoning / relationship lookup:

[0142] For complex query intents such as comparative analysis and attribution analysis, corresponding to the third complexity category, the retrieval sources are knowledge graphs and vector databases. A multi-path recall strategy is adopted, which calls graph traversal algorithms and semantic retrieval in parallel. The graph traversal algorithm is executed in the graph database to mine potential relationship chains and causal paths between entities. At the same time, relevant text paragraphs and chart descriptions are obtained through semantic similarity search. Information from multiple sources is integrated, and the results are reordered and merged to ensure that the answer is both comprehensive and highlights the key points.

[0143] ④ Multi-step reasoning query:

[0144] For query intents with extremely high valuation prediction complexity, corresponding to the third complexity category, the problem is decomposed into multiple sub-problems. Each sub-problem independently executes the aforementioned routing and retrieval process, outputting sub-query results. Each sub-retrieval result is reordered to ensure the relevance of the retrieval results. Finally, a coordinator Large Language Model (LLM) integrates all sub-results and returns the final retrieval answer.

[0145] After generating a candidate fragment pool through multi-path recall, the retrieved fragments are uniformly fused and re-ranked to optimize the relevance of document fragments to the query question. First, the candidate results are deduplicated and integrated, and the candidate fragments are re-ranked and scored based on a pre-trained large model to filter out low-relevance fragments. Furthermore, the Maximum Marginal Relevance (MMR) algorithm is used for diversity re-ranking, which preserves the semantic accuracy of the re-ranking model while introducing diversity.

[0146] Specifically, it includes:

[0147] The pre-trained reranking model bge-reranker is used to score the retrieved fragments, taking into account information such as query question, document fragment type (text, chart, knowledge triples), semantic similarity, keyword matching score, and contextual features, and outputs a reranking score. This reranking score is combined with the relevance label in the self-assessment. Fragments with high reranking scores have a higher probability of becoming highly relevant fragments. Among the top K (e.g., top 20) after reranking, the maximum marginal relevance algorithm is used for diversity reranking, and the final output top N (e.g., top 10) is selected, returning high-quality answers that are relevant to the query question from multiple perspectives.

[0148] For example, suppose a user enters a query: "Please analyze the impact of the Fed's interest rate hikes in 2023 on the stock prices of the technology sector and predict the trend over the next six months." The corresponding processing flow is as follows:

[0149] Input and parsing: The user query inputs the large model deepseek. The model extracts entities and concepts based on the prompt words and identifies key elements such as "Federal Reserve interest rate hike", "technology industry", "stock price", "impact", "prediction" and "next six months".

[0150] Knowledge base search: Combines with a financial knowledge base to retrieve relevant knowledge, such as the Federal Reserve's interest rate hike history, technology industry index data, and policy and regulatory impact analysis.

[0151] Intent Classification: After comprehensive analysis, the model identifies the query intent as "attribution analysis" and "trend prediction".

[0152] Complexity Assessment: The assessment query involves multiple entities (Federal Reserve, technology industry), multiple concepts (policy impact, stock price trend), requires multi-step reasoning (interest rate hike → market sentiment → stock price change → future prediction), and cross-modal data (text report, numerical data), and is therefore classified as "complex".

[0153] Search strategy: Based on complex categories, a multi-path search strategy is adopted, including knowledge graph traversal to find causal relationships, cross-modal fusion to integrate text and numerical data, and deep reasoning to generate search results.

[0154] Output: Returns results with intents of "attribution analysis" and "trend prediction", complexity of "complex", and triggers the adaptive retrieval module to perform subsequent complex retrieval and analysis.

[0155] In one embodiment of the invention, the search results are reordered based on relevance evaluation to return search fragments, and the structured answer is evaluated for support to obtain query results, specifically including:

[0156] Each search segment is scored and labeled; search segments with irrelevant labels are filtered out, and search segments with relevant labels are retained and reordered.

[0157] Calculate the proportion of irrelevant fragments. When the proportion of irrelevant fragments is higher than a preset value, a self-checking mechanism is triggered. The self-checking mechanism includes reconstruction issues and retrieval optimization.

[0158] Verify each structured answer and define a label; filter out structured answers that do not support the label and retain structured answers that support the label.

[0159] A visual chart is generated based on the query results, and the visual chart and the search results are synchronously output to the user.

[0160] It should be noted that, in this embodiment, the self-checking and evaluation process is the core of achieving high reliability. The self-checking and evaluation includes relevance assessment of the search results and support assessment of the structured answers. This process can perform dual checks on the search fragments and structured answers before and after answer generation. Through the dual-dimensional evaluation of the relevance of the search content and the utility of the generated answers, the search strategy is adjusted in real time, such as question rewriting, expanding the recall scope, and switching search algorithms, to reduce missed detections and false detections, and at the same time provide a basis for self-optimization closed loop.

[0161] Specifically, in this embodiment, as Figure 5 The diagram illustrates the relevance assessment process, specifically demonstrating the relevance evaluation of the search content during the self-assessment. The search fragments within the search content are a reordered list of information fragments obtained from adaptive retrieval (each fragment may be a piece of text, a chart description, or a knowledge graph triple). These fragments correspond to different blocks of content from research reports obtained from vector databases or through keyword searches, such as text blocks, chart descriptions, or table summaries. Each search fragment is independently evaluated and assigned a predefined label. The search fragment labels are as follows:

[0162] • [Relevant]: This section directly contains the key facts, data, or arguments needed to answer the user's question.

[0163] • [Partially Relevant]: This fragment is relevant to the query topic but cannot answer the question independently. It needs to be combined with other fragments to provide complete information (e.g., it provides background information or only answers part of the question).

[0164] • [Irrelevant]: This fragment is semantically irrelevant to the user query and is noise information.

[0165] Therefore, different operation steps are performed based on different search segment tags. Specifically, search segments with irrelevant tags are filtered out, while search segments with relevant tags are retained.

[0166] • For [Relevant] fragments: they are passed on first and directly to the subsequent answer generation process.

[0167] • For [Partially Relevant] segments: These are retained and passed on to subsequent answer generation processes, but the large language model will be prompted to "synthesize" this information during generation.

[0168] • For [Irrelevant] fragments: filter them out immediately to prevent noise from interfering with the generation process.

[0169] In practical applications, the relevance classification model can be implemented through large-model classification based on prompt engineering. An example classification prompt is as follows:

[0170] Please determine the relevance of the following search fragments to the user query from the perspective of semantic relevance:

[0171] User query: {query}

[0172] Search fragment: {fragment}

[0173] Rerank score: {rerank_score}

[0174] Please provide the reordering score for the search fragment and select the most appropriate label from the following:

[0175] - [Relevant]: Directly contains key information to answer the question.

[0176] - [Partially Relevant]: Related but cannot answer the question independently.

[0177] - [Irrelevant]: Completely irrelevant

[0178] Please output the category label name and confidence level.

[0179] A user query asked: "Analyze the profitability and growth drivers of ABC Technology's cloud computing business." Examples of different excerpts are given below:

[0180] Example snippet 1 - [Relevant]: Directly contains the key facts, data, or arguments needed to answer the user's question.

[0181] {

[0182] "content": "ABC Technology's gross profit margin for its cloud computing business reached 40% in 2023, an increase of 5 percentage points from 35% in 2022, mainly due to cost reductions resulting from economies of scale and optimized technology architecture."

[0183] "source": "2023 Annual Report, Page 15",

[0184] "relevance_label": "Relevant",

[0185] "confidence_score": 0.95

[0186] }

[0187] Example fragment 2 - [Partially Relevant]: It is related to the query topic, but cannot answer the question independently. It needs to be combined with other fragments to provide complete information.

[0188] {

[0189] "content": "Asia Pacific revenue breakdown: Cloud computing 65%, Enterprise software 25%, Others 10%",

[0190] "source": "Table 2: Revenue Composition by Region",

[0191] "relevance_label": "Partially Relevant",

[0192] "confidence_score": 0.75

[0193] }

[0194] Example snippet 3 - [Irrelevant]: Semantically irrelevant to the user query; it is noise information.

[0195] {

[0196] "content": "ABC Technology - Headquartered in - Nanshan District, Shenzhen",

[0197] "source": "knowledge graph entity relationships",

[0198] "relevance_label": "Irrelevant",

[0199] "confidence_score": 0.08

[0200] }

[0201] Specifically, the query results are output when the number of displayed contents in the search results is higher than or equal to a preset value; otherwise, a self-checking mechanism is triggered. This self-checking mechanism includes refactoring the question and optimizing the search. Specifically, the proportion of the [Irrelevant] fragment is calculated. If the proportion of irrelevant fragments is too high (e.g., exceeding 50%), the quality of the search is judged to be poor, and a self-optimization loop is triggered, such as rewriting the question or re-searching.

[0202] Furthermore, in this embodiment, as Figure 6 The diagram shown illustrates the supportive evaluation process, specifically demonstrating how the retrieval evaluation provides supportive assessment of the retrieval answer. The retrieval answer is generated by fusing a vector database of retrieval fragments and a graph database of a knowledge graph using a large language model. The key facts (entities, data, conclusions) in the retrieval answer are compared item by item with the retrieval evidence, and the authenticity of the answer is determined based on the retrieval content, whether there is any illusion, and whether the answer is credible.

[0203] The overall answer will be evaluated based on the verification results:

[0204] • [Supported] (Fully Supported): All key facts (data, entities, conclusions) in the answer can be clearly supported by the provided search evidence.

[0205] • [Partially Supported]: Some facts in the answer (e.g., ≥50%) are supported by evidence, but there are some unsupported inferences or minor details (these details do not conflict with the evidence, but are not directly supported either).

[0206] • [Not Supported] (Illusion): The answer contains statements that directly contradict the evidence or are completely unsupported by evidence (i.e., the big model is "illusion").

[0207] In practical applications, supportive evaluation labels can be implemented through large model classification based on prompt engineering. An example classification prompt is as follows:

[0208] Based on the provided search evidence, please assess the support level of the generated answer. Output only one of the following three tags:

[0209] [Supported]: All key facts in the answer can be clearly supported by the retrieved evidence;

[0210] [Partially Supported]: At least 50% of the key facts are supported by evidence, but there are inferences without evidence;

[0211] [Not Supported]: Statements that contradict the evidence or are completely unsupported by evidence exist;

[0212] User query: {query};

[0213] The generated answer is: {answer};

[0214] Search for evidence: {evidence};

[0215] Please only output the tag name, without any explanation.

[0216] Specifically, in this embodiment, for the user query "Analyze the revenue growth trend and main driving factors of ABC Technology's cloud computing business", the generated answer is: "According to publicly available data, ABC Technology's cloud computing business achieved strong growth in 2023, with revenue increasing from 12 billion yuan in 2022 to 15 billion yuan, representing a year-on-year growth rate of 25%. The main driving factor for this growth is the rapid expansion of public cloud services, especially the successful signing of several benchmark clients in the financial and government sectors, which boosted market share." The search evidence includes:

[0217] • Vector fragment 1 (from the 2023 financial report): "ABC Technology's cloud computing business revenue reached RMB 15 billion in 2023, a year-on-year increase of 25%. In 2022, the revenue of this business was RMB 12 billion."

[0218] • Vector fragment 2 (from analyst commentary): "The main driver of growth came from increased market share in public cloud services, particularly the acquisition of several benchmark customers in the financial and government sectors."

[0219] • Knowledge graph relationships:

[0220] “(ABC Technology) - [Business] -> (Cloud Computing)”;

[0221] "(Cloud computing) - [Revenue] -> (15 billion yuan, 2023)";

[0222] "(Cloud computing) - [Annual growth rate] -> (25%)";

[0223] "(Cloud computing) - [Key driving factors] -> (Public cloud)";

[0224] "(Public Cloud) - [Advantageous Industries] -> (Finance, Government Affairs)";

[0225] The evaluation process includes:

[0226] • Key Fact 1 (Trend): "Revenue increased from RMB 12 billion in 2022 to RMB 15 billion in 2023, a growth rate of 25%" → The data and evidence are completely consistent.

[0227] • Key Fact 2 (Driver): "The primary driver is the rapid expansion of public cloud services" → Consistent with the evidence that "the primary driver of growth is public cloud services".

[0228] • Key Fact 3 (Driver Details): "Successfully signed several benchmark clients in the financial and government sectors" → Consistent with the evidence that "several benchmark clients were acquired, particularly in the financial and government sectors".

[0229] In summary, all the core data and conclusions in the answer are directly derived from the retrieved evidence, without adding any additional inferences or unfounded information, so the verification result is [Supported].

[0230] Therefore, different operation steps are performed based on different search answer tags. Specifically, search answers with tags that do not support are filtered out, while search answers with tags that support are retained.

[0231] • For [Supported] answers: The answer has been accepted and is visually presented to the user.

[0232] • For [Partially Supported] answers: The answer may be accepted, but a note will be added (e.g., "This answer is partially based on inferences from aggregated information") to remind the user.

[0233] • For [Not Supported] answers: The answer is directly rejected and will not be shown to the user; at the same time, a self-optimization loop is triggered, attempting to re-retrieve or generate the answer.

[0234] Furthermore, in this embodiment, a closed-loop mechanism of "triggering, diagnosis, optimization, iteration, and fine-tuning" can be used to solve problems such as insufficient adaptability of retrieval strategies and answer illusion, thereby achieving continuous improvement in processing performance. The process is as follows:

[0235] ① When a large number of search fragments are evaluated as [Irrelevant] (e.g., more than 50%), or when the generated answer is judged as [Not Supported], the self-correction process will be automatically triggered.

[0236] ② Construct a problem diagnosis model to analyze the causes of problems such as query expression issues and low-quality search results, such as: vague query expression, lack of key entities, and ambiguity.

[0237] ③ Based on the diagnostic results, automatically generate 1-N rewritten and optimized queries. The strategies include: adding key entities, eliminating ambiguity, simplifying sentence structure, and adding contextual constraints.

[0238] ④ Re-execute the adaptive retrieval and generation process using the rewritten query, setting the maximum number of iterations (e.g., 3 times) to avoid infinite loops. If it still fails, output "No relevant content found".

[0239] ⑤ All error correction cases, including the original query, the rewritten query, the retrieval results, and the evaluation results, will be recorded and stored as samples to fine-tune the decision-making capabilities of the intent classification model, the query rewriting model, and the answer scoring model, thereby achieving continuous self-optimization of the entire system.

[0240] Furthermore, in this embodiment, a visual chart is generated based on the query results, and the visual chart and the search answer of the query results are synchronously output to the user terminal. Specifically, a suitable visual chart is automatically matched and generated according to the user's intent and the data type of the query results to intuitively present the analysis results. The most suitable visual chart type, such as line chart, bar chart, pie chart, heatmap, radar chart, etc., is automatically matched according to the query intent and the returned data type. Simultaneously, a visualization library is invoked to dynamically generate interactive charts based on the structured data results returned by the large language model. The text answer generated by the large model is integrated with the automatically generated charts / tables, and presented to the user in a unified, graphic-rich format through the front-end interface.

[0241] The scope of protection of the intelligent analysis method for research reports based on multimodal and adaptive RAG described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0242] Furthermore, existing technologies rely on OCR (Optical Character Recognition) technology for processing. Specifically, OCR-based research report analysis methods uniformly recognize all content in the research report, such as images, tables, and formulas, as text. For text elements such as axis labels, data labels, and legends in charts, the output is arranged according to the scanning order or spatial position, missing information such as the row and column structure and data relationships in the charts. This method cannot understand the semantics of non-text elements such as charts and formulas, nor can it achieve the association and fusion between different modal information.

[0243] The research report intelligent analysis method based on multimodal and adaptive RAG described in this invention integrates visual large model, context enhancement extraction, financial-specific prompt engineering, and adaptive retrieval generation technology. It can deeply understand the content of research reports, correlate and align cross-modal information, and perform intelligent reasoning analysis, achieving a leap from "recognizing text" to "understanding content and generating insights".

[0244] Specifically, taking the same research report on "Silicon Technologies" as an example, it includes a line chart comparing R&D investment ratios and a formula for calculating the technology leadership cycle. A key statement in the report is: "As shown in Figure C, although Silicon Technologies' revenue is relatively small, its R&D investment intensity has surpassed that of international giant Company A since 2021, confirming its 'technology-driven' strategy. According to our model (number of quarters with technology leadership = log2(1 + cumulative R&D investment / industry average)), its technology leadership window is expected to last 6-8 quarters."

[0245] A user asked, "Please verify whether the report's prediction of 'a technology lead time of 6-8 quarters' is reasonable?" A comparison of traditional OCR solutions and the technical solution of this invention is as follows:

[0246] (1). Processing methods and effects of traditional OCR solutions:

[0247] Chart processing: OCR recognizes the line chart as a string of text (such as "Chart C... Silicon Technology 8% 10% 15%... Company A 12% 13% 14%..."), with no corresponding relationship between the data fields.

[0248] Formula processing: The OCR recognizes the formula as ordinary characters "Technology Leading Quarter Number = log2(1 + Cumulative R&D Investment / Industry Average)", unable to understand its mathematical logic, let alone perform the calculation.

[0249] Question and Answer Performance: For such complex user questions, the system is unable to return a response because it cannot extract accurate cumulative R&D investment data from the charts, cannot understand and perform formula calculations.

[0250] (2). The processing method and effects of the present invention:

[0251] Chart Processing: Guided by prompts, the visual big data model transforms line charts into structured JSON data, accurately extracting the percentage of R&D investment for each company each year. Simultaneously, through context enhancement technology, it further... Figure 3 "Strongly correlated with this structured data, the system outputs structured data and semantic descriptions. Furthermore, entities and relationships are extracted from the text content and stored in a knowledge graph."

[0252] Formula processing: The formula is parsed and converted into a computable structured representation, and "cumulative R&D investment" and "industry average" are identified as key variables.

[0253] Information Fusion and Question Answering Effect: The adaptive RAG engine retrieves the predicted conclusion "6-8 quarters" and the corresponding mathematical formula from the main text. Based on the formula variable requirements, it extracts "XinKe Technology's cumulative R&D investment" from the structured chart JSON data and performs formula calculations. The calculation results are compared with the original prediction, generating a direct and accurate verification conclusion: "Based on calculations, the technology lead window based on historical data is approximately X quarters, which is basically consistent with / discrepancies with the report's prediction, due to...".

[0254] It should be noted that traditional OCR solutions primarily address the "digitalization" of unstructured text, while the intelligent research report analysis method based on multimodal and adaptive RAG described in this invention solves the problems of "semanticization" and "computability" of the entire research report. It can transform complete research report data, such as charts and formulas, into machine-readable, queryable, and computable structured data, laying a solid foundation for advanced analysis. It extends from recognizing characters to understanding the semantics of charts and the logic of mathematical formulas. Through context awareness and knowledge graph construction, it integrates, aligns, and links different modalities of information, such as text, data, and formulas, into a unified knowledge network, thereby understanding the inherent connections between different information fragments. Furthermore, by responding to and resolving complex analytical tasks requiring data extraction, mathematical operations, logical reasoning, and cross-validation, it effectively improves the automation level and decision support capabilities of research report analysis.

[0255] This application also provides a research report intelligent analysis system based on multimodal and adaptive RAG. The research report intelligent analysis system based on multimodal and adaptive RAG can implement the research report intelligent analysis method based on multimodal and adaptive RAG described in this application. However, the implementation device of the research report intelligent analysis method based on multimodal and adaptive RAG described in this application includes, but is not limited to, the structure of the research report intelligent analysis system based on multimodal and adaptive RAG listed in this embodiment. All structural modifications and substitutions of the prior art made according to the principles of this application are included within the protection scope of this application.

[0256] Please see Figure 7 In one embodiment, this embodiment provides a research report intelligent analysis system 70 based on multimodal and adaptive RAG, the system comprising:

[0257] The acquisition module 71 is used to acquire research reports and query intentions, wherein the query intentions correspond to the complexity category of the query question;

[0258] The parsing module 72 is used to perform multimodal parsing based on the research report to obtain target data, wherein the data obtained by parsing the research report includes at least one of text data, chart data, table data, and formula data;

[0259] The retrieval module 73 is used to perform adaptive retrieval based on the query intent and the target data, wherein different retrieval strategies are matched based on the complexity category to obtain retrieval results;

[0260] The reordering generation module 74 is used to reorder the search results based on relevance evaluation to return the search fragments. It uses a preset large model to generate structured answers based on the query intent and the search fragments through a custom prompting process.

[0261] Output module 75 is used to perform a supportive evaluation on the structured answer to obtain query results, and extract the query results for visualization and output to the user.

[0262] Since the specific implementation of this embodiment corresponds to the aforementioned method embodiment, the same details will not be repeated here, and those skilled in the art should also understand this. Figure 7 The division of the modules in the embodiments is only a logical functional division. In actual implementation, they can be fully or partially integrated into one or more physical entities. These modules can be fully implemented in software through processing element calls, fully implemented in hardware, or some modules can be implemented in software through processing element calls and some modules can be implemented in hardware.

[0263] In the embodiments provided by this invention, it should be understood that the disclosed apparatus or method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of the apparatus or module or unit may be electrical, mechanical, or other forms.

[0264] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0265] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0266] This invention also provides an electronic device, such as... Figure 8 As shown, the electronic device includes a processor and a memory.

[0267] like Figure 8 As shown, the electronic device includes at least one processor 801, a memory 802, at least one network interface 803, and a user interface 805. The various components in the device are coupled together via a bus system 804. It is understood that the bus system 804 is used to implement communication between these components. In addition to a data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 8 The general will label all buses as bus systems.

[0268] The user interface 805 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0269] It is understood that memory 802 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0270] In this embodiment of the invention, the memory 802 is used to store various types of data to support the operation of the electronic terminal 800. Examples of this data include: any executable program for operation on the electronic terminal 800, such as the operating system 8021 and application programs 8022; the operating system 8021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 8022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention may be included in the application program 8022.

[0271] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 801 or by instructions in software form. The processor 801 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 801 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 801 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0272] In an exemplary embodiment, the electronic terminal 800 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0273] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the method of any of the embodiments described above.

[0274] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0275] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0276] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0277] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0278] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0279] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0280] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0281] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0282] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0283] In summary, this application provides a research report intelligent analysis method, system, and device based on multimodal and adaptive RAG, which achieves unified parsing and semantic association of text, charts, and tables, avoiding information fragmentation; it supports complex tasks ranging from simple queries to multi-hop inference, resulting in more accurate retrieval and more reliable answers; it has a closed-loop optimization mechanism for retrieval evaluation, query rewriting, and model fine-tuning, which can continuously improve processing performance; and it can automatically generate professional charts, improving data presentation and decision-making efficiency. Therefore, this application effectively overcomes the limitations of multimodal fusion in existing technologies, as well as the shortcomings of single retrieval strategies and results, and has high industrial application value.

[0284] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A research report intelligent analysis method based on multimodal and adaptive RAG, characterized in that, include: Obtain research reports and query intents, wherein the query intents correspond to the complexity category of the query question; The target data is obtained by performing multimodal analysis based on the research report, wherein the data obtained by analyzing the research report includes at least one of the following: text data, chart data, table data, and formula data. Adaptive retrieval is performed based on the query intent and the target data. This includes matching different retrieval strategies based on the complexity category to obtain retrieval results. The complexity category includes a first category, and retrieval based on the first category specifically includes: the query question in the first category corresponds to factual and / or numerical queries; a hybrid retrieval strategy using keywords and vectors is employed, including a keyword-first-vector strategy and a vector-first-keyword strategy, wherein keyword-based matching queries and / or vector-based semantic similarity searches are performed; the complexity category includes a second category, and retrieval based on the second category specifically includes: the second category... The query questions correspond to opinion summaries and / or trend judgment queries; a vector retrieval strategy is used for retrieval, wherein the query questions are converted into vectors, semantic similarity searches are performed in a vector database, and multi-source information is integrated; the complexity category includes a third category, and retrieval based on the third category specifically includes: the query questions in the third category correspond to deep reasoning and / or relation queries; a hybrid retrieval strategy of keyword and graph retrieval is used for retrieval, the hybrid retrieval strategy includes multi-path recall, question decomposition, parallel retrieval, and result fusion, wherein queries are performed in a graph database, and / or semantic similarity searches are performed in a vector database, and the question is decomposed into a logically related sequence of sub-questions for querying; The search results are re-ranked based on relevance evaluation to return search fragments. A structured answer is generated based on the query intent and the search fragments using a pre-defined large model and through customized prompt engineering. The structured answer is subjected to a supportive evaluation to obtain the query results, and the query results are extracted and visualized for output to the user.

2. The intelligent analysis method for research reports based on multimodal and adaptive RAG as described in claim 1, characterized in that, The search results are reordered based on relevance to return search fragments, and the structured answers are evaluated for support to produce query results. Specifically, this includes: Each search segment is scored and labeled; search segments with irrelevant labels are filtered out, and search segments with relevant labels are retained and reordered. Calculate the proportion of irrelevant fragments. When the proportion of irrelevant fragments is higher than a preset value, a self-checking mechanism is triggered. The self-checking mechanism includes reconstruction issues and retrieval optimization. Verify each structured answer and define a label; filter out structured answers that do not support the label and retain structured answers that support the label. A visual chart is generated based on the query results, and the visual chart and the search results are synchronously output to the user.

3. The intelligent analysis method for research reports based on multimodal and adaptive RAG as described in claim 1, characterized in that, Based on the aforementioned research report, multimodal analysis was performed to obtain the target data, specifically including: The layout analysis model is used to perform area detection and layout analysis on the research report to identify target elements, which include at least one of text, images, tables, and formulas; Based on the bounding box of the target element, different types of elements are extracted from the research report to obtain text block images, table images, chart images and formula images; Text blocks are obtained based on the text block image; table descriptions are obtained based on the table image; chart descriptions are obtained based on the chart image; and formula descriptions are obtained based on the formula image. Extract the text block, the table description, the chart description, and the formula description to generate a semantic vector, and store the semantic vector and its corresponding original text / description in the vector database of the target data.

4. The intelligent analysis method for research reports based on multimodal and adaptive RAG as described in claim 1, characterized in that, Based on the aforementioned research report, multimodal analysis was performed to obtain the target data, specifically including: The layout analysis model is used to perform area detection and layout analysis on the research report to identify target elements, which include at least one of text, images, tables, and formulas; Based on the bounding box of the target element, different types of elements are extracted from the research report to obtain text block images, table images, chart images and formula images; Text blocks are obtained based on the text block image; table descriptions are obtained based on the table image; chart descriptions are obtained based on the chart image; and formula descriptions are obtained based on the formula image. Semantic content is obtained based on the text block, the table description, the chart description, and the formula description. Entity extraction, relation extraction, and knowledge fusion are performed based on the semantic content to obtain a knowledge graph, and the knowledge graph is stored in the graph database of the target data.

5. A research report intelligent analysis system based on multimodal and adaptive RAG, characterized in that, include: The acquisition module is used to acquire research reports and query intentions, wherein the query intentions correspond to the complexity category of the query question; The parsing module is used to perform multimodal parsing based on the research report to obtain target data, wherein the data obtained by parsing the research report includes at least one of text data, chart data, table data, and formula data; The retrieval module is used to perform adaptive retrieval based on the query intent and the target data. Specifically, it performs retrieval based on different retrieval strategies matched according to the complexity category to obtain retrieval results. The complexity category includes a first category, and retrieval based on the first category specifically includes: the query question of the first category corresponds to factual and / or numerical queries; a hybrid retrieval strategy using keywords and vectors is employed, including a keyword-first-vector strategy and a vector-first-keyword strategy, wherein keyword-based matching queries and / or vector-based semantic similarity searches are performed; the complexity category includes a second category, and retrieval based on the second category specifically includes: the second... The query questions for each category correspond to opinion summaries and / or trend judgments. A vector retrieval strategy is employed, where the query question is converted into a vector, semantic similarity is searched in a vector database, and multi-source information is integrated. The complexity category includes a third category, and retrieval based on this third category specifically includes: the query questions for this third category correspond to deep reasoning and / or relationship queries. A hybrid retrieval strategy combining keywords and graph retrieval is used, including multi-path recall, question decomposition, parallel retrieval, and result fusion. This involves querying in a graph database and / or performing semantic similarity searches in a vector database, as well as decomposing the question into a logically related sequence of sub-questions for querying. The reordering generation module is used to reorder the search results based on relevance evaluation to return search fragments. It uses a preset large model to generate structured answers based on the query intent and the search fragments through customized prompt engineering. The output module is used to perform a supportive evaluation on the structured answer to obtain the query results, and extract the query results for visualization and output to the user.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent analysis method for research reports based on multimodal and adaptive RAG as described in any one of claims 1 to 4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the research report intelligent analysis method based on multimodal and adaptive RAG as described in any one of claims 1 to 4.

Images