Financial chart multi-modal understanding method and system based on graph neural network

Abstract

The invention provides a financial chart multi-modal understanding method and system based on a graph neural network, and the method comprises the steps: carrying out the structural element analysis of a financial chart, dividing the financial chart into a plurality of semantic structure units, and enabling each semantic structure unit to comprise a structure type, a spatial position and text content; taking the semantic structure units as nodes, constructing edges through spatial association and semantic association between the nodes, and constructing a graph structure; performing vectorization coding on the graph structure to generate a full graph structure feature code; a prompt text of each node and a prompt text of each edge are constructed, and structured prompt information Graph-Prompt is generated through integration; the method comprises the following steps: injecting Graph-Prompt into a large multi-mode model; and the model executes a chart understanding task based on Graph-Prompt. The method can automatically, uniformly and accurately characterize the structural relationship between the internal elements of the chart, and effectively injects the structure into the multi-modal large model, so that the model can really understand the internal organization logic of the financial chart.

Description

A Multimodal Understanding Method and System for Financial Charts Based on Graph Neural Networks Technical Field

[0001] This invention belongs to the field of intelligent financial chart analysis technology, and in particular relates to a method and system for multimodal understanding of financial charts based on graph neural networks. Background Technology

[0002] Brokerage research reports often contain financial charts. Current financial chart analysis techniques primarily rely on manual reading of these reports, supplemented by tools such as image recognition and OCR. Common methods involve analysts directly observing the chart content, comparing trends, or interpreting the reasons for changes, or using OCR systems to identify chart titles, legends, axes, and annotations, then inputting the results into a natural language processing system for description generation or question answering. Some institutions also employ multimodal large-scale models to interpret financial charts; however, current technologies mainly rely on overall image feature representation combined with simple textual information to generate conclusions, making it difficult to form a true understanding of the chart's internal structure.

[0003] In practical applications, charts in brokerage research reports are often presented in the form of multiple sub-charts, multiple texts, and multiple annotations. Each chart contains various information areas, including titles, main chart areas, legends, scales, annotations, and data sources. Especially in complex topics such as industry comparisons, sector rotation, and macroeconomic indicators, the logical relationships between different areas within the chart are quite close, making it a highly structured visual document. Existing financial chart analysis equipment (including OCR systems and multimodal models) primarily relies on image visual features and text features to determine the meaning of charts, but it cannot identify the relationships between different elements within the chart. For example, the correspondence between legends and curves, the semantic relationship between titles and main charts, and the hierarchical relationship between data sources and annotations are all difficult to accurately model using existing methods.

[0004] In brokerage charts, there is often a large amount of structured information presented as text, such as "Data source: Wind," "Note: The index is based on xx," and "Figure 1: Industry Prosperity Comparison." These areas have similar rectangular frames, but their semantic roles are completely different. Existing chart understanding models rely solely on pixel features or OCR text, making it impossible to distinguish the functions of different nodes. This easily leads to errors such as misinterpreting legends as captions or recognizing titles as body text, thus interfering with normal analysis, reducing automation efficiency, and making chart understanding results unstable. Ultimately, this may mislead investment decisions.

[0005] Different areas of a chart exhibit clear structural relationships: the title is typically at the top, summarizing the overall image; the main chart area contains key numerical information such as line charts and bar charts; the legend explains the semantic meaning of curves; and annotations usually describe calculation methods, sample sources, or special cases. These relationships constitute a natural graph structure, but current chart understanding technologies lack explicit encoding methods for this structure. The common approach is to input all text areas into the model in positional order, but this linear concatenation fails to represent the true hierarchical relationships, making it difficult for the model to infer the chart's meaning based on its structure.

[0006] Because financial charts are highly structured, and existing multimodal models are insufficient in modeling structural information, a common industry practice is for developers to manually write prompts, adding titles, annotations, and legends as additional text to the prompt template. However, this manual rule-based approach is ill-suited for complex scenarios and fails to depict the relationships between nodes, lacking versatility and further degrading the model's ability to interpret complex charts. Especially in charts with multiple text blocks, subgraphs, or multi-level structures, the simple method of piecing together structural information is highly susceptible to interference from erroneous OCR outputs, leading to incorrect final results.

[0007] Therefore, under the current technological conditions, how to construct a method that can automatically, uniformly and accurately represent the structural relationships between elements within a chart, and effectively inject this structure into a multimodal large model so that the model can truly understand the internal organizational logic of financial charts, has become a major problem that urgently needs to be solved in the field of intelligent financial chart analysis technology. Summary of the Invention

[0008] This invention provides a method and system for multimodal understanding of financial charts based on graph neural networks. It can automatically, uniformly and accurately represent the structural relationships between elements within the chart and effectively inject this structure into a large multimodal model, enabling the model to truly understand the internal organizational logic of the financial chart.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows: a multimodal understanding method for financial charts based on graph neural networks, comprising: S1: performing structural element parsing on the financial chart, dividing it into multiple semantic structural units, each semantic structural unit containing structural type, spatial location, and text content; S2: constructing edges based on the spatial and semantic relationships between semantic structural units as nodes, thereby constructing a graph structure; S3: performing vectorized encoding on the graph structure to generate full graph structural feature encoding; S4: constructing prompt text for each node and each edge, integrating to generate structured prompt information Graph-Prompt; S5: injecting Graph-Prompt into a multimodal large model; S6: the multimodal large model performs chart understanding tasks based on Graph-Prompt.

[0010] Furthermore, in step S1, the structure type is used to identify the semantic category of the semantic structure unit, the spatial location is used to represent the area occupied by the semantic structure unit in normalized coordinates, and the text content is used to include the text content in the identified semantic structure unit.

[0011] Furthermore, the construction of the graph structure in step S2 includes: S201, constructing spatial connection edges based on spatial adjacency relationships to reflect the relative relationships of semantic structural units of different structural types; S202, constructing semantic prior edges based on semantic rules to make the graph structure conform to the organizational logic of real financial charts.

[0012] Furthermore, step S3 includes: S301, constructing initial features of nodes by converting the node's structural type, spatial location, and text content into vectors and concatenating them; S302, calculating the structural embedding of nodes through a graph attention network; S303, compressing all the structural embeddings of nodes together to obtain the full graph structural feature encoding.

[0013] Further, step S4 includes: S401, constructing node prompt text, including the structural type, spatial location, and text content of each node; and the encoding of the full graph structural features; S402, constructing edge prompt text, including the type of edge; S403, integrating the node prompt text and edge prompt text to generate a structured prompt message Graph-Prompt.

[0014] Furthermore, in step S5, the Graph-Prompt injection of the multimodal large model includes one or more of the following: prefix injection, system prompt injection, and graph-text concatenation input.

[0015] Furthermore, the chart understanding task described in step S6 includes chart question answering, chart description generation, trend analysis, and other extended tasks based on financial charts. Examples include data extraction, anomaly detection, cross-chart comparison, etc.

[0016] In another aspect, this invention proposes a multimodal financial chart understanding system based on graph neural networks, comprising: a parsing module: parsing the financial chart into structural elements, dividing it into multiple semantic structural units, each semantic structural unit containing structural type, spatial location, and text content; a graph structure module: constructing edges based on spatial and semantic relationships between semantic structural units as nodes, thereby constructing a graph structure; a vectorization encoding module: performing vectorization encoding on the graph structure to generate full graph structure feature encoding; a prompt text module: constructing prompt text for each node and each edge, integrating to generate structured prompt information Graph-Prompt; an injection module: injecting Graph-Prompt into a multimodal large model; and a task execution module: the multimodal large model executes chart understanding tasks based on Graph-Prompt.

[0017] In another aspect, the present invention provides a computer device, the device including a processor and a memory; the memory is used to store computer programs, and the processor is used to execute corresponding computer program code to implement the above-described method for multimodal understanding of financial charts based on graph neural networks.

[0018] In another aspect, the present invention provides a computer-readable storage medium carrying computer program code, which is invoked by a processor to implement the above-described method for multimodal understanding of financial charts based on graph neural networks.

[0019] Compared with the prior art, the beneficial effects and advantages of the present invention are: (1) it realizes explicit structural modeling of financial charts, which significantly improves the interpretability and structural understanding of the model.

[0020] Existing multimodal large-scale models primarily rely on image-level global visual features for reasoning, failing to explicitly utilize structural information such as graph titles, footnotes, legends, and axis ticks. This invention, through structural element detection, graph structure construction, and graph neural network embedding, transforms the explicit structural information within graphs into parsable graph-prompts, explicitly capturing the logical relationships and organizational structure within the graph. This explicit structural modeling significantly improves the interpretability of the model in financial scenarios, providing question-answering results with a structural basis rather than relying on black-box visual features.

[0021] (2) Constructing spatial and semantic rule edges to improve structural perception capabilities.

[0022] This invention generates edge sets through spatial adjacency relationships and semantic prior rules, enabling the model to perceive the form and semantic relationships of graphs. Compared to relying solely on image convolutional structures or pure visual Transformers, this invention uses explicit graph structure constraints to allow the model to acquire the unique organizational patterns of graphs, thereby improving logical judgment, trend analysis, and text association capabilities.

[0023] (3) Graph neural network node embedding enhances the ability to model local and global structures.

[0024] A graph attention network is used to aggregate node features, integrating node category, location information, and textual information to generate structured node embeddings. These embeddings capture the importance of local neighborhoods and global structural information, enabling the model to more effectively utilize the structural priors of graphs in downstream tasks, thereby improving graph understanding performance.

[0025] (4) The Graph-Prompt injection method is flexible and enhances the adaptability of multimodal large models.

[0026] This invention designs various graph-prompt injection strategies (prefix injection, system prompt injection, and graph-text concatenation input), which can be seamlessly integrated with existing multimodal large models without significant modifications to the model structure. It transforms the structural knowledge of graphs into an interpretable natural language form, making it easier for models to understand and utilize the semantic organization within the graph.

[0027] (5) It has strong versatility and can be extended to different types of financial charts and different model architectures, and has good engineering applicability.

[0028] The structure detection, graph structure construction, graph embedding, and prompt generation in this invention all adopt a modular design. No modification to the model structure is required; only the Graph-Prompt needs to be used as input. This gives the method engineering advantages of low cost, high adaptability, and high transferability.

[0029] (6) Improve the performance of downstream tasks and increase production efficiency.

[0030] Through the aforementioned improvements, this invention significantly enhances performance in tasks such as chart question answering, description generation, and trend analysis. Compared to traditional manual analysis or simple image / text feature fusion methods, this invention reduces human intervention, improves task processing speed and accuracy, thereby increasing the productivity and quality of financial data analysis, report interpretation, and other business operations. Attached Figure Description

[0031] Figure 1 is a flowchart of Embodiment 1 of the present invention; Figure 2 is the original financial chart of Embodiment 2 of the present invention.

[0032] Figure 3 is a structural schematic diagram of Embodiment 3 of the present invention. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0034] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be clearly and completely described below with reference to specific embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1: This example provides a detailed description of the multimodal understanding method for financial charts based on graph neural networks, as shown in Figure 1, including: Step 1: Acquisition of chart sample data and detection of structural elements.

[0036] The process involves acquiring a financial chart image and using a text-image detection model to parse its structural elements, dividing the chart into multiple semantic structural units. The text-image detection model can utilize conventional chart parsing models, such as the MinerU model or the paddleOCR-VL model. Each parsed semantic structural unit contains at least three types of information: structural type, spatial location, and text content.

[0037] Structure type : Used to identify the semantic category of the i-th semantic structure unit. The semantic category may include image caption, image subject, legend, axis title, tick, data source or footnote, etc.

[0038] Spatial location The area occupied by an element is represented by normalized coordinates, where: The bounding box represents the i-th semantic structural unit; : The x-coordinate of the top-left corner of the element (normalized); : The ordinate of the top-left corner of the element (normalized); : The x-coordinate of the bottom right corner of the element (normalized); : The y-coordinate of the bottom right corner of the element (normalized); text content : The text content of the i-th semantic structure unit obtained by the OCR model.

[0039] This forms a set of semantic structural units: I represents the number of semantic structural units obtained from the parsing.

[0040] The structural data generated in this step provides the basic input for subsequent graph structure construction.

[0041] In addition to the three types of information mentioned above, semantic structural units can also contain other information, such as rotation angle. Wait, rotation angle Used to represent the rotation angle of oblique semantic structural units (such as oblique text, slanted coordinate axes, etc.).

[0042] Step 2: Construct a graph structure based on spatial and semantic rules.

[0043] The semantic structural unit is used as a node, and the set of semantic structural units is used as a set of nodes.

[0044] Based on the node set obtained in step 1 Constructing a graph structure The set of edges Describe the spatial and semantic relationships between chart elements.

[0045] Step 2.1: Construct spatial connecting edges based on spatial adjacency relationships.

[0046] Utilizing the spatial location of nodes Based on the center point or the relative relationship between the top, bottom, left, and right sides, construct spatial connecting edges, with rules including but not limited to: If node i and node j satisfy: Then construct the edge: .

[0047] in and These are the neighborhood thresholds for the horizontal and vertical directions, respectively. Depending on the chart structure, adaptive or empirically fixed values ​​can be used. For example, statistical methods can be employed to calculate them. and The result set is used to select a percentile as a threshold, such as the minimum distance of 80%, which ensures that most neighboring nodes can be connected to the edge, while avoiding incorrect connections to nodes that are too far away.

[0048] This represents an edge pointing from node i to node j.

[0049] Spatial connection edges mainly reflect the relative relationships of each node (semantic structural unit), such as the vertical positional relationship between the figure title and the main image, the positional proximity between the main image and the footnote, and the adjacency relationship between the legend and the data area; thus providing a spatial organization basis for the graph structure.

[0050] Step 2.2: Construct semantic prior edges based on semantic rules.

[0051] In addition to spatial relationships, this embodiment further constructs semantic prior edges based on graph domain knowledge. For example: (1) If the node structure type is graph caption, then edges are automatically created: ; (2) If the node structure type is legend, then the edges are automatically created: (3) If the node structure type is axis title, then edges are automatically created: ; For coordinate scale.

[0052] The semantic rule template can be summarized as follows: Table 1: Semantic Rules Table (Partial)

[0053] This rule system uses domain knowledge to make the graph structure more consistent with the organizational logic of real-world graphs, thereby improving the effectiveness of subsequent graph structure coding.

[0054] Step 3: Generate graph structure encoding (GraphEmbedding).

[0055] The graph structure constructed from the set of nodes and edges is vectorized and encoded. This embodiment employs a graph neural network, preferably a graph attention network, to enhance the modeling ability of local structures and important connections.

[0056] Step 3.1: Initial feature construction of nodes.

[0057] Initial features of each node It consists of three parts: Where: TypeEmbed is the trainable vector corresponding to the node structure category; for example, a trainable vector (e.g., dimension 128) can be initialized for each structure category, and the model will automatically adjust the vector value according to the task, such as unit classification, relation reasoning, etc. during training. Trainable vectors can better capture the semantic relationships between structure categories.

[0058] BBoxEmbed maps the spatial coordinates of a node to a vector.

[0059] TextEmbed is a vector of node text content after being transformed by a text encoder (such as BERT or LLaMA-Tokenizer).

[0060] Step 3.2: Graph Attention Network Computation Node Structure Embedding.

[0061] (1) Initial characteristics of nodes Perform multi-level GAT aggregation: ;in: Represents a node In the The feature vector of the layer; Represents a node In the The feature vector of the layer; Indicates the first The learnable weight matrix of the layer; Indicates the first Layer nodes For nodes Attention coefficient; Represents a node The set of neighboring nodes; This represents a non-linear activation function, and commonly used functions such as ReLU, ELU, LeakyReLU, and Sigmoid can be used.

[0062] (2) Attention weights are calculated as follows: ; This represents the attention function (attention mechanism). This indicates that softmax normalization is performed on all neighbors j of node i; Represents a vector of learnable attention parameters; Represents the learnable weight matrix; , This represents the features of nodes i and j in the previous layer.

[0063] (3) Obtain the final structural embedding of each node through multi-layer propagation: ; The final structure embedding representation for node i; Let i be the feature representation of node i at the Lth (last) layer.

[0064] This structure embedding simultaneously includes element semantics, layout position, and its structural role in the overall chart.

[0065] Step 3.3: Compress to obtain the structural features of the entire image .

[0066] The final structure of all nodes is embedded together and compressed to obtain the full graph structure features. .

[0067] Step 4: Generate graph-prompt.

[0068] Based on node structure embedding, type information, text content, and edge structure, this embodiment generates structural prompt text that can be used by the language model.

[0069] Step 4.1: Constructing node hint text.

[0070] Using node information and embedding vectors Generate a structural description in the following format, including all nodes and the structural features of the entire graph: Nodes :type= Position = Content = ...; Structural Feature Vector Summary = .

[0071] Step 4.2: Constructing the side prompt text.

[0072] Convert edge set to edge prompt text Edge information node To the node It can contain edge types, such as spatial connection edges, semantic prior edges, etc.

[0073] Step 4.3: Integrate to generate the final Graph-Prompt.

[0074] The Graph-Prompt takes the following form: Ultimately, it generates structured, readable, and parsable prompts, enabling the model to possess prior cognitive abilities regarding chart structures.

[0075] Step 5: Graph-Prompt injection of multimodal large models.

[0076] The generated Graph-Prompt is injected into a multimodal large language model, such as Qwen2.5vl or internvl3. The injection methods include, but are not limited to, one or more of the following: prefix injection, system prompt injection, and graph-text concatenation input.

[0077] For example, the input format is: ;in For the input image, The question is for input.

[0078] Multimodal large language models extract graph structure, relationships, and semantic information from graphs, thereby enhancing graph understanding capabilities.

[0079] Step 6: The model performs a graph understanding task based on Graph-Prompt.

[0080] Multimodal large language models based on Graph-Prompt can perform graph understanding tasks including but not limited to: graph question answering, graph description generation, and trend analysis. By using structure awareness, model misjudgment can be significantly reduced.

[0081] The method described in this embodiment can automatically, uniformly and accurately represent the structural relationships between elements within a chart, and effectively inject this structure into a multimodal large model, enabling the model to truly understand the internal organizational logic of financial charts.

[0082] Example 2: This example provides an application example of the method described in Example 1.

[0083] Figure 2 shows the original image of a financial chart. The application process of the method described in Example 1 includes the following steps: Step 1: The chart content in Figure 2 is "PMI New Export Orders and Export Year-on-Year Comparison". The chart's structural elements are analyzed using a graph-text detection model, resulting in the following node set: Table 2: Node Set

[0084] Each node It contains three types of information: structure type Node type, indicating the semantic role of the graph; spatial location. : Normalized coordinates of nodes in the chart image; text content The text content recognized by OCR; the node set is: .

[0085] Step 2: Construct a graph structure based on spatial and semantic rules.

[0086] Construct a graph structure G=(V,E), where E contains spatial connection edges and semantic prior edges.

[0087] Step 2.1: Constructing spatial connection edges.

[0088] Determine whether elements are adjacent based on their spatial center point relationship.

[0089] For example: If the diagram is... In the main body of the image Directly above, and satisfying Then construct Spatial connection edge.

[0090] Image subject With footnotes , Adjacent in the vertical direction, construct , Spatial connection edge.

[0091] Step 2.2: Construct semantic prior edges using semantic rules.

[0092] Automatically add semantic prior edges based on prior knowledge in the financial chart domain: image_caption→image for chart title and main image; add semantic prior edges for footnotes as supplementary information to the image, image→image_footnote.

[0093] The final result is an edge set E containing spatial connection edges and semantic prior edges.

[0094] Step 3: Graph structure encoding.

[0095] Step 3.1: Initial feature construction of nodes.

[0096] The initial characteristics of each node consist of three parts:

[0097] Step 3.2: GAT generates structural embeddings.

[0098] After L-layer graph attention calculation, the final node structure embedding is obtained: The embedding vector contains semantic, positional, and structural organization information.

[0099] Step 3.2: After compressing and embedding the structure of all nodes, we obtain... : [22.1453,-3.9857,-48.8765,9.2182,-13.1046,-5.7820,-2.9459,20.9271,9.3375,17.5624,3.7189,-7.4217,-30.2648,24.6696,-15.9343,29.0557].

[0100] Step 4: Generate graph structure hints.

[0101] Based on node structure embedding, type information, text content, and edge structure, this embodiment generates structural prompt text that can be used by the language model.

[0102] Step 4.1: Constructing node hint text.

[0103] Using node information and full-image structural feature encoding, a structural description in the following format is generated: Node information: Node 1: Type=image_caption, Position=[0.03,0.019,0.69,0.077], Content=Figure 1: PMI New Export Orders and Exports Year-on-Year; Node 2: Type=image, Position=[0.038,0.097,0.968,0.803], Content=; Node 3: Type=image_footnote, Position=[0.029,0.862,0.421,0.91], Content=Note: Data as of 2 July 2020; Node 4: Type=image_footnote, Position=[0.033,0.927,0.277,0.974], Content=Source: Wind; Structural Feature Vector Summary: [22.1453,-3.9857,-48.8765,9.2182,-13.1046,-5.7820,-2.9459,20.9271,9.3375,17.5624,3.7189,-7.4217,-30.2648,24.6696,-15.9343,29.0557].

[0104] Step 4.2: Constructing the side prompt text.

[0105] Convert edge set to edge prompt text For example, the edge prompt text is in the form of: Node 1 to Node 2, spatial connection edge; ...

[0106] Step 4.3: Integrate to generate the final Graph-Prompt.

[0107] The Graph-Prompt takes the following form: ; to obtain the final result.

[0108] Step 5: Graph-Prompt injects multimodal models.

[0109] Where P is the final result obtained in step 4; Image is the original image shown in Figure 2; UserQuery is the question "Describe the content of the chart in one sentence".

[0110] Step 6: Model execution chart understanding.

[0111] Multimodal large language models can realize graph question answering, graph description and trend analysis, etc.

[0112] For example, the comparison results obtained based on qwen2.5vl are as follows: Original answer result: This figure shows the year-on-year change trend of PMI new export orders and exports of countries B and A (2014-2020).

[0113] The results obtained using this method: This chart shows the changing trends of China's PMI new export orders index and the year-on-year export growth rates of countries B and A from 2014 to 2020, with the data cutoff date being July 2020 and the data source being Wind.

[0114] The test results based on qwen2.5vl are shown in the table below: Table 3: Test Results

[0115] It is evident that, using this method, the multimodal large language model can truly understand the internal organizational logic of financial charts.

[0116] Example 3: This example proposes a multimodal financial chart understanding system based on graph neural networks, including: a parsing module: parsing the financial chart's structural elements and dividing it into multiple semantic structural units, each of which includes structural type, spatial location, and text content; a graph structure module: using semantic structural units as nodes and constructing edges based on spatial and semantic relationships between nodes to build a graph structure; a vectorization encoding module: performing vectorization encoding on the graph structure to generate full graph structure feature encoding; a prompt text module: constructing prompt text for each node and each edge, integrating them to generate a structured prompt information Graph-Prompt; an injection module: injecting Graph-Prompt into a multimodal large model; and a task execution module: the multimodal large model executes chart understanding tasks based on Graph-Prompt.

[0117] In the parsing module, the structure type is used to identify the semantic category of the semantic structure unit, the spatial location is used to represent the area occupied by the semantic structure unit in normalized coordinates, and the text content is used to include the text content in the identified semantic structure unit.

[0118] The graph structure module includes: a spatial connection edge construction unit, which constructs spatial connection edges based on spatial adjacency relationships to reflect the relative relationships of semantic structure units of different structure types; and a semantic prior edge construction unit, which constructs semantic prior edges based on semantic rules to make the graph structure conform to the organizational logic of real financial charts.

[0119] The vectorization encoding module includes: a node initial feature unit, which constructs the initial features of nodes by converting the node's structural type, spatial location, and text content into vectors and concatenating them; a node structure embedding unit, which calculates the structural embedding of nodes through a graph attention network; and a compression unit, which compresses the structural embeddings of all nodes together to obtain the full graph structure feature encoding.

[0120] The prompt text module includes: a node prompt unit: constructing node prompt text, including the structural type, spatial location, and text content of each node; and the encoding of the full graph structural features; an edge prompt unit: constructing edge prompt text, including the edge type; and an integration unit: integrating the node prompt text and edge prompt text to generate structured prompt information Graph-Prompt.

[0121] The Graph-Prompt injection module injects multimodal large models, including one or more of the following: prefix injection, system prompt injection, and graph-text concatenation input.

[0122] The chart understanding tasks described in the task execution module include chart Q&A, chart description generation, trend analysis, and other extended tasks based on financial charts.

[0123] The financial chart multimodal understanding system based on graph neural networks proposed in this embodiment can realize the financial chart multimodal understanding method based on graph neural networks described in Embodiments 1 and 2, and has the same technical effect as Embodiments 1 and 2.

[0124] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for multimodal understanding of financial charts based on graph neural networks, characterized in that, include: S1: Perform structural element analysis on the financial chart, dividing it into multiple semantic structural units. Each semantic structural unit contains structural type, spatial location, and text content. S2: Use the semantic structural units as nodes and construct edges based on the spatial and semantic relationships between nodes to build a graph structure. S3: Perform vectorized encoding on the graph structure to generate full graph structure feature encoding. S4. Construct the prompt text for each node and each edge, and integrate it to generate a structured prompt message Graph-Prompt; S5. Inject Graph-Prompt into the multimodal large model; S6. The multimodal large model performs graph understanding tasks based on Graph-Prompt.

2. The method for multimodal understanding of financial charts based on graph neural networks according to claim 1, characterized in that, In step S1, the structure type is used to identify the semantic category of the semantic structure unit, the spatial location is used to represent the area occupied by the semantic structure unit in normalized coordinates, and the text content is used to include the text content in the identified semantic structure unit.

3. The method for multimodal understanding of financial charts based on graph neural networks according to claim 1, characterized in that, The construction of the graph structure in step S2 includes: S201, constructing spatial connection edges based on spatial adjacency relationships to reflect the relative relationships of semantic structural units of different structural types; S202, constructing semantic prior edges based on semantic rules to make the graph structure conform to the organizational logic of real financial charts.

4. The method for multimodal understanding of financial charts based on graph neural networks according to claim 1, characterized in that, Step S3 includes: S301, constructing initial features of nodes by converting the node's structural type, spatial location, and text content into vectors and concatenating them; S302, calculating the structural embedding of nodes through a graph attention network; S303, compressing all the structural embeddings of nodes together to obtain the full graph structural feature encoding.

5. The method for multimodal understanding of financial charts based on graph neural networks according to claim 1, characterized in that, Step S4 includes: S401, constructing node prompt text, including the structural type, spatial location, and text content of each node; and the encoding of the full graph structural features; S402, constructing edge prompt text, including the edge type; S403, integrating the node prompt text and edge prompt text to generate a structured prompt message Graph-Prompt.

6. The method for multimodal understanding of financial charts based on graph neural networks according to claim 1, characterized in that, In step S5, the Graph-Prompt injection of multimodal large models includes one or more of the following: prefix injection, system prompt injection, and graph-text concatenation input.

7. The method for multimodal understanding of financial charts based on graph neural networks according to claim 1, characterized in that, The chart understanding task described in step S6 includes chart question and answer, chart description generation, trend analysis, and other extended tasks based on financial charts.

8. A multimodal understanding system for financial charts based on graph neural networks, characterized in that, It includes: a parsing module: parsing the structural elements of financial charts and dividing them into multiple semantic structural units, each of which contains structural type, spatial location, and text content; a graph structure module: using semantic structural units as nodes and constructing edges based on spatial and semantic relationships between nodes to build a graph structure; and a vectorization encoding module: performing vectorization encoding on the graph structure to generate full graph structure feature encoding. The tooltip module constructs tooltip text for each node and edge, integrating it to generate a structured tooltip, Graph-Prompt; the injection module injects Graph-Prompt into the multimodal large model; and the task execution module executes graph understanding tasks based on Graph-Prompt in the multimodal large model.

9. A computer device, the device comprising a processor and a memory; the memory being used to store computer programs, characterized in that, The processor is used to execute corresponding computer program code to implement the multimodal understanding method for financial charts based on graph neural networks as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It contains computer program code, which is invoked by a processor to implement the multimodal understanding method for financial charts based on graph neural networks as described in any one of claims 1-7.

Images