Structured chart analysis method and system based on large and small model collaboration, and medium

Abstract

The invention belongs to the technical field of chart data analysis, and particularly relates to a structured chart analysis method and system based on large and small model collaboration and a medium. The method comprises the steps of S1, obtaining a to-be-analyzed chart and a semantic description text corresponding to the to-be-analyzed chart; s2, inputting the to-be-analyzed chart, the semantic description text corresponding to the to-be-analyzed chart and the question text related to the to-be-analyzed chart into a pre-trained chart analysis model to obtain probability distribution of an answer corresponding to the question and a corresponding reasoning process; the chart analysis model is trained according to the following mode that a large model serves as a teacher model, the chart analysis model serves as a student model, and the chart analysis model is trained in a knowledge distillation mode. The technical problems that an existing chart understanding technology is poor in reasoning ability and high in computing resource consumption are solved.

Description

Technical Field

[0001] This invention belongs to the field of chart data analysis technology, specifically relating to a structured chart analysis method, system, and medium based on the collaboration of large and small models. Background Technology

[0002] With the rapid development of large language models and multimodal models, AI (Artificial Intelligence) has made significant progress in visual question answering and image generation. However, some models still have significant shortcomings in the automatic understanding and analysis of structured charts (such as histograms, line charts, pie charts, scatter plots, etc.).

[0003] 1) Insufficient training data and missing semantic structure. As a highly structured form of information visualization, charts typically present key semantics through axis labels, legend titles, and data point trends, requiring a combination of visual and semantic understanding. Compared to the large-scale multimodal training sets widely constructed in the natural image domain (such as face recognition and object detection), publicly available data resources for chart semantic learning are still very limited. Existing chart datasets (such as PlotQA and ChartQA) are significantly insufficient in terms of sample size, chart type, and domain distribution, making it difficult to cover diverse chart contexts such as bar charts, pie charts, and line charts, and failing to meet the training needs of large models in chart structure understanding and language generation tasks. Furthermore, the manual annotation cost for constructing and generating chart-text question-answering data with multiple types, styles, and high semantic matching is high.

[0004] 2) Existing models are based on static image understanding and lack the ability to reason about chart semantics. Multimodal large models generally treat charts as static images, such as analyzing landscapes, objects, and portraits in images. However, they lack structure perception and reasoning ability when faced with questions such as "What trend does this chart reflect?" or "In which year did this data peak?"

[0005] 3) High model training and deployment costs hinder their widespread application to diverse chart analysis needs. While large multimodal models like ChatGPT and Deepseek possess powerful semantic understanding and image generation capabilities, their sheer size, resource consumption, and training costs are extremely high, especially when introducing multimodal inputs such as charts and images, further increasing fine-tuning and inference costs. On the other hand, while lightweight models offer certain deployment advantages, their performance is limited when facing complex tasks such as chart semantic recognition, trend and change inference. They often rely on repeated fine-tuning and training for specific chart domains, exhibiting weak generalization ability and poor transfer adaptability, making it difficult to meet the diverse chart analysis needs of different fields such as finance, healthcare, and education.

[0006] 4) Existing reliable research mainly focuses on high-risk scenarios such as medical diagnosis and autonomous driving, with limited application in chart understanding tasks. Current methods have limited ability to process structural information such as trends, uncertain fluctuations, and local extrema in charts, and lack reliable reasoning mechanisms for semantic consistency. Therefore, in tasks such as chart question answering, trend analysis, and interpretation generation, models may still output logically inconsistent or distorted confidence levels, making it difficult to ensure the reliability of the final answer. Summary of the Invention

[0007] The purpose of this invention is to provide a structured chart analysis method, system, and medium based on the collaboration of large and small models, in order to solve the technical problems of insufficient reasoning ability and high consumption of computing resources in existing chart understanding technologies.

[0008] To address the aforementioned technical problems, the present invention provides a structured chart analysis method based on size-model collaboration, comprising: a structured chart analysis method based on size-model collaboration, the method comprising:

[0009] S1. Obtain the chart to be analyzed and its corresponding semantic description text;

[0010] S2. Input the chart to be analyzed, its corresponding semantic description text, and the question text related to the chart into a pre-trained chart analysis model to obtain the probability distribution of the answer to the question and its corresponding reasoning process.

[0011] The chart analysis model is trained in the following way: using a large model as the teacher model and the chart analysis model as the student model, the chart analysis model is trained through knowledge distillation.

[0012] The beneficial effects of the above technical solution are as follows: When training the chart analysis model, this invention introduces a collaborative optimization mechanism between large and small models, enabling the smaller model (chart analysis model) to inherit the reasoning capabilities of the larger model. Furthermore, it reduces computational overhead through an efficient reasoning process, avoiding the excessive computational resource consumption problem of traditional large models and improving the actual deployment efficiency of the model. Simultaneously, by learning the reasoning path of the large model through knowledge distillation, interpretability and transparency of reasoning results are enhanced, allowing users to understand the reasoning process and its basis, greatly improving reasoning ability. This invention solves the technical problems of insufficient reasoning ability and high computational resource consumption in existing chart understanding technologies.

[0013] Furthermore, the training process of the chart analysis model includes:

[0014] (1) Input the charts in the training data and their corresponding semantic description texts, as well as the questions and answers related to the charts, into the large model to obtain the soft label probability distribution corresponding to the question and the reasoning process corresponding to the soft label probability distribution output by the large model.

[0015] (2) The chart analysis model is trained using the input and output data of the large model in (1).

[0016] Furthermore, the loss function used to train the chart analysis model includes an inference distribution consistency loss function. :

[0017]

[0018] in, This indicates the calculation of KL divergence; The soft label probability distribution output by the teacher model; The predicted probability distribution output by the student model; CE This indicates the calculation of cross-entropy; The predicted labels output by the student model. The true label for the answer.

[0019] Furthermore, in (1), the chart and its corresponding semantic description text, along with the questions and answers related to the chart, are input into the large model in the form of cross-modal fusion features;

[0020] The cross-modal fusion feature is obtained as follows: visual features in the chart are extracted by the image feature extraction module to obtain a visual representation vector; semantic features in the semantic description text corresponding to the chart and the question and answer text related to the chart are extracted by the semantic feature extraction module to obtain a language embedding representation vector; the visual representation vector and the language embedding representation vector are mapped to a unified semantic space to obtain the cross-modal fusion feature.

[0021] Furthermore, the chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to extract visual features from the chart to be analyzed to obtain the corresponding visual representation vector. The semantic feature extraction module is used to extract semantic features from the semantic description text corresponding to the chart to be analyzed and the question text related to the chart to obtain the corresponding language embedding representation vector. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The inference unit is used to perform inference based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and its corresponding inference process.

[0022] Furthermore, the loss function used to train the graph analysis model includes the cross-modal semantic alignment loss function L. al :

[0023]

[0024] Among them, f v This is the image feature extraction module; x1 represents the image; f t x2 represents the semantic feature extraction module; x2 represents the semantic text associated with the chart.

[0025] Furthermore, the method also includes: S3, calculating the confidence level of the answer based on the probability distribution obtained in S2; if the confidence level exceeds a preset confidence level threshold, then directly outputting the final answer based on the probability distribution; if the confidence level does not exceed the confidence level threshold, then outputting ambiguous explanatory text and prompting the user that the output result is uncertain.

[0026] Furthermore, the confidence level Con is obtained in the following manner:

[0027]

[0028]

[0029] Where C represents the number of answer tags; The i-th answer output by the chart analysis model The probability of.

[0030] Furthermore, the training data in (1) is a chart of perception data in an intelligent perception scenario automatically generated by a large model, along with its corresponding text description and questions and answers related to the chart.

[0031] Furthermore, the image feature extraction module is a convolutional neural network; the semantic feature extraction module is a Transformer.

[0032] This invention also provides a technical solution for a structured chart analysis system based on size-model collaboration: a structured chart analysis system based on size-model collaboration, comprising a processor, the processor being used to execute a computer program to implement the steps of the structured chart analysis method based on size-model collaboration as described below:

[0033] S1. Obtain the chart to be analyzed and its corresponding semantic description text;

[0034] S2. Input the chart to be analyzed, its corresponding semantic description text, and the question text related to the chart into a pre-trained chart analysis model to obtain the probability distribution of the answer to the question and its corresponding reasoning process.

[0035] The chart analysis model is trained in the following way: using a large model as the teacher model and the chart analysis model as the student model, the chart analysis model is trained through knowledge distillation.

[0036] The beneficial effects of the above technical solution are as follows: When training the chart analysis model, this invention introduces a collaborative optimization mechanism between large and small models, enabling the smaller model (chart analysis model) to inherit the reasoning capabilities of the larger model. Furthermore, it reduces computational overhead through an efficient reasoning process, avoiding the excessive computational resource consumption problem of traditional large models and improving the actual deployment efficiency of the model. Simultaneously, by learning the reasoning path of the large model through knowledge distillation, interpretability and transparency of reasoning results are enhanced, allowing users to understand the reasoning process and its basis, greatly improving reasoning ability. This invention solves the technical problems of insufficient reasoning ability and high computational resource consumption in existing chart understanding technologies.

[0037] Furthermore, the training process of the chart analysis model includes:

[0038] (1) Input the charts in the training data and their corresponding semantic description texts, as well as the questions and answers related to the charts, into the large model to obtain the soft label probability distribution corresponding to the question and the reasoning process corresponding to the soft label probability distribution output by the large model.

[0039] (2) The chart analysis model is trained using the input and output data of the large model in (1).

[0040] Furthermore, the loss function used to train the chart analysis model includes an inference distribution consistency loss function. :

[0041]

[0042] in, This indicates the calculation of KL divergence; The soft label probability distribution output by the teacher model; The predicted probability distribution output by the student model; This indicates the calculation of cross-entropy; The predicted labels output by the student model. The true label for the answer.

[0043] Furthermore, in (1), the chart and its corresponding semantic description text, along with the questions and answers related to the chart, are input into the large model in the form of cross-modal fusion features;

[0044] The cross-modal fusion feature is obtained as follows: visual features in the chart are extracted by the image feature extraction module to obtain a visual representation vector; semantic features in the semantic description text corresponding to the chart and the question and answer text related to the chart are extracted by the semantic feature extraction module to obtain a language embedding representation vector; the visual representation vector and the language embedding representation vector are mapped to a unified semantic space to obtain the cross-modal fusion feature.

[0045] Furthermore, the chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to extract visual features from the chart to be analyzed to obtain the corresponding visual representation vector. The semantic feature extraction module is used to extract semantic features from the semantic description text corresponding to the chart to be analyzed and the question text related to the chart to obtain the corresponding language embedding representation vector. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The inference unit is used to perform inference based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and its corresponding inference process.

[0046] Furthermore, the loss function used to train the graph analysis model includes the cross-modal semantic alignment loss function L. al :

[0047]

[0048] Among them, f v This is the image feature extraction module; x1 represents the image; f t x2 represents the semantic feature extraction module; x2 represents the semantic text associated with the chart.

[0049] Furthermore, the method also includes: S3, calculating the confidence level of the answer based on the probability distribution obtained in S2; if the confidence level exceeds a preset confidence level threshold, then directly outputting the final answer based on the probability distribution; if the confidence level does not exceed the confidence level threshold, then outputting ambiguous explanatory text and prompting the user that the output result is uncertain.

[0050] Furthermore, the confidence level Con is obtained in the following manner:

[0051]

[0052]

[0053] Where C represents the number of answer tags; The i-th answer output by the chart analysis model The probability of.

[0054] Furthermore, the training data in (1) is a chart of perception data in an intelligent perception scenario automatically generated by a large model, along with its corresponding text description and questions and answers related to the chart.

[0055] Furthermore, the image feature extraction module is a convolutional neural network; the semantic feature extraction module is a Transformer.

[0056] The present invention also provides a technical solution for a computer-readable storage medium: a computer-readable storage medium internally storing a computer program, the computer program being executed by a processor to implement the steps of the structured graph analysis method based on size model collaboration as described below:

[0057] S1. Obtain the chart to be analyzed and its corresponding semantic description text;

[0058] S2. Input the chart to be analyzed, its corresponding semantic description text, and the question text related to the chart into a pre-trained chart analysis model to obtain the probability distribution of the answer to the question and its corresponding reasoning process.

[0059] The chart analysis model is trained in the following way: using a large model as the teacher model and the chart analysis model as the student model, the chart analysis model is trained through knowledge distillation.

[0060] The beneficial effects of the above technical solution are as follows: When training the chart analysis model, this invention introduces a collaborative optimization mechanism between large and small models, enabling the smaller model (chart analysis model) to inherit the reasoning capabilities of the larger model. Furthermore, it reduces computational overhead through an efficient reasoning process, avoiding the excessive computational resource consumption problem of traditional large models and improving the actual deployment efficiency of the model. Simultaneously, by learning the reasoning path of the large model through knowledge distillation, interpretability and transparency of reasoning results are enhanced, allowing users to understand the reasoning process and its basis, greatly improving reasoning ability. This invention solves the technical problems of insufficient reasoning ability and high computational resource consumption in existing chart understanding technologies.

[0061] Furthermore, the training process of the chart analysis model includes:

[0062] (1) Input the charts in the training data and their corresponding semantic description texts, as well as the questions and answers related to the charts, into the large model to obtain the soft label probability distribution corresponding to the question and the reasoning process corresponding to the soft label probability distribution output by the large model.

[0063] (2) The chart analysis model is trained using the input and output data of the large model in (1).

[0064] Furthermore, the loss function used to train the chart analysis model includes an inference distribution consistency loss function. :

[0065]

[0066] in, This indicates the calculation of KL divergence; The soft label probability distribution output by the teacher model; The predicted probability distribution output by the student model; This indicates the calculation of cross-entropy; The predicted labels output by the student model. The true label for the answer.

[0067] Furthermore, in (1), the chart and its corresponding semantic description text, along with the questions and answers related to the chart, are input into the large model in the form of cross-modal fusion features;

[0068] The cross-modal fusion feature is obtained as follows: visual features in the chart are extracted by the image feature extraction module to obtain a visual representation vector; semantic features in the semantic description text corresponding to the chart and the question and answer text related to the chart are extracted by the semantic feature extraction module to obtain a language embedding representation vector; the visual representation vector and the language embedding representation vector are mapped to a unified semantic space to obtain the cross-modal fusion feature.

[0069] Furthermore, the chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to extract visual features from the chart to be analyzed to obtain the corresponding visual representation vector. The semantic feature extraction module is used to extract semantic features from the semantic description text corresponding to the chart to be analyzed and the question text related to the chart to obtain the corresponding language embedding representation vector. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The inference unit is used to perform inference based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and its corresponding inference process.

[0070] Furthermore, the loss function used to train the graph analysis model includes the cross-modal semantic alignment loss function L. al :

[0071]

[0072] Among them, f v This is the image feature extraction module; x1 represents the image; ft x2 represents the semantic feature extraction module; x2 represents the semantic text associated with the chart.

[0073] Furthermore, the method also includes: S3, calculating the confidence level of the answer based on the probability distribution obtained in S2; if the confidence level exceeds a preset confidence level threshold, then directly outputting the final answer based on the probability distribution; if the confidence level does not exceed the confidence level threshold, then outputting ambiguous explanatory text and prompting the user that the output result is uncertain.

[0074] Furthermore, the confidence level Con is obtained in the following manner:

[0075]

[0076]

[0077] Where C represents the number of answer tags; The i-th answer output by the chart analysis model The probability of.

[0078] Furthermore, the training data in (1) is a chart of perception data in an intelligent perception scenario automatically generated by a large model, along with its corresponding text description and questions and answers related to the chart.

[0079] Furthermore, the image feature extraction module is a convolutional neural network; the semantic feature extraction module is a Transformer. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating an implementation method for the structured graph analysis method based on size model collaboration of the present invention.

[0081] Figure 2 This is a schematic diagram of the perception confidence data of a mobile robot during nine time periods in continuous operation, as an embodiment of the structured graph analysis method based on size model collaboration of the present invention. Detailed Implementation

[0082] This invention introduces a collaborative optimization mechanism between large and small models during the training of the chart analysis model. This allows the smaller model (chart analysis model) to inherit the reasoning capabilities of the larger model, while reducing computational overhead through an efficient reasoning process. This avoids the problem of excessive computational resource consumption in traditional large models and improves the actual deployment efficiency of the model. Simultaneously, by learning the reasoning path of the larger model through knowledge distillation, interpretability and transparency of reasoning results are enhanced, enabling users to understand the reasoning process and its basis, thus significantly improving reasoning ability. This invention solves the technical problems of insufficient reasoning ability and high computational resource consumption in existing chart understanding technologies.

[0083] Implementation of a structured graph analysis method based on size model collaboration:

[0084] Currently, chart understanding technology still faces significant challenges. First, the scale of training data is limited and the cost of constructing it is high. As a highly structured visual expression, charts rely on structural elements such as axes, scales, legends, and trend changes for semantics. However, existing public datasets (such as PlotQA and ChartQA) are insufficient in terms of sample size, chart types, and domain coverage, making it difficult to support large-scale models in fully learning the semantics of complex charts. Furthermore, manually constructing high-quality "chart-text-question-answer" training corpora is extremely costly. Second, existing chart understanding models often treat charts as ordinary static images, lacking the ability to deeply semantically reason about structural patterns such as numerical trends, stage changes, and year-on-year relationships. This makes it difficult to meet the requirements of scenarios such as scientific research management, financial analysis, and medical statistics, which demand rigorous trend insight and interpretation capabilities. In addition, while large-scale multimodal models possess strong semantic expression capabilities, they consume enormous training and inference resources, and lightweight models struggle to maintain stable performance in cross-domain analysis, limiting the practical application of the technology. Most importantly, current methods generally lack reliable evaluation, causing models to output incorrect answers when faced with data fluctuations, outliers, or multiple solutions, making it difficult to meet the stringent requirements for results in some fields.

[0085] This implementation provides a structured graph analysis method based on large and small model collaboration. This method performs multimodal graph analysis and cross-modal reasoning through large-model collaborative optimization. The large model assists in generating training samples and inference knowledge, while the small model achieves capability compression and optimization through distillation learning, thereby ensuring efficient deployment and stable operation while maintaining inference accuracy. This implementation uses an intelligent sensing scenario as an example, such as... Figure 1 As shown, the method includes the following steps:

[0086] Step 1: Data generation and construction.

[0087] First, there's the automatic generation of chart data and text descriptions. Using large models like ChatGPT, diverse chart data and their corresponding descriptions are automatically generated. Bar charts, line charts, and other similar formats are all included in the chart category, while axis labels and legend titles are contained within the chart data. Furthermore, question-and-answer pairs for the charts are automatically generated; this multimodal data is used for subsequent training.

[0088] This implementation takes the perception process of an embodied intelligence or robotic system as an example. It utilizes a predefined perception evaluation index system, combined with a large model generation auxiliary mechanism, to generate intelligent perception operation datasets for multiple time periods through large model diagnosis. This dataset covers key indicators such as perception confidence, multi-sensor consistency, target tracking stability, and perception error changes, and is used to characterize the state evolution process of the intelligent perception system under different environmental conditions and operational stages.

[0089] Based on this, various types of charts, such as line charts, bar charts, and pie charts, are automatically generated from the perceived data to describe the trend characteristics of the perceived indicators over time or in the environment. At the same time, the large model automatically generates semantic description text, a set of reasoning questions, and standard answers corresponding to the charts, thereby forming a complete training corpus resource, which provides data support for subsequent collaborative distillation of large and small models and enhancement of reasoning capabilities.

[0090] This step realizes an integrated training corpus construction path for intelligent perception scenarios, encompassing "perceptual data—graphical representation—language semantics." The generated perceptual corpus has broad coverage, is applicable to various operating environments and perception tasks, and requires no manual annotation, significantly reducing the cost of training data construction. Through this mechanism, the model can effectively learn and master core capabilities such as judging trends in perception quality changes, identifying phased degradation or abnormal fluctuations, distinguishing between short-term environmental disturbances and long-term performance changes, and analyzing the relationship between environmental complexity and perception stability.

[0091] like Figure 2 As shown, nine time periods were collected during the continuous operation of a mobile robot. Figure 2 Perceived confidence data (horizontal axis) Figure 2 (vertical axis).

[0092] To enhance the model's training and understanding capabilities, the system also automatically generates a text description and inference task corresponding to the chart using a larger model, as shown in the example below:

[0093] Example of automatic chart description: The system maintains a high level of perception confidence in the initial stage, then shows a significant decline in the third time period, which may be related to increased environmental occlusion or changes in lighting conditions; in the fourth to sixth time periods, the perception confidence gradually recovers and remains stable, indicating that the multimodal perception fusion mechanism is functioning; after a slight fluctuation in the seventh time period, the overall perception quality continues to improve and reaches a high stable level in the ninth time period, showing typical staged degradation-recovery evolution characteristics.

[0094] Automatically generated questions - example answers:

[0095] Question: What is the overall operational trend of this sensing system?

[0096] Answer: Perceived confidence initially decreased, then recovered, and eventually maintained a stable upward trend.

[0097] Question: At which stage is perceived quality lowest? What are the possible reasons?

[0098] Answer: The third time period may be affected by environmental obstruction or changes in perception conditions.

[0099] Step 2: Multimodal feature encoding.

[0100] To address the visual features and textual descriptions of charts, this invention employs a feature encoding module to uniformly process information from each modality. Specifically, a deep convolutional neural network (CNN) is used for image encoding to extract the visual features of the chart image; while a Transformer-based model is used for text encoding to obtain the semantic features of the chart description. After encoding, the features from each modality are integrated through a multimodal fusion mechanism to generate a unified feature representation, providing input for subsequent inference units.

[0101] In this step, the system performs bimodal semantic encoding on the chart images and their accompanying text descriptions and question-and-answer data.

[0102] Specifically, for input charts such as line charts and bar charts, the system uses an image feature extraction module (such as a convolutional neural network) with chart understanding capabilities to extract structural visual features, including axis proportions, numerical points, trend slopes, contrast relationships, outliers, and inflection points. This is combined with OCR recognition of labels and numerical characters within the chart to form a visual representation (i.e., a visual representation vector).

[0103]

[0104] Among them, z v For the final visual representation; f v This is an image feature extraction module used to extract visual features; x1 represents the chart image.

[0105] Correspondingly, the text information associated with the charts is semantically encoded. This text includes the chart's narrative text and questions and standard answers automatically constructed by the system. The narrative text summarizes the trends and phases reflected in the chart, the questions guide the model to focus on key analytical dimensions, and the answers provide clear semantic conclusions about the chart's trends or key points. The system parses this text using a large semantic model (i.e., a semantic feature extraction module) (such as Transformer), focusing on extracting semantic components directly corresponding to the chart's visual content. These include trend-related words (such as decline, recovery, acceleration, stabilization, etc.), phase or time-related entities, and conclusive sentence structures, thereby obtaining the corresponding linguistic embedding representations, i.e.:

[0106]

[0107] Among them, z t For the final language embedding representation; f t x1 represents the semantic model used to extract semantic features; x2 represents the semantic text associated with the chart.

[0108] Then, the visual representation z is obtained through the feature fusion module. v With semantic embedding representation z t They are mapped together to a unified semantic space to obtain cross-modal fusion features.

[0109] It should be noted that the language embedding does not simply correspond to the complete question text itself, but focuses on the semantic components in the chart description text and answer text that are related to visual trends, stage changes and key conclusions; the question text mainly plays a semantic guiding role, and its sentence structure is not used as a direct constraint object for cross-modal alignment.

[0110] To achieve alignment between different modalities at the trend semantic level, this step introduces a cross-modal semantic alignment loss function to implement cross-modal consistency constraints, as described below:

[0111]

[0112] By leveraging cross-modal consistency constraints, the semantic alignment between image trends and textual descriptions, as well as question-and-answer conclusions, is significantly improved. For example, when a line in a chart exhibits a phased decline followed by a gradual recovery, the corresponding textual description and the semantics of "phased decline" and "gradual recovery" contained in the answer will be mapped to the same semantic region, thereby enhancing the consistency and discriminative ability of cross-modal representations.

[0113] Step 3: Collaborative learning and distillation of large and small models.

[0114] The core innovation of this invention lies in the collaborative optimization mechanism between large and small models. The large model acts as the teacher model, generating soft labels, attention distributions, and inference chains to guide the training of the small model. Through distillation learning, the small model inherits the knowledge of the large model, while cross-modal consistency constraints ensure the consistency between graph / image features and text descriptions within the high-dimensional embedding space.

[0115] This step, based on the cross-modal unified semantic representation in step two, introduces a large-scale model collaborative distillation mechanism to achieve effective transfer of high-performance reasoning capabilities to lightweight models.

[0116] Specifically, the large model and the small model share the cross-modal fusion features generated in step two as input, thereby ensuring that they can reason and learn in the same semantic space.

[0117] Among them, the large model, acting as the teacher model, performs in-depth analysis and high-order reasoning on the graph structure information and text semantic information represented by the fused representation, and outputs the soft label distribution. The soft-label distribution describes the uncertainty and preference distribution of the model's inference results in the form of continuous probability. It covers multi-dimensional semantic signals such as trend judgment, difference analysis, growth rate comparison, stage change characteristics, and future development trend prediction, rather than just giving a single discrete conclusion.

[0118] Simultaneously, the teacher model further generates explanatory reasoning chains (i.e., reasoning processes) corresponding to the aforementioned soft labels. These chains, presented in natural language or structured statements, explicitly describe the logical mapping relationship between visual features of charts and inference conclusions. For example, the teacher model can output the explanatory statement: "The growth rate of the indicator accelerated significantly between 2022 and 2023, which may be related to increased external resource investment and improved system operating conditions." This explanatory reasoning chain is used to construct a transferable cross-modal logical paradigm, clearly indicating the key stages, inflection points of change, and potential causal relationships that the model focuses on.

[0119] As a smaller model of the student model (i.e., the graph analysis model), it uses the same fusion representation as the teacher model as input during the training phase. By fitting the soft label distribution output by the teacher model and its corresponding inference paradigm (i.e., the inference process), it learns the teacher model's strategies in cross-modal semantic alignment, trend structure recognition, and inference decision-making, and then outputs the predicted distribution of the student model. And the corresponding reasoning process.

[0120] In this process, the reasoning focus and semantic attention patterns reflected in the teacher model at important trend nodes and inflection points are implicitly transmitted to the student model, enabling it to gradually form a reasoning path and decision preferences consistent with the teacher model.

[0121] To achieve the aforementioned collaborative distillation process, the optimization objective of the small model consists of a distribution matching term and a supervision constraint term, and its loss function L... kd Defined as:

[0122]

[0123] in, This indicates the calculation of KL divergence. The soft label probability distribution output by the teacher model. The predicted probability distribution output by the student model; L kd The first term aligns the prediction distribution of the student model with the inference distribution of the teacher model by constraining the Kullback-Leibler divergence, so as to inherit its higher-order reasoning ability and uncertainty modeling features. This indicates the calculation of cross-entropy. The predicted labels output by the student model. For the actual labels in the answer text; L kd The second term, the over-cross-entropy loss, ensures that the student model's output remains consistent with the reference answer, thus preventing semantic drift or inference bias caused by relying solely on soft supervision.

[0124] Taking the example of "a line graph showing the change in depth estimation error of a certain embodied intelligent system at different observation distances," the teacher model identifies a significant increase in the slope of the perception error growth at longer distances and generates an explanatory inference: "The accelerated growth of error in this stage may be related to sensor resolution limitations and a decrease in signal-to-noise ratio." During the distillation training process, the student model learns this type of trend and explanatory logic, enabling it not only to determine the direction of error change but also to further analyze the rate of error growth and its correlation with potential perception conditions.

[0125] In summary, based on the descriptions in steps two and three, the structure of the chart analysis model can be briefly explained as follows: The chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to obtain the corresponding visual representation vector based on the chart to be analyzed. The semantic feature extraction module is used to obtain the corresponding language embedding representation vector based on the semantic description text corresponding to the chart to be analyzed and the question related to the chart. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The inference unit (i.e., the inference module) is used to perform inference based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and its corresponding inference process.

[0126] Step 4: Cross-modal consistent reasoning constraint mechanism.

[0127] This implementation constrains visual and linguistic encoding through cross-modal consistency loss, enabling the model to achieve unified logical judgments across the multimodal evidence chain when determining trend direction, growth rate, and change structure, thereby avoiding erroneous inferences caused by single-modal bias. To achieve this goal, this invention employs the following consistency loss function L. con Optimize:

[0128]

[0129] Where λ1 and λ2 are both weighting coefficients. L con The first term is used to constrain the consistency between visual and text embeddings, L con The second constraint ensures consistency in the distribution of inferences, such as trend judgments, guaranteeing a strict correspondence between the final question and answer and the reference answer. During actual training, the system simultaneously receives charts and their accompanying text descriptions as common input. When the visual and linguistic branches produce differing judgments on the same trend event, the consistency constraint will proactively adjust the learning direction.

[0130] For example, when a line graph of a perception index of an embodied intelligence system shows a trend of "perceptual quality continuously improving over the last four time periods, with a significant increase in the rate of improvement at a certain stage," and the corresponding text description indicates that "the improvement in perception performance at this stage is significantly higher than in the previous stage," if the visual branch correctly captures the "accelerated improvement" trend, while the language branch incorrectly judges "the change tends to stabilize" or "the improvement is not obvious," the consistency loss term will increase accordingly. This prompts the model to strengthen its focus on key change intervals and conclusive semantics in subsequent iterations, gradually bringing the visual trend representation and language inference results closer together. Through the aforementioned cross-modal consistency constraint mechanism, the system can establish a stable consistency mapping relationship between visual trend recognition and language semantic inference, thereby improving the reliability and robustness of multimodal perception inference results and providing a unified and interpretable semantic foundation for subsequent perception state analysis and interpretation in complex environments.

[0131] Step 5: Credible Reasoning and Result Output.

[0132] After training, the inference unit performs tasks such as trend type identification, key year detection, and extreme point extraction based on fused features. Subsequently, the natural language generation module outputs structured explanatory text, providing trend conclusions, key evidence, and analysis of the reasons for changes. Simultaneously, this invention introduces a credibility assessment mechanism, calculating information entropy from the probability distribution of the output trend categories to obtain a trend judgment confidence index, thereby improving the transparency and reliability of the model's output results.

[0133] When a model is quite confident in its judgment of a certain trend, its corresponding probability is significantly higher than that of other candidate results, indicating that a relatively consistent judgment has been formed within the model. Conversely, when the probabilities of multiple candidate results are close, it indicates that the model lacks sufficient confidence in the current conclusion.

[0134] After model training is complete, inference can be performed using the trained small model, entering the credible inference stage. First, trend type identification, key year detection, and significant inflection point extraction are performed. Then, the natural language generation module generates structured explanatory text, which includes trend conclusions, key evidence explanations, and potential causal analysis.

[0135] To enhance the transparency and reliability of the analysis results, this invention introduces a credibility assessment mechanism, utilizing the probability distribution of output trend categories. Calculate information entropy As an indicator of uncertainty, it is defined as follows:

[0136]

[0137] Where C represents the number of trend categories (such as rising, falling, stable, fluctuating, etc.) (i.e., labels). The lower the information entropy H, the more concentrated the model's decisions are, and the higher its credibility; conversely, the higher the entropy, the higher the uncertainty of the model, and the lower its credibility.

[0138] Furthermore, this embodiment defines a credibility index based on the aforementioned information entropy to quantify the credibility index. The credibility index Con is defined as follows:

[0139]

[0140] After the credibility assessment is completed, the system executes an automatic strategy based on the confidence level: when the confidence level is higher than the preset threshold, the conclusion is output directly; when the confidence level is lower, conservative suggestions and review prompts are output, or the user is prompted to provide more information, in order to avoid wrong decisions and improve the reliability of the system in real-world scenarios.

[0141] After obtaining the confidence index, adaptive decision-making is performed: when the confidence level is higher than a preset threshold τ, the model directly outputs the trend result and explanatory text; when the confidence level is lower than the threshold, a conservative output strategy is automatically triggered, including providing ambiguous or interval trend explanations (such as ambiguous explanatory text like "regional increase"), prompting the user that the output result has uncertainty, giving a review prompt, or suggesting supplementing more relevant input information before further analysis. This mechanism enables reliable inference in risk-sensitive scenarios, avoids misleading judgments due to low-confidence output, and thus improves the system's credibility and robustness in chart analysis.

[0142] The purpose of this mechanism is to allow the model to "speak according to its own level of confidence," avoiding seemingly certain but actually unreliable conclusions when there is insufficient evidence or unstable judgment. By matching the output method with the probability distribution state inside the model, this invention can effectively reduce the risk of misleading judgments, and is particularly suitable for graph analysis and decision support scenarios with high reliability requirements, thereby improving the overall credibility and robustness of the system.

[0143] Implementation of a structured graph analysis system based on size model collaboration:

[0144] A structured graph analysis system based on size-model collaboration includes a processor, which executes a computer program to implement the steps of the structured graph analysis method based on size-model collaboration as described above. The specific structured graph analysis method based on size-model collaboration has been described in sufficient detail above and will not be repeated here. The structured graph analysis method based on size-model collaboration includes the following steps:

[0145] S1. Obtain the chart to be analyzed and its corresponding semantic description text.

[0146] S2. Input the chart to be analyzed, its corresponding semantic description text, and the question text related to the chart into a pre-trained chart analysis model to obtain the probability distribution of the answer to the question and its corresponding reasoning process.

[0147] The chart analysis model is trained in the following way: using a large model as the teacher model and the chart analysis model as the student model, the chart analysis model is trained through knowledge distillation.

[0148] Furthermore, the training process of the chart analysis model includes: (1) inputting the charts and their corresponding semantic description texts and the questions and answers related to the charts in the training data into the large model to obtain the soft label probability distribution corresponding to the question and the reasoning process corresponding to the soft label probability distribution output by the large model; (2) training the chart analysis model with the input data and output data of the large model in (1).

[0149] Furthermore, the loss function used to train the chart analysis model includes an inference distribution consistency loss function. :

[0150]

[0151] in, This indicates the calculation of KL divergence; The soft label probability distribution output by the teacher model; The predicted probability distribution output by the student model; This indicates the calculation of cross-entropy; The predicted labels output by the student model. The true label for the answer.

[0152] Further, in (1), the chart and its corresponding semantic description text, as well as the questions and answers related to the chart, are input into the large model in the form of cross-modal fusion features; the cross-modal fusion features are obtained as follows: visual features in the chart are extracted by the image feature extraction module to obtain a visual representation vector; semantic features in the semantic description text corresponding to the chart and the questions and answers related to the chart are extracted by the semantic feature extraction module to obtain a language embedding representation vector; the visual representation vector and the language embedding representation vector are mapped to a unified semantic space to obtain the cross-modal fusion features.

[0153] Furthermore, the chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to extract visual features from the chart to be analyzed to obtain the corresponding visual representation vector. The semantic feature extraction module is used to extract semantic features from the semantic description text corresponding to the chart to be analyzed and the question text related to the chart to obtain the corresponding language embedding representation vector. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The inference unit is used to perform inference based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and its corresponding inference process.

[0154] Furthermore, the loss function used to train the graph analysis model includes the cross-modal semantic alignment loss function L. al :

[0155]

[0156] Among them, f v This is the image feature extraction module; x1 represents the image; f t x2 represents the semantic feature extraction module; x2 represents the semantic text associated with the chart.

[0157] Furthermore, the method also includes: S3, calculating the confidence level of the answer based on the probability distribution obtained in S2; if the confidence level exceeds a preset confidence level threshold, then directly outputting the final answer based on the probability distribution; if the confidence level does not exceed the confidence level threshold, then outputting ambiguous explanatory text and prompting the user that the output result is uncertain.

[0158] Furthermore, the confidence level Con is obtained in the following manner:

[0159]

[0160]

[0161] Where C represents the number of answer tags; The i-th answer output by the chart analysis model The probability of.

[0162] Furthermore, the training data in (1) is a chart of perception data in an intelligent perception scenario automatically generated by a large model, along with its corresponding text description and questions and answers related to the chart.

[0163] Furthermore, the image feature extraction module is a convolutional neural network; the semantic feature extraction module is a Transformer.

[0164] Specifically, the processor can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can also be a processor supporting the Advanced Reduced Instruction Set Machine (ARM) architecture.

[0165] Implementation of computer-readable storage media:

[0166] A computer-readable storage medium stores a computer program internally, the computer program being processed to perform the steps of the size-model-based collaborative structured graph analysis method described below. The specific size-model-based collaborative structured graph analysis method has been described sufficiently in detail in the above-described embodiments and will not be repeated here. The size-model-based collaborative structured graph analysis method includes the following steps:

[0167] S1. Obtain the chart to be analyzed and its corresponding semantic description text.

[0168] S2. Input the chart to be analyzed, its corresponding semantic description text, and the question text related to the chart into a pre-trained chart analysis model to obtain the probability distribution of the answer to the question and its corresponding reasoning process.

[0169] The chart analysis model is trained in the following way: using a large model as the teacher model and the chart analysis model as the student model, the chart analysis model is trained through knowledge distillation.

[0170] Furthermore, the training process of the chart analysis model includes: (1) inputting the charts and their corresponding semantic description texts and the questions and answers related to the charts in the training data into the large model to obtain the soft label probability distribution corresponding to the question and the reasoning process corresponding to the soft label probability distribution output by the large model; (2) training the chart analysis model with the input data and output data of the large model in (1).

[0171] Furthermore, the loss function used to train the chart analysis model includes an inference distribution consistency loss function. :

[0172]

[0173] in, This indicates the calculation of KL divergence; The soft label probability distribution output by the teacher model; The predicted probability distribution output by the student model; This indicates the calculation of cross-entropy; The predicted labels output by the student model. The true label for the answer.

[0174] Further, in (1), the chart and its corresponding semantic description text, as well as the questions and answers related to the chart, are input into the large model in the form of cross-modal fusion features; the cross-modal fusion features are obtained as follows: visual features in the chart are extracted by the image feature extraction module to obtain a visual representation vector; semantic features in the semantic description text corresponding to the chart and the questions and answers related to the chart are extracted by the semantic feature extraction module to obtain a language embedding representation vector; the visual representation vector and the language embedding representation vector are mapped to a unified semantic space to obtain the cross-modal fusion features.

[0175] Furthermore, the chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to extract visual features from the chart to be analyzed to obtain the corresponding visual representation vector. The semantic feature extraction module is used to extract semantic features from the semantic description text corresponding to the chart to be analyzed and the question text related to the chart to obtain the corresponding language embedding representation vector. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The inference unit is used to perform inference based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and its corresponding inference process.

[0176] Furthermore, the loss function used to train the graph analysis model includes the cross-modal semantic alignment loss function L. al :

[0177]

[0178] Among them, f v This is the image feature extraction module; x1 represents the image; f t x2 represents the semantic feature extraction module; x2 represents the semantic text associated with the chart.

[0179] Furthermore, the method also includes: S3, calculating the confidence level of the answer based on the probability distribution obtained in S2; if the confidence level exceeds a preset confidence level threshold, then directly outputting the final answer based on the probability distribution; if the confidence level does not exceed the confidence level threshold, then outputting ambiguous explanatory text and prompting the user that the output result is uncertain.

[0180] Furthermore, the confidence level Con is obtained in the following manner:

[0181]

[0182]

[0183] Where C represents the number of answer tags; The i-th answer output by the chart analysis model The probability of.

[0184] Furthermore, the training data in (1) is a chart of perception data in an intelligent perception scenario automatically generated by a large model, along with its corresponding text description and questions and answers related to the chart.

[0185] Furthermore, the image feature extraction module is a convolutional neural network; the semantic feature extraction module is a Transformer.

[0186] Specifically, the computer-readable storage medium can be volatile memory or non-volatile memory, or may include both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. For example, Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), SynchLink DRAM (SLDRAM), or Direct Rambus RAM (DRRAM).

[0187] This invention has the following characteristics:

[0188] This invention effectively addresses the challenges of existing chart dataset construction, insufficient reasoning capabilities, high computational resource consumption, and weak interpretability by introducing a large-scale model collaborative optimization mechanism, automatic chart generation, multimodal feature fusion, and cross-modal inference technology. First, by automatically generating charts and corresponding question-answer pairs using a large model, the manual annotation problem in dataset construction is solved, achieving low-cost, high-quality training data generation. In the inference process, multimodal feature fusion and consistency constraints are implemented in parallel to ensure semantic consistency between chart images and text, enabling the model to better understand the structural information and trend changes within the charts. The large-scale model collaborative optimization framework utilizes distillation learning, allowing the small model to inherit the reasoning capabilities of the large model, while reducing computational overhead through an efficient inference process, avoiding the excessive computational resource consumption of traditional large models and improving the actual deployment efficiency of the model. The cross-modal inference module integrates chart visual features with text descriptions to achieve complex reasoning tasks such as trend analysis and extreme value judgment, generating natural language answers. Simultaneously, the interpretability of the inference path enhances the transparency of the inference results, allowing users to understand the reasoning process and its basis.

[0189] From a social perspective, this invention significantly reduces the manual cost of dataset construction by automatically generating chart data and question-and-answer samples, thus promoting intelligent data analysis in industries such as finance, healthcare, and education. The natural language generation module enables users, even those lacking specialized knowledge, to achieve a deeper understanding of the information behind the charts.

[0190] From an economic perspective, this invention effectively reduces enterprises' technological investment and labor costs in the field of chart analysis by automating the chart generation and data annotation process. Enterprises can achieve significant savings in time and resources through this invention. Furthermore, in large-scale data processing scenarios, the collaborative optimization mechanism between large and small models ensures inference accuracy and efficiency while reducing reliance on high-performance computing resources, providing a cost-effective solution for various industries.

[0191] From a technical perspective, this invention significantly enhances chart understanding and cross-modal reasoning capabilities. Through multimodal feature fusion and consistency constraints, the model achieves a more accurate understanding of chart data structure, trend changes, and data reasoning, thereby improving reasoning accuracy. Under the collaborative optimization mechanism of large and small models, the smaller model inherits the reasoning capabilities of the larger model while maintaining lower computational overhead, thus improving reasoning efficiency.

[0192] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still make modifications to the technical solutions described in the foregoing embodiments without creative effort, or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A structured graph analysis method based on size model collaboration, characterized in that, The method includes: S1. Obtain the chart to be analyzed and its corresponding semantic description text; S2. Input the chart to be analyzed, its corresponding semantic description text, and the question text related to the chart into a pre-trained chart analysis model to obtain the probability distribution of the answer to the question and its corresponding reasoning process. The chart analysis model is trained in the following way: using a large model as the teacher model and the chart analysis model as the student model, the chart analysis model is trained through knowledge distillation.

2. The structured graph analysis method based on size model collaboration according to claim 1, characterized in that, The training process of the chart analysis model includes: (1) Input the charts in the training data and their corresponding semantic description texts, as well as the questions and answers related to the charts, into the large model to obtain the soft label probability distribution corresponding to the question and the reasoning process corresponding to the soft label probability distribution output by the large model. (2) The chart analysis model is trained using the input and output data of the large model in (1).

3. The structured graph analysis method based on size model collaboration according to claim 2, characterized in that, The loss function used to train the chart analysis model includes the inference distribution consistency loss function. : ； in, This indicates the calculation of KL divergence; The soft label probability distribution output by the teacher model; The predicted probability distribution output by the student model; This indicates the calculation of cross-entropy; The predicted labels output by the student model. The true label for the answer.

4. The structured graph analysis method based on size model collaboration according to claim 2, characterized in that, In (1), the chart and its corresponding semantic description text, as well as the questions and answers related to the chart, are input into the large model in the form of cross-modal fusion features; The cross-modal fusion feature is obtained as follows: visual features in the chart are extracted by the image feature extraction module to obtain a visual representation vector; semantic features in the semantic description text corresponding to the chart and the question and answer text related to the chart are extracted by the semantic feature extraction module to obtain a language embedding representation vector; the visual representation vector and the language embedding representation vector are mapped to a unified semantic space to obtain the cross-modal fusion feature.

5. The structured graph analysis method based on size model collaboration according to claim 1, characterized in that, The chart analysis model includes a feature encoding unit and an inference unit. The feature encoding unit includes an image feature extraction module, a semantic feature extraction module, and a feature fusion module. The image feature extraction module is used to extract visual features from the chart to be analyzed to obtain the corresponding visual representation vector. The semantic feature extraction module is used to extract semantic features from the semantic description text corresponding to the chart to be analyzed and the question text related to the chart to obtain the corresponding language embedding representation vector. The feature fusion module maps the visual representation vector and the language embedding representation vector to a unified semantic space to obtain cross-modal fusion features. The reasoning unit is used to perform reasoning based on the cross-modal fusion features to obtain the probability distribution of the answer corresponding to the question and the corresponding reasoning process.

6. The structured graph analysis method based on size model collaboration according to claim 4 or 5, characterized in that, The loss function to which the graph analysis model is adapted comprises a cross-modal semantic alignment loss function L al : ； wherein f v is an image feature extraction module; x1 represents an image; f t is a semantic feature extraction module; x2 represents a semantic text associated with the graph.

7. The structured graph analysis method based on size model collaboration according to claim 1, characterized in that, The method further includes: S3, calculating the confidence level of the answer based on the probability distribution obtained in S2; if the confidence level exceeds a preset confidence level threshold, the final answer is directly output based on the probability distribution; if the confidence level does not exceed the confidence level threshold, an ambiguous explanation text is output, and the user is prompted that the output result is uncertain.

8. The structured graph analysis method based on size model collaboration according to claim 7, characterized in that, The confidence level Con is obtained in the following way: ； ； Where C represents the number of answer tags; The i-th answer output by the chart analysis model The probability of.

9. A structured graph analysis system based on size-model collaboration, comprising a processor, characterized in that, The processor is used to execute a computer program to implement the steps of the structured graph analysis method based on size model collaboration as described in any one of claims 1 to 8.

10. A computer-readable storage medium, wherein a computer program is stored internally, characterized in that, The computer program is executed by a processor to implement the steps of the structured graph analysis method based on size model collaboration as described in any one of claims 1 to 8.

Images