A Multidimensional Chart-Based Summary and Evaluation Method and System Aligned with Human Experts

By combining expert structures and HSIC-optimized multi-dimensional chart summarization evaluation methods, this approach addresses the issues of high-cost annotation and insufficient fine-grained evaluation in existing technologies. It achieves efficient and robust chart-to-text evaluation, enhancing the professionalism and interpretability of the evaluation.

CN121365151BActive Publication Date: 2026-11-17PEKING UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511282682.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-09-09
Publication Date
2026-11-17
Estimated Expiration
2045-09-09

AI Technical Summary

Technical Problem

Existing chart-to-text evaluation methods rely on costly reference text annotations, making it difficult to identify factual errors and logical flaws in the generated text. Furthermore, they lack fine-grained evaluation and cannot effectively align with human cognitive processes, limiting their application scenarios with high accuracy and security requirements.

Method used

A multi-dimensional chart-based evaluation method with a hybrid expert structure is adopted, including expert models with four dimensions: fidelity, comprehensiveness, simplicity, logic, and analysis. Combined with CLIP encoder and scoring head, the scoring module achieves independence and stability through the HSIC optimization mechanism, and a lightweight architecture supports efficient deployment.

Benefits of technology

It achieves fine-grained, interpretable graphical summary evaluation, improves the professionalism and interpretability of the evaluation, reduces deployment costs, enhances the robustness and stability of the model, and supports efficient inference on a single 24GB GPU.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121365151B_ABST
    Figure CN121365151B_ABST
Patent Text Reader

Abstract

This invention discloses a multi-dimensional chart summary evaluation method and system aligned with human experts, belonging to the field of artificial intelligence technology. The method includes: acquiring a target chart image, chart context text, and chart summary; feeding the image, chart context text, and chart summary into expert models for each dimension to obtain a first score result for that dimension; feeding the image, chart context text, and chart summary into a shared expert model to obtain a second score result for that dimension; and obtaining a score result for the chart summary based on the first and second score results. This invention enables fine-grained, interpretable, and efficient evaluation of chart summaries.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a multi-dimensional chart summary and evaluation method and system aligned with human experts. Background Technology

[0002] Charts, as an important form of information representation, are widely used in various fields such as academic papers and business analysis. The Figure-to-Text (F2T) task aims to transform the key information contained in an image into a text description, thereby improving the structure and comprehensibility of the information. This technology not only helps achieve more efficient information retrieval in data-intensive fields but also provides a more user-friendly way for visually impaired people to access content. However, due to the semantic compression and ambiguity inherent in charts, the generated text often suffers from inaccurate facts, incomplete content coverage, and weak logical reasoning. Therefore, there is an urgent need to introduce automated quality assessment methods. Effective quality assessment of Figure-to-Text conversion is crucial for the development of this task. Currently, evaluation methods for Figure-to-Text tasks can be mainly divided into two categories: reference-based methods and reference-free methods.

[0003] Evaluation methods based on reference text (such as BLEU, ROUGE, CIDEr, and BERTScore) are commonly used for graph-to-text tasks. These methods measure generation quality by comparing the semantic similarity between the generated text and the reference text. These methods are simple to implement, applicable to various text generation scenarios, and offer good reproducibility and efficiency. However, they heavily rely on high-quality reference text, which typically requires professional annotation, leading to high annotation costs. Furthermore, these methods primarily rely on shallow semantic similarity calculations, making it difficult to effectively identify factual errors, logical flaws, and incomplete reasoning in the generated text.

[0004] With the development of multimodal large language models (MLLMs), no-reference evaluation methods for graph-to-text tasks have been significantly enhanced. These methods typically rely on pre-trained models, generating evaluation scores or text explanations through customized prompts (instructions). Current research attempts to incorporate contextual scoring templates to enhance evaluation robustness and employ lightweight multimodal models to reduce deployment costs. However, these methods remain sensitive to input instructions and sample distribution, making them difficult to apply stably in large-scale evaluation tasks. Furthermore, lightweight models have limited performance, failing to meet accuracy requirements, while high-performance large closed-source models suffer from high invocation costs and version instability.

[0005] In addition to the challenges mentioned above, most existing evaluation methods only provide overall scores at the sample level, lacking fine-grained evaluations on key dimensions such as content quality, logical structure, conciseness, and analytical depth, thus limiting the interpretability of the evaluation results. Furthermore, these methods still differ significantly from expert human evaluation, failing to effectively align the task context with human cognitive processes, limiting their application in generative modeling scenarios with high requirements for accuracy, security, and controllability. Therefore, how to construct an efficient, robust, fine-grained, and human-cognitive automatic evaluation method has become a key challenge in current graph-to-text evaluation research. Summary of the Invention

[0006] To address the aforementioned challenges, this invention proposes a multi-dimensional chart summary evaluation method and system aligned with human expert evaluation criteria. This method aligns with human expert judgment standards, enabling fine-grained, interpretable, and efficient evaluation of chart summaries.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following:

[0008] A multi-dimensional chart-based summary evaluation method aligned with human experts, the method comprising:

[0009] Retrieve the target chart image, chart context text, and chart summary;

[0010] The image, chart context text, and chart summary are fed into the expert models for each dimension to obtain the first score result for that dimension. These dimensions include fidelity, comprehensiveness, simplicity, logic, and analysis. Each dimension's expert model includes a CLIP encoder. spe An aligned linear projection layer and a scoring head

[0011] The image, chart context text, and chart summary are fed into a shared expert model to obtain a second score result for the corresponding dimension; the shared expert model includes: a CLIP encoder. share and five rating heads

[0012] Based on the first and second rating results, the rating results summarized in the chart are obtained.

[0013] Furthermore, the contextual text of the images and charts, as well as the chart summaries, are fed into the expert models for each dimension to obtain the first score results for the corresponding dimension, including:

[0014] Based on CLIP encoder E spe The image, chart context text, and chart summary are encoded into feature vectors v, respectively. img , Feature vector vtext and eigenvector v summary ;

[0015] The feature vector v img With feature vector v text Perform concatenation to obtain the feature vector v it ;

[0016] Using an aligned linear projection layer on this feature vector v it Perform projection mapping to obtain the feature vector v summary The corresponding eigenvector z it ;

[0017] Based on the rating head Calculate the eigenvector v summary With eigenvector z it The similarity between them is used to obtain the first score result for the corresponding dimension.

[0018] Furthermore, the process of training the fidelity dimension expert model includes:

[0019] Construct a fidelity dimension dataset, which includes: a chart sample, a chart summary of the chart sample, and the true fidelity dimension score y of the chart summary. i ;

[0020] The image of the chart sample, the chart context text, and the chart summary are fed into the fidelity dimension expert model to obtain a predicted score.

[0021] Constructing the minimum mean square error function N represents the number of graph samples contained in a training batch;

[0022] Constructing the expert alignment loss function in, This represents all the true scores y in this training batch. i The average value, This represents all predicted scores for this training batch. The average value;

[0023] Based on the minimum mean square error function Align loss function with experts Obtain the trained fidelity dimension expert model.

[0024] Furthermore, the image, chart context text, and chart summary are fed into the shared expert model to obtain the second scoring results for the corresponding dimensions, including:

[0025] Based on CLIP encoder E shareThe image, chart context text, and chart summary are encoded into feature vectors v, respectively. img , Feature vector v text and eigenvector v summary ;

[0026] The feature vector v img , Feature vector v text and eigenvector v summary Perform concatenation to obtain the feature vector v its ;

[0027] The feature vector v its Enter each rating header separately The second score result for the corresponding dimension is obtained.

[0028] Furthermore, the second scoring result of the corresponding dimension in, These represent the rating headers respectively. The first layer weight matrix and the second layer weight matrix, These represent the rating headers respectively. The first-level bias matrix and the second-level bias matrix.

[0029] Furthermore, the process of training a shared expert model includes:

[0030] Construct fidelity dimension datasets, comprehensive dimension datasets, concise dimension datasets, logical dimension datasets, and analytical dimension datasets;

[0031] Training CLIP encoder E share Then, freeze the CLIP encoder. share Parameters;

[0032] Based on the fidelity dimension dataset, comprehensiveness dimension dataset, conciseness dimension dataset, logicality dimension dataset, and analytical dimension dataset, adjust the corresponding scoring heads respectively. The parameters in.

[0033] Furthermore, the corresponding scoring heads were adjusted based on the fidelity dimension dataset, comprehensiveness dimension dataset, conciseness dimension dataset, logicality dimension dataset, and analytical dimension dataset, respectively. The parameters include:

[0034] Constructing the HSIC loss function Among them, W1 (i) W1 represents the first-level weight matrix of the i-th dimension rating head. (j) This represents the first-level weight matrix of the rating head for the j-th dimension. n represents the first-layer weight matrix W1(i) With the first layer weight matrix W1 (j) The total number of rows, p, q, r, s are the first-layer weight matrix W1 (i) and the first layer weight matrix W1 (j) row index, K pq L pq K pr L qr K pr L qs The Gram matrix is ​​the kernel of the radial basis functions;

[0035] Based on HSIC loss function For the first layer weight matrix W1 (i) and the first layer weight matrix W1 (j) Optimize.

[0036] Furthermore, based on the first and second rating results, the rating results summarized in the chart are obtained, including:

[0037] The first and second scores for each dimension are weighted to obtain the score for that dimension.

[0038] The scores for each dimension are weighted to obtain the overall score summary in this chart.

[0039] A multi-dimensional chart-based summary and evaluation system aligned with human experts, the system comprising:

[0040] The data acquisition module is used to obtain the chart summary of the target chart and extract the image and chart context text of the target chart;

[0041] The first scoring module is used to input the contextual text of images and charts, as well as the chart summaries, into the expert models for each dimension to obtain the first scoring result for that dimension. These dimensions include fidelity, comprehensiveness, simplicity, logic, and analysis. Each dimension's expert model includes a CLIP encoder. spe An aligned linear projection layer and a scoring head

[0042] The second scoring module is used to feed the image, chart context text, and chart summary into the shared expert model to obtain the second scoring result for the corresponding dimension; the shared expert model includes: a CLIP encoder. share and five rating heads

[0043] The rating result generation module is used to obtain the rating result summarized by the chart based on the first rating result and the second rating result.

[0044] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the multi-dimensional chart summary evaluation method aligned with human experts as described above.

[0045] Compared with the prior art, the present invention has at least the following beneficial effects.

[0046] 1) Aligning with human evaluation and supporting multi-dimensional fine-grained assessment: The F2TEval model proposed in this invention introduces five evaluation dimensions: fidelity, comprehensiveness, simplicity, logic and analytical quality, which fully align with the human expert evaluation standards and significantly improves the interpretability and professionalism of the evaluation of image-to-text generated content.

[0047] 2) Lightweight design and efficient deployment: This invention adopts a lightweight architecture with an overall parameter count of 0.9B. Only 0.3B parameters are activated for single-dimensional scoring. It supports deployment and efficient inference on a single 24GB GPU, and has good practicality and scalability.

[0048] 3) Employing a hybrid expert structure to enhance the independence of scoring modules: By designing a Mixture of Experts (MoE) structure and setting up five dimensional experts and one shared expert, the scoring process is modularized and parallelized, effectively improving the independence and generalization ability among the scoring dimensions. The Hilbert-Schmidt Independence Criterion (HSIC) is innovatively introduced as an optimization mechanism to achieve independent mapping of each dimension's scoring modules in the matrix space, effectively reducing gradient interference between dimensions and improving model robustness and stability. Attached Figure Description

[0049] Figure 1 A flowchart summarizing the evaluation method using multidimensional charts aligned with human experts. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0051] The multi-dimensional chart summary evaluation method proposed in this invention, aligned with human experts, is based on the F2TEval model. The F2TEval model is an automatic evaluation model for multi-dimensional chart-to-text (F2T) conversion, constructed based on a Mixture of Experts (MoE) structure (see [link]). Figure 1The F2TEval model, with the left side representing the dimensional expert module and the first-stage training process, and the right side representing the shared expert module and the second-stage training process, enables a fine-grained, interpretable, and human-standard-aligned scoring mechanism for five dimensions: Faithfulness, Comprehensiveness, Conciseness, Logicality, and Analysis. The model consists of two main technical components: (1) Dimensional expert module: independently trained for each scoring dimension, learning the corresponding cross-modal semantic associations; (2) Shared expert module: featuring a multi-head output structure, jointly trained, and incorporating the Hilbert-Schmidt Independence Criterion (HSIC) to facilitate decoupled representation learning.

[0052] The reason for employing two types of expert modules in this invention is that while dimensional experts can independently learn the semantic features of each scoring dimension, they struggle to capture global features at the sample level and cannot reasonably calibrate cross-dimensional scoring results. Therefore, a shared expert module is introduced to uniformly correct the scores across the five dimensions through a reweighting mechanism. Considering the challenge of multimodal large language models in distinguishing dimension-specific semantics under shared representations, the HSIC optimization method can be used to enhance the independence between scoring heads, reduce feature redundancy, and thus improve overall evaluation accuracy and stability.

[0053] 1) Dimension Expert Module: Experts for each dimension The module is trained independently for a specific evaluation dimension d∈D. It consists of a pre-trained CLIP encoder, a lightweight projection layer, and a scoring function. Inputs include an image I, contextual text T (title and contextual information), and a generated graph summary S. The expert module outputs a predicted score. Image I, text T, and summary S are encoded into feature vectors v using a CLIP encoder. img ,v text ,v summary Then v img ,v text splicing process: v it =[v img ;v text Then align the feature dimensions: z it =D(v) it ), where D(v it This is a dimension-aligned linear projection layer. Finally, the cross-modal alignment similarity is calculated:

[0054]

[0055] Each expert uses the minimum mean squared error (MSE) Expert-aligned loss function Joint training is used to ensure the accuracy of predictions and consistency with ranking. For a batch containing N samples, the true score is y. i The predicted score is The loss term is defined as follows:

[0056]

[0057] in and λ represents the mean of the actual score and the predicted score in the batch, respectively. ali It is a hyperparameter used to balance the two loss terms.

[0058] 2) Shared Expert and HSIC Optimization: This invention introduces a shared expert module to jointly learn a general scoring pattern applicable to all five evaluation dimensions. Unlike dimensional experts, which focus on independently modeling each evaluation dimension, the shared expert is trained end-to-end, using shared image and text representations and equipped with a multi-head output structure. This design is flexible, supports cross-dimensional knowledge transfer, and is compatible with hybrid expert (MoE) structures. The shared expert consists of a CLIP encoder and five independent MLP output heads, each consisting of a two-layer feedforward network with nonlinearity. First, this invention uses the CLIP encoder to extract features from the image I, the contextual text T (title and contextual information), and the generated chart summary S, and concatenates them to obtain the vector v. its :

[0059]

[0060] To ensure that each scoring head focuses on learning different semantic signals, this invention uses a first-layer weight W1 (d) The HSIC method is introduced for optimization to encourage statistical independence of representations between dimensions and reduce redundancy. The specific formula is as follows:

[0061]

[0062]

[0063] Among them, W1 (i) W1 represents the first-level weight matrix of the i-th (i-dimensional) rating head. (j) This represents the first-level weight matrix of the j-th (j-dimensional) rating head. p, q, r, s are matrices W1. (i) and W1 (j) The row index; tr(.) represents the trace of the matrix. σ is the kernel bandwidth hyperparameter. Kpq With L pq This is the Gram matrix of the radial basis function (RBF) kernel. (Central matrix) Defined as: Among them, I n Represents an n-dimensional identity matrix. Let be a column vector containing all elements equal to 1. The total loss function is: Where λ hsic To adjust the weights, the final HSIC loss function is calculated as follows:

[0064]

[0065] End-to-end optimization of the HSIC loss term is achieved through backpropagation. Unlike traditional optimization methods based on orthogonality or covariance, which typically rely on assumptions of linear independence, the HSIC method measures statistical dependencies between representations in the reproducing kernel Hilbert space, capturing nonlinear and higher-order correlation features between representations. Guided by this gradient, the entire weight matrix effectively reduces its dependence on the weights of other rating heads, thereby promoting functional diversity among the rating heads. This mechanism enables each rating head to learn and encode an independent semantic subspace, thus enhancing the decoupling of information representations between different evaluation dimensions.

[0066] 3) Summary and evaluation using charts.

[0067] The final score for each dimension is calculated by combining the dimension's expert prediction score and the shared expert prediction:

[0068]

[0069] Where w d Let σ represent the learnable gating parameters (for each dimension) and denote the sigmoid function. Figure 1 Chinese w f w m w n w l w a y represents the scoring weights for each dimension. f y m y n y l y a The scores represent the scores from the shared expert scorers for the corresponding dimension.

[0070] Finally, the scores for each dimension. Weighting is applied to obtain the evaluation results summarized in the chart.

[0071] In summary, F2TEval designs five fine-grained evaluation dimensions: Faithfulness, Comprehensiveness, Conciseness, Logicality, and Analysis, to enhance the interpretability of evaluation results and their consistency with human evaluation standards. F2TEval is an open-source, lightweight, and reference-free evaluation model. Given that multi-dimensional scoring mechanisms may cause gradient interference during training, F2TEval employs a Mixture of Experts (MoE) structure. By introducing a non-linear decoupling mechanism and the Hilbert-Schmidt Independence Criterion (HSIC), feature mapping for different evaluation dimensions is implemented in matrix space, allowing each dimension's score to be completed by an independent sub-module, thereby improving the independence and generalization ability among the scoring dimensions.

[0072] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A multi-dimensional chart-based summary and evaluation method aligned with human experts, characterized in that, The method includes: Retrieve the target chart image, chart context text, and chart summary; The image, chart context text, and chart summary are fed into the expert models for each dimension to obtain the first score result for that dimension. These dimensions include fidelity, comprehensiveness, simplicity, logic, and analytical ability. Each dimension's expert model includes a CLIP encoder. An aligned linear projection layer and a scoring head ; The image, chart context text, and chart summary are fed into a shared expert model to obtain a second score result for the corresponding dimension; the shared expert model includes: a CLIP encoder. and five rating heads ; Based on the first rating result and the second rating result, the rating result summarized in the chart is obtained; Specifically, the contextual text of the images and charts, as well as the chart summaries, are fed into the expert models for each dimension to obtain the first score results for the corresponding dimension, including: Based on CLIP encoder Image, chart context text, and chart summary are encoded into feature vectors respectively. eigenvectors and eigenvectors ; eigenvectors With feature vectors Perform concatenation to obtain the feature vector. ; Using an aligned linear projection layer on this feature vector Perform projection mapping to obtain the feature vector Corresponding feature vector ; Based on the rating head Calculate eigenvectors With feature vectors The similarity between them is used to obtain the first score result for the corresponding dimension; Specifically, the image, chart context text, and chart summary are fed into a shared expert model to obtain the second scoring results for the corresponding dimensions, including: Based on CLIP encoder Image, chart context text, and chart summary are encoded into feature vectors respectively. eigenvectors and eigenvectors ; eigenvectors eigenvectors and eigenvectors Perform concatenation to obtain the feature vector. ; eigenvectors Enter each rating header separately This yields the second score for the corresponding dimension.

2. The method according to claim 1, characterized in that, The process of training an expert model for the fidelity dimension includes: Construct a fidelity dimension dataset, which includes: a chart sample, a chart summary of the chart sample, and the true fidelity dimension scores of the chart summary. ; The image of the chart sample, the chart context text, and the chart summary are fed into the fidelity dimension expert model to obtain a predicted score. ; Constructing the minimum mean square error function , This indicates the number of graph samples contained in a training batch; Constructing the expert alignment loss function ;in, This represents all the true scores in this training batch. The average value, This represents all predicted scores for this training batch. The average value; Based on the minimum mean square error function Align loss function with experts Obtain the trained fidelity dimension expert model.

3. The method according to claim 1, characterized in that, The second rating result of the corresponding dimension ;in, , These represent the rating headers respectively. The first layer weight matrix and the second layer weight matrix, , These represent the rating headers respectively. The first-level bias matrix and the second-level bias matrix.

4. The method according to claim 3, characterized in that, The process of training a shared expert model includes: Construct fidelity dimension datasets, comprehensive dimension datasets, concise dimension datasets, logical dimension datasets, and analytical dimension datasets; Training CLIP encoder Then, freeze the CLIP encoder. Parameters; Based on the fidelity dimension dataset, comprehensiveness dimension dataset, conciseness dimension dataset, logicality dimension dataset, and analytical dimension dataset, adjust the corresponding scoring heads respectively. The parameters in.

5. The method according to claim 4, characterized in that, Based on the fidelity dimension dataset, comprehensiveness dimension dataset, conciseness dimension dataset, logicality dimension dataset, and analytical dimension dataset, adjust the corresponding scoring heads respectively. The parameters include: Constructing the HSIC loss function ;in, Indicates the first The first layer weight matrix of the scoring heads for each dimension Indicates the first The first layer weight matrix of the scoring heads for each dimension , Represents the first layer weight matrix With the first layer weight matrix Total number of rows , , , It is the first layer weight matrix and the first layer weight matrix row index, , , , , , The Gram matrix is ​​the kernel of the radial basis functions; Based on HSIC loss function For the first layer weight matrix and the first layer weight matrix Optimize.

6. The method according to claim 1, characterized in that, Based on the first and second rating results, the rating results summarized in this chart are obtained, including: The first and second scores for each dimension are weighted to obtain the score for that dimension. The scores for each dimension are weighted to obtain the overall score summary in this chart.

7. A multi-dimensional chart-based summary and evaluation system aligned with human experts, characterized in that, The system includes: The data acquisition module is used to obtain the chart summary of the target chart and extract the image and chart context text of the target chart; The first scoring module is used to input the contextual text of images and charts, as well as the chart summaries, into the expert models for each dimension to obtain the first scoring result for that dimension. These dimensions include fidelity, comprehensiveness, simplicity, logic, and analytical quality. Each dimension's expert model includes a CLIP encoder. An aligned linear projection layer and a scoring head ; The second scoring module is used to feed the image, chart context text, and chart summary into the shared expert model to obtain the second scoring result for the corresponding dimension; the shared expert model includes: a CLIP encoder. and five rating heads ; The rating result generation module is used to obtain the rating result summarized by the chart based on the first rating result and the second rating result; Specifically, the image and chart context text and chart summary are fed into the expert models for each dimension to obtain the first score result for the corresponding dimension, including: Based on CLIP encoder Image, chart context text, and chart summary are encoded into feature vectors respectively. eigenvectors and eigenvectors ; eigenvectors With feature vectors Perform concatenation to obtain the feature vector. ; Using an aligned linear projection layer on this feature vector Perform projection mapping to obtain the feature vector Corresponding feature vector ; Based on the rating head Calculate eigenvectors With feature vectors The similarity between them is used to obtain the first score result for the corresponding dimension; Specifically, the image, chart context text, and chart summary are fed into a shared expert model to obtain the second scoring results for the corresponding dimensions, including: Based on CLIP encoder Image, chart context text, and chart summary are encoded into feature vectors respectively. eigenvectors and eigenvectors ; eigenvectors eigenvectors and eigenvectors Perform concatenation to obtain the feature vector. ; eigenvectors Enter each rating header separately This yields the second score for the corresponding dimension.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the multi-dimensional chart summary evaluation method aligned with human experts as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Project pre-evaluation method and device based on multi-dimensional neural network

    CN117574912A

  • Chart replacement text quality automatic evaluation method and system based on large language model

    CN119646460A