Multi-dimensional chart summary evaluation method and system for human expert alignment

Abstract

The invention discloses a multi-dimensional chart summary evaluation method and system for human expert alignment, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a target chart image, a chart context text and a chart summary; respectively sending the image, the chart context text and the chart summary into the expert model of each dimension to obtain a first scoring result of the corresponding dimension; respectively sending the image, the chart context text and the chart summary into a shared expert model to obtain a second scoring result of the corresponding dimension; and obtaining a scoring result of the chart summary according to the first scoring result and the second scoring result. According to the method, fine-grained, explainable and efficient evaluation of chart summarization can be realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a multi-dimensional chart summary evaluation method and system for human expert alignment. BACKGROUND

[0002] As an important form of information representation, charts are widely used in various fields such as academic papers and business analysis. The figure-to-text (F2T) task aims to convert the key information contained in the image into a textual description, thereby improving the structured and understandable nature of the information. This technology not only helps to achieve more efficient information retrieval in data-intensive fields, but also provides a more friendly way for visually impaired people to access content. However, due to the semantic compression and ambiguous expression of charts, the generated text often has problems such as inaccurate facts, incomplete content, and weak logical reasoning, so it is necessary to introduce an automated quality evaluation method. Effective evaluation of chart-to-text quality is crucial for the development of this task. Currently, the evaluation methods for chart-to-text tasks can be divided into two categories: reference-based methods and reference-free methods.

[0003] Reference-based evaluation methods (such as BLEU, ROUGE, CIDEr, and BERTScore) are commonly used for chart-to-text tasks. These methods measure the semantic similarity between the generated text and the reference text to evaluate the quality of the generated text. This type of method is easy to implement, suitable for various text generation scenarios, and has good repeatability and efficiency. However, this method is highly dependent on high-quality reference texts, which usually need to be annotated by professionals, resulting in high annotation costs. In addition, this method is mainly based on shallow text semantic similarity calculation, making it difficult to effectively identify factual errors, logical flaws, and reasoning omissions in the generated text.

[0004] With the development of multi-modal large language models (MLLMs), reference-free evaluation methods for chart-to-text tasks have also been significantly enhanced. This type of method usually relies on pre-trained models to generate evaluation scores or text explanations through customized prompts (instructions). Current research attempts to combine context to design scoring templates to enhance evaluation robustness and use lightweight multi-modal models to reduce deployment costs. However, this type of method is still sensitive to input instructions and sample distribution, making it difficult to be stable and applicable in large-scale evaluation tasks. In addition, the performance of lightweight models is limited, making it difficult to meet the accuracy requirements, while large closed-source models have high calling costs and unstable versions.

[0005] In addition to the above challenges, most existing evaluation methods only provide overall scores at the sample level, lack fine-grained evaluation in key dimensions such as content quality, logical structure, conciseness and analysis depth, and limit the interpretability of evaluation results. At the same time, there is still a significant gap between these methods and expert manual evaluation, which fails to effectively align the task context and human cognitive process, limiting its application in high-precision, safety and controllability requirements of the generated modeling scene. Therefore, how to build an efficient, robust, fine-grained and human cognitive judgment process aligned automatic evaluation method has become a key challenge in current chart-to-text evaluation research. SUMMARY

[0006] To solve the above challenges, the present application proposes a human expert aligned multi-dimensional chart summary evaluation method and system. This method aligns with the expert evaluation criteria, and can achieve fine-grained, interpretable and efficient evaluation of chart summaries.

[0007] To achieve the above purpose, the technical scheme of the present application includes the following contents.

[0008] A human expert aligned multi-dimensional chart summary evaluation method, the method comprising:

[0009] Obtaining a target chart image, chart context text and chart summary;

[0010] Sending the image, chart context text and chart summary into each dimension expert model respectively to obtain the first score result of the corresponding dimension; wherein the dimensions include fidelity dimension, comprehensiveness dimension, conciseness dimension, logicality dimension and analysis dimension, and each dimension expert model comprises: a CLIP encoder E spe , an alignment linear projection layer and a score head

[0011] Sending the image, chart context text and chart summary into a shared expert model respectively to obtain the second score result of the corresponding dimension; the shared expert model comprises: a CLIP encoder E share and five score heads

[0012] According to the first score result and the second score result, the score result of the chart summary is obtained.

[0013] Further, sending the image, chart context text and chart summary into each dimension expert model respectively to obtain the first score result of the corresponding dimension, comprising:

[0014] Based on the CLIP encoder E spe The image, chart context text and chart summary are respectively encoded into feature vectors v img , feature vector vtext and feature vector v summary ;

[0015] Splicing the feature vector v img and the feature vector v text , to obtain the feature vector v it ;

[0016] Projecting and mapping the feature vector v it using the alignment linear projection layer, to obtain the feature vector z summary corresponding to the feature vector v it ;

[0017] Based on the scoring head , calculate the similarity between the feature vector v summary and the feature vector z it to obtain the first scoring result of the corresponding dimension.

[0018] Further, the process of training the fidelity dimension expert model includes:

[0019] Construct a fidelity dimension dataset, which includes: a chart sample, a chart summary of the chart sample, and a fidelity dimension true score y i of the chart summary.

[0020] Send the image of the chart sample, the chart context text, and the chart summary into the fidelity dimension expert model to obtain the predicted score

[0021] Construct a least mean square error function N represents the number of chart samples included in a training batch.

[0022] Construct an expert alignment loss function wherein, represents the average of all true scores y i of the training batch, represents the average of all predicted scores of the training batch.

[0023] Based on the least mean square error function and the expert alignment loss function , obtain the trained fidelity dimension expert model.

[0024] Further, send the image, the chart context text, and the chart summary into the shared expert model respectively to obtain the second scoring result of the corresponding dimension, including:

[0025] Based on the CLIP encoder E shareEncode the image, chart context text and chart summary into feature vectors v img , respectively text and v summary , respectively

[0026] Concatenate the feature vectors v img , v text and v summary to obtain a feature vector v its ;

[0027] Input the feature vector v its into each scoring head to obtain a second scoring result of the corresponding dimension.

[0028] Further, the second scoring result of the corresponding dimension wherein, denote the first layer weight matrix and the second layer weight matrix of the scoring head , respectively, denote the first layer bias matrix and the second layer bias matrix of the scoring head , respectively.

[0029] Further, the process of training the shared expert model comprises:

[0030] constructing the fidelity dimension dataset, the comprehensiveness dimension dataset, the conciseness dimension dataset, the logicality dimension dataset and the analysis dimension dataset;

[0031] training the CLIP encoder E share , and then freezing the parameters of the CLIP encoder E share ;

[0032] adjusting the parameters in the corresponding scoring head based on the fidelity dimension dataset, the comprehensiveness dimension dataset, the conciseness dimension dataset, the logicality dimension dataset and the analysis dimension dataset, respectively.

[0033] Further, adjusting the parameters in the corresponding scoring head based on the fidelity dimension dataset, the comprehensiveness dimension dataset, the conciseness dimension dataset, the logicality dimension dataset and the analysis dimension dataset, respectively, comprises:

[0034] constructing a HSIC loss function wherein, W1 (i) denotes the first layer weight matrix of the i-th dimension scoring head, W1 (j) denotes the first layer weight matrix of the j-th dimension scoring head, n denotes the first layer weight matrix W1(i) the total number of rows of the first layer weight matrix W1 (j) , p, q, r, s are row indexes of the first layer weight matrix W1 (i) , and the first layer weight matrix W1 (j) , K pq , L pq , K pr , L qr , K pr , L qs is a Gram matrix of a radial basis function kernel;

[0035] based on the HSIC loss function optimizing the first layer weight matrix W1 (i) and the first layer weight matrix W1 (j) .

[0036] Further, according to the first score result and the second score result, a score result of the chart summary is obtained, including:

[0037] weighting the first score result and the second score result of each dimension to obtain a score result of the dimension;

[0038] weighting the score results of the dimensions to obtain the score result of the chart summary.

[0039] A human expert aligned multi-dimensional chart summary evaluation system, the system comprising:

[0040] a data acquisition module for acquiring a chart summary of a target chart and extracting an image and chart context text of the target chart;

[0041] a first scoring module for feeding the image, chart context text and chart summary into respective dimension expert models to obtain a first score result of the corresponding dimension; wherein the dimensions include fidelity dimension, comprehensiveness dimension, conciseness dimension, logic dimension and analysis dimension, and each dimension expert model comprises a CLIP encoder E spe , an alignment linear projection layer and a scoring head

[0042] a second scoring module for feeding the image, chart context text and chart summary into a shared expert model to obtain a second score result of the corresponding dimension; the shared expert model comprises a CLIP encoder E share and five scoring heads

[0043] a score result generation module for obtaining a score result of the chart summary according to the first score result and the second score result.

[0044] An electronic device, comprising: a processor and a memory storing computer program instructions; the processor implements the human expert aligned multi-dimensional chart summary evaluation method of any one of the above when executing the computer program instructions.

[0045] Compared with the prior art, the present application has at least the following beneficial effects.

[0046] 1) Align artificial evaluation, support multi-dimensional fine-grained evaluation: the F2TEval model proposed in the present application introduces five evaluation dimensions of fidelity, comprehensiveness, conciseness, logic and analysis, fully aligns with the artificial expert evaluation standard, and significantly improves the interpretability and professionalism of the evaluation of the generated content of the graph to text.

[0047] 2) Lightweight design, efficient deployment: the present application adopts a lightweight architecture, the overall parameter quantity is 0.9B, and only 0.3B parameters are activated for single-dimensional scoring, which supports deployment and efficient inference on a single 24GB GPU, and has good practicality and scalability.

[0048] 3) Adopting a mixed expert structure, enhancing the independence of the scoring module: by designing a Mixture of Experts (MoE) structure and setting five dimension experts and one shared expert, the modularization and parallelization of the scoring process are realized, and the independence and generalization ability between the scoring dimensions are effectively improved. The Hilbert-Schmidt Independence Criterion (HSIC) is innovatively introduced as an optimization mechanism to realize the independent mapping of each dimension scoring module in the matrix space, effectively reducing the gradient interference between dimensions, and improving the robustness and stability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 Flowchart of the human expert aligned multi-dimensional chart summary evaluation method. DETAILED DESCRIPTION

[0050] The present application will be further described in detail below in conjunction with the accompanying drawings, and the examples given are only used to explain the present application and are not intended to limit the scope of the present application.

[0051] The human expert aligned multi-dimensional chart summary evaluation method proposed in the present application is realized based on the F2TEval model, which is a multi-dimensional chart to text (F2T) automatic evaluation model, which is constructed based on a mixed expert structure (Mixture of Experts, MoE) (see Figure 1), where the left side is the dimension-specific module and the first-stage training process, and the right side is the shared module and the second-stage training process. The F2TEval model can realize a fine-grained, interpretable, and aligned artificial standard scoring mechanism for five dimensions: Faithfulness, Comprehensiveness, Conciseness, Logicality, and Analysis. The model consists of two main technical components: (1) Dimension-specific module: independently trained for each scoring dimension, learning the corresponding cross-modal semantic correlation; (2) Shared module: with a multi-head output structure, joint training, and the introduction of Hilbert-Schmidt Independence Criterion (HSIC) to promote decoupled representation learning.

[0052] The reason why the present application adopts two types of expert modules is that the dimension-specific expert can independently learn the semantic features of each scoring dimension, but it is difficult to capture the global features at the sample level and cannot reasonably calibrate the cross-dimensional score results. Therefore, the shared expert module is introduced to uniformly correct the scores of the five dimensions through a re-weighting mechanism. Considering the challenge that the multi-modal large language model cannot distinguish dimension-specific semantics under shared representation, the HSIC optimization method can be used to enhance the independence between scoring heads, reduce feature redundancy, and thus improve the overall evaluation accuracy and stability.

[0053] 1) Dimension-specific module: Each dimension-specific expert is independently trained for a specific evaluation dimension d∈D. This module consists of a pre-trained CLIP encoder, a lightweight projection layer, and a scoring function. The input includes the image I, the context text T (title and context information), and the generated chart summary S. The output of the expert module is the predicted score The image I, the text T, and the summary S are encoded into feature vectors v img ,v text ,v summary respectively using the CLIP encoder. Then, v img ,v text are concatenated: v it = [v img ; v text ], and then the feature dimensions are aligned: z it = D(v it ), where D(v it ) is the dimension alignment linear projection layer. Finally, the cross-modal alignment similarity is calculated:

[0054]

[0055] Each expert is trained by minimizing the mean squared error (MSE) Alignment loss function with experts Joint training to ensure the accuracy of the prediction and the consistency of the ranking. For a batch containing N samples, the true scores are y i , and the predicted scores are The loss terms are defined as follows:

[0056]

[0057] where and are the mean of the true scores and the predicted scores in the batch, respectively, λ ali is a hyperparameter used to balance the two loss terms.

[0058] 2) Shared expert and HSIC optimization: The present invention introduces a shared expert module to jointly learn a general scoring pattern applicable to all five evaluation dimensions. Unlike dimension-specific experts that focus on independent modeling of each evaluation dimension, the shared expert is trained in an end-to-end manner, using shared image and text representations, and equipped with a multi-head output structure. This design is flexible, supports cross-dimensional knowledge transfer, and is compatible with the MoE structure. The shared expert consists of a CLIP encoder and five independent MLP output heads, each consisting of a two-layer feedforward network with nonlinearity. First, the present invention uses the CLIP encoder to extract features from the image I, the context text T (title and context information), and the generated chart summary S, and concatenates them to obtain the vector v its :

[0059]

[0060] To ensure that each scoring head focuses on learning different semantic signals, the present invention introduces the HSIC method to optimize the first layer weight W1 (d) to encourage statistical independence between dimensions and reduce redundancy. The specific formula is as follows:

[0061]

[0062]

[0063] where W1 (i) represents the first layer weight matrix of the i-th (i-dimensional) scoring head. W1 (j) represents the first layer weight matrix of the j-th (j-dimensional) scoring head. p, q, r, s are the row indices of the matrices W1 (i) and W1 (j) ; tr(.) represents the trace of the matrix. σ is the kernel bandwidth hyperparameter. Kpq with L pq is the Gram matrix of the Radial Basis Function (RBF) kernel. The center matrix is defined as: where I n is the n-dimensional identity matrix, is a column vector with all elements being 1. The total loss function is: where λ hsic is the adjustment weight. The final HSIC loss function is calculated as:

[0064]

[0065] The end-to-end optimization of the HSIC loss term is achieved by backpropagation. Unlike traditional optimization methods based on orthogonality or covariance, which usually rely on the assumption of linear independence, the HSIC method measures the statistical dependence between representations in the reproducing kernel Hilbert space, which can capture the nonlinear and high-order correlation characteristics between representations. Through the guidance of this gradient, the entire weight matrix can effectively reduce the dependence on other rating head weights, thereby promoting the functional diversity between rating heads. This mechanism enables each rating head to learn and encode an independent semantic subspace, thereby enhancing the information representation decoupling ability between different evaluation dimensions.

[0066] 3) Chart summary evaluation.

[0067] The final score of each dimension is calculated by the combination of the dimension expert predicted score and the shared expert predicted score:

[0068]

[0069] where w d is the (dimensional) learnable gating parameter, and σ represents the sigmoid function. Figure 1 w f , w m , w n , w l , w a are the scoring weights of each dimension, y f , y m , y n , y l , y a represent the scores of the shared expert scoring head of the corresponding dimension.

[0070] Finally, the scores of each dimension are weighted to obtain the evaluation results of the chart summary.

[0071] In summary, F2TEval designs five fine-grained evaluation dimensions, including Faithfulness, Comprehensiveness, Conciseness, Logicality and Analysis, to enhance the interpretability of the evaluation results and the consistency with human evaluation criteria. F2TEval is an open-source, lightweight and non-reference evaluation model. Since the multi-dimensional scoring mechanism may cause gradient interference in the training process, F2TEval designs a Mixture of Experts (MoE). By introducing a nonlinear decoupling mechanism and Hilbert-Schmidt Independence Criterion (HSIC), the feature mapping of different evaluation dimensions is realized in the matrix space, so that the scores of each dimension are completed by independent sub-modules, thereby improving the independence and generalization ability between each scoring dimension.

[0072] Although specific embodiments of the present application are disclosed to illustrate the present application, their purpose is to assist in understanding the present application and to enable it to be put into practice, it being understood that various changes, substitutions and modifications can be made by those skilled in the art without departing from the spirit and scope of the present application and of the appended claims. The present application should therefore not be considered as limited to the embodiments disclosed, but rather the scope of the application is defined by the scope of the claims.

Claims

1. A method for human expert aligned multi-dimensional chart summary evaluation, characterized in that, The method comprises: obtaining a target chart image, chart context text and a chart summary; The image, chart context text and chart summary are respectively sent into each dimension expert model to obtain a first score result of the corresponding dimension; wherein, the dimensions include fidelity dimension, comprehensiveness dimension, conciseness dimension, logic dimension and analysis dimension, each dimension expert model includes: a CLIP encoder E spe , an alignment linear projection layer and a score head The image, chart context text and chart summary are respectively sent into a shared expert model to obtain a second score result of the corresponding dimension; the shared expert model comprises: a CLIP encoder E share and five score heads obtaining a score result of the chart summary according to the first score result and the second score result.

2. The method of claim 1, wherein, sending the image, the chart context text and the chart summary into respective dimension expert models to obtain a first score result of a corresponding dimension, comprising: Based on the CLIP encoder E spe Encoding the image, chart context text, and chart summary into feature vectors v img , v text , and v summary respectively ; The feature vector v img is obtained by performing the stitching processing on the feature vectors v text . it ; aligning linear projection layer to the feature vector v it performing projection mapping to obtain a feature vector z summary corresponding to the feature vector v it ; Based on the score head Computing a feature vector v summary Similarity between the feature vector z it To get the first score result of the corresponding dimension.

3. The method of claim 1, wherein, a process of training the fidelity dimension expert model, comprising: constructing a fidelity dimension dataset comprising: a chart sample, a chart summary of the chart sample, and a fidelity dimension ground truth score y of the chart summary i ; feeding the image of the chart sample, the chart contextual text, and the chart summary into the fidelity dimension expert model to obtain a predicted score constructing a minimum mean square error function N denotes the number of chart samples contained in a training batch; Constructing expert alignment loss function where, denotes the average of all true scores y i of the training batch, denotes the average of all predicted scores of the training batch. Based on a least mean square error function and an expert alignment loss function Obtain the trained fidelity dimension expert model.

4. The method of claim 1, wherein, sending the image, the chart context text and the chart summary into a shared expert model to obtain a second score result of a corresponding dimension, comprising: Based on the CLIP encoder E share Encoding the image, chart context text, and chart summary into feature vectors v img , v text , and v summary , respectively The feature vector v img is obtained by splicing the feature vector v text , the feature vector v summary , and the feature vector v its . The feature vector v its Each score head is input respectively The second score result corresponding to the dimension is obtained.

5. The method of claim 4, wherein, the second score result of the corresponding dimension wherein W1 (d) , W2 (d) respectively represent a first layer weight matrix and a second layer weight matrix of the scoring head respectively represent a first layer bias matrix and a second layer bias matrix of the scoring head ​​ 6. The method of claim 5, wherein, a process of training the shared expert model, comprising: constructing a fidelity dimension dataset, a comprehensiveness dimension dataset, a conciseness dimension dataset, a logic dimension dataset and an analysis dimension dataset; Training a CLIP encoder E share Afterwards, the parameters of the CLIP encoder E share are frozen; Adjust the parameters in the corresponding score header based on the fidelity dimension dataset, the comprehensiveness dimension dataset, the conciseness dimension dataset, the logicality dimension dataset, and the analyticity dimension dataset, respectively ​ 7. The method of claim 6, wherein, Adjust parameters in the corresponding score header in based on the fidelity dimension dataset, the comprehensiveness dimension dataset, the conciseness dimension dataset, the logicality dimension dataset, and the analyticity dimension dataset, respectively.​ Constructing the HSIC loss function where W1 (i) denotes the first layer weight matrix of the i-th dimension head, W1 (j) denotes the first layer weight matrix of the j-th dimension head, n denotes the total number of rows of the first layer weight matrix W1 (i) and the first layer weight matrix W1 (j) , p, q, r, s are the row indices of the first layer weight matrix W1 (i) and the first layer weight matrix W1 (j) , K pq , L pq , K pr , L qr , K pr , L qs is the Gram matrix of the radial basis function kernel; Based on the HSIC loss function to the first layer weight matrix W1 (i) and the first layer weight matrix W1 (k) are optimized.

8. The method of claim 1, wherein, obtaining a score result of the chart summary according to the first score result and the second score result, comprising: weighting the first score result and the second score result of each dimension to obtain a score result of the dimension; weighting the score results of the dimensions to obtain a score result of the chart summary.

9. A human expert aligned multi-dimension chart summary evaluation system, characterized by, The system comprises: a data acquisition module configured to obtain a chart summary of a target chart and extract an image and chart context text of the target chart; The first scoring module is configured to send the image, the chart context text and the chart summary into respective dimension expert models respectively to obtain first scoring results of corresponding dimensions; wherein, the dimensions include a fidelity dimension, a comprehensiveness dimension, a conciseness dimension, a logic dimension and an analysis dimension, and each dimension expert model includes: a CLIP encoder E spe , an alignment linear projection layer and a scoring head The second scoring module is configured to send the image, the chart context text and the chart summary into a shared expert model respectively to obtain a second scoring result of a corresponding dimension. share and five scoring heads a score result generation module configured to obtain a score result of the chart summary according to the first score result and the second score result.

10. An electronic device, comprising: The electronic device comprises a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the multi-dimension chart summary evaluation method of human expert alignment according to any one of claims 1-8.

Images