Chart replacement text quality automatic evaluation method and system based on large language model

Abstract

The invention discloses a method and system for automatically evaluating the quality of a chart substitution text based on a large language model, and relates to the field of large language models and the field of chart substitution texts.The method comprises the steps that a chart substitution text quality evaluation scheme is obtained through demand investigation; the method comprises the following steps: manually constructing a chart to replace a text evaluation standard data set to carry out data preprocessing, and carrying out Finetune training and performance optimization based on a GLM4-9B model. According to the automatic chart substitution text quality evaluation method, evaluation scheme design is carried out based on the chart barrier-free reading criterion, the prediction score of the migration model obtained through training on the text quality is highly related to the manual evaluation score, and a high-reliability automatic chart substitution text quality evaluation result can be provided; the method can be applied to text quality evaluation in the graph replacement text field.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method and system for automatically evaluating the quality of text replacement for a diagram based on a large language model. Background Art

[0002] Chart alternative text is text used to describe the content of a chart. It is usually used to help visually impaired people understand the content of the chart and plays an important role in promoting the development of information accessibility. In recent years, the development of natural language processing technology has promoted the advancement of automatic chart alternative text generation technology. The current multimodal large language model can receive chart images input by users and provide descriptions of the chart content. However, due to the lack of reliable automatic evaluation methods for the quality of chart alternative text, the quality of chart alternative text generated by these large language models cannot be effectively verified.

[0003] The existing evaluation methods for chart alternative text mainly include manual evaluation methods and automated evaluation methods. Manual evaluation methods are not only inefficient, but also easily affected by subjective factors; while mainstream automated text quality evaluation methods (such as BLEU, ROUGE, etc.) mainly evaluate the quality of alternative text based on the content similarity between the reference text and the generated text, which has a low correlation with manual evaluation, and such scores are not of reference significance. In recent years, large language models based on the Transformer architecture have made major breakthroughs in natural language understanding, but in terms of automated evaluation of chart alternative text quality, due to the lack of professional background knowledge in the field of accessible chart understanding, it is impossible to reliably evaluate the quality of chart alternative text. Therefore, it is urgent to invent an automated evaluation method for chart alternative text quality based on a large language model to solve the problem that the current quality evaluation of chart alternative text cannot be automated. Summary of the invention

[0004] The present invention aims to address the deficiencies of the prior art and to propose a method and system for automatically evaluating the quality of text replacements for graphs based on a large language model.

[0005] The object of the present invention is achieved through the following technical solutions: A method for automatically evaluating the quality of graph replacement text based on a large language model, comprising the following steps:

[0006] S1. Determine the evaluation dimensions and weights of text alternatives for graphics based on the accessible graphics reading guidelines and the graphics reading needs of visually impaired users;

[0007] S2, obtaining the chart to be evaluated for image preprocessing, constructing the chart to be evaluated alternative text, reference chart alternative text, and manual evaluation scores as a data set, and performing text data preprocessing on the text data of the data set;

[0008] S3, perform model post-processing on the GLM4 model, customize the output prediction value of the model; obtain a GLM4 model whose input is the pre-processed data and whose output is the prediction value defined in the post-processing; input the pre-processed data set into the post-processed GLM4 model and perform fine-tuning training, test the trained migration model and tune the model performance; wherein the output prediction value of the GLM4 model is obtained by calculating the n-gram overlap between the reference chart replacement text and the chart replacement text to be evaluated, as well as the weight of each evaluation dimension;

[0009] S4. Input the chart replacement text into the optimized GLM4 model to obtain the final evaluation score.

[0010] Furthermore, the S1 comprises the following sub-steps:

[0011] Investigate the chart reading needs of visually impaired users, conduct user surveys, set the evaluation dimensions of chart alternative text to chart title, chart type, chart coordinate axis, chart specific value, color information description and chart trend analysis, and obtain the user importance score of each dimension through the survey results, and obtain the weight W of each dimension through the user importance score of each dimension i :

[0012]

[0013] Among them, A i is the mean of user importance scores for each dimension.

[0014] Furthermore, the construction of the chart replacement text to be evaluated, the reference chart replacement text, and the manual evaluation score specifically includes: the chart replacement text to be evaluated is obtained by inputting the chart image into a large language model; the reference chart replacement text is manually produced for the chart image based on the evaluation dimension, and the manual evaluation score is obtained by manually evaluating the chart replacement text to be evaluated.

[0015] Furthermore, the manual evaluation criteria specifically include:

[0016] In the chart type dimension, one point is awarded for correctly giving the chart type; in the icon title dimension, one point is awarded for correctly describing the chart title; no point is awarded if there is no description or the chart title description is incomplete; in the chart axis dimension, 0.25 points are awarded for correctly describing the horizontal axis title, label range, vertical axis title, and label range; in the chart specific value dimension, the score = the number of data points in the chart drawing area that are correctly described / the total number of data points in the chart drawing area; in the color information description dimension, the score = the number of colors that are correctly described / the total number of colors in the chart; in the chart trend analysis dimension, 1 point is awarded for a relevant description of the data trend presented in the chart; no point is awarded if there is no description or the description is irrelevant to the chart data.

[0017] Furthermore, the image preprocessing includes correction of non-horizontal images, image size standardization, and smoothing;

[0018] The text data preprocessing includes text cleaning, word segmentation, removal of special characters, standardization, and feature splicing; the feature splicing is to use a pre-trained word embedding model to convert the alternative text of the chart to be evaluated, the alternative text of the reference chart, and the manual evaluation score into a numerical vector representation, and splice them into a new feature vector; the standardization process is specifically:

[0019]

[0020] Among them, S normalized is the normalized manual evaluation score, S is the original manual evaluation score, and S min is the minimum value of the manual evaluation score; S max is the maximum value of the manual evaluation score.

[0021] Furthermore, the feature concatenation includes: combining the standardized manual evaluation score with the text feature; constructing a feature vector X, which includes the embedding vector T of the replacement text of the chart to be evaluated and the embedding vector Q of the replacement text of the reference chart, as well as the manual evaluation score S normalized ;

[0022] X=[T;Q;S normalized ]

[0023] The “;” indicates a vector concatenation operation.

[0024] Furthermore, the model post-processing is specifically as follows:

[0025] Use the ROUGE evaluation method to measure the n-gram overlap between the reference chart alternative text and the chart alternative text to be evaluated, and combine the evaluation dimension weights to customize the output prediction value E of the model:

[0026]

[0027] Among them, R1 represents the n-gram precision, which indicates the n-gram overlap ratio between the alternative text to be evaluated and the reference alternative text; P i W represents the precision, which is the ratio of the number of word vectors correctly described in the alternative text to be evaluated to the total number of word vectors in the alternative text to be evaluated; i represents the weight of each key evaluation dimension, r i Recall is the sum of the number of correctly described word vectors in each evaluation dimension of the alternative text to be evaluated to the total number of word vectors in each evaluation dimension of the reference alternative text; α and β are balancing coefficients used to adjust the weight relationship between n-gram accuracy, precision, and recall of key evaluation dimensions.

[0028] Furthermore, the loss function used in the fine-tuning training is:

[0029]

[0030] Where L(θ) is the loss function, N is the number of samples in the dataset, Si is the actual human evaluation score of the i-th sample, Zi is the predicted score of the i-th sample, and θ is the model parameter.

[0031] Furthermore, the testing and model performance tuning include:

[0032] The performance of the migration model was evaluated using the validation set in the graph-to-text evaluation dataset, and the difference between the predicted score and the actual manual evaluation score was calculated using the correlation index, which is:

[0033]

[0034] Where ρ is the Pearson correlation coefficient; S i is the actual manual evaluation score of the ith sample, is the average of the actual manual evaluation scores; Z i is the predicted score of the ith sample, is the average of the predicted scores; N is the number of samples;

[0035] Tune the migration model based on the model test performance, including hyperparameter adjustment and algorithm optimization.

[0036] According to another aspect of the specification, there is also provided a system for automatically evaluating the quality of text alternatives to diagrams based on a large language model, the system comprising: a data set construction module, a data preprocessing module, a model training module, and a model testing and performance tuning module;

[0037] The data set construction module is used to collect high-quality reference chart alternative texts, chart alternative texts to be evaluated, and manual evaluation scores to construct a chart alternative text evaluation data set;

[0038] The data preprocessing module is used to preprocess the data set, including text cleaning, word segmentation, removal of special characters, standardization, and feature splicing; and divide the data set into a training set, a validation set, and a test set for model training;

[0039] The model training module selects GLM4-9B as the basic model, and defines the model output prediction value E in combination with the reading preference of visually impaired users; customizes the loss function L(θ), uses the chart instead of the text evaluation data set to perform Finetune training on the GLM4 model, and continuously adjusts the model weights to minimize the loss function;

[0040] The model testing and performance tuning module uses a graph to replace the validation set in the text evaluation data set to evaluate the performance of the migration model, and uses a correlation index to calculate the difference between the model prediction score and the actual manual evaluation score. The migration model is tuned according to the model test performance.

[0041] Beneficial effects of the present invention:

[0042] 1. Based on the actual needs of visually impaired users and combined with the current design criteria for text content of chart alternatives in the field of information accessibility, this paper proposes a new 6-dimensional evaluation scheme for the quality of chart alternative text.

[0043] 2. The present invention provides an automated chart replacement text quality assessment method for the current chart replacement text research field, which combines mainstream evaluation indicators based on text content similarity with indicators in the field of chart information accessibility, and can provide efficient and reliable quality assessment results for chart replacement text.

[0044] 3. The method for automatically evaluating the quality of text replacements for charts based on a large language model proposed in the present invention can be deployed and run on different computing devices, and can quickly build an efficient, professional and reliable automatic evaluation system for the quality of text replacements for charts.

[0045] 4. The present invention can solve the problem of lack of reliable automated evaluation methods in the current field of graph replacement text generation, and further promote relevant research in the field of automated generation of graph replacement text.

[0046] 5. The present invention constructs a currently scarce chart alternative text evaluation dataset that is completely derived from the real world. The construction of reference chart alternative texts and manual evaluation scores is faithful to the chart accessibility guidelines, has high reference significance, and can provide dataset support for chart alternative text evaluation work in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the automated quality assessment method for graph alternative text based on a large language model.

[0048] Figure 2 It is a schematic diagram of an automatic evaluation device for the quality of graph alternative text based on a large language model. DETAILED DESCRIPTION

[0049] The present invention is described in detail below with reference to the accompanying drawings. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. All other embodiments obtained by ordinary technicians in the field without creative work based on the embodiments of the present invention are within the scope of protection of the present invention.

[0050] See also Figure 1 The method for automatically evaluating the quality of graph replacement text based on a large language model of the present invention comprises the following steps:

[0051] Step 1: Design an evaluation plan

[0052] We conducted a literature survey in the field and initially set five evaluation dimensions based on the current accessibility standards in the field of chart reading. We conducted an offline survey on the chart reading needs of visually impaired users to understand the types of information they prefer to obtain when reading charts. The survey data showed that more than 80% of users have a need to obtain color information. Therefore, we added a color information description evaluation dimension to the original five evaluation dimensions. Based on the average importance rating of the six content dimensions by users, we obtained the weight percentages W of the six evaluation dimensions. i , establish a reliable chart alternative text quality assessment scheme. The six assessment dimensions are: chart title, chart type, chart coordinate axis, chart specific value, color information description, and chart trend analysis. The weight of each dimension is W i The calculation formula is as follows:

[0053]

[0054] Among them, A i is the mean of user importance scores for each dimension.

[0055] Step 2: Construct a data set: collect reference chart alternative texts, chart alternative texts to be evaluated, and manual evaluation scores to construct a chart alternative text evaluation data set; preprocess the text data in the chart alternative text evaluation data set, and the preprocessing includes text cleaning, word segmentation, removal of special characters, standardization, and feature splicing; divide the chart alternative text evaluation data set into a training set, a validation set, and a test set.

[0056] In this embodiment, constructing the data set further includes:

[0057] Collect different types of chart images and perform image preprocessing, including correction of non-horizontal images, image size standardization, and smoothing; input the chart images into the large language model to obtain the chart replacement text to be evaluated output by the large language model; manually create reference chart replacement text for the chart images, and manually evaluate the chart replacement text to be evaluated to obtain a manual evaluation score S:

[0058]

[0059] Among them, Wi represents the weight of each of the six key evaluation dimensions, s i Represents the manual evaluation scores of each dimension.

[0060] The manual evaluation criteria for each dimension are as follows:

[0061]

[0062]

[0063] In this embodiment, the standardization process is to standardize the manual evaluation scores in the data set, specifically:

[0064]

[0065] Among them, S normalized is the normalized manual evaluation score, S is the original manual evaluation score, and S min is the minimum value of the manual evaluation score; S max is the maximum value of the manual evaluation score. The feature concatenation includes: combining the standardized manual evaluation score with the text feature. Constructing a feature vector X, combining the embedding vector T of the replacement text of the chart to be evaluated and the embedding vector Q of the replacement text of the reference chart, as well as the standardized manual evaluation score S normalized Make stitching.

[0066] X=[T;Q;S normalized ]

[0067] The above charts and images are all from the real world and have practical significance. The Chinese charts are from the China Economic and Social Big Data Research Platform, Beijing Justice Bureau, and China Internet Network Information Center, and the English charts are from the Pew Research Center (https: / / www.pewresearch.org).

[0068] Step 3: Model selection and model post-processing: Select the GLM4-9B model as the basic model; in order to optimize the model's prediction results and make it better suited to specific application scenarios, we customized the model's original output. Use the ROUGE evaluation method to measure the n-gram overlap between the reference chart alternative text and the chart alternative text to be evaluated; at the same time, fully consider the chart reading preferences of visually impaired users, combined with the evaluation dimension weights in step 1. The prediction value E output by the model is as follows:

[0069]

[0070] Where R1 represents the n-gram overlap ratio between the alternative text to be evaluated and the reference alternative text; P i represents the ratio of the number of word vectors correctly described in the alternative text to be evaluated to the total number of word vectors in the alternative text to be evaluated; Wi represents the weight of each of the six key evaluation dimensions, r i Represents the recall rate, which is the cumulative ratio of the number of correctly described word vectors in each evaluation dimension of the alternative text to be evaluated to the total number of word vectors in each evaluation dimension of the reference alternative text. α and β are balance coefficients used to adjust the weight relationship between n-gram accuracy, precision and recall of key evaluation dimensions. In this embodiment, based on the optimal model training effect, the selected balance coefficient value 0<α+β<0.4.

[0071] Step 4: Finetune training and model performance tuning: Use the graph-to-text evaluation dataset to fine-tune the GLM4 model, and test and optimize the model performance of the trained migration model;

[0072] In this embodiment, the step 4 further includes customizing the loss function:

[0073] Define the loss function to make the model-predicted score Z as close as possible to the actual manual evaluation score S. The loss function is:

[0074]

[0075] Where L(θ) is the loss function, N is the number of samples in the dataset, Si is the actual human evaluation score of the i-th sample, Zi is the predicted score of the i-th sample, and θ is the model parameter.

[0076] In this embodiment, the step 4 also includes model training:

[0077] Based on the training set divided in step 2, the processed feature vector X in step 2 is used as the model input, and reasonable model training parameters are set to perform model training. The prediction value score of the original output of the model is calculated based on the customized model prediction value formula E in step 3. In this embodiment, the first training parameters are:

[0078] epochs=50; batch_size=20; learning_rate=2e -5 ;weight_decay=0.01

[0079] In this embodiment, the step 4 also includes model testing performance:

[0080] The performance of the migration model was evaluated using the validation set in the graph-to-text evaluation dataset, and the Pearson correlation coefficient was used to calculate the difference between the predicted score and the actual manual evaluation score. The correlation index in this embodiment is:

[0081]

[0082] Where ρ is the Pearson correlation coefficient; S i is the actual manual evaluation score of the ith sample, is the average of the actual manual evaluation scores; Z i is the predicted score of the ith sample, is the mean of the predicted scores; N is the number of samples.

[0083] In this embodiment, the step 4 also includes model performance tuning:

[0084] Try different optimization algorithms (such as Adam, SGD, etc.) and compare their effects. The closer the Pearson correlation coefficient ρ of the model is to 1, the better the training effect. Try different hyperparameter combinations, adjust the learning rate function, batch size, number of training rounds, regularization weights, compare the model training results of different hyperparameter combinations, and automatically save the model with the best training effect.

[0085] Step 5: Use the optimized model to evaluate the quality of the chart alternative text and obtain an evaluation score.

[0086] Corresponding to the aforementioned embodiment of a method for automatically evaluating the quality of text alternatives to diagrams based on a large language model, the present invention further provides an embodiment of a system for automatically evaluating the quality of text alternatives to diagrams based on a large language model.

[0087] According to another aspect of an embodiment of the present invention, the present invention specification further provides a system for automatically evaluating the quality of text alternatives to diagrams based on a large language model, comprising:

[0088] The dataset construction module is used to collect high-quality reference chart alternative texts, chart alternative texts to be evaluated, and manual evaluation scores to construct a chart alternative text evaluation dataset;

[0089] Data preprocessing module: preprocess the data set, including text cleaning, word segmentation, removal of special characters, standardization, and feature splicing; and divide the data set into training set, validation set, and test set for model training.

[0090] Model training module: GLM4-9B is selected as the basic model, and the model prediction value E is customized in combination with the reading preference of visually impaired users; the loss function L(θ) is customized, and the GLM4 model is trained with Finetune using the chart-based text evaluation dataset, and the model weights are continuously adjusted to minimize the loss function.

[0091] Model testing and performance tuning module: Use the graph to replace the validation set in the text evaluation dataset to evaluate the performance of the migration model, and use the correlation index to calculate the difference between the model prediction score and the actual manual evaluation score. Tune the migration model according to the model test performance, including: hyperparameter adjustment, optimization algorithm (such as Adam, SGD, etc.).

[0092] Corresponding to the aforementioned embodiment of a method for automatically evaluating the quality of text alternatives to diagrams based on a large language model, the present invention further provides an embodiment of an apparatus for automatically evaluating the quality of text alternatives to diagrams based on a large language model.

[0093] See also Figure 2 An automatic evaluation device for text replacement quality of a diagram based on a large language model is provided in an embodiment of the present invention, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement an automatic evaluation method for text replacement quality of a diagram based on a large language model in the above embodiment.

[0094] An embodiment of the automatic text quality assessment device for chart replacement based on a large language model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 2As shown, it is a hardware structure diagram of any device with data processing capability where a diagram replacement text quality automatic evaluation device based on a large language model provided by the present invention is located. Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0095] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0096] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0097] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for automatically evaluating the quality of text replacement for a diagram based on a large language model in the above embodiment is implemented.

[0098] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0099] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for automatically evaluating the quality of text replacement for a diagram based on a large language model is implemented.

[0100] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0101] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A method for automatically evaluating the quality of text alternatives to diagrams based on a large language model, characterized in that: The following steps are involved: S1. Determine the evaluation dimensions and weights of text alternatives for graphics based on the accessible graphics reading guidelines and the graphics reading needs of visually impaired users; S2, obtaining the chart to be evaluated for image preprocessing, constructing the chart to be evaluated alternative text, reference chart alternative text, and manual evaluation scores as a data set, and performing text data preprocessing on the text data of the data set; S3, perform model post-processing on the GLM4 model and customize the output prediction value of the model; Get a GLM4 model with preprocessed data as input and predicted values ​​defined in postprocessing as output; input the preprocessed data set into the postprocessing GLM4 model and perform fine-tuning training, test the trained migration model and optimize model performance; The output prediction value of the GLM4 model is calculated by the n-gram overlap between the reference chart alternative text and the chart alternative text to be evaluated, as well as the weights of each evaluation dimension; S4. Input the chart replacement text into the optimized GLM4 model to obtain the final evaluation score.

2. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 1, characterized in that: The S1 comprises the following sub-steps: Investigate the chart reading needs of visually impaired users, conduct user surveys, set the evaluation dimensions of chart alternative text to chart title, chart type, chart coordinate axis, chart specific value, color information description and chart trend analysis, and obtain the user importance score of each dimension through the survey results, and obtain the weight W of each dimension through the user importance score of each dimension i : Among them, A i is the mean of user importance scores for each dimension.

3. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 1, characterized in that: The construction of the chart replacement text to be evaluated, the reference chart replacement text, and the manual evaluation score specifically includes: the chart replacement text to be evaluated is obtained by inputting the chart image into a large language model; the reference chart replacement text is manually produced for the chart image based on the evaluation dimension, and the manual evaluation score is obtained by manually evaluating the chart replacement text to be evaluated.

4. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 2, characterized in that: The manual evaluation criteria specifically include: In the chart type dimension, one point is awarded for correctly giving the chart type; in the icon title dimension, one point is awarded for correctly describing the chart title; no point is awarded if there is no description or the chart title description is incomplete; in the chart axis dimension, 0.25 points are awarded for correctly describing the horizontal axis title, label range, vertical axis title, and label range; in the chart specific value dimension, the score = the number of data points in the chart drawing area that are correctly described / the total number of data points in the chart drawing area; in the color information description dimension, the score = the number of colors that are correctly described / the total number of colors in the chart; in the chart trend analysis dimension, 1 point is awarded for a relevant description of the data trend presented in the chart; no point is awarded if there is no description or the description is irrelevant to the chart data.

5. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 1, characterized in that: The image preprocessing includes correction of non-horizontal images, image size standardization, and smoothing; The text data preprocessing includes text cleaning, word segmentation, removal of special characters, standardization, and feature splicing; the feature splicing is to use a pre-trained word embedding model to convert the alternative text of the chart to be evaluated, the alternative text of the reference chart, and the manual evaluation score into a numerical vector representation, and splice them into a new feature vector; the standardization process is specifically: Among them, S normalized is the normalized manual evaluation score, S is the original manual evaluation score, and S min is the minimum value of the manual evaluation score; S max is the maximum value of the manual evaluation score.

6. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 5, characterized in that: The feature concatenation includes: combining the standardized manual evaluation score with the text feature; constructing a feature vector X, which includes the embedding vector T of the replacement text of the chart to be evaluated and the embedding vector Q of the replacement text of the reference chart, as well as the manual evaluation score S normalized ; X=[T;Q;S normalized ] Where ";" represents the vector concatenation operation.

7. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 1, characterized in that: The model post-processing is specifically as follows: Use the ROUGE evaluation method to measure the n-gram overlap between the reference chart alternative text and the chart alternative text to be evaluated, and combine the evaluation dimension weights to customize the output prediction value E of the model: Among them, R1 represents the n-gram precision, which indicates the n-gram overlap ratio between the alternative text to be evaluated and the reference alternative text; P i W represents the precision, which is the ratio of the number of word vectors correctly described in the alternative text to be evaluated to the total number of word vectors in the alternative text to be evaluated; i represents the weight of each key evaluation dimension, r i Recall is the sum of the number of correctly described word vectors in each evaluation dimension of the alternative text to be evaluated to the total number of word vectors in each evaluation dimension of the reference alternative text; α and β are balancing coefficients used to adjust the weight relationship between n-gram accuracy, precision, and recall of key evaluation dimensions.

8. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 1, characterized in that: The loss function used in the fine-tuning training is: Where L(θ) is the loss function, N is the number of samples in the dataset, Si is the actual human evaluation score of the i-th sample, Zi is the predicted score of the i-th sample, and θ is the model parameter.

9. The method for automatically evaluating the quality of text replacement for a diagram based on a large language model according to claim 1, characterized in that: The testing and model performance tuning include: The performance of the migration model was evaluated using the validation set in the graph-to-text evaluation dataset, and the difference between the predicted score and the actual manual evaluation score was calculated using the correlation index, which is: Where ρ is the Pearson correlation coefficient; S i is the actual manual evaluation score of the ith sample, is the average of the actual manual evaluation scores; Z i is the predicted score of the ith sample, is the average of the predicted scores; N is the number of samples; Tune the migration model based on the model test performance, including hyperparameter adjustment and algorithm optimization.

10. A system for implementing the method according to any one of claims 1 to 9, characterized in that: The system includes: a data set construction module, a data preprocessing module, a model training module, and a model testing and performance tuning module; The data set construction module is used to collect high-quality reference chart alternative texts, chart alternative texts to be evaluated, and manual evaluation scores to construct a chart alternative text evaluation data set; The data preprocessing module is used to preprocess the data set, including text cleaning, word segmentation, removal of special characters, standardization, and feature splicing; and divide the data set into a training set, a validation set, and a test set for model training; The model training module selects GLM4-9B as the basic model, and customizes the model prediction value E in combination with the reading preference of visually impaired users; customizes the loss function L(θ), uses the chart instead of the text evaluation data set to perform Finetune training on the GLM4 model, and continuously adjusts the model weights to minimize the loss function; The model testing and performance tuning module uses a graph to replace the validation set in the text evaluation data set to evaluate the performance of the migration model, and uses a correlation index to calculate the difference between the model prediction score and the actual manual evaluation score; the migration model is tuned according to the model test performance.

Images