Chart analysis model training method and device, storage medium and computer equipment

Abstract

The application discloses a chart analysis model training method and device, a storage medium and computer equipment, relates to the technical field of chart processing, and mainly pays more attention to numerical features in charts in the model training process, and clearly defines the optimization targets of a feature extraction module and a mathematical representation extraction module. The application comprises the following steps: constructing a preset initial chart analysis model, wherein the preset initial chart analysis model comprises an initial visual feature extraction module and an initial mathematical representation extraction module; obtaining a plurality of sample chart description texts with actual mathematical representation labels, and generating a plurality of sample charts corresponding to each sample chart description text; based on the plurality of sample charts corresponding to each sample chart description text, separately training the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction as a benchmark to obtain a preset chart analysis model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

Description

Technical Field

[0001] This invention relates to the field of chart processing technology, and in particular to a training method, apparatus, storage medium, and computer equipment for a chart parsing model. Background Technology

[0002] Charts are a visual representation of data, and analyzing charts can help us accurately extract key data from complex visual elements.

[0003] In current graph processing, analyzing the mathematical representations within graphs allows for a deeper understanding of the development trends, underlying mechanisms, and interrelationships of research objects, providing a solid foundation for theoretical construction. However, in traditional joint training models, the optimization objectives of the feature extraction module and the mathematical representation extraction module are not clearly distinguished. The mathematical representation extraction module may over-rely on textual semantics while neglecting precise numerical alignment. This training method struggles to meet the learning needs of different parts of the model, resulting in poor model training performance. Summary of the Invention

[0004] This invention provides a training method, apparatus, storage medium, and computer device for a chart analysis model. The main advantage lies in its ability to focus more on the numerical features in the chart during the model training process, clarify the optimization objectives of the feature extraction module and the mathematical representation extraction module, thereby improving the training effect of the chart analysis model.

[0005] According to a first aspect of the present invention, a method for training a graph parsing model is provided, comprising:

[0006] Construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module;

[0007] Obtain multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each of the sample chart description texts;

[0008] Based on multiple sample charts corresponding to the description text of each sample chart, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately with numerical extraction as the benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0009] Optionally, the method of training the initial visual feature extraction module separately based on numerical extraction includes:

[0010] N sets of data pairs are determined, wherein each set of data pairs consists of a first sample chart description text selected from multiple sample chart description texts and a first sample chart generated for the first sample chart description text;

[0011] The text feature vectors corresponding to the first sample chart description text in the N sets of data pairs are determined respectively, and the numerical visual feature vectors corresponding to the first sample chart in the N sets of data pairs are extracted respectively using the initial visual feature extraction module.

[0012] Calculate the same-group similarity between each text feature vector and the numerical visual feature vector in the same data pair, and calculate the different-group similarity between each text feature vector and the numerical visual feature vector in different data pairs.

[0013] Based on the same-group similarity and the different-group similarity, the parameters of the initial visual feature extraction module are adjusted to obtain the trained initial visual feature extraction module.

[0014] Optionally, the method of training the initial mathematical representation extraction module separately based on numerical extraction includes:

[0015] Select the second sample chart description text and its corresponding second sample chart required for training the initial mathematical representation extraction module from multiple sample chart description texts, and determine the first actual chart corresponding to the second sample chart description text;

[0016] The trained initial visual feature extraction module extracts the first image feature vector corresponding to the first actual chart and the second image feature vector corresponding to each second sample chart.

[0017] The first image feature vector and multiple second image feature vectors are fused to obtain a fused image feature vector. The fused image feature vector is then analyzed using the initial mathematical representation extraction module to obtain a predictive mathematical representation.

[0018] Based on the difference between the actual mathematical representation and the predicted mathematical representation corresponding to the second sample chart description text, the loss function corresponding to the initial mathematical representation extraction module is determined, and the initial mathematical representation extraction module is iteratively trained based on the loss function to obtain the trained initial mathematical representation extraction module.

[0019] Optionally, the plurality of second image feature vectors include at least the first sub-image feature vector and the second sub-image feature vector;

[0020] The process of fusing the first image feature vector and multiple second image feature vectors to obtain a fused image feature vector includes:

[0021] Based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, a feature matrix is ​​constructed, and the covariance matrix corresponding to the feature matrix is ​​determined.

[0022] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain multiple eigenvalues ​​and eigenvectors corresponding to each eigenvalue. Based on the magnitude of the eigenvalues, a preset number of eigenvalues ​​are selected as principal component eigenvalues ​​for each eigenvalue, and a projection matrix is ​​formed by the eigenvectors corresponding to each principal component eigenvalue.

[0023] Based on the projection matrix, the feature matrix is ​​projected to obtain a fused feature matrix, and based on the fused feature matrix, the fused image feature vector is determined.

[0024] Optionally, the step of fusing the first image feature vector and multiple second image feature vectors to obtain a fused image feature vector includes:

[0025] The first image feature vector, the first image sub-feature vector, and the second image sub-feature vector are subjected to feature-level fusion processing to obtain a feature fusion vector.

[0026] Element-level fusion processing is performed on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element-fused vector.

[0027] The first image feature vector, the first image sub-feature vector, and the second image sub-feature vector are subjected to low-order fusion processing to obtain a low-order fused vector.

[0028] The feature fusion vector, the element fusion vector, and the low-order fusion vector are transformed using a preset transformation function to obtain the fused image feature vector.

[0029] Optionally, generating multiple sample charts corresponding to each of the sample chart description texts includes:

[0030] Each sample chart description text is input into a preset chart generator to generate a chart, resulting in multiple sample charts corresponding to each sample chart description text.

[0031] Optionally, after training the initial visual feature extraction module and the initial mathematical representation extraction module separately based on multiple sample charts corresponding to each sample chart description text, using numerical extraction as a benchmark, to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module, the method further includes;

[0032] An initial lightweight chart parsing model is constructed, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module;

[0033] Obtain the lightweight chart description text required for lightweighting the preset chart parsing model, and generate multiple lightweight charts corresponding to the lightweight chart description text;

[0034] Based on the second actual chart corresponding to the lightweight chart description text and multiple lightweight charts, the differences in image feature extraction between the trained initial visual feature extraction module and the lightweight visual feature extraction module, as well as the differences in mathematical representation analysis between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module are determined.

[0035] Based on the differences in image feature extraction, a chart feature loss function is determined, and based on the differences in mathematical representation analysis, a mathematical representation loss function is determined.

[0036] Based on the chart feature loss function and the mathematical representation loss function, a joint loss function is determined, and based on the joint loss function, the initial lightweight chart parsing model is iteratively trained to obtain a lightweight chart parsing model.

[0037] Optionally, based on the second actual chart corresponding to the lightweight chart description text and multiple lightweight charts, the differences in image feature extraction between the trained initial visual feature extraction module and the lightweight visual feature extraction module, as well as the differences in mathematical representation analysis between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module, are determined, including:

[0038] The trained initial visual feature extraction module extracts the first weighted image feature vector corresponding to the second actual chart and the second weighted image feature vector corresponding to multiple lightweight charts, respectively. The first weighted image feature vector and the multiple second weighted image feature vectors are then fused to obtain a weighted image fusion feature vector. The lightweight visual feature extraction module extracts the lightweight chart feature vector corresponding to the second actual chart, and the difference between the lightweight chart feature vector and the weighted image fusion feature vector is taken as the image feature extraction difference.

[0039] The initial mathematical representation extraction module, after training, performs graph parsing on the weighted image fusion feature vector to obtain a weighted graph mathematical representation. The lightweight mathematical representation extraction module then performs graph parsing on the lightweight graph feature vector to obtain a lightweight graph mathematical representation. The difference between the weighted graph mathematical representation and the lightweight graph mathematical representation is taken as the mathematical representation parsing difference.

[0040] Optionally, based on multiple sample charts corresponding to each sample chart description text, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately using numerical extraction as a benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module. The method further includes:

[0041] Get the chart to be parsed;

[0042] The chart to be parsed is input into the preset chart parsing model. The trained initial visual feature extraction module extracts numerical features from the chart to be parsed to obtain a numerical visual feature vector. The trained initial mathematical representation extraction module then performs chart parsing on the numerical visual feature vector to obtain the mathematical representation of the chart to be parsed.

[0043] According to a second aspect of the present invention, a training apparatus for a graph parsing model is provided, comprising:

[0044] A construction unit is used to construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module;

[0045] The generation unit is used to acquire multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each of the sample chart description texts;

[0046] The training unit is used to train the initial visual feature extraction module and the initial mathematical representation extraction module separately based on multiple sample charts corresponding to each sample chart description text, using numerical extraction as a benchmark, to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0047] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method for the above-described graph analysis model.

[0048] According to a fourth aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for training the graph analysis model.

[0049] According to the training method, apparatus, storage medium, and computer equipment of the graph parsing model provided by the present invention, compared with the current traditional joint training mode, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module separately by using multiple sample graph description texts with actual mathematical representation labels. This enables training based on the respective optimization objectives of the initial visual feature extraction module and the initial mathematical representation extraction module, allowing for independent adjustment of the optimization strategies of the feature extraction module and the mathematical representation extraction module. This avoids the gradient vanishing problem caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module during joint training. Therefore, the present invention can improve the graph parsing model. Training effect; Furthermore, this invention trains the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, enabling these modules to focus more on the data essence in the charts during training, thereby improving the accuracy of the trained model in extracting mathematical representations from the charts. Simultaneously, this invention utilizes multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training with multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thus exhibiting better generalization ability when faced with unseen chart types. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0051] Figure 1 A flowchart illustrating a training method for a graph parsing model provided in an embodiment of the present invention is shown.

[0052] Figure 2 This invention provides a schematic diagram illustrating multiple types of charts corresponding to the same chart description text.

[0053] Figure 3 A flowchart illustrating another training method for a graph parsing model provided in an embodiment of the present invention is shown.

[0054] Figure 4 This diagram illustrates a process for training a visual feature extraction module according to an embodiment of the present invention.

[0055] Figure 5 This diagram illustrates a process for training a mathematical representation extraction module according to an embodiment of the present invention.

[0056] Figure 6 This diagram illustrates a process for lightweighting a preset chart parsing model according to an embodiment of the present invention.

[0057] Figure 7 This diagram illustrates a chart parsing method provided by an embodiment of the present invention.

[0058] Figure 8 This diagram illustrates the structure of a training device for a graph parsing model provided in an embodiment of the present invention.

[0059] Figure 9 This diagram illustrates the structure of a training device for another graph parsing model provided in an embodiment of the present invention.

[0060] Figure 10 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0061] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0062] Currently, in the traditional joint training model, the optimization objectives of the feature extraction module and the mathematical representation extraction module are not clearly distinguished. The mathematical representation extraction module may over-rely on text semantics and ignore the precise alignment of numerical values. This training method is difficult to meet the learning needs of each part of the model, resulting in poor model training performance.

[0063] To address the aforementioned problems, embodiments of the present invention provide a training method for a graph parsing model, such as... Figure 1 As shown, the method includes:

[0064] 101. Construct a preset initial chart parsing model, which includes an initial visual feature extraction module and an initial mathematical representation extraction module.

[0065] In this invention, the initial visual feature extraction module can be an initial visual encoder, and the initial mathematical representation extraction module can be an initial decoder. In this embodiment, a preset initial chart parsing model based on the initial visual feature extraction module and the initial mathematical representation extraction module is first constructed. Then, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately. This ensures that the initial visual feature extraction module and the initial mathematical representation extraction module are trained according to their respective optimization objectives, avoiding the gradient vanishing problem caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module when they are jointly trained. This ensures that both the feature extraction module and the mathematical representation extraction module can meet their respective optimization objectives, thereby improving the training effect of the chart parsing model in this embodiment.

[0066] 102. Obtain multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each sample chart description text.

[0067] In this embodiment of the invention, the actual mathematical representation label refers to the real data content (mathematical text content) contained in the chart; the sample chart description text refers to the mathematical description text of the chart, and the format can be set according to needs, such as "|age|number of people|\n|13|4|\n|14|5|\n|15|4|\n|16|3|"; various sample charts include table charts, bar charts, column charts, scatter charts, pie charts, etc. For example, if the description text of a sample chart is: "In a certain class, the number of people aged 13 is 4, the number of people aged 14 is 5, the number of people aged 15 is 4, and the number of people aged 1 / 6 is 3", then the different types of sample charts corresponding to this description text are as follows: Figure 2 As shown.

[0068] In this embodiment of the invention, in order to enable the chart parsing model to learn the commonalities and differences between different types of charts, thereby increasing the generalization ability of the chart parsing model on unseen chart types, when training the chart parsing model, it is first necessary to generate multiple types of sample charts for the same sample chart description text. Based on this, the method includes: inputting each sample chart description text into a preset chart generator to generate charts, thereby obtaining multiple sample charts corresponding to each sample chart description text.

[0069] Specifically, to improve the chart generation accuracy of the preset chart generator, it is first necessary to train and construct the preset chart generator. Based on this, the method includes: constructing a preset initial chart generator and obtaining a sample dataset, wherein the sample dataset includes chart description text and various types of actual charts corresponding to the chart description text; dividing the sample dataset into a training set and a test set; training the preset initial chart generator using the training set; testing the trained preset initial chart generator using the test set; and selecting the preset initial chart generator that meets the test conditions as the preset chart generator.

[0070] The above embodiments provide a method for constructing and training a preset chart generator. First, a preset initial chart generator is constructed. Second, a sample dataset is downloaded from an official website or a specified data source. Ensure the sample dataset contains all necessary files, including chart description text and various types of actual charts corresponding to that text. Convert the files to a format that the preset initial chart generator can understand. Finally, train and test the generator. Specifically, the dataset can be divided first: using random or specific strategies (such as stratified sampling), the dataset is divided into a training set and a test set. The model is then trained using the training set, and metrics such as loss value and mAP are monitored during training to evaluate generator performance. Training parameters, such as learning rate, optimizer, and regularization, are adjusted as needed to optimize training effectiveness. Finally, the generator is tested: the trained generator is tested using the test set to evaluate its performance on unseen data. Metrics such as mAP, precision, and recall on the test set are calculated and recorded. If the generator performance does not meet requirements, the training phase can be returned for further iterations or adjustments. Finally, a preset chart generator that meets the requirements is obtained, and this preset chart generator is used to generate various types of sample charts based on the sample chart description text. This embodiment of the invention uses a preset chart generator to generate various types of charts, which can improve the efficiency and accuracy of chart generation.

[0071] 103. Based on the multiple sample charts corresponding to the description text of each sample chart, train the initial visual feature extraction module and the initial mathematical representation extraction module separately with numerical extraction as the benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0072] In this embodiment of the invention, a first sample chart description text and its corresponding multiple first sample charts are extracted from each sample chart description text to train the initial visual feature extraction module. Simultaneously, a second sample chart description text and its corresponding multiple second sample charts are extracted from each sample chart description text to train the initial mathematical representation extraction module. Then, the initial visual feature extraction module is trained using the first sample chart description text and its corresponding multiple first sample charts based on numerical feature extraction, and the initial mathematical representation extraction module is trained using the second sample chart description text and its corresponding multiple second sample charts based on numerical extraction, thereby obtaining the chart parsing model. This embodiment of the invention, by independently training the feature extraction module and the mathematical representation extraction module, enables each module to be independently optimized according to its respective task requirements. Therefore, both the trained feature extraction module and the mathematical representation extraction module satisfy their respective optimization objectives, avoiding the problem that the optimization objectives of the feature extraction module and the mathematical representation extraction module may conflict during joint training, leading to difficulties in the model converging to the optimal solution. Furthermore, by training the feature extraction module and the mathematical representation extraction module based on numerical extraction, this embodiment of the invention enables the trained chart parsing model to focus more on the data essence within the chart, thereby improving the training effect of the chart parsing model. Additionally, by adding more diverse chart features, this embodiment of the invention enhances the decoding capability of the mathematical representation extraction module.

[0073] According to the training method of the chart parsing model provided by the present invention, compared with the current traditional joint training mode, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module separately by using multiple sample chart description texts with actual mathematical representation labels. This enables training based on the optimization objectives of the initial visual feature extraction module and the initial mathematical representation extraction module, and allows for separate adjustment of the optimization strategies of the feature extraction module and the mathematical representation extraction module. This avoids the gradient vanishing problem caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module in joint training, thereby improving the training effect of the chart parsing model. At the same time, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, which enables the initial visual feature extraction module and the initial mathematical representation extraction module to pay more attention to the data essence in the chart during the training process, thereby improving the accuracy of the trained model in extracting mathematical representations from the chart. Furthermore, the present invention uses multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training with multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thereby exhibiting better generalization ability when faced with unseen chart types.

[0074] Furthermore, to better illustrate the process of training the chart parsing model described above, and as a refinement and extension of the above embodiments, this invention provides another method for training a chart parsing model, such as... Figure 3 As shown, the method includes:

[0075] 201. Construct a preset initial chart parsing model, which includes an initial visual feature extraction module and an initial mathematical representation extraction module.

[0076] 202. Obtain multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each sample chart description text.

[0077] 203. Based on the multiple sample charts corresponding to the description text of each sample chart, train the initial visual feature extraction module and the initial mathematical representation extraction module separately with numerical extraction as the benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0078] In an embodiment of the present invention, the method for training the initial visual feature extraction module based solely on numerical extraction includes: determining N sets of data pairs, wherein each set of data pairs consists of a first sample chart description text selected from multiple sample chart description texts and a first sample chart generated for the first sample chart description text; determining the text feature vectors corresponding to the first sample chart description texts in the N sets of data pairs respectively, and extracting the numerical visual feature vectors corresponding to the first sample charts in the N sets of data pairs respectively using the initial visual feature extraction module; calculating the same-group similarity between each text feature vector and the numerical visual feature vectors in the same set of data pairs respectively, and calculating the different-group similarity between each text feature vector and the numerical visual feature vectors in different sets of data pairs respectively; adjusting the parameters of the initial visual feature extraction module based on the same-group similarity and the different-group similarity to obtain the trained initial visual feature extraction module.

[0079] Specifically, one sample chart description text is randomly selected from multiple sample chart description texts as the first sample chart description text. Then, one sample chart is selected from the various types of sample charts generated based on this first sample chart description text as the first sample chart. The first sample chart description text and the first sample chart constitute a data pair. Thus, following the above method, N data pairs can be determined. Then, proceed as follows... Figure 4The method shown trains the initial visual feature extraction module (initial visual encoder) based on numerical feature extraction. Specifically, it uses a pre-trained text feature extraction module to extract text feature vectors T1, T2, ..., T from the first sample chart description text in N data pairs. N And using the initial visual feature extraction module, extract the numerical visual feature vectors I1, I2, ..., I from the first sample chart in each of the N data pairs. N Calculate the similarity between the text feature vectors and numerical visual feature vectors in the same data pair, i.e., the similarity within the same pair I1·T1, I2·T2, ..., I... N ·T N Simultaneously, the similarity between the text feature vector in each data pair and the numerical visual feature vector in other data pairs is calculated, i.e., the inter-group similarity I1·T2, I1·T3, ..., I1·T N ;I2·T1, I2·T3,..., I2·T N ;I3·T1, I3·T2,..., I3·T N ;....;I N ·T1、I N ·T2、...、I N ·T N-1 If at least one of the similarity scores within the same group or between different groups fails to meet the requirements (i.e., the similarity score within the same group is less than the first preset similarity threshold, or the similarity score between different groups is greater than the second preset similarity threshold, where both the first and second preset similarity thresholds are set according to actual needs), the parameters of the initial visual feature extraction module are adjusted until both the similarity scores within the same group and between different groups meet the requirements (i.e., the similarity score within the same group is greater than or equal to the first preset similarity threshold, and the similarity score between different groups is less than or equal to the second preset similarity threshold), at which point the adjustment of the parameters of the initial visual feature extraction module stops. This embodiment of the invention trains the initial visual feature extraction module based on numerical feature extraction, enabling the trained feature extraction module to focus more on the essence of the data in the chart, thereby improving the accuracy of numerical feature extraction from the chart.

[0080] In an embodiment of the present invention, the method for training the initial mathematical representation extraction module based solely on numerical extraction includes: selecting a second sample chart description text and its corresponding second sample chart from multiple sample chart description texts for training the initial mathematical representation extraction module, and determining a first actual chart corresponding to the second sample chart description text; extracting a first image feature vector corresponding to the first actual chart and a second image feature vector corresponding to each second sample chart through the trained initial visual feature extraction module; fusing the first image feature vector and multiple second image feature vectors to obtain a fused image feature vector, and performing chart parsing on the fused image feature vector through the initial mathematical representation extraction module to obtain a predicted mathematical representation; determining a loss function corresponding to the initial mathematical representation extraction module based on the difference between the actual mathematical representation corresponding to the second sample chart description text and the predicted mathematical representation, and iteratively training the initial mathematical representation extraction module based on the loss function to obtain the trained initial mathematical representation extraction module.

[0081] Wherein, the first actual chart refers to the accurate chart corresponding to the description text of the second sample chart. The first actual chart can be any type of chart, and this embodiment of the invention does not specifically limit it. Specifically, several texts are randomly selected from multiple sample chart description texts as the second sample chart description text, and various types of sample charts generated for the second sample chart description text are used as the second sample chart. Then, according to... Figure 5 The method shown trains the initial mathematical representation extraction module (decoder) based on numerical extraction. That is, it first utilizes the trained initial visual feature extraction module (decoder) described above. Figure 5A visual encoder (with snowflakes in the image) extracts a first image feature vector with numerical features from the first actual chart, and extracts a second image feature vector with numerical features corresponding to each second sample chart. If the second sample chart includes a first sub-chart and a second sub-chart, the second image feature vector includes the first sub-image feature vector (image feature vector 1) corresponding to the first sub-chart and the second sub-image feature vector (image feature vector 2) corresponding to the second sub-chart. Then, the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector are fused. Thus, multiple fused image feature vectors corresponding to the description text of the second sample charts can be obtained in the above manner. The fused image feature vectors corresponding to the description text of the multiple second sample charts and the actual mathematical representation labels constitute a training set. The initial mathematical representation extraction module (the mathematical representation extraction module with flames in the figure) is iteratively trained using this training set. During the training process, the fused image feature vectors are used as input data and the actual mathematical representation labels are used as output data. The loss value, mAP, and other indicators of the initial mathematical representation extraction module are monitored during the training process to evaluate the performance of the mathematical representation extraction module. Training parameters, such as learning rate, optimizer, and regularization, are adjusted as needed to optimize training performance, ultimately resulting in a mathematical representation extraction module with the required accuracy. It should be noted that the second image feature vector in this embodiment is not limited to image feature vector 1 and image feature vector 2, and may include more feature vectors. The number of vectors is not specifically limited in this embodiment.

[0082] In the above embodiments, the method for fusing the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector includes: constructing a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determining the covariance matrix corresponding to the feature matrix; performing eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and eigenvectors corresponding to each eigenvalue; selecting a preset number of eigenvalues ​​as principal component eigenvalues ​​based on the magnitude of the eigenvalues, and constructing a projection matrix from the eigenvectors corresponding to each principal component eigenvalue; projecting the feature matrix based on the projection matrix to obtain a fused feature matrix, and determining the fused image feature vector based on the fused feature matrix.

[0083] Specifically, for example, if the feature vector of the first image is f1(a1,b1), the feature vector of the first sub-image is f2(a2,b2), and the feature vector of the second sub-image is f3(a3,b3), then the constructed feature matrix X is as follows:

[0084]

[0085] Next, for each column of the feature matrix X, calculate its mean and standard deviation. Divide the difference between each element of the feature matrix X and its corresponding mean by its standard deviation to obtain each standardized element of the feature matrix X. This yields the standardized matrix Z. Then, determine the covariance matrix C of matrix Z as follows:

[0086]

[0087] Here, m represents the total number of image feature vectors. Further, the covariance matrix is ​​decomposed into eigenvalues ​​to obtain multiple eigenvalues ​​and their corresponding eigenvectors. The eigenvalues ​​are then sorted in ascending order. The first k eigenvalues ​​are selected from the sorted eigenvalues, and their corresponding eigenvectors form a projection matrix P. Based on the projection matrix P, the standardized matrix Z is projected onto the principal component space to obtain the fused feature matrix Y, where Y = ZP. Finally, each element in the fused feature matrix Y constitutes the fused image feature vector.

[0088] In another embodiment of the present invention, in order to perform fusion processing on multiple image feature vectors, the method further includes: performing feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector; performing element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element fusion vector; performing low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector; and performing transformation processing on the feature fusion vector, the element fusion vector, and the low-order fusion vector using a preset transformation function to obtain the fused image feature vector.

[0089] Specifically, if the first image feature vector is (a1, a2), the first sub-image feature vector is (b1, b2), and the second sub-image feature vector is (c1, c2), the specific fusion processing method includes: performing feature-level fusion between different feature vectors, that is, performing a Hadamard product on all elements of the vectors, and then performing a convolution transformation under a certain weight w1 to obtain the feature fusion vector f(w1*(a1*b1*c1, a2*b2*c2, a3*b3*c3)); at the same time, performing element-level fusion on all feature vector data, that is, performing a Hadamard product on each element of the vectors. Then, different weight values ​​are assigned to the results of each product, followed by a linear transformation, resulting in an element-wise fused vector of f(w2*a1*b1*c1,w3*a2*b2*c2,w4*a3*b3*c3). Furthermore, all feature vectors undergo low-order fusion processing, and weight coefficients are assigned to the fused results, followed by another linear transformation, resulting in a low-order fused vector of f(w5(a1,a2,b1,b2,c1,c2)). Finally, a preset transformation function is used to transform the above feature fused vector, element-wise fused vector, and low-order fused vector to obtain the fused image feature vector. The preset transformation function is determined according to actual needs, and this invention does not impose specific limitations on it. It should be noted that the above examples are merely illustrative and do not limit the embodiments of this application.

[0090] In another embodiment of the present invention, multiple image feature vectors (f) can also be processed based on an average pooling layer (avg_pool) and a self-attention mechanism (self_attn). 1, The images f2 and f3 are fused together, and the resulting image features T is T = avg_pool(self_atten([f 1, f2,f3])).

[0091] This invention, through the fusion processing of multiple image feature vectors, can automatically or explicitly combine different features to generate new feature combinations. These combined features may contain complex nonlinear relationships between the original features, enabling the mathematical representation extraction module to capture more refined and richer information in the data. In other words, it can make full use of the relationships between various data, extract more latent features, and improve the training effect of the subsequent mathematical representation extraction module.

[0092] Furthermore, to reduce the energy consumption and computational resources of the chart parsing model during chart parsing, it is necessary to lightweight the preset chart parsing model. Based on this, the method includes: constructing an initial lightweight chart parsing model, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module; obtaining lightweight chart description text required for lightweighting the preset chart parsing model, and generating multiple lightweight charts corresponding to the lightweight chart description text; based on the second actual chart corresponding to the lightweight chart description text and the multiple lightweight charts, determining the image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and the mathematical representation parsing difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module; determining a chart feature loss function based on the image feature extraction difference, and determining a mathematical representation loss function based on the mathematical representation parsing difference; determining a joint loss function based on the chart feature loss function and the mathematical representation loss function, and iteratively training the initial lightweight chart parsing model based on the joint loss function to obtain the lightweight chart parsing model. The method for determining the differences in image feature extraction and mathematical representation analysis includes: extracting a first weighted image feature vector corresponding to the second actual chart and a second weighted image feature vector corresponding to multiple lightweight charts through the trained initial visual feature extraction module; fusing the first weighted image feature vector and the multiple second weighted image feature vectors to obtain a weighted image fusion feature vector; extracting a lightweight chart feature vector corresponding to the second actual chart through the lightweight visual feature extraction module; and using the difference between the lightweight chart feature vector and the weighted image fusion feature vector as the image feature extraction difference; performing chart analysis on the weighted image fusion feature vector through the trained initial mathematical representation extraction module to obtain a weighted chart mathematical representation; performing chart analysis on the lightweight chart feature vector through the lightweight mathematical representation extraction module to obtain a lightweight chart mathematical representation; and using the difference between the weighted chart mathematical representation and the lightweight chart mathematical representation as the mathematical representation analysis difference.

[0093] The lightweight visual feature extraction module can be a lightweight visual encoder, and the lightweight mathematical representation extraction module can be a lightweight decoder. Specifically, a lightweight sample dataset is acquired, which includes chart description text (chart mathematical text), i.e., lightweight chart description text. A preset chart generator is used to generate multiple types of charts for each lightweight chart description text, i.e., lightweight charts, and the accurate second actual chart corresponding to each lightweight chart description text is determined. Then, according to... Figure 6The method shown performs lightweight processing on the preset chart parsing model, wherein, Figure 6 The visual feature extraction module (visual encoder) and mathematical representation extraction module (decoder) marked with snowflakes are components of the preset chart parsing model. The visual feature extraction module 1 (visual encoder 1) and mathematical representation extraction module 1 (decoder 1) marked with flames are components of the initial lightweight chart parsing model. Specifically, the visual feature extraction module 1 marked with flames is a lightweight visual feature extraction module, and the mathematical representation extraction module 1 marked with flames is a lightweight mathematical representation extraction module. During model lightweighting, the parameters of the visual feature extraction module and mathematical representation extraction module in the preset chart parsing model remain fixed. Training is only performed on the parameters of the visual feature extraction module 1 and mathematical representation extraction module 1 in the initial lightweight chart parsing model. Multiple image feature vectors extracted by the visual feature extraction module in the preset chart parsing model for the same chart description text are fused according to the aforementioned feature fusion method. The difference between the fused image features and the image features extracted by the lightweight visual feature extraction module is determined, and the chart feature loss function is determined based on this difference. Furthermore, for the same lightweight chart description text, the image features extracted by the lightweight visual feature extraction module are input into the lightweight mathematical representation extraction module. Simultaneously, the image features extracted by the visual feature extraction module in the preset chart parsing model are input into its corresponding mathematical representation extraction module. Then, based on the difference between the output of the lightweight mathematical representation extraction module and the output of the mathematical representation extraction module in the preset chart parsing model, a mathematical representation loss function is determined. Next, the weight coefficients corresponding to the chart feature loss function and the mathematical representation loss function are determined separately. Based on these weight coefficients, the chart feature loss function and the mathematical representation loss function are added together to obtain a joint loss function. Finally, the initial lightweight chart parsing model is iteratively trained using the joint loss function. During training, the differences between the image features and mathematical representations extracted by the initial lightweight chart parsing model and those extracted by the preset chart parsing model are monitored until the differences meet the requirements (e.g., less than a certain value), at which point iterative training stops. Finally, the initial lightweight chart parsing model at the end of training is determined as the final model for chart parsing. By employing visual loss and output prediction loss, a single image is input. The visual feature extraction module 1 and the mathematical representation extraction module 1 also possess the ability to encode and decode diverse statistical graph features. This embodiment of the invention, through lightweight model processing, can save computational resources in the graph parsing process, reduce energy consumption, improve graph parsing efficiency, and reduce the complexity of graph parsing.

[0094] It should be noted that the embodiments of the present invention do not impose specific limitations on the functional forms of the chart feature loss function and the mathematical representation loss function. For example, the chart feature loss function can be the mean absolute error loss function, and the mathematical representation loss function can be the binary cross-entropy loss function.

[0095] 204. Obtain the chart to be parsed.

[0096] 205. Input the chart to be parsed into the preset chart parsing model. The initial visual feature extraction module after training extracts the numerical features of the chart to be parsed to obtain the numerical visual feature vector. The initial mathematical representation extraction module after training performs chart parsing on the numerical visual feature vector to obtain the mathematical representation of the chart to be parsed.

[0097] Specifically, after constructing the preset chart parsing model, such as Figure 7 As shown, the initial visual feature extraction module after training is... Figure 7 The visual encoder 1 in the model, and the initial mathematical representation extraction module after training are: Figure 7 Decoder 1 in the model. To parse a chart, the chart is directly input into a preset chart parsing model, which outputs the mathematical representation of the chart. Since the preset chart parsing model is trained separately on the visual feature extraction module and the mathematical representation extraction module using numerical extraction as a benchmark, it focuses more on the data essence within the chart, improving parsing accuracy. Furthermore, the visual feature extraction module and the mathematical representation extraction module can serve as a base model; the input and output data can be changed subsequently, and after fine-tuning the model, different applications can be implemented, such as chart understanding and chart question answering. Additionally, after outputting the mathematical representation through the preset chart parsing model, this embodiment of the invention can also redraw statistical charts based on this mathematical representation.

[0098] According to another training method for a chart parsing model provided by this invention, compared with the current traditional joint training mode, this invention trains the initial visual feature extraction module and the initial mathematical representation extraction module separately using multiple sample chart description texts with actual mathematical representation labels. This enables training based on the respective optimization objectives of the initial visual feature extraction module and the initial mathematical representation extraction module, allowing for separate adjustments to the optimization strategies of the feature extraction module and the mathematical representation extraction module. This avoids the gradient vanishing problem caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module during joint training, thereby improving the training effect of the chart parsing model. Simultaneously, by training the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, this invention enables the initial visual feature extraction module and the initial mathematical representation extraction module to focus more on the data essence in the chart during training, thereby improving the accuracy of the trained model in extracting mathematical representations from the chart. Furthermore, this invention utilizes multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training with multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thus exhibiting better generalization ability when faced with unfamiliar chart types.

[0099] Furthermore, as Figure 1 In specific implementation, embodiments of the present invention provide a training device for a graph parsing model, such as... Figure 8 As shown, the device includes: a construction unit 31, a generation unit 32, and a training unit 33.

[0100] The construction unit 31 can be used to construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module.

[0101] The generation unit 32 can be used to obtain multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each sample chart description text.

[0102] The training unit 33 can be used to train the initial visual feature extraction module and the initial mathematical representation extraction module separately based on numerical extraction, according to multiple sample charts corresponding to each sample chart description text, to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0103] In specific application scenarios, in order to train the initial visual feature extraction module separately, such as Figure 9As shown, the training unit 33 includes a determination module 331, a feature extraction module 332, a calculation module 333, and a parameter adjustment module 334.

[0104] The determining module 331 can be used to determine N sets of data pairs, wherein each set of data pairs consists of a first sample chart description text selected from multiple sample chart description texts and a first sample chart generated for the first sample chart description text.

[0105] The feature extraction module 332 can be used to determine the text feature vector corresponding to the first sample chart description text in the N sets of data pairs, and use the initial visual feature extraction module to extract the numerical visual feature vector corresponding to the first sample chart in the N sets of data pairs.

[0106] The calculation module 333 can be used to calculate the same-group similarity between each text feature vector and the numerical visual feature vector in the same group of data pairs, and to calculate the different-group similarity between each text feature vector and the numerical visual feature vector in different groups of data pairs.

[0107] The parameter adjustment module 334 can be used to adjust the parameters of the initial visual feature extraction module based on the same-group similarity and the different-group similarity to obtain the trained initial visual feature extraction module.

[0108] In specific application scenarios, in order to train the initial mathematical representation extraction module separately, the training unit 33 also includes a feature fusion module 335 and a training module 336.

[0109] The determining module 331 can be used to select, from multiple sample chart description texts, the second sample chart description text required for training the initial mathematical representation extraction module and its corresponding second sample chart, and to determine the first actual chart corresponding to the second sample chart description text.

[0110] The feature extraction module 332 can also be used to extract the first image feature vector corresponding to the first actual chart and the second image feature vector corresponding to each second sample chart through the trained initial visual feature extraction module.

[0111] The feature fusion module 335 can be used to fuse the first image feature vector and multiple second image feature vectors to obtain a fused image feature vector, and to perform graph analysis on the fused image feature vector through the initial mathematical representation extraction module to obtain a predictive mathematical representation.

[0112] The training module 336 can be used to determine the loss function corresponding to the initial mathematical representation extraction module based on the difference between the actual mathematical representation and the predicted mathematical representation corresponding to the second sample chart description text, and to iteratively train the initial mathematical representation extraction module based on the loss function to obtain the trained initial mathematical representation extraction module.

[0113] In specific application scenarios, multiple second image feature vectors include at least a first sub-image feature vector and a second sub-image feature vector. To fuse the first image feature vector and multiple second image feature vectors, the feature fusion module 335 can specifically be used to construct a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determine the covariance matrix corresponding to the feature matrix; perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and eigenvectors corresponding to each eigenvalue; select a preset number of eigenvalues ​​as principal component eigenvalues ​​based on the magnitude of the eigenvalues, and construct a projection matrix using the eigenvectors corresponding to each principal component eigenvalue; project the feature matrix based on the projection matrix to obtain a fused feature matrix, and determine the fused image feature vector based on the fused feature matrix.

[0114] In specific application scenarios, in order to fuse the first image feature vector and multiple second image feature vectors, the feature fusion module 335 can specifically be used to perform feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector; perform element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element fusion vector; perform low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector; and use a preset transformation function to transform the feature fusion vector, the element fusion vector, and the low-order fusion vector to obtain the fused image feature vector.

[0115] In specific application scenarios, in order to generate multiple sample charts corresponding to each sample chart description text, the generation unit 32 can be used to input each sample chart description text into a preset chart generator to generate charts, thereby obtaining multiple sample charts corresponding to each sample chart description text.

[0116] In specific application scenarios, in order to perform lightweight processing on the preset chart parsing model, the device further includes: a lightweight unit 34.

[0117] The lightweight unit 34 can be used to construct an initial lightweight chart parsing model, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module; obtain lightweight chart description text required for lightweighting the preset chart parsing model, and generate multiple lightweight charts corresponding to the lightweight chart description text; based on the second actual chart corresponding to the lightweight chart description text and the multiple lightweight charts, determine the image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and the mathematical representation parsing difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module; based on the image feature extraction difference, determine a chart feature loss function, and based on the mathematical representation parsing difference, determine a mathematical representation loss function; based on the chart feature loss function and the mathematical representation loss function, determine a joint loss function, and based on the joint loss function, iteratively train the initial lightweight chart parsing model to obtain the lightweight chart parsing model.

[0118] In specific application scenarios, to extract differences in image feature extraction and mathematical representation analysis, the lightweight unit 34 can be specifically used to extract the first weighted image feature vector corresponding to the second actual chart and the second weighted image feature vector corresponding to multiple lightweight charts through the trained initial visual feature extraction module, and to fuse the first weighted image feature vector and the multiple second weighted image feature vectors to obtain a weighted image fusion feature vector. The lightweight visual feature extraction module then extracts the lightweight chart feature vector corresponding to the second actual chart, and the difference between the lightweight chart feature vector and the weighted image fusion feature vector is used as the image feature extraction difference. The trained initial mathematical representation extraction module performs chart analysis on the weighted image fusion feature vector to obtain a weighted chart mathematical representation, and the lightweight mathematical representation extraction module performs chart analysis on the lightweight chart feature vector to obtain a lightweight chart mathematical representation. The difference between the weighted chart mathematical representation and the lightweight chart mathematical representation is used as the mathematical representation analysis difference.

[0119] In specific application scenarios, in order to parse the charts, the device further includes a chart parsing unit 35.

[0120] The chart parsing unit 35 can be used to obtain a chart to be parsed; input the chart to be parsed into the preset chart parsing model, extract numerical features from the chart to be parsed through the trained initial visual feature extraction module to obtain a numerical visual feature vector, and perform chart parsing on the numerical visual feature vector through the trained initial mathematical representation extraction module to obtain the mathematical representation of the chart to be parsed.

[0121] It should be noted that other corresponding descriptions of the functional modules involved in the training device for a graph parsing model provided in this embodiment of the invention can be found in [reference]. Figure 1 The corresponding description of the method shown will not be repeated here.

[0122] Based on the above, Figure 1 Accordingly, this embodiment of the invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the following steps: constructing a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; acquiring multiple sample chart description texts with actual mathematical representation labels, and generating multiple sample charts corresponding to each sample chart description text; based on the multiple sample charts corresponding to each sample chart description text, separately training the initial visual feature extraction module and the initial mathematical representation extraction module with numerical extraction as a benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0123] Based on the above, Figure 1 The method shown and as Figure 8 The embodiment of the device shown in the invention also provides a physical structure diagram of a computer device, such as... Figure 10 As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor. Both the memory 42 and the processor 41 are mounted on a bus 43. When the processor 41 executes the program, it performs the following steps: constructing a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; acquiring multiple sample chart description texts with actual mathematical representation labels, and generating multiple sample charts corresponding to each sample chart description text; based on the multiple sample charts corresponding to each sample chart description text, separately training the initial visual feature extraction module and the initial mathematical representation extraction module using numerical extraction as a benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0124] Through the technical solution of this invention, the initial visual feature extraction module and the initial mathematical representation extraction module are trained independently using multiple sample chart description texts with actual mathematical representation labels. This allows training based on the respective optimization objectives of the initial visual feature extraction module and the initial mathematical representation extraction module, and the optimization strategies of the feature extraction module and the mathematical representation extraction module can be adjusted separately. This avoids the gradient vanishing problem caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module during joint training, thereby improving the training effect of the chart parsing model. At the same time, by training the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, this invention enables the initial visual feature extraction module and the initial mathematical representation extraction module to pay more attention to the data essence in the chart during training, thereby improving the accuracy of the trained model in extracting mathematical representations from the chart. Furthermore, this invention utilizes multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training with multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thereby exhibiting better generalization ability when faced with unfamiliar chart types.

[0125] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a graph parsing model, characterized in that, The method comprises the steps of: constructing a preset initial chart analysis model, wherein the preset initial chart analysis model comprises an initial visual feature extraction module and an initial mathematical representation extraction module; obtaining a plurality of sample chart description texts with actual mathematical representation labels, and generating a plurality of sample charts corresponding to each of the sample chart description texts; training the initial visual feature extraction module and the initial mathematical representation extraction module separately based on the plurality of sample charts corresponding to each of the sample chart description texts, and taking numerical extraction as a benchmark to obtain a preset chart analysis model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

2. The method of claim 1, wherein, The method for training the initial visual feature extraction module separately based on numerical extraction comprises the steps of: determining N groups of data pairs, wherein each group of data pairs is composed of a first sample chart description text selected from the plurality of sample chart description texts and a first sample chart generated for the first sample chart description text; determining a text feature vector corresponding to the first sample chart description text in each group of data pairs, and extracting a numerical visual feature vector corresponding to the first sample chart in each group of data pairs by using the initial visual feature extraction module; calculating the intragroup similarity between each text feature vector and the numerical visual feature vector in the same group of data pairs, and calculating the intergroup similarity between each text feature vector and the numerical visual feature vector in the different group of data pairs; adjusting the parameters of the initial visual feature extraction module based on the intragroup similarity and the intergroup similarity to obtain the trained initial visual feature extraction module.

3. The method of claim 1, wherein, The method for training the initial mathematical representation extraction module separately based on numerical extraction comprises the steps of: selecting second sample chart description texts and corresponding second sample charts required for training the initial mathematical representation extraction module from the plurality of sample chart description texts, and determining a first actual chart corresponding to the second sample chart description texts; extracting a first image feature vector corresponding to the first actual chart and a second image feature vector corresponding to each second sample chart by using the trained initial visual feature extraction module; performing fusion processing on the first image feature vector and the plurality of second image feature vectors to obtain a fusion image feature vector, and performing chart analysis on the fusion image feature vector by using the initial mathematical representation extraction module to obtain a predicted mathematical representation; determining a loss function corresponding to the initial mathematical representation extraction module based on the difference between the actual mathematical representation corresponding to the second sample chart description text and the predicted mathematical representation, and iteratively training the initial mathematical representation extraction module based on the loss function to obtain the trained initial mathematical representation extraction module.

4. The method of claim 3, wherein, The plurality of second image feature vectors at least comprise a first sub-image feature vector and a second sub-image feature vector. The fusion processing on the first image feature vector and the plurality of second image feature vectors to obtain a fusion image feature vector comprises the steps of: constructing a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determining a covariance matrix corresponding to the feature matrix; performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvalues and an eigenvector corresponding to each eigenvalue, selecting a preset number of eigenvalues from each of the eigenvalues as principal component eigenvalues based on the size of the eigenvalues, and constructing a projection matrix from the eigenvector corresponding to each of the principal component eigenvalues; projecting the feature matrix based on the projection matrix to obtain a fused feature matrix, and determining the fused image feature vector based on the fused feature matrix.

5. The method of claim 4, wherein, The fusion processing of the first image feature vector and the plurality of second image feature vectors to obtain the fused image feature vector includes: performing feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector; performing element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element fusion vector; performing low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector; performing transformation processing on the feature fusion vector, the element fusion vector, and the low-order fusion vector using a preset transformation function to obtain the fused image feature vector.

6. The method of claim 1, wherein, The generating of a plurality of sample charts corresponding to each of the sample chart description texts includes: inputting each of the sample chart description texts into a preset chart generator for chart generation to obtain a plurality of sample charts corresponding to each of the sample chart description texts.

7. The method of claim 1, wherein, After training the initial visual feature extraction module and the initial mathematical representation extraction module based on the plurality of sample charts corresponding to each of the sample chart description texts and taking numerical extraction as a benchmark to obtain a preset chart analysis model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module, the method further includes: constructing an initial lightweight chart analysis model, wherein the initial lightweight chart analysis model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module; obtaining lightweight chart description texts required for lightweighting the preset chart analysis model, and generating a plurality of lightweight charts corresponding to the lightweight chart description texts; determining an image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and a mathematical representation analysis difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module based on the second actual chart corresponding to the lightweight chart description texts and the plurality of lightweight charts; determining a chart feature loss function based on the image feature extraction difference, and determining a mathematical representation loss function based on the mathematical representation analysis difference; Determine a joint loss function based on the chart feature loss function and the mathematical representation loss function, and iteratively train the initial lightweight chart parsing model based on the joint loss function to obtain a lightweight chart parsing model.

8. The method of claim 7, wherein, Based on the second actual chart corresponding to the lightweight chart description text, a plurality of lightweight charts, determine the image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and the mathematical representation analysis difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module, including: Through the trained initial visual feature extraction module, extract the first heavy image feature vector corresponding to the second actual chart and the second heavy image feature vector corresponding to a plurality of lightweight charts, respectively, and perform fusion processing on the first heavy image feature vector and a plurality of second heavy image feature vectors to obtain a heavy image fusion feature vector, and extract a lightweight chart feature vector corresponding to the second actual chart through the lightweight visual feature extraction module, and the difference between the lightweight chart feature vector and the heavy image fusion feature vector is taken as the image feature extraction difference; Through the trained initial mathematical representation extraction module, the heavy image fusion feature vector is parsed to obtain a heavy chart mathematical representation, and the lightweight chart mathematical representation is obtained by parsing the lightweight chart feature vector through the lightweight mathematical representation extraction module, and the difference between the heavy chart mathematical representation and the lightweight chart mathematical representation is taken as the mathematical representation analysis difference.

9. The method of claim 1, wherein, Based on a plurality of sample charts corresponding to each sample chart description text, the initial visual feature extraction module and the initial mathematical representation extraction module are trained individually based on numerical extraction as a benchmark to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module. Obtain a chart to be parsed; Input the chart to be parsed into the preset chart parsing model, extract numerical visual feature vectors through the trained initial visual feature extraction module, and parse the numerical visual feature vectors through the trained initial mathematical representation extraction module to obtain the mathematical representation of the chart to be parsed.

10. A training device for a graph parsing model, characterized in that, Including: A construction unit is configured to construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; A generation unit is configured to obtain a plurality of sample chart description texts with actual mathematical representation labels, and generate a plurality of sample charts corresponding to each sample chart description text. The training unit is configured to train the initial visual feature extraction module and the initial mathematical representation extraction module separately based on the multiple sample charts corresponding to each sample chart description text, and obtain a preset chart analysis model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 9.

12. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 9.

Images