Training method and device of chart analysis model, storage medium and computer equipment

Abstract

The invention discloses a chart analysis model training method and device, a storage medium and computer equipment, relates to the technical field of chart processing, and mainly aims to pay more attention to numerical features in a chart in a model training process and clarify optimization targets of a feature extraction module and a mathematical representation extraction module. Comprising the steps that a preset initial chart analysis model is constructed, and the preset initial chart analysis model comprises an initial visual feature extraction module and an initial mathematical representation extraction module; obtaining a plurality of sample chart description texts with actual mathematical representation labels, and generating a plurality of sample charts corresponding to each sample chart description text; and based on various sample charts corresponding to each sample chart description text, independently training an initial visual feature extraction module and an initial mathematical representation extraction module by taking numerical value extraction as a reference to obtain a preset chart analysis model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

Description

Technical Field

[0001] The present invention relates to the field of chart processing technology, and in particular to a training method, device, storage medium and computer equipment for a chart parsing model. Background Art

[0002] Charts are visual representations of data, and analyzing charts can help us accurately extract key data from complex visual elements.

[0003] In current graph processing, analyzing the mathematical representations within graphs allows for a deeper understanding of the research object's development trends, underlying mechanisms, and interrelationships, providing a solid foundation for theoretical development. However, in traditional joint training models, the optimization objectives of the feature extraction module and the mathematical representation extraction module are not clearly distinguished. The mathematical representation extraction module may overly rely on textual semantics and neglect precise numerical alignment. This training approach struggles to meet the learning needs of all parts of the model, resulting in poor model training results. Summary of the Invention

[0004] The present invention provides a training method, device, storage medium and computer equipment for a chart parsing model, which mainly focuses on the numerical features in the chart during the model training process, clarifies the optimization goals of the feature extraction module and the mathematical representation extraction module, and thus improves the training effect of the chart parsing model.

[0005] According to a first aspect of the present invention, a method for training a graph parsing model is provided, comprising:

[0006] Constructing a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module;

[0007] Acquire multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each of the sample chart description texts;

[0008] Based on the multiple sample charts corresponding to the text describing each sample chart, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately based on numerical extraction to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0009] Optionally, the method of training the initial visual feature extraction module based on numerical extraction alone includes:

[0010] Determine N groups of data pairs, wherein each group of data pairs consists of a first sample chart description text selected from a plurality of sample chart description texts and a first sample chart generated based on the first sample chart description text;

[0011] Determine the text feature vectors corresponding to the first sample chart description texts in the N groups of data pairs respectively, and use the initial visual feature extraction module to extract the numerical visual feature vectors corresponding to the first sample charts in the N groups of data pairs respectively;

[0012] Calculate the same-group similarity between each text feature vector and the numerical visual feature vector in the same group of data pairs, and calculate the different-group similarity between each text feature vector and the numerical visual feature vector in the different groups of data pairs;

[0013] Based on the same-group similarity and the different-group similarity, the parameters of the initial visual feature extraction module are adjusted to obtain the trained initial visual feature extraction module.

[0014] Optionally, the method of training the initial mathematical representation extraction module based on numerical extraction alone includes:

[0015] Selecting a second sample chart description text and its corresponding second sample chart required for training the initial mathematical representation extraction module from a plurality of sample chart description texts, and determining a first actual chart corresponding to the second sample chart description text;

[0016] extracting a first image feature vector corresponding to the first actual chart and a second image feature vector corresponding to each second sample chart respectively through the trained initial visual feature extraction module;

[0017] fusing the first image feature vector and a plurality of second image feature vectors to obtain a fused image feature vector, and performing a graphical analysis on the fused image feature vector using the initial mathematical representation extraction module to obtain a predicted mathematical representation;

[0018] Based on the difference between the actual mathematical representation corresponding to the second sample graph description text and the predicted mathematical representation, a loss function corresponding to the initial mathematical representation extraction module is determined, and based on the loss function, the initial mathematical representation extraction module is iteratively trained to obtain the trained initial mathematical representation extraction module.

[0019] Optionally, the plurality of second image feature vectors include at least a first sub-image feature vector and a second sub-image feature vector;

[0020] The fusing the first image feature vector and the plurality of second image feature vectors to obtain a fused image feature vector includes:

[0021] constructing a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determining a covariance matrix corresponding to the feature matrix;

[0022] Performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvalues ​​and an eigenvector corresponding to each eigenvalue, selecting a preset number of eigenvalues ​​from each of the eigenvalues ​​as principal component eigenvalues ​​based on the magnitude of the eigenvalues, and forming a projection matrix using the eigenvectors corresponding to each of the principal component eigenvalues;

[0023] The feature matrix is ​​projected based on the projection matrix to obtain a fusion feature matrix, and the fusion image feature vector is determined based on the fusion feature matrix.

[0024] Optionally, the fusing the first image feature vector and the plurality of second image feature vectors to obtain a fused image feature vector includes:

[0025] performing feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector;

[0026] performing element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element-fused vector;

[0027] performing low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector;

[0028] The feature fusion vector, the element fusion vector, and the low-order fusion vector are transformed using a preset transformation function to obtain the fused image feature vector.

[0029] Optionally, generating a plurality of sample charts corresponding to each sample chart description text includes:

[0030] Each sample chart description text is input into a preset chart generator to generate a chart, thereby obtaining a plurality of sample charts corresponding to each sample chart description text.

[0031] Optionally, based on a plurality of sample charts corresponding to each sample chart description text, the initial visual feature extraction module and the initial mathematical representation extraction module are individually trained based on numerical extraction to obtain a preset chart parsing model having the trained initial visual feature extraction module and the trained initial mathematical representation extraction module, the method further includes:

[0032] Constructing an initial lightweight chart parsing model, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module;

[0033] Acquiring lightweight chart description text required for lightweighting the preset chart parsing model, and generating a plurality of lightweight charts corresponding to the lightweight chart description text;

[0034] Determining, based on a second actual chart corresponding to the lightweight chart description text and a plurality of lightweight charts, image feature extraction differences between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and mathematical representation parsing differences between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module;

[0035] determining a chart feature loss function based on the image feature extraction differences, and determining a mathematical representation loss function based on the mathematical representation analysis differences;

[0036] A joint loss function is determined based on the graph feature loss function and the mathematical representation loss function, and the initial lightweight graph parsing model is iteratively trained based on the joint loss function to obtain a lightweight graph parsing model.

[0037] Optionally, based on a second actual chart corresponding to the lightweight chart description text and multiple lightweight charts, determining an image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and a mathematical representation parsing difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module includes:

[0038] extracting a first heavy-quantized image feature vector corresponding to the second actual chart and second heavy-quantized image feature vectors corresponding to multiple lightweight charts through the trained initial visual feature extraction module, fusing the first heavy-quantized image feature vector and the multiple second heavy-quantized image feature vectors to obtain a heavy-quantized image fusion feature vector, extracting a lightweight chart feature vector corresponding to the second actual chart through the lightweight visual feature extraction module, and using the difference between the lightweight chart feature vector and the heavy-quantized image fusion feature vector as the image feature extraction difference;

[0039] The re-quantized image fusion feature vector is subjected to chart analysis by the trained initial mathematical representation extraction module to obtain a re-quantized chart mathematical representation. The lightweight chart feature vector is subjected to chart analysis by the lightweight mathematical representation extraction module to obtain a lightweight chart mathematical representation. The difference between the re-quantized chart mathematical representation and the lightweight chart mathematical representation is used as the mathematical representation analysis difference.

[0040] Optionally, based on a plurality of sample charts corresponding to each sample chart description text, the initial visual feature extraction module and the initial mathematical representation extraction module are individually trained based on numerical extraction, and after obtaining a preset chart parsing model including the trained initial visual feature extraction module and the trained initial mathematical representation extraction module, the method further includes:

[0041] Get the chart to be parsed;

[0042] The chart to be analyzed is input into the preset chart analysis model, and the numerical features of the chart to be analyzed are extracted by the trained initial visual feature extraction module to obtain a numerical visual feature vector. The numerical visual feature vector is analyzed by the trained initial mathematical representation extraction module to obtain a mathematical representation of the chart to be analyzed.

[0043] According to a second aspect of the present invention, there is provided a training device for a graph parsing model, comprising:

[0044] A construction unit, configured to construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module;

[0045] a generating unit, configured to obtain a plurality of sample chart description texts with actual mathematical representation labels, and generate a plurality of sample charts corresponding to each of the sample chart description texts;

[0046] A training unit is used to train the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction as a benchmark based on multiple sample charts corresponding to the text describing each sample chart, so as to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0047] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned training method of the graph parsing model when executed by a processor.

[0048] According to a fourth aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned training method for the graph parsing model when executing the program.

[0049] According to a training method, device, storage medium and computer equipment of a chart parsing model provided by the present invention, compared with the current traditional joint training mode, the present invention uses multiple sample chart description texts with actual mathematical representation labels to train the initial visual feature extraction module and the initial mathematical representation extraction module separately, which can realize training based on the respective optimization targets of the initial visual feature extraction module and the initial mathematical representation extraction module, and can adjust the optimization strategies of the feature extraction module and the mathematical representation extraction module respectively, avoiding the problem of gradient disappearance caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module in the joint training, thereby improving the performance of the chart parsing model. Training effect; at the same time, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, which can enable the initial visual feature extraction module and the initial mathematical representation extraction module to pay more attention to the data nature in the chart during the training process, thereby improving the accuracy of the trained model in extracting the mathematical representation in the chart; at the same time, the present invention uses multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module, and can enable the chart parsing model to learn the commonalities and differences between different chart types through training of multiple types of charts, thereby showing better generalization ability when facing chart types that have not been seen before. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0051] Figure 1 A flowchart of a method for training a graph parsing model provided by an embodiment of the present invention is shown;

[0052] Figure 2 A schematic diagram showing various types of charts corresponding to the same chart description text provided by an embodiment of the present invention is shown;

[0053] Figure 3 A flowchart of another method for training a graph parsing model provided by an embodiment of the present invention is shown;

[0054] Figure 4 A schematic diagram of a process for training a visual feature extraction module provided by an embodiment of the present invention is shown;

[0055] Figure 5 A schematic diagram of a process for training a mathematical representation extraction module provided by an embodiment of the present invention is shown;

[0056] Figure 6 A schematic diagram of a process for lightweighting a preset chart parsing model provided by an embodiment of the present invention is shown;

[0057] Figure 7 A schematic diagram of a chart parsing method provided by an embodiment of the present invention is shown;

[0058] Figure 8 A schematic structural diagram of a training device for a chart parsing model provided by an embodiment of the present invention is shown;

[0059] Figure 9 A schematic structural diagram of another diagram parsing model training device provided by an embodiment of the present invention is shown;

[0060] Figure 10 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0061] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0062] At present, in the traditional joint training mode, the optimization objectives of the feature extraction module and the mathematical representation extraction module are not clearly distinguished. The mathematical representation extraction module may over-rely on text semantics and ignore the precise alignment of numerical values. This training method is difficult to meet the learning needs of various parts of the model, resulting in poor model training results.

[0063] In order to solve the above problems, an embodiment of the present invention provides a training method for a graph parsing model, such as Figure 1 As shown, the method includes:

[0064] 101. Construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module.

[0065] Among them, the initial visual feature extraction module can be an initial visual encoder, and the initial mathematical representation extraction module can be an initial decoder. For the embodiment of the present invention, a preset initial chart parsing model based on the initial visual feature extraction module and the initial mathematical representation extraction module is first constructed, and then the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately, which can ensure that the initial visual feature extraction module and the initial mathematical representation extraction module are trained according to their respective corresponding optimization targets, and avoid the problem that the gradients of the feature extraction module and the mathematical representation extraction module interfere with each other when the initial visual feature extraction module and the initial mathematical representation extraction module are jointly trained, resulting in the disappearance of the gradient, and ensure that the feature extraction module and the mathematical representation extraction module can meet their respective optimization targets, so that the embodiment of the present invention can improve the training effect of the chart parsing model.

[0066] 102. Acquire multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each sample chart description text.

[0067] For the embodiment of the present invention, the actual mathematical representation label refers to the real data content (the content of the mathematical text) in the chart; the sample chart description text refers to the mathematical description text of the chart, and the format can be set according to the needs, such as "|age|number of people (people)|\n|13|4|\n|14|5|\n|15|4|\n|16|3|"; a variety of sample charts include table charts, column charts, bar charts, scatter charts, pie charts, etc. For example, if a sample chart description text is: the number of people aged 13 in a class is 4, the number of people aged 14 is 5, the number of people aged 15 is 4, and the number of people aged 1 / 6 is 3, then the different types of sample charts corresponding to the sample chart description text are as follows: Figure 2 shown.

[0068] For an embodiment of the present invention, in order to enable the chart parsing model to learn the commonalities and differences between different types of charts, thereby increasing the generalization ability of the chart parsing model on unseen chart types, when training the chart parsing model, it is first necessary to generate multiple types of sample charts for the same sample chart description text. Based on this, the method includes: inputting each of the sample chart description texts into a preset chart generator for chart generation, and obtaining multiple sample charts corresponding to each of the sample chart description texts.

[0069] Specifically, in order to improve the chart generation accuracy of the preset chart generator, it is first necessary to train and construct the preset chart generator. Based on this, the method includes: constructing a preset initial chart generator and obtaining a sample data set, wherein the sample data set includes a chart description text and multiple types of actual charts corresponding to the chart description text; dividing the sample data set into a training set and a test set, using the training set to train the preset initial chart generator, and using the test set to test the trained preset initial chart generator, and using the preset initial chart generator that meets the test conditions as the preset chart generator.

[0070] In the above embodiment, a method for building and training a preset chart generator is provided. First, a preset initial chart generator is built, and then a sample data set is downloaded from the official website or a designated data source. Make sure that the sample data set contains all necessary files, including chart description text and various types of actual charts corresponding to the chart description text. Convert the file into a format that the preset initial chart generator can understand. Finally, train and test the generator. Specifically, the data set can be divided first: use random or specific strategies (such as stratified sampling) to divide the data set into a training set and a test set. Then use the training set to train the model, monitor the loss value, mAP and other indicators during the training process to evaluate the performance of the generator. Adjust the training parameters as needed, such as learning rate, optimizer, regularization, etc., to optimize the training effect. Finally, test the generator: use the test set to test the trained generator to evaluate its performance on unseen data. Calculate and record indicators such as mAP, precision, recall rate on the test set. If the generator performance does not meet the requirements, you can return to the training stage for more iterations or adjustments. Finally, a preset chart generator that meets the requirements is obtained, and the preset chart generator is used to generate various types of sample charts based on the sample chart description text. The embodiment of the present invention generates various types of charts by using the preset chart generator, which can improve the generation efficiency and generation accuracy of the charts.

[0071] 103. Based on the multiple sample charts corresponding to the description text of each sample chart, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately based on numerical extraction to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0072] In the embodiment of the present invention, a first sample chart description text and its corresponding multiple first sample charts are extracted from each sample chart description text for training the initial visual feature extraction module alone, and a second sample chart description text and its corresponding multiple second sample charts are extracted from each sample chart description text for training the initial mathematical representation extraction module alone. Then, the first sample chart description text and its corresponding multiple first sample charts are used to train the initial visual feature extraction module based on numerical feature extraction, and the second sample chart description text and its corresponding multiple second sample charts are used to train the initial mathematical representation extraction module based on numerical extraction, thereby obtaining a chart parsing model. The embodiment of the present invention independently trains the feature extraction module and the mathematical representation extraction module, so that the feature extraction module and the mathematical representation extraction module are independently optimized according to their respective task requirements. Therefore, the trained feature extraction module and the mathematical representation extraction module both meet their respective optimization objectives, avoiding the problem that the optimization objectives of the feature extraction module and the mathematical representation extraction module may conflict with each other during joint training, resulting in the model being difficult to converge to the optimal solution. Furthermore, by training the feature extraction module and the mathematical representation extraction module based on numerical extraction, the embodiments of the present invention enable the trained chart parsing model to better focus on the essence of the data in the chart, thereby improving the training effect of the chart parsing model. Furthermore, by adding more diverse chart features, the embodiments of the present invention enhance the decoding capabilities of the mathematical representation extraction module.

[0073] According to a training method for a chart parsing model provided by the present invention, compared with the current traditional joint training mode, the present invention uses multiple sample chart description texts with actual mathematical representation labels to separately train the initial visual feature extraction module and the initial mathematical representation extraction module. This can achieve training based on the optimization targets of the initial visual feature extraction module and the initial mathematical representation extraction module, and can adjust the optimization strategies of the feature extraction module and the mathematical representation extraction module separately to avoid the problem of gradient disappearance caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module during joint training. Therefore, the present invention can improve the training effect of the chart parsing model. At the same time, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, which can enable the initial visual feature extraction module and the initial mathematical representation extraction module to pay more attention to the data nature in the chart during training, thereby improving the accuracy of the trained model in extracting mathematical representations in the chart. At the same time, the present invention uses multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training on multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thereby demonstrating better generalization ability when facing unseen chart types.

[0074] Furthermore, in order to better illustrate the process of training the above-mentioned chart parsing model, as a refinement and extension of the above-mentioned embodiment, the embodiment of the present invention provides another training method of the chart parsing model, such as Figure 3 As shown, the method includes:

[0075] 201. Construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module.

[0076] 202. Acquire multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each sample chart description text.

[0077] 203. Based on the multiple sample charts corresponding to the description text of each sample chart, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately based on numerical extraction to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0078] For an embodiment of the present invention, a method for training an initial visual feature extraction module based on numerical extraction includes: determining N groups of data pairs, wherein each group of data pairs consists of a first sample chart description text selected from multiple sample chart description texts and a first sample chart generated for the first sample chart description text; respectively determining the text feature vectors corresponding to the first sample chart description texts in the N groups of data pairs, and using the initial visual feature extraction module to respectively extract the numerical visual feature vectors corresponding to the first sample charts in the N groups of data pairs; respectively calculating the same-group similarity between each text feature vector and the numerical visual feature vector in the same group of data pairs, and respectively calculating the different-group similarity between each text feature vector and the numerical visual feature vector in the different groups of data pairs; based on the same-group similarity and the different-group similarity, adjusting the parameters of the initial visual feature extraction module to obtain the trained initial visual feature extraction module.

[0079] Specifically, a sample chart description text is randomly selected from multiple sample chart description texts as the first sample chart description text, and a sample chart is randomly selected from multiple types of sample charts generated based on the sample chart description text as the first sample chart. The first sample chart description text and the first sample chart constitute a set of data pairs. Thus, according to the above method, N sets of data pairs can be determined. Then, according to the following method, Figure 4The initial visual feature extraction module (initial visual encoder) is trained based on numerical feature extraction in the manner shown, that is, the pre-trained text feature extraction module is used to extract the text feature vectors T1, T2, ..., T of the first sample chart description text in N groups of data pairs. N , and use the initial visual feature extraction module to extract the numerical visual feature vectors I1, I2, ..., I of the first sample chart in the N groups of data pairs respectively N , respectively calculate the similarity between the text feature vector and the numerical visual feature vector in the same set of data pairs, that is, the similarity of the same group I1·T1, I2·T2, ..., I N ·T N At the same time, the similarities between the text feature vectors in each data pair and the numerical visual feature vectors in other data pairs are calculated, namely, the heterogeneous similarities I1·T2, I1·T3, ..., I1·T N ;I2·T1, I2·T3,..., I2·T N ;I3·T1, I3·T2,..., I3·T N ;....;I N T1, I N T2, ..., I N ·T N-1 , finally, if at least one of the same-group similarity and the different-group similarity does not meet the requirements (such as the same-group similarity is less than the first preset similarity threshold, or the different-group similarity is greater than the second preset similarity threshold, wherein the first preset similarity threshold and the second preset similarity threshold are both set according to actual needs), the parameters of the initial visual feature extraction module are adjusted until both the same-group similarity and the different-group similarity meet the requirements (such as the same-group similarity is greater than or equal to the first preset similarity threshold, and the different-group similarity is less than or equal to the second preset similarity threshold), then the adjustment of the parameters of the initial visual feature extraction module is stopped. The embodiment of the present invention trains the initial visual feature extraction module based on numerical feature extraction, so that the trained feature extraction module pays more attention to the nature of the data in the chart, thereby improving the extraction accuracy of the numerical features in the chart.

[0080] For an embodiment of the present invention, a method for training an initial mathematical representation extraction module based on numerical extraction alone includes: selecting a second sample chart description text and its corresponding second sample chart required for training the initial mathematical representation extraction module from multiple sample chart description texts, and determining the first actual chart corresponding to the second sample chart description text; extracting the first image feature vector corresponding to the first actual chart and the second image feature vector corresponding to each second sample chart through the trained initial visual feature extraction module; fusing the first image feature vector and multiple second image feature vectors to obtain a fused image feature vector, and performing chart analysis on the fused image feature vector through the initial mathematical representation extraction module to obtain a predicted mathematical representation; determining the loss function corresponding to the initial mathematical representation extraction module based on the difference between the actual mathematical representation corresponding to the second sample chart description text and the predicted mathematical representation, and iteratively training the initial mathematical representation extraction module based on the loss function to obtain the trained initial mathematical representation extraction module.

[0081] The first actual chart refers to the accurate chart corresponding to the second sample chart description text. The first actual chart can be any type of chart, and the embodiment of the present invention does not specifically limit this. Specifically, a number of texts are randomly selected from multiple sample chart description texts as the second sample chart description text, and multiple types of sample charts generated for the second sample chart description text are used as the second sample chart. Then, according to the following Figure 5 The initial mathematical representation extraction module (decoder) is trained based on numerical extraction. That is, first, the initial visual feature extraction module ( Figure 5The visual encoder with snowflakes in the figure extracts the first image feature vector with numerical features of the first actual chart, and extracts the second image feature vector with numerical features corresponding to each second sample chart. If the second sample chart includes a first sub-chart and a second sub-chart, the second image feature vector includes a first sub-image feature vector (image feature vector 1) corresponding to the first sub-chart and a second sub-image feature vector (image feature vector 2) corresponding to the second sub-chart. The first image feature vector, the first sub-image feature vector, and the second sub-image feature vector are then fused. Thus, according to the above method, a plurality of fused image feature vectors corresponding to the description texts of the second sample charts can be obtained. The fused image feature vectors corresponding to the description texts of the plurality of second sample charts and the actual mathematical representation labels constitute a training set. The initial mathematical representation extraction module (the mathematical representation extraction module with flames in the figure) is iteratively trained using the training set. During the training process, the fused image feature vector is used as input data and the actual mathematical representation label is used as output data. The loss value, mAP and other indicators of the initial mathematical representation extraction module during the training process are monitored to evaluate the performance of the mathematical representation extraction module. Training parameters such as the learning rate, optimizer, and regularization are adjusted as needed to optimize the training effect, ultimately training a mathematical representation extraction module that meets the required accuracy. It should be noted that the second image feature vector in the embodiment of the present invention is not limited to image feature vector 1 and image feature vector 2, but may also include more feature vectors, and the embodiment of the present invention does not specifically limit the number of vectors.

[0082] In the above embodiment, the method of fusing the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector includes: constructing a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determining the covariance matrix corresponding to the feature matrix; performing eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and eigenvectors corresponding to each eigenvalue, and based on the size of the eigenvalue, selecting a preset number of eigenvalues ​​in each of the eigenvalues ​​as principal component eigenvalues, and forming a projection matrix from the eigenvectors corresponding to each principal component eigenvalue; projecting the feature matrix based on the projection matrix to obtain a fused feature matrix, and determining the fused image feature vector based on the fused feature matrix.

[0083] Specifically, for example, if the first image feature vector is f1(a1, b1), the first sub-image feature vector is f2(a2, b2), and the second sub-image feature vector is f3(a3, b3), then the constructed feature matrix X is as follows:

[0084]

[0085] Then, for each column in the feature matrix X, calculate its mean and standard deviation, divide each element in the feature matrix X by the difference between its corresponding mean and standard deviation, and obtain each standardized element in the feature matrix X. This will give the standardized matrix Z, and then determine the covariance matrix C of the matrix Z as follows:

[0086]

[0087] Here, m is the total number of image eigenvectors. Furthermore, the covariance matrix is ​​subjected to eigenvalue decomposition to obtain multiple eigenvalues ​​and their corresponding eigenvectors. The eigenvalues ​​are then sorted from smallest to largest. The first k eigenvalues ​​are selected from these sorted eigenvalues, and the eigenvectors corresponding to these first k eigenvalues ​​form the projection matrix P. Based on the projection matrix P, the normalized matrix Z is projected onto the principal component space to obtain the fused feature matrix Y, where Y = ZP. Ultimately, each element in the fused feature matrix Y constitutes the fused image feature vector.

[0088] In another embodiment of the present invention, in order to perform fusion processing on multiple image feature vectors, the method also includes: performing feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector; performing element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element fusion vector; performing low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector; and using a preset transformation function to transform the feature fusion vector, the element fusion vector, and the low-order fusion vector to obtain the fused image feature vector.

[0089] Specifically, if the first image feature vector is (a1, a2), the first sub-image feature vector is (b1, b2), and the second sub-image feature vector is (c1, c2), the specific fusion processing method includes: performing feature-level fusion between different feature vectors, that is, performing Hadamard product on all elements between vectors, and then performing convolution transformation under a certain weight w1 to obtain a feature fusion vector of f(w1*(a1*b1*c1, a2*b2*c2, a3*b3*c3)); at the same time, performing element-level fusion on all feature vector data, that is, performing Hadamard product on each element between vectors. After that, assign different weight values ​​to the results of each product, and then perform linear transformation, and the obtained element fusion vector is f(w2*a1*b1*c1,w3*a2*b2*c2,w4*a3*b3*c3); in addition, all feature vectors are subjected to low-order fusion processing, and then the results after fusion processing are assigned weight coefficients, and then perform linear transformation, and the obtained low-order fusion vector is f(w5(a1,a2,b1,b2,c1,c2)); finally, the above feature fusion vectors, element fusion vectors, and low-order fusion vectors are transformed using the preset transformation function to obtain the fused image feature vector. Among them, the preset transformation function is determined according to actual needs, and the present invention does not make specific limitations on this. It should be noted that the above examples are only illustrative and do not limit the embodiments of the present application.

[0090] In another embodiment of the present invention, multiple image feature vectors (f 1, f2, f3) are fused, and the fused image feature T is T = avg_pool(self_atten([f 1, f2,f3])).

[0091] The embodiments of the present invention can automatically or explicitly combine different features by fusing multiple image feature vectors to generate new feature combinations. These combined features may contain complex nonlinear relationships between the original features, so that the mathematical representation extraction module can capture more detailed and rich information in the data, that is, it can make full use of the relationship between various data, extract more implicit features, and improve the subsequent training effect of the mathematical representation extraction module.

[0092] Furthermore, in order to reduce the energy consumption and computing resources of the chart parsing model when parsing a chart, it is necessary to perform lightweight processing on the preset chart parsing model. Based on this, the method includes: constructing an initial lightweight chart parsing model, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module; obtaining a lightweight chart description text required for lightweighting the preset chart parsing model, and generating a plurality of lightweight charts corresponding to the lightweight chart description text; based on a second actual chart and a plurality of lightweight charts corresponding to the lightweight chart description text, determining the image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and the mathematical representation analysis difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module; determining a chart feature loss function based on the image feature extraction difference, and determining a mathematical representation loss function based on the mathematical representation analysis difference; determining a joint loss function based on the chart feature loss function and the mathematical representation loss function, and iteratively training the initial lightweight chart parsing model based on the joint loss function to obtain a lightweight chart parsing model. Among them, the method for determining the difference in image feature extraction and the difference in mathematical representation analysis includes: extracting the first re-quantized image feature vector corresponding to the second actual chart and the second re-quantized image feature vectors corresponding to multiple lightweight charts through the trained initial visual feature extraction module, and fusing the first re-quantized image feature vector and the multiple second re-quantized image feature vectors to obtain a re-quantized image fusion feature vector, extracting the lightweight chart feature vector corresponding to the second actual chart through the lightweight visual feature extraction module, and taking the difference between the lightweight chart feature vector and the re-quantized image fusion feature vector as the image feature extraction difference; performing chart analysis on the re-quantized image fusion feature vector through the trained initial mathematical representation extraction module to obtain a re-quantized chart mathematical representation, performing chart analysis on the lightweight chart feature vector through the lightweight mathematical representation extraction module to obtain a lightweight chart mathematical representation, and taking the difference between the re-quantized chart mathematical representation and the lightweight chart mathematical representation as the mathematical representation analysis difference.

[0093] Among them, the lightweight visual feature extraction module can be a lightweight visual encoder, and the lightweight mathematical representation extraction module can be a lightweight decoder. Specifically, a lightweight sample data set is obtained, which includes a chart description text (chart mathematical text), that is, a lightweight chart description text, and a preset chart generator is used to generate multiple types of charts for each lightweight chart description text, that is, lightweight charts, and determine the accurate second actual chart corresponding to each lightweight chart description text. Then, according to the following Figure 6The method shown is used to perform lightweight processing on the preset chart analysis model, wherein, Figure 6 The visual feature extraction module (visual encoder) and the mathematical representation extraction module (decoder) marked with snowflakes are components of the preset chart parsing model, and the visual feature extraction module 1 (visual encoder 1) and the mathematical representation extraction module 1 (decoder 1) marked with flames are components of the initial lightweight chart parsing model, that is, the visual feature extraction module 1 marked with flames is a lightweight visual feature extraction module, and the mathematical representation extraction module 1 marked with flames is a lightweight mathematical representation extraction module. During the model lightweighting process, the parameters of the visual feature extraction module and the mathematical representation extraction module in the preset chart parsing model remain unchanged, and only the parameters of the visual feature extraction module 1 and the mathematical representation extraction module 1 in the initial lightweight chart parsing model are trained. According to the above-mentioned feature fusion method, the multiple image feature vectors extracted by the visual feature extraction module in the preset chart parsing model for the chart corresponding to the same chart description text are fused, and the difference between the fused image features and the image features extracted by the lightweight visual feature extraction module is determined, and the chart feature loss function is determined based on the difference. Furthermore, for the same lightweight chart description text, the image features extracted by the lightweight visual feature extraction module are input into the lightweight mathematical representation extraction module. At the same time, the image features extracted by the visual feature extraction module in the preset chart parsing model are input into the corresponding mathematical representation extraction module. Then, based on the difference between the output of the lightweight mathematical representation extraction module and the output of the mathematical representation extraction module in the preset chart parsing model, the mathematical representation loss function is determined. Then, the weight coefficients corresponding to the chart feature loss function and the mathematical representation loss function are determined respectively, and based on the weight coefficients, the chart feature loss function and the mathematical representation loss function are added to obtain a joint loss function. Finally, the initial lightweight chart parsing model is iteratively trained using the joint loss function. During the training process, the difference between the image features and mathematical representations extracted by the initial lightweight chart parsing model and the image features and mathematical representations extracted by the preset chart parsing model is monitored until the difference meets the requirement (such as being less than a certain value) and the iterative training stops. Finally, the initial lightweight chart parsing model at the end of training is determined as the final model for parsing the chart. By using visual loss and output prediction loss, a single image can be input, and the visual feature extraction module 1 and the mathematical representation extraction module 1 can also have the ability to encode and decode diverse statistical graph features. By lightweighting the model, the embodiments of the present invention can save computing resources and reduce energy consumption during the chart parsing process, improve chart parsing efficiency, and reduce the complexity of chart parsing.

[0094] It should be noted that the embodiment of the present invention does not specifically limit the functional form of the graph feature loss function and the mathematical representation loss function. For example, the graph feature loss function can be a mean absolute error loss function, and the mathematical representation loss function can be a binary cross entropy loss function.

[0095] 204. Obtain the chart to be parsed.

[0096] 205. Input the chart to be analyzed into the preset chart analysis model, perform numerical feature extraction on the chart to be analyzed through the trained initial visual feature extraction module to obtain a numerical visual feature vector, perform chart analysis on the numerical visual feature vector through the trained initial mathematical representation extraction module to obtain a mathematical representation of the chart to be analyzed.

[0097] Specifically, after building the preset chart parsing model, Figure 7 As shown, the initial visual feature extraction module after training is Figure 7 The visual encoder 1 in the training is the initial mathematical representation extraction module. Figure 7 Decoder 1 in. If a chart to be parsed is to be parsed, the chart to be parsed is directly input into the preset chart parsing model, and the preset chart parsing model can output the mathematical representation of the chart to be parsed. Since the preset chart parsing model is obtained by separately training the visual feature extraction module and the mathematical representation extraction module based on numerical extraction, the preset chart parsing model can pay more attention to the essence of the data in the chart when parsing the chart, and can improve the parsing accuracy of the chart. Furthermore, the visual feature extraction module and the mathematical representation extraction module can be used as base models, and the input and output data can be changed subsequently. After fine-tuning the model, different model applications can be realized, such as chart understanding, chart question and answer, etc. At the same time, after the embodiment of the present invention outputs the mathematical representation through the preset chart parsing model, it can also redraw the statistical chart based on the mathematical representation.

[0098] According to another training method for a chart parsing model provided by the present invention, compared with the current traditional joint training mode, the present invention uses multiple sample chart description texts with actual mathematical representation labels to separately train the initial visual feature extraction module and the initial mathematical representation extraction module. This can achieve training based on the optimization targets of the initial visual feature extraction module and the initial mathematical representation extraction module, and can adjust the optimization strategies of the feature extraction module and the mathematical representation extraction module separately to avoid the problem of gradient disappearance caused by mutual interference between the gradients of the feature extraction module and the mathematical representation extraction module during joint training. Therefore, the present invention can improve the training effect of the chart parsing model. At the same time, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, which can enable the initial visual feature extraction module and the initial mathematical representation extraction module to pay more attention to the data nature in the chart during training, thereby improving the accuracy of the trained model in extracting mathematical representations in the chart. At the same time, the present invention uses multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training on multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thereby demonstrating better generalization ability when facing unseen chart types.

[0099] Further, as Figure 1 The specific implementation of the present invention provides a training device for a graph parsing model, such as Figure 8 As shown, the device includes: a construction unit 31, a generation unit 32, and a training unit 33.

[0100] The construction unit 31 may be used to construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module.

[0101] The generating unit 32 may be configured to obtain a plurality of sample chart description texts with actual mathematical representation labels, and generate a plurality of sample charts corresponding to each of the sample chart description texts.

[0102] The training unit 33 can be used to train the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction as a benchmark based on multiple sample charts corresponding to the text describing each sample chart, and obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0103] In specific application scenarios, in order to train the initial visual feature extraction module separately, such as Figure 9As shown, the training unit 33 includes a determination module 331 , a feature extraction module 332 , a calculation module 333 , and a parameter adjustment module 334 .

[0104] The determination module 331 may be configured to determine N groups of data pairs, wherein each group of data pairs is composed of a first sample chart description text selected from a plurality of sample chart description texts and a first sample chart generated based on the first sample chart description text.

[0105] The feature extraction module 332 can be used to respectively determine the text feature vectors corresponding to the first sample chart description text in the N groups of data pairs, and use the initial visual feature extraction module to respectively extract the numerical visual feature vectors corresponding to the first sample chart in the N groups of data pairs.

[0106] The calculation module 333 can be used to calculate the same-group similarity between each text feature vector and the numerical visual feature vector in the same group of data pairs, and to calculate the different-group similarity between each text feature vector and the numerical visual feature vector in the different groups of data pairs.

[0107] The parameter adjustment module 334 may be configured to adjust the parameters of the initial visual feature extraction module based on the same-group similarity and the different-group similarity to obtain the trained initial visual feature extraction module.

[0108] In a specific application scenario, in order to train the initial mathematical representation extraction module separately, the training unit 33 further includes a feature fusion module 335 and a training module 336 .

[0109] The determination module 331 may be configured to select a second sample chart description text and its corresponding second sample chart required for training the initial mathematical representation extraction module from a plurality of sample chart description texts, and determine a first actual chart corresponding to the second sample chart description text.

[0110] The feature extraction module 332 may also be configured to extract a first image feature vector corresponding to the first actual chart and a second image feature vector corresponding to each second sample chart using the trained initial visual feature extraction module.

[0111] The feature fusion module 335 can be used to fuse the first image feature vector and multiple second image feature vectors to obtain a fused image feature vector, and perform graphical analysis on the fused image feature vector through the initial mathematical representation extraction module to obtain a predicted mathematical representation.

[0112] The training module 336 can be used to determine the loss function corresponding to the initial mathematical representation extraction module based on the difference between the actual mathematical representation corresponding to the second sample graph description text and the predicted mathematical representation, and iteratively train the initial mathematical representation extraction module based on the loss function to obtain the trained initial mathematical representation extraction module.

[0113] In a specific application scenario, multiple second image feature vectors include at least a first sub-image feature vector and a second sub-image feature vector; in order to perform fusion processing on the first image feature vector and multiple second image feature vectors, the feature fusion module 335 can be specifically used to construct a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determine the covariance matrix corresponding to the feature matrix; perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and eigenvectors corresponding to each eigenvalue, and based on the size of the eigenvalue, select a preset number of eigenvalues ​​in each of the eigenvalues ​​as principal component eigenvalues, and form a projection matrix by the eigenvectors corresponding to each principal component eigenvalue; based on the projection matrix, project the feature matrix to obtain a fused feature matrix, and determine the fused image feature vector based on the fused feature matrix.

[0114] In a specific application scenario, in order to perform fusion processing on the first image feature vector and multiple second image feature vectors, the feature fusion module 335 can be specifically used to perform feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector; perform element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element fusion vector; perform low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector; use a preset transformation function to transform the feature fusion vector, the element fusion vector, and the low-order fusion vector to obtain the fused image feature vector.

[0115] In a specific application scenario, in order to generate multiple sample charts corresponding to each sample chart description text, the generation unit 32 can be specifically used to input each of the sample chart description texts into a preset chart generator for chart generation, thereby obtaining multiple sample charts corresponding to each of the sample chart description texts.

[0116] In a specific application scenario, in order to perform lightweight processing on the preset chart analysis model, the device further includes: a lightweight unit 34.

[0117] The lightweight unit 34 can be used to construct an initial lightweight chart parsing model, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module; obtain a lightweight chart description text required for lightweighting the preset chart parsing model, and generate multiple lightweight charts corresponding to the lightweight chart description text; based on the second actual chart and multiple lightweight charts corresponding to the lightweight chart description text, determine the image feature extraction difference between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and the mathematical representation analysis difference between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module; based on the image feature extraction difference, determine a chart feature loss function, and based on the mathematical representation analysis difference, determine a mathematical representation loss function; based on the chart feature loss function and the mathematical representation loss function, determine a joint loss function, and based on the joint loss function, iteratively train the initial lightweight chart parsing model to obtain a lightweight chart parsing model.

[0118] In a specific application scenario, in order to extract the difference in image feature extraction and the difference in mathematical representation analysis, the lightweight unit 34 can be specifically used to extract the first re-quantized image feature vector corresponding to the second actual chart and the second re-quantized image feature vectors corresponding to multiple lightweight charts through the trained initial visual feature extraction module, and fuse the first re-quantized image feature vector and the multiple second re-quantized image feature vectors to obtain a re-quantized image fusion feature vector, extract the lightweight chart feature vector corresponding to the second actual chart through the lightweight visual feature extraction module, and use the difference between the lightweight chart feature vector and the re-quantized image fusion feature vector as the image feature extraction difference; perform chart analysis on the re-quantized image fusion feature vector through the trained initial mathematical representation extraction module to obtain a re-quantized chart mathematical representation, perform chart analysis on the lightweight chart feature vector through the lightweight mathematical representation extraction module to obtain a lightweight chart mathematical representation, and use the difference between the re-quantized chart mathematical representation and the lightweight chart mathematical representation as the mathematical representation analysis difference.

[0119] In a specific application scenario, in order to parse the chart, the device further includes: a chart parsing unit 35 .

[0120] The chart parsing unit 35 can be used to obtain a chart to be parsed; input the chart to be parsed into the preset chart parsing model, perform numerical feature extraction on the chart to be parsed through the trained initial visual feature extraction module to obtain a numerical visual feature vector, and perform chart parsing on the numerical visual feature vector through the trained initial mathematical representation extraction module to obtain a mathematical representation of the chart to be parsed.

[0121] It should be noted that for other corresponding descriptions of the functional modules involved in the training device for a graph parsing model provided by an embodiment of the present invention, please refer to Figure 1 The corresponding description of the method shown will not be repeated here.

[0122] Based on the above Figure 1 The method shown, accordingly, an embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: constructing a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; obtaining a plurality of sample chart description texts with actual mathematical representation labels, and generating a plurality of sample charts corresponding to each of the sample chart description texts; based on the plurality of sample charts corresponding to each of the sample chart description texts, separately training the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, and obtaining a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0123] Based on the above Figure 1 The method shown and Figure 8 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 10 As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43. When the processor 41 executes the program, the following steps are implemented: constructing a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; obtaining a plurality of sample chart description texts with actual mathematical representation labels, and generating a plurality of sample charts corresponding to each of the sample chart description texts; based on the plurality of sample charts corresponding to each of the sample chart description texts, separately training the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction, and obtaining a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

[0124] According to the technical solution of the present invention, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately by using multiple sample chart description texts with actual mathematical representation labels. This allows the initial visual feature extraction module and the initial mathematical representation extraction module to be trained based on their respective optimization targets. The optimization strategies of the feature extraction module and the mathematical representation extraction module can be adjusted separately to avoid the problem of gradient disappearance caused by gradient interference between the feature extraction module and the mathematical representation extraction module during joint training. As a result, the present invention can improve the training effect of the chart parsing model. At the same time, the present invention trains the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction. This allows the initial visual feature extraction module and the initial mathematical representation extraction module to pay more attention to the data nature in the chart during training, thereby improving the accuracy of the trained model in extracting mathematical representations in the chart. At the same time, the present invention uses multiple types of charts corresponding to the same chart description text to train the feature extraction module and the mathematical representation extraction module. Through training on multiple types of charts, the chart parsing model can learn the commonalities and differences between different chart types, thereby exhibiting better generalization ability when facing unseen chart types.

[0125] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0126] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for training a graph parsing model, characterized in that: include: Constructing a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; Acquire multiple sample chart description texts with actual mathematical representation labels, and generate multiple sample charts corresponding to each of the sample chart description texts; Based on the multiple sample charts corresponding to the text describing each sample chart, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately based on numerical extraction to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

2. The method according to claim 1, characterized in that The method of training the initial visual feature extraction module based on numerical extraction alone includes: Determine N groups of data pairs, wherein each group of data pairs consists of a first sample chart description text selected from a plurality of sample chart description texts and a first sample chart generated based on the first sample chart description text; Determine the text feature vectors corresponding to the first sample chart description texts in the N groups of data pairs respectively, and use the initial visual feature extraction module to extract the numerical visual feature vectors corresponding to the first sample charts in the N groups of data pairs respectively; Calculate the same-group similarity between each text feature vector and the numerical visual feature vector in the same group of data pairs, and calculate the different-group similarity between each text feature vector and the numerical visual feature vector in the different groups of data pairs; Based on the same-group similarity and the different-group similarity, the parameters of the initial visual feature extraction module are adjusted to obtain the trained initial visual feature extraction module.

3. The method according to claim 1, characterized in that The method of training the initial mathematical representation extraction module based on numerical extraction alone includes: Selecting a second sample chart description text and its corresponding second sample chart required for training the initial mathematical representation extraction module from a plurality of sample chart description texts, and determining a first actual chart corresponding to the second sample chart description text; extracting a first image feature vector corresponding to the first actual chart and a second image feature vector corresponding to each second sample chart respectively through the trained initial visual feature extraction module; fusing the first image feature vector and a plurality of second image feature vectors to obtain a fused image feature vector, and performing a graphical analysis on the fused image feature vector using the initial mathematical representation extraction module to obtain a predicted mathematical representation; Based on the difference between the actual mathematical representation corresponding to the second sample graph description text and the predicted mathematical representation, a loss function corresponding to the initial mathematical representation extraction module is determined, and based on the loss function, the initial mathematical representation extraction module is iteratively trained to obtain the trained initial mathematical representation extraction module.

4. The method according to claim 3, characterized in that The plurality of second image feature vectors include at least a first sub-image feature vector and a second sub-image feature vector; The fusing the first image feature vector and the plurality of second image feature vectors to obtain a fused image feature vector includes: constructing a feature matrix based on the first image feature vector, the first sub-image feature vector, and the second sub-image feature vector, and determining a covariance matrix corresponding to the feature matrix; Performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvalues ​​and an eigenvector corresponding to each eigenvalue, selecting a preset number of eigenvalues ​​from each of the eigenvalues ​​as principal component eigenvalues ​​based on the magnitude of the eigenvalues, and forming a projection matrix using the eigenvectors corresponding to each of the principal component eigenvalues; The feature matrix is ​​projected based on the projection matrix to obtain a fusion feature matrix, and the fusion image feature vector is determined based on the fusion feature matrix.

5. The method according to claim 4, characterized in that The fusing the first image feature vector and the plurality of second image feature vectors to obtain a fused image feature vector includes: performing feature-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a feature fusion vector; performing element-level fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain an element-fused vector; performing low-order fusion processing on the first image feature vector, the first image sub-feature vector, and the second image sub-feature vector to obtain a low-order fusion vector; The feature fusion vector, the element fusion vector, and the low-order fusion vector are transformed using a preset transformation function to obtain the fused image feature vector.

6. The method according to claim 1, characterized in that The generating of a plurality of sample charts corresponding to each sample chart description text includes: Each sample chart description text is input into a preset chart generator to generate a chart, thereby obtaining a plurality of sample charts corresponding to each sample chart description text.

7. The method according to claim 1, characterized in that After separately training the initial visual feature extraction module and the initial mathematical representation extraction module based on a plurality of sample charts corresponding to each sample chart description text using numerical extraction as a benchmark to obtain a preset chart parsing model having the trained initial visual feature extraction module and the trained initial mathematical representation extraction module, the method further includes: Constructing an initial lightweight chart parsing model, wherein the initial lightweight chart parsing model includes a lightweight visual feature extraction module and a lightweight mathematical representation extraction module; Acquiring lightweight chart description text required for lightweighting the preset chart parsing model, and generating a plurality of lightweight charts corresponding to the lightweight chart description text; Determining, based on a second actual chart corresponding to the lightweight chart description text and a plurality of lightweight charts, image feature extraction differences between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and mathematical representation parsing differences between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module; determining a chart feature loss function based on the image feature extraction differences, and determining a mathematical representation loss function based on the mathematical representation analysis differences; A joint loss function is determined based on the graph feature loss function and the mathematical representation loss function, and the initial lightweight graph parsing model is iteratively trained based on the joint loss function to obtain a lightweight graph parsing model.

8. The method according to claim 7, characterized in that Determining, based on a second actual chart corresponding to the lightweight chart description text and multiple lightweight charts, image feature extraction differences between the trained initial visual feature extraction module and the lightweight visual feature extraction module, and mathematical representation parsing differences between the trained initial mathematical representation extraction module and the lightweight mathematical representation extraction module, including: extracting a first heavy-quantized image feature vector corresponding to the second actual chart and second heavy-quantized image feature vectors corresponding to multiple lightweight charts through the trained initial visual feature extraction module, fusing the first heavy-quantized image feature vector and the multiple second heavy-quantized image feature vectors to obtain a heavy-quantized image fusion feature vector, extracting a lightweight chart feature vector corresponding to the second actual chart through the lightweight visual feature extraction module, and using the difference between the lightweight chart feature vector and the heavy-quantized image fusion feature vector as the image feature extraction difference; The re-quantized image fusion feature vector is subjected to chart analysis by the trained initial mathematical representation extraction module to obtain a re-quantized chart mathematical representation. The lightweight chart feature vector is subjected to chart analysis by the lightweight mathematical representation extraction module to obtain a lightweight chart mathematical representation. The difference between the re-quantized chart mathematical representation and the lightweight chart mathematical representation is used as the mathematical representation analysis difference.

9. The method according to claim 1, characterized in that Based on a plurality of sample charts corresponding to the description text of each sample chart, the initial visual feature extraction module and the initial mathematical representation extraction module are trained separately based on numerical extraction, and after obtaining a preset chart parsing model including the trained initial visual feature extraction module and the trained initial mathematical representation extraction module, the method further includes: Get the chart to be parsed; The chart to be analyzed is input into the preset chart analysis model, and the numerical features of the chart to be analyzed are extracted by the trained initial visual feature extraction module to obtain a numerical visual feature vector. The numerical visual feature vector is analyzed by the trained initial mathematical representation extraction module to obtain a mathematical representation of the chart to be analyzed.

10. A training device for a graph parsing model, characterized in that: include: A construction unit, configured to construct a preset initial chart parsing model, wherein the preset initial chart parsing model includes an initial visual feature extraction module and an initial mathematical representation extraction module; a generating unit, configured to obtain a plurality of sample chart description texts with actual mathematical representation labels, and generate a plurality of sample charts corresponding to each of the sample chart description texts; A training unit is used to train the initial visual feature extraction module and the initial mathematical representation extraction module based on numerical extraction as a benchmark based on multiple sample charts corresponding to the text describing each sample chart, so as to obtain a preset chart parsing model with the trained initial visual feature extraction module and the trained initial mathematical representation extraction module.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images