Chart data analysis method, apparatus and device, and computer program product

Abstract

The embodiment of the invention discloses a chart data analysis method and device, equipment and a computer program product, and belongs to the field of artificial intelligence. The method comprises the steps of obtaining a to-be-recognized image, wherein the to-be-recognized image is a recognized image containing an analyzable chart; performing image quality enhancement on the to-be-identified image to obtain an enhanced image; extracting a chart image of the analyzable chart from the enhanced image; and based on the chart type of the chart contained in the chart image, performing structured data analysis on the chart in the chart image to obtain structured data of the chart in the chart image. By adopting the scheme provided by the embodiment of the invention, the accuracy of the structured data extracted from the chart contained in the image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a method, device, equipment and computer program product for analyzing data of a chart. Background Art

[0002] As a structured data display method, charts have the advantages of being intuitive and efficient in expressing information, making it easy for users to quickly obtain the information they want.

[0003] Since charts in pictures are often presented in visual form, extracting charts in pictures into structured data is of great significance for improving the efficiency of information utilization and mining the potential value of data. Summary of the invention

[0004] The present application provides a data analysis method, device, equipment and computer program product for a chart. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides a data parsing method for a chart, the method comprising:

[0006] Acquire an image to be identified, wherein the image to be identified is an image containing a parsable chart;

[0007] Performing image quality enhancement on the image to be identified to obtain an enhanced image;

[0008] extracting a chart image of the parseable chart from the enhanced image;

[0009] Based on the chart type of the chart included in the chart image, structured data parsing is performed on the chart in the chart image to obtain structured data of the chart in the chart image.

[0010] On the other hand, an embodiment of the present application provides a data parsing device for a chart, the device comprising:

[0011] An image acquisition module, used to acquire an image to be identified, wherein the image to be identified is an image containing a parsable chart;

[0012] An image processing module, used for enhancing the image quality of the image to be identified to obtain an enhanced image;

[0013] an image extraction module for extracting a chart image of the parseable chart from the enhanced image;

[0014] The parsing module is used to perform structured data parsing on the chart in the chart image based on the chart type of the chart contained in the chart image to obtain structured data of the chart in the chart image.

[0015] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the data parsing method of the chart as described in the above aspects.

[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the data parsing method of the chart as described in the above aspects.

[0017] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the data analysis method of the chart as described in the above aspects.

[0018] The embodiments of the present application optimize different stages in the chart data parsing pipeline. In the image preprocessing stage, by enhancing the image quality of an image containing a parsable chart and extracting a chart image of the parsable chart from the obtained enhanced image, the image quality of the chart image is improved, which in turn helps to improve the accuracy of the data parsed from the chart image. In the data parsing stage, by identifying the chart type of the chart contained in the chart image and performing structured data parsing on the chart in the chart image based on the chart type, the structured data of the chart is obtained, thereby achieving targeted structured data extraction for different types of charts, which helps to improve the accuracy of the structured data finally extracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a computer system provided by an embodiment of the present application is shown;

[0020] Figure 2 is a schematic diagram of the stages of a chart data parsing pipeline shown in an exemplary embodiment of the present application;

[0021] Figure 3 A flow chart showing a data parsing method for a chart provided by an exemplary embodiment of the present application is shown;

[0022] Figure 4 is a schematic diagram of an implementation of a feature extraction network pre-training process shown in an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an implementation of a classification network fine-tuning process shown in an exemplary embodiment of the present application;

[0024] Figure 6 A flowchart of an image correction process provided by an exemplary embodiment of the present application is shown;

[0025] Figure 7 is a schematic diagram of an implementation of a trapezoidal correction and a curved surface correction process shown in an exemplary embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of a moiré removal model shown in an exemplary embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of an exemplary embodiment of the present application showing a basic encoding and decoding module performing a feature encoding and decoding process;

[0028] Fig.10 A flowchart of a moiré removal model training process provided by an exemplary embodiment of the present application is shown;

[0029] Fig.11 is a schematic diagram of an implementation of a loss determination process of a moiré removal model provided by an exemplary embodiment of the present application;

[0030] Fig.12 is a schematic diagram of an implementation of a graph parsing process shown in an exemplary embodiment of the present application;

[0031] Fig.13 is a schematic diagram of an implementation of an encoder encoding process shown in an exemplary embodiment of the present application;

[0032] Fig.14 is a schematic diagram of an implementation of a decoding process of a decoder shown in an exemplary embodiment of the present application;

[0033] Fig.15 A structural block diagram of a data parsing device for a chart provided by another exemplary embodiment of the present application is shown;

[0034] Fig.16 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0036] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0037] Please refer to Figure 1 , which shows a schematic diagram of a computer system provided in an embodiment of the present application, the computer system includes a terminal 110 and a server 120.

[0038] The terminal 110 is an electronic device having a function of extracting chart data from an image. The electronic device may be a smart phone, a tablet computer, a personal computer, etc. In some embodiments, the electronic device may provide a function of extracting chart data from an image through a system or an application (native or third-party application).

[0039] Optionally, the terminal 110 can use the function of extracting chart data from images for locally taken photos, network downloaded pictures, screenshots, or documents containing pictures. For example, the terminal 110 can use a camera to take a photo containing a chart, and then use the function of extracting chart data from images provided by the system to extract structured data of the chart in the photo.

[0040] The server 120 is a server that provides services for extracting chart data from images, and the server may be a single physical server, a server cluster composed of servers, or a cloud computing center. Optionally, the server 120 may be a server that provides services for the system running on the terminal 110, or a background server of an application running on the terminal 110.

[0041] In some embodiments, Figure 2 As shown, the chart data parsing pipeline can include the following stages:

[0042] Algorithm input stage: The image to be identified is obtained in various ways, such as locally taken photos, images downloaded from the Internet, screenshots, or images in documents as images to be identified.

[0043] Rough classification stage: This stage is used to determine whether there is a chart that supports structured data analysis in the image to be identified. If so, it will enter the next stage. If not, it will terminate. The significance of this stage is to preliminarily screen the image to avoid the waste of processing resources caused by performing structured data analysis on any image.

[0044] Image quality enhancement stage: This stage is used to enhance the image quality of the images that support structured data parsing determined in the coarse classification stage. The image quality enhancement methods may include trapezoidal correction, surface correction, moiré removal, deblurring, etc. The purpose of image quality enhancement is to ensure the input image quality (especially for photographic images) during subsequent structured data parsing and improve the accuracy of structured data parsing.

[0045] Sub-classification stage: This stage is used to divide the charts in the image by bounding boxes (bboxes) and to sub-classify the charts in each bounding box. The sub-classification is used to confirm whether the charts in the image meet the specifications of the subsequent structured parsing algorithm in more detail, so as to avoid the problem of poor parsing results caused by parsing out-of-specification charts. The sub-classification types can include text, mind maps, flowcharts, bar charts, pie charts, line charts, tables, formulas, etc.

[0046] Data parsing stage: This stage is used to parse the chart images that meet the specifications of the structured parsing algorithm divided in the previous stage to obtain the structured data of the chart contained in the chart image. In some embodiments, the structured parsing algorithm is implemented based on multiple models, and different models are used to perform structured data parsing on charts of different subdivision types. Optionally, text is parsed using OCR (Optical Character Recognition), mind maps, flow charts, bar charts, pie charts, line charts Figure 1 The graph of the class uses the graph parsing model to parse structured data, the table uses the table parsing model to parse structured data, and the formula uses the formula parsing model to parse.

[0047] Algorithm post-processing stage: This stage is used to post-process the parsed structured data based on downstream tasks, such as data assembly or application-level rendering.

[0048] In a possible implementation, all operations in the chart data parsing pipeline except the algorithm input stage are performed by the server 120. Figure 1 As shown, after the terminal 110 obtains the image to be identified through local photography, documents or pictures, the image to be identified is uploaded to the server 120. The server 120 extracts the structured data of the chart contained in the image to be identified and feeds it back to the terminal 110, so that the terminal 110 can reconstruct and edit the chart based on the structured data.

[0049] In another possible implementation, operations in stages other than the algorithm input stage in the chart data parsing pipeline are performed jointly by the terminal 110 and the server 120. For example, the terminal 110 can perform rough classification locally, and upload the images that support structured data parsing determined in the rough classification stage to the server 120, and the server 120 performs operations in subsequent stages. Of course, the terminal 110 can also perform operations in more other stages, such as image quality enhancement can be performed by the terminal 110, and accordingly, the server 120 can perform operations in fewer stages.

[0050] In other possible implementations, when the terminal 110 has strong computing capabilities, operations at each stage in the chart data parsing pipeline may be executed by the terminal 110 .

[0051] In the following embodiments, for the convenience of description, the data analysis method of the chart is executed by a computer device (which can be executed by a terminal alone, a server alone, or both interactively) as an example for description.

[0052] Please refer to Figure 3 , which shows a flow chart of a data parsing method for a chart provided by an exemplary embodiment of the present application. The method is executed by a computer device, and the method may include the following steps.

[0053] Step 301 : obtaining an image to be identified, where the image to be identified is an image containing a parsable chart.

[0054] In a possible implementation, the computer device roughly classifies the images to identify images containing parseable charts and images not containing parseable charts. The images containing parseable charts are subsequently determined as images to be identified, and subsequent analysis processes are performed on the images to be identified.

[0055] In some embodiments, the parseable chart includes a specified type of chart, table, and formula. The specified type of chart may include a mind map, a bar chart, a pie chart, a line chart, an architecture chart, etc. The embodiments of the present application do not limit the specific types of parseable charts.

[0056] Optionally, the image to be identified includes at least one parseable chart, and the image to be identified may also include unparseable charts other than parseable charts, such as charts other than specified types.

[0057] Step 302: enhance the image quality of the image to be recognized to obtain an enhanced image.

[0058] When the image to be identified is a photographed image, due to the influence of factors such as shooting angle, shooting jitter, and interference fringes, the chart in the image to be identified may have problems such as deformation, blur, and moiré, which affects the accuracy of subsequent structured data analysis. Therefore, in this embodiment, before performing structured data analysis, the computer device first enhances the image quality of the image to be identified.

[0059] Optionally, the image quality enhancement method includes at least one of image correction (including trapezoidal correction and curved surface correction), de-moiré, and de-blurring. Image correction is used to eliminate the influence of the deformation of the chart caused by the shooting angle in the document shooting scene; de-moiré is used to eliminate the interference fringes (moiré) generated during the shooting process; and de-blurring is used to improve the clarity of the image.

[0060] In some embodiments, the computer device can selectively enhance the quality of the image to be recognized according to the image source of the image to be recognized. For example, for a photo of text, the computer device performs image correction, de-moiré and de-blurring on the photo; for a screenshot of a document, the computer device performs de-blurring on the screenshot (screenshots usually do not have image deformation and moiré problems).

[0061] Step 303: extracting a chart image of a parseable chart from the enhanced image.

[0062] Since the original image to be identified may contain multiple parseable charts and may contain unparseable charts, in order to improve the accuracy of analysis, the computer device extracts the chart image containing independent parseable charts from the added image.

[0063] When the image to be identified contains multiple parseable charts, the computer device extracts multiple chart images from the enhanced image.

[0064] In some embodiments, the computer device uses a target detection method to perform bounding box segmentation on the enhanced image, and then performs image segmentation on the enhanced image according to the bounding box to obtain a chart image.

[0065] In a possible implementation, when extracting a chart image, the computer device may also set a chart type label for the chart image according to the chart type of the parseable chart contained in the chart image, so as to subsequently perform structured data parsing based on the label using a matching structured data parsing method.

[0066] Optionally, the computer device performs target detection on the enhanced image through a target detection model to determine the bounding boxes corresponding to each chart.

[0067] In some embodiments, the computer device applies non-maximum suppression (NMS) to the output of the target detection model, removes redundant objects caused by dense and multi-scale target detection by setting the score and NMS threshold, and outputs labels and bounding boxes. The score and NMS threshold can be implemented through a statistical scheme, that is, the score and NMS threshold are set by the percentile of the predicted output in the statistical test set.

[0068] Step 304 , based on the chart type of the chart contained in the chart image, perform structured data analysis on the chart in the chart image to obtain structured data of the chart in the chart image.

[0069] Since charts of different chart types have different characteristics, for example, a graph is usually composed of a legend, graph nodes, and edges connecting the graph nodes, while a table is usually composed of table dividing lines and cells, in order to improve the accuracy of structured data parsing, in one possible implementation, for different chart types, the computer device uses different methods to parse the structured data of the chart.

[0070] In a possible implementation, the computer device pre-trains a graph parsing model and a table parsing model, so that the graph parsing model is used to parse the structured data of the graph, and the table parsing model is used to parse the structured data of the table. Of course, for different types of graphs, the computer device trains different graph parsing models to further improve the parsing accuracy of structured data of specific types of graphs.

[0071] In summary, the embodiments of the present application optimize different stages in the chart data parsing pipeline. Among them, in the image preprocessing stage, by enhancing the image quality of the image containing the parseable chart and extracting the chart image of the parseable chart from the obtained enhanced image, the image quality of the chart image is improved, which in turn helps to improve the accuracy of the data parsed from the chart image. In the data parsing stage, by identifying the chart type of the chart contained in the chart image and performing structured data parsing on the chart in the chart image based on the chart type, the structured data of the chart is obtained, thereby achieving targeted structured data extraction for different types of charts, which helps to improve the accuracy of the structured data finally extracted.

[0072] The process of acquiring the image to be recognized

[0073] In one possible implementation, a computer device is provided with an image classification model trained based on training set images (including images containing parseable charts and images not containing parseable charts), and the image classification model is used to classify images containing parseable charts and images not containing parseable charts.

[0074] When an input image is acquired, the computer device classifies the input image through an image classification model, thereby determining an image to be identified based on the image classification result output by the image classification model.

[0075] In some embodiments, the classification of input images can be performed by a server (i.e., an image classification model is deployed on the cloud), or by a terminal (i.e., an image classification model is deployed on the terminal side), thereby filtering the input images that do not contain parsable graphs on the terminal side to avoid uploading the images that do not contain parsable graphs to the cloud.

[0076] Optionally, the image classification model can be based on Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Feature Pyramid Networks (FPN), etc., which is not limited to the embodiments of the present application.

[0077] Since the input image types are complex and diverse, it is essentially an open set recognition (OSR), which has the problems of difficulty in constructing unknown class data sets and difficulty in feature extraction and clustering. Therefore, in order to improve the accuracy of image classification, compared with the data-driven image classification model, in the embodiment of the present application, the image classification task is divided into two subtasks: feature extraction pre-training and feature classification fine-tuning.

[0078] In a possible design, the image classification model includes a feature extraction network and a classification network, and the feature extraction network is obtained through pre-training, and the classification network is obtained by fine-tuning after the feature extraction network pre-training is completed. Accordingly, the training process of the image classification model may include the following steps:

[0079] Step 1: During the pre-training process, feature extraction is performed on sample images and sample mixed images through a feature extraction network to obtain sample image features and sample mixed image features. The sample mixed image is obtained by scaling and mixing different sample images.

[0080] The sample image is an image containing a parsable chart. Optionally, each sample image contains a single parsable chart.

[0081] In a possible implementation, the computer device expands the training data by resizing and mixing (resizemix) to achieve the effect of simulating location category samples. The computer device resizes the sample images (such as scaling), and then mixes the resized sample images to obtain a sample mixed image, which includes multiple parseable charts.

[0082] Optionally, the sample mixed image classification label corresponding to the sample mixed image is a soft label (soft label) obtained by determining the sample image classification label based on the sample image and the area ratio of the resized sample image in the sample mixed image.

[0083] In some embodiments, the sample image is further subjected to image enhancement processing. Optionally, the computer device adopts a random data enhancement method, randomly selects a set of enhancement operations from a predefined operation pool, and randomly applies different strengths to each enhancement operation.

[0084] For the sample image and the sample mixed image, the computer device performs feature extraction through a feature extraction network to obtain sample image features corresponding to the sample image and sample mixed image features corresponding to the sample mixed image.

[0085] In one possible design, the MobileNet network is used as the feature extraction network in the embodiment of the present application, and the fully connected layer of the original classification head of the network is modified to a linear layer with spectral normalization (SpectralNorm) as the feature projection layer. Among them, spectral normalization was originally used to fight against the generative network, which is used to perform singular value decomposition on the parameters of each layer in the neural network, limit the maximum singular value to 1, and divide it by its maximum singular value after each weight update, effectively limiting the drastic impact of numerical changes and making the gradient more stable when the weight is updated.

[0086] Indicatively, Figure 4 As shown, the computer device performs random data enhancement and resizemix processing on the sample image 401 to obtain the enhanced sample image 401 and the sample mixed image 402. The feature extraction network 403 performs feature extraction to obtain the sample image feature 404 and the sample mixed image feature 405.

[0087] Step 2: Based on the classification loss, the sample image features are subjected to gradient backpropagation perturbation processing to obtain the sample perturbation image features. The classification loss is determined based on the classification result of the sample image features by the auxiliary classification network.

[0088] In a possible implementation, during the pre-training phase of the feature extraction network, the computer device adds an auxiliary classification network after the feature extraction network, uses the auxiliary classification network to perform image classification based on sample image features, obtains a classification result, and determines the classification loss corresponding to the sample image based on the classification result and the sample image classification label corresponding to the sample image. The classification loss may be a cross entropy loss.

[0089] Furthermore, the computer device uses the classification loss to perform gradient backpropagation perturbation processing on the sample image features, that is, the gradient of the classification loss is used as the perturbation of the sample image features. The perturbation of the gradient backpropagation perturbation process can be expressed as:

[0090]

[0091] Among them, g represents the gradient of classification loss and ε is a hyperparameter that controls the size of perturbation.

[0092] In some embodiments, the computer device adds the sample image feature to the disturbance to obtain the sample perturbation image feature. The sample perturbation image feature will be subsequently used in the subsequent loss function calculation for pre-training the feature extraction network.

[0093] Gradient perturbation can enhance the robustness of neural networks against adversarial attacks. By adding perturbations to sample image features, the model will learn features that are resistant to these perturbations during training, allowing the model to learn more robust feature representations to resist the deception of adversarial samples, thereby improving the robustness and generalization of the model.

[0094] Optionally, the auxiliary classification network may be a multi-layer perceptron (MLP).

[0095] Indicatively, Figure 4 As shown, the computer device uses the auxiliary classification network 406 to classify the sample image features output by the feature extraction network 403, and determines the classification loss 407 of the sample image 401 based on the classification result and the sample image label of the sample image 401. Further, the computer device uses the classification loss 407 to perform gradient back propagation perturbation processing on the sample image feature 404 to obtain a sample perturbed image feature 408.

[0096] Step 3: Determine the feature extraction loss based on the sample image features, the sample mixed image features and the sample perturbation image features.

[0097] In order to improve the accuracy and robustness of feature extraction, the feature extraction loss used to train the feature extraction network consists of two parts: the main feature extraction loss and the perturbation adversarial loss. In a possible implementation, this step may include the following sub-steps:

[0098] Sub-step 1: Determine the Info NCE (Information Noise Contrastive Estimation) loss based on the sample image features and the sample perturbation image features.

[0099] The Info NCE loss is established to enable the feature extraction network to better distinguish between original samples and perturbed samples (i.e., sample image features and sample perturbed image features). In addition, since the perturbed samples are obtained by gradient backpropagation perturbation, the feature extraction network can generate classification-oriented image features as much as possible while distinguishing between the two.

[0100] Optionally, the Info NCE loss can be expressed as:

[0101]

[0102] Among them, q is the sample image feature of the sample image, k + is a positive sample (i.e., sample perturbation image feature), k i is a negative sample, K is the number of samples, and τ is a temperature hyperparameter used to control the smoothness of the distribution.

[0103] Indicative, such as Figure 4 As shown, the computer device determines Info NCE loss 409 based on sample image features 404 and sample perturbation image features 408 .

[0104] Sub-step 2: Determine the SupCon loss based on the sample image features, the sample mixture image features, the sample image classification label of the sample image, and the sample mixture image classification label of the sample mixture image.

[0105] Inspired by supervised contrastive learning, the open set recognition problem is essentially to tighten the decision boundary of the same category as much as possible and to widen the distance between different categories as much as possible. Due to the use of resizemix data enhancement, it is difficult to use the traditional supervised contrastive learning loss function to calculate the sample mixed image. Therefore, in the embodiment of this application, a more generalized loss is used for calculation, which can be expressed as:

[0106]

[0107] Where y represents the sample label (the sample image classification label of the sample image and the sample mixed image classification label of the sample mixed image), s(·) represents the similarity calculation between labels (the introduction of label similarity makes full use of the soft labels of the sample mixed image), z represents the sample features (sample image features and sample mixed image features), and τ is the temperature hyperparameter used to control the smoothness of the distribution.

[0108] The above loss SupCon loss can tighten the decision boundary when the model makes decisions in the same category, while increasing the distance between different categories, so as to better distinguish rejected samples. In addition, the pre-training stage follows the idea of ​​multi-task learning, using cross entropy loss, InfoNCE loss and SupCon loss for joint optimization, which improves the training quality of the feature extraction network.

[0109] Indicative, such as Figure 4 As shown, the computer device determines the SupCon loss 412 based on the sample image feature 404 , the sample mixed image feature 405 , the sample image classification label 410 of the sample image 401 , and the sample mixed image classification label 411 of the sample mixed image 402 .

[0110] Sub-step 3: Determine the Info NCE loss and SupCon loss as feature extraction losses.

[0111] In a possible implementation, the computer device determines the sum of the Info NCE loss and the SupCon loss as the feature extraction loss of the feature extraction network. Optionally, the computer device determines the weighted sum of the two as the feature extraction loss.

[0112] Step 4: Train the feature extraction network based on the feature extraction loss.

[0113] Optionally, the computer device uses gradient descent and back propagation algorithms to train the feature extraction network.

[0114] In some embodiments, when the feature extraction loss converges, or the number of training rounds reaches a threshold, the computer device determines that the feature extraction network training is completed.

[0115] Indicative, such as Figure 4 As shown, the computer device trains the feature extraction network 403 based on the Info NCE loss and the SupCon loss 412.

[0116] Step 5: After the feature extraction network is pre-trained, the feature extraction network is used to extract features from the sample image to obtain sample image features.

[0117] After the pre-training of the feature extraction network is completed, the computer device uses the feature extraction results of the trained feature extraction network as the input of the classification network to fine-tune the classification network. In a possible implementation, when fine-tuning the classification network, the computer device uses the sample image features corresponding to the sample image as the input of the classification network, that is, fine-tunes the classification network based on the classification results of the sample image by the classification network.

[0118] Step 6: Based on the sample image features, classification is performed through the classification network to obtain a sample classification result of the sample image.

[0119] In one possible implementation, similar to the feature extraction network, the classification network is newly added with a spectral normalization fully connected layer as a classification head, and the classification task is trained on the sample image. Of course, in other possible implementations, the computer device may not introduce a new structure, directly load the original network weights, and only update the network parameters of the auxiliary classification network.

[0120] Step 7: Fine-tune the classification network based on the sample classification results and the sample image classification labels of the sample images.

[0121] In a possible implementation, the computer device determines the image classification loss based on the sample classification result and the sample image classification label, and fine-tunes the classification network based on the image classification loss. In the process of fine-tuning the classification network, the parameters of the feature extraction network are frozen, that is, the parameters of the feature extraction network are not updated during the process of fine-tuning the classification network.

[0122] Optionally, the computer device image classification model correctly classifies the sample on the validation set and uses the percentile as the classification threshold. In the subsequent image classification process, when the threshold of the image belonging to the image containing a parseable chart is lower than the classification threshold, it is determined that the image does not contain a parseable chart.

[0123] Indicative, such as Figure 5 As shown, after the feature extraction network training is completed, the computer device extracts features of the sample image 501 through the feature extraction network 502 to obtain sample image features 503, and then classifies the image based on the sample image features 503 through the classification network 504 to obtain a sample classification result 505. After determining the image classification loss 507 based on the sample classification result 505 and the sample image classification label 506 corresponding to the sample image 501, the computer device fine-tunes the classification network 504 based on the image classification loss 507 and freezes the parameters of the feature extraction network 502.

[0124] In some embodiments, in order to further improve the generalization performance of the model, the computer device can add some samples of unknown categories to the training set. During each training process, for each known category, the cosine similarity matrix of the unknown category samples and the known category samples is calculated, and sampling with replacement is performed to obtain the top N difficult negative samples to add to the training, where N is the corresponding batch size divided by the total number of categories.

[0125] The introduction of hard negative samples can effectively improve the effect of the contrastive learning paradigm and obtain better generalization performance. At the same time, simple unknown samples will lead to oversimplification of model fitting, which will affect the performance of the model. Therefore, the hard negative sample introduction strategy can effectively avoid this problem.

[0126] In this embodiment, the computer device uses two tasks, namely, a pre-trained feature extraction network and a fine-tuned classification network, to train the image classification model, which can solve the problem of difficulty in constructing unknown class data sets and difficulty in feature extraction and clustering during open set recognition, and helps to improve the accuracy of image classification. In addition, during the training of the feature extraction network, the training data is expanded by resizemixing the sample image to generate a sample mixed image; by performing gradient backpropagation perturbation on the sample image features and introducing the Info NCE loss between the sample image features and the sample perturbation image features, it helps to improve the robustness of the feature extraction process.

[0127] In one possible design, the operations of training the image classification model and classifying the input image can be performed by the server. In another possible design, the image classification model can be trained by the server and then deployed on the terminal side (the image classification model adopts a lightweight model architecture), and the terminal classifies the input image to reduce the processing pressure of the server.

[0128] In addition, in addition to being used in the coarse classification stage to distinguish between images containing parseable charts and images that do not contain parseable charts, in a possible implementation, the computer device can also use the image classification model to perform image classification on chart images to determine whether the chart contained in the chart image is a parseable chart.

[0129] Furthermore, the computer device may use the image classification model to sub-classify the chart contained in the chart image, so as to determine the sub-classification type of the contained chart, for example, identifying the chart as a graph, a table, a formula, etc., or further identifying the sub-classified chart type.

[0130] Generally, the captured images usually have problems such as deformation, moiré and blur, which will affect the subsequent analysis of chart data. Therefore, the computer device performs at least one of image correction processing, de-moiré processing and de-blurring processing on the image to be identified to obtain an enhanced image. Among them, image correction processing is used to correct deformation, de-moiré processing is used to remove interfering moiré in the image, and de-blurring processing is used to improve image clarity.

[0131] Image correction processing

[0132] Image correction usually includes trapezoidal correction and surface correction, which are often used to correct photographed documents. In the embodiment of the present application, a two-stage image correction method is proposed, wherein the first stage is used to detect image vertices and implement trapezoidal correction based on the vertices; the second stage is used to estimate deep information and calculate offset fields, and implement surface correction through pixel offset regression.

[0133] In a possible implementation, Figure 6 As shown, the image correction process may include the following steps.

[0134] Step 601, performing image vertex detection on the image to be identified through the target detection network to obtain the corrected image vertices in the image to be identified, where the corrected image vertices are the vertices of the image region to be corrected in the image.

[0135] In related technologies, trapezoidal correction usually includes multi-stage tasks such as image edge detection, probabilistic straight line fitting, and image intersection calculation. Line fitting and quadrilateral selection are based on post-processing logic, and complex edge segmentation scenarios will bring problems such as time-consuming fitting, unreasonable threshold selection, and unstable output results.

[0136] In the embodiment of the present application, the multi-stage task is converted into a corrected image vertex detection task, and end-to-end corrected image vertex detection is achieved by pre-training a target detection network for identifying corrected image vertices in an image.

[0137] In one possible design, the target detection network is based on a lightweight target network and a newly enhanced quantized semantic segmentation head, thereby simultaneously performing target detection and image segmentation based on a semantic segmentation algorithm. During the training of the target detection network, the computer device uses the semantic segmentation results of the semantic segmentation head to assist in the training of the target detection task.

[0138] In a possible implementation, the target detection network has a vertex output head and an image segmentation head, wherein the vertex output head is used to output corrected image vertices, and the image segmentation head is used to output an image region to be corrected in the image.

[0139] Optionally, the corrected image vertices are four vertices of the photographed document, and the image area to be corrected is a text area in the image.

[0140] Optionally, the target detection network can use YOLO, SSD, R-CNN, etc. as the backbone network, which is not limited in the embodiments of the present application.

[0141] In some embodiments, the training process of the object detection network may include the following steps:

[0142] Step 1: Input the sample image into the target detection network to obtain the sample corrected image vertices output by the vertex output head in the target detection network and the sample image area to be corrected output by the image segmentation head in the target detection network.

[0143] The sample image has true values ​​of the corrected image vertices and true values ​​of the image area to be corrected, wherein the true values ​​of the corrected image vertices are used to indicate the positions of the corrected image vertices in the sample image, such as the vertex coordinates of multiple document vertices; the true values ​​of the image area to be corrected are used to indicate the image area to be corrected marked in the sample image.

[0144] In a possible implementation, the computer device simultaneously performs target detection and image segmentation on the sample image through semantic segmentation (for example, based on the BiSeNetV2 semantic segmentation algorithm) to obtain the sample corrected image vertices and the sample image area to be corrected.

[0145] Step 2: Determine the vertex detection loss based on the sample corrected image vertices, the true values ​​of the corrected image vertices of the sample image, the sample image area to be corrected, and the true values ​​of the image area to be corrected of the sample image.

[0146] In a possible design, the vertex detection loss consists of three parts of loss: 1. Corrected image vertex position loss; 2. Edge segmentation loss of the image region to be corrected; 3. Mask prediction loss of the image region to be corrected.

[0147] Among them, the corrected image vertex position loss is used to characterize the difference between the sample corrected image vertices obtained by target detection and the true values ​​of the corrected image vertices; the edge segmentation loss of the image area to be corrected is used to characterize the difference between the edge of the sample image area to be corrected obtained by image segmentation and the edge indicated by the true value of the image area to be corrected; the mask prediction loss of the image area to be corrected is used to characterize the mask difference between the sample image area to be corrected obtained by image segmentation and the image area to be corrected represented by the true value of the image area to be corrected.

[0148] In one possible implementation, the vertex detection loss determination process may include the following sub-steps:

[0149] Sub-step 1: Determine the vertex position loss based on the rectified image vertex and the true value of the rectified image vertex.

[0150] In a possible implementation, the computer device determines the true value of the rectified image vertex corresponding to each rectified image vertex, thereby determining the L2 loss between the two as the vertex position loss.

[0151] Since the corrected image vertices output by the vertex output head may be missing, in some embodiments, when the predicted corrected image vertices are missing, the computer device determines the coordinates of the missing corrected image vertices as preset values, such as (0,0).

[0152] Sub-step 2: determining edge segmentation loss based on the sample region edge of the sample image region to be corrected and the region edge true value of the true value of the image region to be corrected.

[0153] In one possible implementation, the computer device determines the edge of the sample area to be corrected based on the sample area to be corrected output by the image segmentation head, and determines the edge segmentation loss based on the difference between the edge of the sample area and the true value of the area edge of the true value of the image area to be corrected.

[0154] Optionally, the edge segmentation loss adopts diceloss.

[0155] Sub-step 3: determining a regional mask loss based on the sample regional mask of the sample image region to be corrected and the regional mask true value of the true value of the image region to be corrected.

[0156] In one possible implementation, a computer device determines a sample region mask based on a sample image region to be corrected output by an image segmentation head, and determines a region mask true value indicated by a region true value of the image region to be corrected, thereby determining a region mask loss based on the difference between the masks.

[0157] Optionally, the region mask loss adopts diceloss, and the dice system in diceloss is determined based on the IoU (Intersection over Union) of the sample region mask and the true value of the region mask.

[0158] Sub-step 4: Determine the vertex position loss, edge segmentation loss, and region mask loss as the vertex detection loss.

[0159] Further, the computer device determines the sum of the vertex position loss, the edge segmentation loss, and the region mask loss as the vertex detection loss. Optionally, the vertex detection loss is the weighted sum of the three losses, wherein the weights of different loss items can be fixed weights or dynamically adjusted at different training stages.

[0160] Step 3: Train the object detection network based on vertex detection loss.

[0161] In one possible implementation, the computer device trains the target detection network through gradient descent and back propagation algorithms with the goal of minimizing vertex loss.

[0162] Vertex position prediction is the main task, and edge prediction and mask prediction are auxiliary tasks. During the training process, the vertex output head and image segmentation head of the target detection network are trained synchronously. In the application process, the image segmentation head corresponding to the auxiliary task does not participate in the calculation, that is, the computer device performs image vertex detection on the image to be identified through the target detection network, and obtains the corrected image vertices output by the vertex output head.

[0163] Step 602: determine a perspective transformation matrix based on the vertices of the corrected image, and perform trapezoidal correction on the image to be recognized based on the perspective transformation matrix.

[0164] In a possible implementation, the computer device determines the perspective transformation matrix based on the original coordinates and the target coordinates of the corrected image vertices. For example, when four corrected image vertices are detected, the computer device determines the linear transformation equations corresponding to each corrected image vertex (used to transform the corrected image vertex from the original coordinates to the target coordinates) based on the original coordinates and the target coordinates of the four corrected image vertices, thereby constructing a 3×3 perspective transformation matrix based on the linear transformation equation group.

[0165] Furthermore, the computer device prompts and corrects the pixel points in the image area to be corrected in the image to be identified based on the perspective transformation matrix.

[0166] In some embodiments, after determining the vertices of the corrected image, the computer device needs to perform some post-processing. The post-processing process may include:

[0167] 1. Determine the vertices of the corrected image through non-maximum suppression (NMS);

[0168] 2. Determine the rationality of the polygon formed by the vertices of the corrected image;

[0169] 3. According to the vertex coordinates of the corrected image vertices, determine whether the image to be identified needs to be filled (padding);

[0170] 4. Get the image background color of the image to be recognized and fill the image. If there is filling in the two upper left directions, update the vertex coordinates of the corrected image vertices.

[0171] Step 603, performing curved surface correction on the image to be recognized after trapezoidal correction to obtain an enhanced image.

[0172] In one possible implementation, a computer device performs edge cropping on the image to be recognized after trapezoidal correction, and inputs the cropped image into a surface correction network to obtain a surface correction offset field, thereby using the surface correction offset field to perform pixel offset to achieve surface correction and obtain a surface corrected image (if de-moiré processing and deblurring processing are not performed, the surface corrected image is an enhanced image).

[0173] Optionally, the tasks of the surface correction network include depth information estimation and correction point offset regression, wherein depth information estimation serves as an auxiliary task to assist the training of the main task of correction point offset regression.

[0174] In some embodiments, the surface correction network can use PaperEdge, UVDoc, etc. as the backbone network, which is not limited in the embodiments of the present application.

[0175] Indicatively, Figure 7 As shown, in the trapezoidal correction stage, the computer device outputs the corrected image vertices 704 in the image to be identified 701 through the vertex output head corresponding to the corrected image vertex detection task 703 in the target detection network (the image segmentation task 702 is used to assist in training during the training process), and then uses the perspective matrix 705 determined based on the corrected image vertices 704 to perform trapezoidal correction on the image to be identified 701 to obtain a trapezoidal corrected image 706.

[0176] In the surface correction stage, the computer device outputs the correction offset field 709 through the branch corresponding to the correction point offset regression task 708 in the surface correction network (the depth of field information estimation task 707 is used to assist in training during the training process), and then uses the correction offset field 709 to perform surface correction on the trapezoidal correction image 706 to obtain a surface correction image 710.

[0177] In this embodiment, the computer device uses an end-to-end corrected image vertex detection algorithm to determine the corrected image vertices in the image to be identified, and then performs trapezoidal correction based on the determined corrected image vertices. Since the end-to-end corrected image vertex detection algorithm is data-driven, it can improve the efficiency of trapezoidal correction on the one hand, and avoid the output instability problem caused by multi-stage tasks on the other hand, thereby improving the trapezoidal correction quality of the image to be identified.

[0178] Moiré removal

[0179] In a possible implementation, the computer device performs moiré removal processing on the image to be identified using a moiré removal model to obtain a moiré-removed image (without subsequent image correction and deblurring processing, the moiré-removed image is the enhanced image).

[0180] In some embodiments, the input of the moiré removal model may be an image to be recognized, or an image after image correction, or an image after deblurring.

[0181] In one possible design, the moire removal model adopts a codec structure, in which the encoder and decoder are jump-connected and are composed of basic codec modules based on transformers. With the powerful knowledge compression and mapping learning capabilities of transformers, the model can be easily expanded with the data scale, and development optimization for different parameter models can be achieved.

[0182] Indicative, such as Figure 8 As shown, the moiré removal model 810 adopts a codec structure, including an encoder 811 and a decoder 812, and the network layer between the encoder 811 and the decoder 812 is skipped. After the image to be recognized 820 is input into the moiré removal model 810, the moiré removal model 810 outputs a de-moiré image 830.

[0183] In a possible design, the basic codec module is called TBB (Transformer-Boosted Block). Based on the traditional codec module, the basic codec module uses transformers to perform feature enhancement (boost) of SA (Spatial Attention) and CA (Channel Attention) in parallel with Spatial and Channel as tokens.

[0184] In a possible implementation, the basic encoding and decoding module includes a first feature processing branch and a second feature processing branch, wherein the first feature processing branch may also be referred to as an SA branch, and the second feature processing branch may also be referred to as a CA branch.

[0185] In some embodiments, the first feature processing branch is used to perform spatial dimension reduction and feature stretching processing on the input features to obtain a spatial representation; perform spatial attention processing on the spatial representation through a transformer; and perform feature restoration on the spatial representation after the spatial attention processing to obtain a spatial attention (Spatial Attention) feature.

[0186] Optionally, the input features are first subjected to PixelShuffleDown for spatial dimensionality reduction, and then stretched into SpatialTokens (i.e., spatial representation) through SpatialFlatten and Transpose so as to be fed into the transformer for spatial attention processing and equal-length sequence output.

[0187] After spatial attention processing, the processed SpatialTokens are first processed by SpatialUnFlatten+Transpose (restored to the previous shape), and then spatially upsampled to the spatial attention features at the original resolution through PixelShuffleUp.

[0188] Indicative, such as Fig. 9 As shown, the feature 901 output by the codec block is first spatially downsampled in the first feature processing branch to obtain the spatial downsampled feature 902, and then the spatial representation 903 is obtained by feature stretching. The spatial representation 903 is processed by the transformer based on spatial attention to obtain the spatial representation 904 after the spatial attention processing. Furthermore, the spatial representation 904 after the spatial attention processing is feature restored to obtain the spatial restored feature 905, and the spatial restored feature 905 is spatially upsampled to obtain the spatial attention feature.

[0189] In some embodiments, the second feature processing branch is used to perform channel dimension reduction and feature stretching processing on the input features to obtain channel representation; perform channel attention processing on the channel representation through transformer; perform feature restoration on the channel representation after channel attention processing to obtain channel attention features.

[0190] Optionally, the input features are first subjected to 1*1 convolution for channel dimension reduction, and then converted into ChannelTokens (i.e. channel representation) through SpatialFlatten so as to be sent to the transformer for channel attention processing and equal-length sequence output.

[0191] After channel attention processing, the processed ChannelTokens are first processed by SpatialUnFlatten (restoring them to their previous shape), and then the channel dimension is increased through 1*1 convolution to obtain the channel attention features under the original number of channels.

[0192] Indicative, such as Fig. 9 As shown, the feature 901 output by the codec block is first subjected to channel dimension reduction in the second feature processing branch to obtain a channel dimension reduction feature 906, and then subjected to feature stretching to obtain a channel representation 907. The channel representation 907 is subjected to transformer processing based on channel attention to obtain a channel representation 908 after channel attention processing. Further, the channel representation 908 after channel attention processing is subjected to feature restoration to obtain a channel restoration feature 909, and the channel restoration feature 909 is subjected to channel dimension increase to obtain a channel attention feature.

[0193] After performing spatial attention and channel attention processing through the dual feature processing channel, the basic codec module performs residual connection on the spatial attention features, channel attention features and input features. The features obtained by the residual connection are the output features of the basic codec block.

[0194] Indicatively, Fig. 9 As shown, the spatial attention feature, channel attention feature and feature 901 are subjected to residual connection processing to obtain output feature 910.

[0195] Since the basic codec block has upsampling, downsampling and identity mapping requirements in different coding and decoding stages, in a possible implementation, the basic codec module in the encoder is also used to perform identity mapping or downsampling processing on the features obtained by residual connection;

[0196] The basic encoding and decoding module in the decoder is also used to perform identity mapping or upsampling on the features obtained by residual connection.

[0197] Optionally, when the basic codec module is located at the last layer of a certain encoding stage in the encoder, the basic codec module is used to downsample the features obtained by the residual connection (PixelShuffleDown); when the basic codec module is located at the last layer of a certain decoding stage in the decoder, the basic codec module is used to upsample the features obtained by the residual connection (PixelShuffleUp); when the basic codec module is located at a non-last layer of the encoding and decoding stage, the basic codec module is used to perform identity mapping on the features obtained by the residual connection.

[0198] Indicatively, Fig. 9 As shown, in the encoder downsampling stage, the output feature 910 is downsampled to obtain the downsampled output feature 913, which is used as the input of the next encoder layer; in the decoder upsampling stage, the output feature 910 is upsampled to obtain the upsampled output feature 911, which is used as the input of the next decoder layer; in other cases, the output feature 910 is identically mapped to obtain the constant mapping output feature 912, which is used as the input of the next layer.

[0199] Training process of moiré removal model

[0200] In order to remove moiré from an image while ignoring the influence of lens distortion, in a possible implementation, when training a moiré removal model, in addition to supervision in the spatial domain, supervision is also performed in the frequency domain. The computer device may perform supervision in high frequency (moiré has a greater influence on high frequency components) and low frequency (lens distortion has a greater influence on low frequency components) respectively.

[0201] In a possible implementation, Fig.10 As shown, the training process of the moiré removal model may include the following steps:

[0202] Step 1001, input a sample image into a moiré removal model to obtain a multi-scale sample de-moiré image output by the moiré removal model, wherein the sample de-moiré images of different scales are generated based on different scale features output by a decoder in the moiré removal model.

[0203] In a possible implementation, in order to fully utilize the multi-scale feature extraction features of the moiré removal model, after the computer device inputs the sample image into the moiré removal model, the model, in addition to outputting a sample de-moiré image of the same size as the sample image, also outputs sample de-moiré images of other sizes based on features of other scales after the moiré removal model.

[0204] In some embodiments, the multi-scale sample de-moiré images include a sample de-moiré image that is consistent with the original image size and at least one sample de-moiré image that is smaller than the original image size, wherein the original image size is the image size of the sample image.

[0205] Indicatively, Fig.11 As shown, during the training process, after the sample image 1101 is input into the moiré removal model 1104, the moiré removal model 1104 outputs a first sample de-moiré image 1105 having the same size as the sample image 1101, and a second sample de-moiré image 1106 and a third sample de-moiré image 1107 having a size smaller than the sample image 1101. The second sample de-moiré image 1106 is generated based on the output feature of the second layer decoder in the moiré removal model 1104, and the third sample de-moiré image 1107 is generated based on the output feature of the first layer decoder in the moiré removal model 1104.

[0206] Step 1002, based on the true value of the multi-scale sample de-moiré image corresponding to the sample image and the multi-scale sample de-moiré image, a multi-scale spatial domain loss is determined, where the multi-scale spatial domain loss is used to characterize the difference of the multi-scale images in the spatial domain.

[0207] In order to facilitate the determination of the spatial difference of images at multiple scales, in some embodiments, the computer device downsamples the sample demoiré image true value corresponding to the sample image to obtain the multi-scale sample demoiré image true value, wherein the size of the multi-scale sample demoiré image true value is consistent with the size of the multi-scale sample demoiré image.

[0208] Indicatively, Fig.11As shown, the computer device downsamples the first sample demoiré image true value 1108 of the original size (consistent with the size of the first sample demoiré image 1105) to obtain the second sample demoiré image true value 1109 (consistent with the size of the second sample demoiré image 1106); the computer device further downsamples the second sample demoiré image true value 1109 to obtain the third sample demoiré image true value 1110 (consistent with the size of the third sample demoiré image 1107).

[0209] In one possible implementation, the computer device determines the spatial loss at the scale based on the true value of the sample demoiré image at the same scale and the multi-scale sample demoiré image, and then determines the multi-scale spatial loss based on the spatial losses at different sizes.

[0210] Optionally, the spatial domain loss may include a pixel-level L1 loss and a feature-perceptual loss LP.

[0211] Indicatively, Fig.11 As shown, the computer device determines the multi-scale spatial loss of the multi-scale sample demoiré image true value and the multi-scale sample demoiré image based on the pixel-level L1 loss and feature perception loss LP between the first sample demoiré image true value 1108 and the first sample demoiré image 1105, based on the pixel-level L1 loss and feature perception loss LP between the second sample demoiré image true value 1109 and the second sample demoiré image 1106, and based on the pixel-level L1 loss and feature perception loss LP between the third sample demoiré image true value 1110 and the third sample demoiré image 1107.

[0212] Step 1003, based on the multi-scale down-sampled image corresponding to the sample image, the true value of the multi-scale sample de-moiré image and the multi-scale sample de-moiré image, a multi-scale frequency domain loss is determined, where the multi-scale frequency domain loss is used to characterize the difference of the multi-scale image in the frequency domain.

[0213] In a possible implementation, similar to the multi-scale spatial domain loss, the computer device calculates the frequency domain loss at different scales to obtain the multi-scale frequency domain loss. Moreover, at each scale, the frequency domain loss includes a frequency domain low-pass loss for guiding low-frequency feature learning, and a frequency domain high-pass loss for guiding high-frequency feature learning.

[0214] In some embodiments, determining the frequency domain loss at the same scale may include:

[0215] 1. Based on the sample image and the sample de-moiré image at the same scale, the frequency domain low-pass loss is determined. The frequency domain low-pass loss is used to guide the learning of the low-frequency features of the sample image.

[0216] Since the impact of lens distortion is mainly reflected in the low-frequency components, when lens distortion is ignored, the de-moiréd image output by the model needs to be learned from the original image without de-moiréd, in order to learn the low-frequency characteristics of lens distortion in the image.

[0217] In some embodiments, a computer device performs a fast Fourier transform (FFT) on a sample image and a sample de-moiréed image at the same scale to determine the difference between the two in the frequency domain; and performs a fast Fourier transform and a low-pass filter on the difference between the sample image and the sample de-moiréed image in the spatial domain to obtain the difference between the two in low-frequency features, thereby determining the frequency domain low-pass loss between the sample image and the sample de-moiréed image based on the difference in low-frequency features and the difference in frequency domain.

[0218] The sample images of different scales are obtained by downsampling the original sample images. Fig.11 As shown, the computer device downsamples the sample image 1101 of the original size to obtain a first downsampled sample image 1102 of the same size as the second sample de-moiré image 1106. Further, the first downsampled sample image 1102 is downsampled to obtain a second downsampled sample image 1103 of the same size as the third sample de-moiré image 1107.

[0219] In a possible implementation, the frequency domain low-pass loss between the sample image and the sample de-moiré image at the same scale can be expressed as:

[0220] L M-P =H LowPass (FFT|MP|)·|FFT(M)-FFT(P)| 2

[0221] Among them, L M-P is the frequency domain low-pass loss, H LowPass is a low-pass filter, M and P are the sample image and the sample de-moiré image of the same scale, respectively, and FFT stands for fast Fourier transform.

[0222] Indicative, such as Fig.11 As shown, in order to ignore the lens distortion, the low frequency of the sample de-moiré image should be learned from the sample image. The computer device calculates the frequency domain low-pass loss between the sample image 1101 and the first sample de-moiré image 1105, calculates the frequency domain low-pass loss between the first down-sampled sample image 1102 and the second sample de-moiré image 1106, and calculates the frequency domain low-pass loss between the second down-sampled sample image 1103 and the third sample de-moiré image 1107, thereby obtaining a multi-scale frequency domain low-pass loss 1113.

[0223] 2. Based on the true value of the sample de-moiré image and the sample de-moiré image at the same scale, determine the frequency domain high-pass loss, which is used to guide the learning of the high-frequency features of the true value of the sample de-moiré image.

[0224] Since the impact of moiré is mainly reflected in the high-frequency component, for moiré removal, the moiré-removed image output by the model needs to be learned from the true value of the moiré-removed image to learn the high-frequency characteristics of the moiré in the image.

[0225] In some embodiments, a computer device performs a fast Fourier transform (FFT) on a sample de-moiréd image and a true value of the sample de-moiréd image at the same scale to determine a difference between the two in the frequency domain; and performs a fast Fourier transform and a high-pass filter on the difference between the sample de-moiréd image and the true value of the sample de-moiréd image in the spatial domain to obtain a difference in high-frequency features between the two, thereby determining a high-pass loss in the frequency domain between the sample de-moiréd image and the true value of the sample de-moiréd image based on the difference in high-frequency features and the difference in the frequency domain.

[0226] In a possible implementation, the frequency domain high-pass loss between the sample de-moiré image and the true value of the sample de-moiré image at the same scale can be expressed as:

[0227] L T-P =H HighPass (FFT|TP|)·|FFT(T)-FFT(P)| 2

[0228] Among them, L T-P is the frequency domain high-pass loss, H HighPass is a high-pass filter, T and P are the true value of the sample de-moiré image and the sample de-moiré image of the same scale, respectively, and FFT represents fast Fourier transform.

[0229] Indicative, such as Fig.11 As shown, in order to remove moiré, the high frequency of the sample demoiré image should be learned from the sample demoiré image true value. The computer device calculates the frequency domain high-pass loss between the first sample demoiré image 1105 and the first sample demoiré image true value 1108, calculates the frequency domain high-pass loss between the second sample demoiré image 1106 and the second sample demoiré image true value 1109, and calculates the frequency domain high-pass loss between the third sample demoiré image 1107 and the third sample demoiré image true value 1110, thereby obtaining a multi-scale frequency domain high-pass loss 1112.

[0230] Step 1004: training a moiré removal model based on multi-scale spatial domain loss and multi-scale frequency domain loss.

[0231] In one possible implementation, the computer device determines the multi-scale total loss based on the multi-scale spatial domain loss and the multi-scale frequency domain loss, and then trains the moiré removal model based on the multi-scale total loss through gradient descent and back propagation algorithms.

[0232] Optionally, the computer device performs loss weighting on the multi-scale spatial domain loss and the multi-scale frequency domain loss to obtain a multi-scale total loss, wherein the loss weights of the multi-scale spatial domain loss and the multi-scale frequency domain loss may be fixed values, or may be dynamically adjusted at different training stages.

[0233] Indicatively, Fig.11 As shown, the computer device trains the moiré removal model 1104 based on the multi-scale spatial domain loss 1111 , the multi-scale frequency domain low-pass loss 1113 , and the multi-scale frequency domain high-pass loss 1112 .

[0234] In this embodiment, the moiré removal model makes use of the knowledge compression and mapping learning capabilities of the transformer, so that the model can be easily expanded with the data scale; and the basic encoding and decoding module in the moiré removal model enhances the channel attention and spatial attention of the features, which helps to improve the accuracy of feature expression, and thus helps to improve the de-moiré effect.

[0235] In addition, during the model training process, based on the prior knowledge of the frequency domain characteristics of moiré and lens distortion, frequency domain auxiliary supervision is designed on the basis of the spatial domain to achieve moiré removal while retaining the lens distortion of the original image, thereby improving the model's focus on the moiré removal task and the model's moiré removal quality.

[0236] For different types of parseable charts, the computer device uses a corresponding parsing model to parse the structured data based on the type of chart to obtain corresponding structured data.

[0237] In some embodiments, when the chart type is a graph, the graph in the chart image is parsed for structured data using a graph parsing model to obtain graph structured data.

[0238] When the chart type is a table, the computer device performs structured data parsing on the table in the chart image through a table parsing model to obtain table structured data.

[0239] When the chart type is a formula, the computer device performs structured data parsing on the formula in the chart image through a formula parsing model to obtain formula structured data.

[0240] Structured data analysis of graphs

[0241] The structured data parsing of graphs includes graph structure detection, relationship matching and OCR text parsing. In a possible implementation, the structured data parsing of graphs is implemented based on (Detection Transformer, DETR), and the core includes two parts: structure detection and relationship detection.

[0242] Although graphs come in a variety of types, they are essentially composed of nodes with specific relationships. Therefore, computer devices can obtain graph structured data by extracting the nodes in the graph and the node relationships between the nodes.

[0243] In a possible implementation, the graph parsing model may adopt an encoder-decoder structure, in which the encoder is responsible for extracting graph features, and the decoder is responsible for extracting nodes and node relationships based on the output of the encoder.

[0244] Optionally, when parsing structured data, the computer device extracts features of the graph through the encoder of the graph parsing model to obtain graph features of the graph; based on the graph features, the computer device performs node target detection and node relationship extraction through the decoder of the graph parsing model to obtain node information of the nodes in the graph and node relationships between the nodes.

[0245] In some embodiments, the node information includes the rectangular box position of the node corresponding to the rectangular box and the node type; the node relationship is represented by a coordinate pair.

[0246] For different types of graphs, the node-to-node relationship can include the following situations:

[0247] 1. When the graph is a mind map, the nodes are mind map nodes, and the node relationship pairs are used to represent the superior-subordinate relationship between mind map nodes;

[0248] 2. When the diagram is a flow chart, the nodes are process nodes (such as process boxes), and the node relationship pairs are used to represent the order relationship between process nodes;

[0249] 3. In the case of a bar graph, the nodes are the columns and the legends, and the node relationship pairs are used to represent the corresponding relationship between the columns and the legends;

[0250] 4. In the case of a pie chart, the nodes are sectors and legends, and the node relationship pairs are used to represent the corresponding relationship between the sectors and the legends;

[0251] 5. In the case of a line graph, the nodes include turning points (which can be determined by segmenting the line graph) and legends (such as the horizontal and vertical coordinates of the line graph, and the meanings of lines of different colors). The node relationship is the correspondence between the line representing the turning point and the legend;

[0252] 6. When the diagram is an architecture diagram, the nodes include architecture elements, and the node relationship pairs are used to represent the hierarchical relationships between different architecture elements.

[0253] Correspondingly, for different types of graphs, the computer device combines the graph parsing model with OCR text extraction. The structured parsing process of the graph is as follows: Fig.12 shown.

[0254] In a possible implementation, in order to improve feature expression capability, the encoder may use a convolutional neural network that can output multi-scale feature maps as a backbone network, such as ResNet.

[0255] Feature maps of different scales have different corresponding receptive fields. Among them, feature maps of lower scales (i.e., shallower layers) have smaller receptive fields and can capture detailed information in the image, such as edges and corners. Feature maps of higher scales (i.e., deeper layers) have larger receptive fields and can capture more extensive contextual information, which is suitable for identifying large-scale structures and patterns in images.

[0256] In order to improve the quality of the graph features extracted by the encoder, in a possible implementation, the encoder performs multi-scale feature extraction on the graph. After obtaining the multi-scale features, the computer device performs self-attention processing on the minimum scale features in the multi-scale features to obtain the full-image features, thereby performing feature fusion on the full-image features with features of other scales in the multi-scale features to obtain the graph features of the graph.

[0257] Among them, the smallest scale feature has a larger receptive field and can capture the features of the whole image level. Accordingly, after the whole image features after self-attention processing are fused with the features of other scales, the features of other scales can be fused into the whole image features, improving the feature expression of feature maps at each scale.

[0258] Schematically, take the feature extraction of the flowchart as an example, Fig.13 As shown, the flowchart 1301 inputs the backbone network 1302 of the encoder to perform multi-scale feature extraction, and after obtaining the graph features 1303 of three scales, the self-attention module 1304 performs self-attention processing on the minimum scale feature to obtain the full-graph feature 1305. The computer device performs feature fusion on the full-graph feature 1305 with the graph features of the other two scales through the feature fusion module 1306 to obtain the multi-scale graph feature 1307 that fuses the full-graph feature, and inputs it into the decoder 1308 for subsequent node and node relationship detection.

[0259] Regarding the design of the decoder in the graph parsing model, in one possible design, the graph parsing model includes a first decoder and a second decoder, wherein the first decoder is used to output node relationship pairs based on the graph features output by the encoder, and the second decoder is used to output node information of each node based on the graph features output by the encoder.

[0260] In order to reduce the computational complexity during decoding, in this embodiment, a deformable attention mechanism is introduced to effectively handle spatial changes in the image and improve the model's ability to understand the image content. Since images are usually converted into a large number of pixel points or feature point sequences, in the standard Transformer model, the attention mechanism considers the interactions between all elements in the input sequence, which will result in huge computational costs. The deformable attention mechanism allows the model to focus only on a portion of the key area of ​​the image instead of the entire image, which not only reduces the amount of calculation, but also improves the efficiency and effect of image processing. Since the deformable attention allows the model to dynamically select a small number of key sampling positions in each attention head for attention calculation, and these positions are dynamically determined based on the input data, the model can be more focused on important information in the image, improving the performance and generalization ability of the model.

[0261] In one possible implementation, during the decoding process, the computer device performs multi-head self-attention processing on the graph features through the first decoder to obtain multi-head self-attention features, and then performs deformable attention processing on the graph features and the multi-head self-attention features through the first decoder to obtain node relationship pairs.

[0262] The computer device performs multi-head self-attention processing on the graph features through the second decoder to obtain multi-head self-attention features, and then performs deformable attention processing on the graph features and the multi-head self-attention features through the second decoder to obtain node information.

[0263] Among them, the multi-head self-attention processing process can be expressed as:

[0264] MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,…,head h )W O

[0265]

[0266] Among them, Q, K, and V are query, key, and value matrices respectively. are the linear transformation matrices corresponding to the query, key, and value of each head i, respectively.

[0267] The multi-head self-attention mechanism allows the model to capture information in different representation subspaces in parallel. Each head learns a different representation, which is then transformed by the linear transformation W. O splicing, integrating this information to form the final output.

[0268] In one possible implementation, the computer may use Multi-Scale Deformable Attention (MSD-DA) for variable attention processing. The multi-scale variable attention module can sample on multiple feature maps to better utilize the multi-scale features of the image. The variable attention processing process can be expressed as:

[0269]

[0270] Where m is the index of the attention head, M is the total number of attention heads, l is the index of the feature scale, L is the total number of feature scales, k is the index of the sampling point, K is the total number of sampling points, Δp mlqk is the sampling point offset, A mlqk is the corresponding attention weight, Represents a multi-scale feature map, and its operation with the subsequent sampling points is defined as sampling in the feature map according to the position of the sampling points.

[0271] In some embodiments, the graph parsing model adopts a multi-layer decoder structure, and the calculation result of the previous layer of decoders will be used as the input of the next layer of decoders. The calculation result will be used as a query to participate in the calculation of the next layer of decoders.

[0272] In some embodiments, the input of the first layer decoder can be selected from the graph features output by the encoder. Illustratively, the computer device inputs the graph features output by the encoder into a feature evaluation module, which can parse the information in the feature vector to obtain the estimated coordinate pairs and the corresponding confidences, thereby selecting the features corresponding to the estimated coordinate pairs with high confidence from the graph features according to the confidence levels and inputting them into the decoder. The number of vectors selected from the graph features is related to the maximum number of detection nodes and the number of relationship extractions supported by the graph parsing model. For example, 300 vectors are selected from the graph features as the input to the first layer decoder.

[0273] In one possible design, each layer of decoders is also provided with a prediction head, which is responsible for reading the output of the decoder and calculating the prediction result, which means that each layer of decoders has the ability to output the final result. In the embodiment of the present application, the prediction model of node relationship pairs and confidence is modeled as an iterative optimization process, where the encoder first generates a rough prediction, and then optimizes it layer by layer in the decoder.

[0274] In some embodiments, starting from the second decoder layer, the input Query is provided by the previous layer, and the correction value of the coordinate pair is calculated through the same calculation process. This correction value is based on the coordinate pair prediction result of the previous layer to correct the coordinate pair.

[0275]

[0276] in, is the prediction of the coordinate pair by the d-th layer decoder, The coordinate pairs generated by the decoder of layer d-1, Represents the correction value predicted by the decoder of the current layer (output by the prediction head of the d-1th layer decoder), and σ is the sigmoid function.

[0277] During the model operation, the coordinates are transformed by the function g(x), and the correction value is corrected on this basis, so that its value range is more suitable for the model operation results. The calculation process is: the original coordinates are transformed by the inverse sigmoid function g(x), and then the correction value is added to obtain the corrected value, for example Then the new coordinates are obtained by restoring them through the sigmoid function. The gradient of the original coordinate part is as follows: Will be blocked and will not participate in the back propagation of gradients.

[0278] Indicatively, Fig.14 As shown, after the flowchart 1401 is encoded by the encoder 1402 to obtain the graph feature 1403, the computer device determines the initial Query 1404 based on the graph feature 1403, and then inputs the initial Query 1404 into the self-attention module 1406 of the first layer decoder to obtain the multi-head self-attention feature output by the self-attention module 1406 (starting from the second layer decoder, the input is the query output by the previous layer decoder). The first layer decoder performs deformable attention processing on the graph feature 1403 and the multi-head self-attention feature through the deformable attention module 1407 to obtain the decoder output 1409 of the first layer decoder. After the decoder output 1409 is predicted by the prediction head 1408, the coordinate pair correction value 1410 and the corresponding confidence are obtained. The computer device corrects the coordinate pair initial value 1405 based on the coordinate pair correction value 1410 (starting from the second layer decoder, the coordinate pair correction value 1410 is used to correct the coordinate pair obtained after the previous layer correction), and further determines the next Query (the input of the next layer decoder) based on the corrected coordinate pair 1411. For the last layer of decoder, the confidence 1412 output by the prediction head 1408 in the decoder and the coordinate pair 1411 obtained by the last correction will be output as the prediction result 1413.

[0279] Similar to the structure of the first decoder, the second decoder also has a multi-layer decoder, and a second prediction head is set in each layer of the decoder. The second prediction head is used to predict node information (correction value) based on the result of the deformable attention output and output the corresponding confidence. Correspondingly, the second decoder finally outputs the node information and the corresponding confidence. The specific decoding process can refer to the first decoder, and this embodiment will not be repeated here.

[0280] In most cases, each coordinate point in the node relationship pair output by the graph parsing model will fall at the center of a node (determined based on the node information output by the model), and at this time, it can be determined which two specific nodes the node relationship pair belongs to. If the coordinates in the node relationship pair are inaccurate, that is, the coordinate point does not fall within any node, the computer device can perform node allocation based on the node type of the nodes around the coordinate point and the distance information from the coordinate point to the surrounding nodes. For example, the coordinate point can be assigned to the node that is closest to the coordinate point and has a matching node type. Among them, node type matching refers to the matching of the association relationship between the node types of the two nodes in the coordinate pair after the nodes are assigned.

[0281] In a possible implementation, when a first prediction head is provided in the first decoder and the first prediction head is used to output a node relationship pair correction value (the node relationship pair correction value is used to correct the node relationship pair) and a first confidence level, the training process of the first decoder may include the following steps.

[0282] 1. During the training process, the node relationship loss is determined based on the sample node relationship pairs output by the first decoder and the true value of the node information.

[0283] In a possible implementation, the computer device uses the Hungarian Algorithm to evaluate the prediction accuracy of the node relationship pairs.

[0284] The computer device uses the Hungarian algorithm to assign coordinate pairs indicated by the predicted sample node relationships, that is, to assign a unique node bounding rectangle (bounding rectangle) in the true value of the node information to each node predicted coordinate in the coordinate pair.

[0285] In a possible implementation, the computer device allocates a first node bounding rectangle and a second node bounding rectangle to the first node predicted coordinates and the second node predicted coordinates represented by the sample node relationship from the true value of the node information by using the Hungarian algorithm, wherein the first node bounding rectangle corresponds to the first node predicted coordinates, and the second node bounding rectangle corresponds to the second node predicted coordinates.

[0286] For example, for the predicted coordinates of the parent node and the predicted coordinates of the child node in the coordinate pair, the computer device uses the Hungarian algorithm to assign a parent node bounding rectangle to the predicted coordinates of the parent node and assign a child node bounding rectangle to the predicted coordinates of the child node, where the parent and child node bounding rectangles are both the node bounding rectangle true values ​​of the nodes in the node information true value.

[0287] Further, the computer device determines the node relationship loss based on the first node predicted coordinates, the second node predicted coordinates, the first node circumscribed rectangle, and the second node circumscribed rectangle.

[0288] In a possible implementation, the computer device may determine a calculation method for node relationship loss based on GIoU (Generalized IoU).

[0289] Among them, when the predicted coordinates of the first node are (px,py), the predicted coordinates of the second node are (sx,sy), and the circumscribed rectangle of the first node is And the circumscribed rectangle of the second node is When , the bounding rectangle of the first node is moved to the predicted coordinates of the first node, and the bounding rectangle of the first node after the move is recorded as P, and the bounding rectangle of the second node is moved to the predicted coordinates of the second node, and the bounding rectangle of the second node after the move is recorded as S. The node relationship loss can be expressed as:

[0290]

[0291] C represents the minimum enclosing rectangle of A and B

[0292] Among them, α and β are adjustable scaling factors, and a greater weight is given to difficult samples whose node prediction coordinates are outside the node circumscribed rectangle. At the same time, the ratio between α and β can be adjusted according to the model effect, so that the first decoder pays more attention to the first node prediction coordinates or the second node prediction coordinates.

[0293] Of course, the computer device may also use other methods to calculate the node relationship loss (such as MSE), which is not limited in the embodiments of the present application.

[0294] It should be noted that, when the first decoder includes a multi-layer structure, the computer device executes the above-mentioned allocation and loss calculation process for the output of each layer of the decoder, which will not be described in detail in this embodiment.

[0295] 2. Determine a confidence loss based on the first confidence output by the first decoder.

[0296] In a possible implementation, for the first confidence output by the first prediction head in the first decoder, the computer device determines the confidence loss based on the first confidence. Wherein, when the first decoder includes a multi-layer structure, the computer device performs the loss calculation process for the first confidence output by the first prediction head in each layer of the decoder, which is not described in detail in this embodiment.

[0297] Optionally, the calculation of confidence loss can refer to Generalized Focal Loss, so that the confidence prediction result can better reflect the prediction quality of the node relationship pair.

[0298] 3. Train the first decoder based on node relationship loss and confidence loss.

[0299] In one possible implementation, the computer device determines a weighted total loss based on the node relationship loss and the confidence loss, thereby training the first decoder based on the weighted total loss.

[0300] Indicatively, Fig.14 As shown, during the training process, the computer device performs node matching through the Hungarian algorithm based on the prediction result 1413 (including the predicted sample node relationship pair and confidence) and the node information true value 1414, and obtains the optimal matching result 1415, and then calculates the loss based on the optimal matching result, and uses the calculated loss to train the first decoder.

[0301] In a possible implementation, when a second prediction head is provided in the second decoder and the second prediction head is used to output a node information correction value (the node information correction value is used to correct the node information) and a second confidence level, the training process of the second decoder may include the following steps.

[0302] 1. During the training process, the node information loss is determined based on the sample node information output by the first decoder and the true value of the node information.

[0303] In a possible implementation, the computer device may determine the target detection loss between the sample node information and the true value of the node information as the node information loss. The target detection loss may include a node type loss (clsloss) and a bounding rectangle loss (bbox loss), wherein the bounding rectangle loss may be an IoU loss. The embodiment of the present application does not limit the specific calculation method of the node information loss.

[0304] Optionally, the node type loss and the bounding rectangle loss may have different loss weights, so that the second decoder pays more attention to the node type or bounding rectangle prediction.

[0305] It should be noted that, when the second decoder includes a multi-layer structure, the computer device executes the above loss calculation process for the output of each layer of the decoder, which is not described in detail in this embodiment.

[0306] 2. Determine a confidence loss based on a second confidence output by the second decoder.

[0307] In a possible implementation, for the second confidence output by the second prediction head in the second decoder, the computer device determines the confidence loss based on the second confidence. Wherein, when the second decoder includes a multi-layer structure, the computer device performs the loss calculation process for the second confidence output by the second prediction head in each layer of the decoder, which is not described in detail in this embodiment.

[0308] Optionally, the calculation of confidence loss can refer to Generalized Focal Loss, so that the confidence prediction result can better reflect the prediction quality of node information.

[0309] 3. Train the second decoder based on node information loss and confidence loss.

[0310] In one possible implementation, the computer device determines a weighted total loss based on the node information loss and the confidence loss, thereby training the second decoder based on the weighted total loss.

[0311] In this embodiment, the computer device adopts a graph parsing model of a codec structure to perform graph parsing, and uses the encoder to perform multi-scale feature extraction and full-graph feature fusion, which improves the feature quality of the graph features output by the encoder and helps to improve the subsequent decoding quality; the decoder performs node relationship pair and node information prediction in parallel, and incorporates node relationship pair correction and confidence mechanism into the prediction process, which helps to improve the prediction accuracy of node information.

[0312] Structured data parsing of tables

[0313] The structured data parsing of the table includes table structure information parsing and OCR text information parsing, and the output result is a reconstructible table structure and text information. Among them, the table structure information parsing is used to obtain the cell position, cell size, topological relationship between cells in the table, etc.

[0314] In a possible implementation, the table structured information parsing adopts a regression-merging paradigm, which is divided into two parts: a dividing line regression task and a cell merging task. Accordingly, the table parsing model used for table parsing is composed of a dividing line regression network and a cell merging network.

[0315] During the process of parsing the table, the computer device performs segmentation line regression on the table through the segmentation line regression network of the table parsing model to obtain segmentation line coordinates. The segmentation lines include row segmentation lines and column segmentation lines. Then, based on the segmentation line coordinates and the table, the cell merging network of the table parsing model is used to merge cells to obtain cell information.

[0316] In one possible design, the segmentation line regression network uses transformer-based DETR to perform segmentation line regression. Taking row regression as an example, the computer device first performs reference point probability prediction on the row pooling features, determines the row separation area through probability prediction, and uses the midpoint of the row separation area as the reference point (reference point) of each row segmentation line; then, the feature vector set composed of visual and spatial information corresponding to these reference points is used as the query feature (query) input into the decoder (Transformer Decoder), and the corresponding row segmentation line coordinate point is regressed. The regression method of the column segmentation line is similar to that of the row segmentation line, which will not be repeated in this embodiment.

[0317] In one possible design, the cell merging network takes the table image and the output of the segmentation line regression network as input, and outputs the cells that need to be merged to restore cells across multiple rows or columns. Optionally, the input of the cell merging network includes: the table image, the row and column probability prediction matrix output by the segmentation line regression network, and the binary mask (7 pixel wide line) of the center of the row and column area. The output is 4 matrices, namely the upper, lower, left, and right merge matrices, which are used to characterize the merging of cells in various directions.

[0318] Furthermore, the upper merge matrix and the lower merge matrix are merged, and the left merge matrix and the right merge matrix are merged to obtain two merge matrices, and then the D-probs matrix (a matrix indicating whether it is merged from top to bottom) and the R-probs (a matrix indicating whether it is merged from left to right) are obtained through ROI (Region Of Interest) average pooling. For example, when the number of rows and columns output by the segmentation line regression network is r and c respectively, the size of D-probs is (r-1)*c, and the size of R-probs is r*(c-1). The subsequent computer device merges cells according to the D-probs matrix and R-probs.

[0319] In another possible design, the cell merging network divides the table into rows and columns according to the row and column segmentation lines output by the segmentation line regression network to obtain N*M cells. For the cell bounding boxes of each of the N*M cells, RoIAlign is used to extract the cell features of each cell (such as 7*7*C) from the image features of the table image, and then a multidimensional cell feature (such as 512d) is generated through the MLP layer, and finally an N*M*512 cell feature map is obtained. Furthermore, based on the cell feature map, the cell merging network performs feature fusion on the features near the cell to obtain an enhanced cell feature map. When determining whether the cell needs to be merged, the cell merging network inputs the cell features of the adjacent cells in the enhanced cell feature map into the classifier to obtain the cell merging classification result output by the classifier (binary classification, merge or not merge).

[0320] It should be noted that the solutions at different stages provided in the above embodiments can be combined to obtain different embodiments, which will not be described in detail in this embodiment.

[0321] Please refer to Fig.15 , which shows a structural block diagram of a data parsing device for a chart provided by an exemplary embodiment of the present application. The device includes:

[0322] An image acquisition module 1501 is used to acquire an image to be identified, where the image to be identified is an image containing a parsable chart;

[0323] The image processing module 1502 is used to enhance the image quality of the image to be identified to obtain an enhanced image;

[0324] An image extraction module 1503, for extracting a chart image of the parseable chart from the enhanced image;

[0325] The parsing module 1504 is configured to perform structured data parsing on the chart in the chart image based on the chart type of the chart contained in the chart image, so as to obtain structured data of the chart in the chart image.

[0326] Optionally, the image acquisition module 1501 is used to:

[0327] Performing image classification on the input image by an image classification model, wherein the image classification model is used to classify images that contain the parseable graph and images that do not contain the parseable graph;

[0328] The image to be identified is determined based on the image classification result output by the image classification model.

[0329] Optionally, the image classification model includes a feature extraction network and a classification network, and the feature extraction network is obtained through pre-training, and the classification network is obtained by fine-tuning after the feature extraction network pre-training is completed;

[0330] The device also includes a training module, which is used to:

[0331] During the pre-training process, the feature extraction network is used to extract features from the sample image and the sample mixed image to obtain sample image features and sample mixed image features, wherein the sample mixed image is obtained by scaling and mixing different sample images;

[0332] Performing gradient backpropagation perturbation processing on the sample image features based on the classification loss to obtain sample perturbation image features, wherein the classification loss is determined based on the classification result of the sample image features by the auxiliary classification network;

[0333] Determining feature extraction loss based on the sample image features, the sample mixed image features, and the sample perturbation image features;

[0334] training the feature extraction network based on the feature extraction loss;

[0335] When the feature extraction network is pre-trained, extracting features from the sample image through the feature extraction network to obtain features of the sample image;

[0336] Based on the sample image features, classification is performed through the classification network to obtain a sample classification result of the sample image;

[0337] The classification network is fine-tuned based on the sample classification result and the sample image classification label of the sample image.

[0338] Optional, training module for:

[0339] Determining Info NCE loss based on the sample image feature and the sample perturbation image feature;

[0340] Determining a SupCon loss based on the sample image feature, the sample mixed image feature, the sample image classification label of the sample image, and the sample mixed image classification label of the sample mixed image;

[0341] The Info NCE loss and the SupCon loss are determined as the feature extraction loss.

[0342] Optionally, the image processing module 1502 is used to:

[0343] At least one of image correction processing, moiré removal processing and deblurring processing is performed on the image to be recognized to obtain the enhanced image.

[0344] Optionally, the image processing module 1502 is used to:

[0345] Performing image vertex detection on the image to be identified through a target detection network to obtain corrected image vertices in the image to be identified, wherein the corrected image vertices are vertices of the image region to be corrected in the image;

[0346] Determining a perspective transformation matrix based on the vertices of the corrected image, and performing trapezoidal correction on the image to be identified based on the perspective transformation matrix;

[0347] The image to be recognized after trapezoidal correction is subjected to curved surface correction to obtain the enhanced image.

[0348] Optionally, the target detection network has a vertex output head and an image segmentation head, the vertex output head is used to output the corrected image vertices, and the image segmentation head is used to output the image area to be corrected in the image;

[0349] The device further comprises a training module for:

[0350] Inputting the sample image into the target detection network, obtaining the sample corrected image vertices output by the vertex output head in the target detection network, and the sample image region to be corrected output by the image segmentation head in the target detection network;

[0351] Determine vertex detection loss based on the sample corrected image vertices, the true values ​​of the corrected image vertices of the sample image, the sample image region to be corrected, and the true value of the image region to be corrected of the sample image;

[0352] The object detection network is trained based on the vertex detection loss.

[0353] Optionally, the training module is used to:

[0354] Determining vertex position loss based on the rectified image vertex and the true value of the rectified image vertex;

[0355] Determine edge segmentation loss based on a sample region edge of the sample to-be-corrected image region and a region edge true value of a true value of the to-be-corrected image region;

[0356] Determine a regional mask loss based on a sample regional mask of the sample image region to be corrected and a regional mask true value of a true value of the image region to be corrected;

[0357] The vertex position loss, the edge segmentation loss, and the region mask loss are determined as the vertex detection loss.

[0358] Optionally, the image processing module 1502 is used to:

[0359] The image to be identified is subjected to moiré removal processing by a moiré removal model to obtain the enhanced image, wherein the moiré removal model adopts a codec structure, the encoder and the decoder in the codec structure are jump-connected, and the encoder and the decoder in the codec structure are composed of basic codec modules based on transformer.

[0360] Optionally, the basic encoding and decoding module includes a first feature processing branch and a second feature processing branch;

[0361] The first feature processing branch is used to perform spatial dimension reduction and feature stretching processing on the input features to obtain a spatial representation; perform spatial attention processing on the spatial representation through a transformer; and perform feature restoration on the spatial representation after the spatial attention processing to obtain a spatial attention feature;

[0362] The second feature processing branch is used to perform channel dimension reduction processing and feature stretching processing on the input feature to obtain a channel representation; perform channel attention processing on the channel representation through a transformer; and perform feature restoration on the channel representation after the channel attention processing to obtain a channel attention feature;

[0363] The basic encoding and decoding module is used to perform residual connection on the spatial attention feature, the channel attention feature and the input feature.

[0364] Optionally, the basic encoding and decoding module in the encoder is further used to perform identity mapping or downsampling processing on the features obtained by residual connection;

[0365] The basic encoding and decoding module in the decoder is also used to perform identity mapping or upsampling processing on the features obtained by residual connection.

[0366] Optionally, the device further comprises a training module, which is used to:

[0367] Inputting a sample image into the moiré removal model to obtain a multi-scale sample de-moiré image output by the moiré removal model, wherein the sample de-moiré images of different scales are generated based on different scale features output by the decoder in the moiré removal model;

[0368] Determine a multi-scale spatial domain loss based on a true value of a multi-scale sample de-moiré image corresponding to the sample image and the multi-scale sample de-moiré image, wherein the multi-scale spatial domain loss is used to characterize the difference of the multi-scale images in the spatial domain;

[0369] Determine a multiscale frequency domain loss based on a multiscale downsampled image corresponding to the sample image, a true value of the multiscale sample de-moiré image, and the multiscale sample de-moiré image, wherein the multiscale frequency domain loss is used to characterize the difference of the multiscale image in the frequency domain;

[0370] The moiré removal model is trained based on the multi-scale spatial domain loss and the multi-scale frequency domain loss.

[0371] Optionally, the training module is used to:

[0372] Based on the sample image and the sample de-moiré image at the same scale, determining a frequency domain low-pass loss, wherein the frequency domain low-pass loss is used to guide learning of low-frequency features of the sample image;

[0373] Based on the true value of the sample demoiré image and the sample demoiré image at the same scale, a frequency domain high-pass loss is determined, and the frequency domain high-pass loss is used to guide the learning of high-frequency features of the true value of the sample demoiré image.

[0374] Optionally, the parsing module 1504 is used to:

[0375] In the case where the chart type is a graph, performing structured data parsing on the graph in the chart image by using a graph parsing model to obtain graph structured data;

[0376] When the chart type is a table, the table in the chart image is parsed for structured data using a table parsing model to obtain table structured data.

[0377] Optionally, the graph parsing model adopts a codec structure;

[0378] The parsing module 1504 is used to:

[0379] Extracting features from the graph through an encoder of the graph parsing model to obtain graph features of the graph;

[0380] Based on the graph features, node target detection and node relationship extraction are performed through the decoder of the graph parsing model to obtain node information of the nodes in the graph and node relationships between the nodes.

[0381] Optionally, the parsing module 1504 is used to:

[0382] Performing multi-scale feature extraction on the image by the encoder to obtain multi-scale features;

[0383] Performing self-attention processing on the smallest scale feature in the multi-scale features to obtain full-image features;

[0384] The full-image feature is fused with features of other scales in the multi-scale feature to obtain the image feature of the image.

[0385] Optionally, the parsing module 1504 is used to:

[0386] Performing multi-head self-attention processing on the graph feature through the first decoder to obtain a multi-head self-attention feature; performing deformable attention processing on the graph feature and the multi-head self-attention feature through the first decoder to obtain the node relationship pair;

[0387] The second decoder performs multi-head self-attention processing on the graph features to obtain multi-head self-attention features; the second decoder performs deformable attention processing on the graph features and the multi-head self-attention features to obtain the node information.

[0388] Optionally, a first prediction head is provided in the first decoder, and the first prediction head is used to output a node relationship pair correction value and a first confidence, and the node relationship pair correction value is used to correct the node relationship pair;

[0389] The device also includes a training module for:

[0390] During the training process, determining the node relationship loss based on the sample node relationship pairs and the true value of the node information output by the first decoder;

[0391] Determining a confidence loss based on the first confidence output by the first decoder;

[0392] The first decoder is trained based on the node relationship loss and the confidence loss.

[0393] Optionally, the training module is used to:

[0394] By using the Hungarian algorithm, a first node circumscribed rectangle and a second node circumscribed rectangle are respectively assigned to the first node predicted coordinates and the second node predicted coordinates represented by the sample node relationship from the true value of the node information;

[0395] The node relationship loss is determined based on the first node prediction coordinates, the second node prediction coordinates, the first node circumscribed rectangle, and the second node circumscribed rectangle.

[0396] Optionally, a second prediction head is provided in the second decoder, and the second prediction head is used to output a node information correction value and a second confidence, and the node information correction value is used to correct the node information;

[0397] The device also includes a training module for:

[0398] During the training process, determining the node information loss based on the sample node information output by the second decoder and the true value of the node information;

[0399] Determining a confidence loss based on the second confidence output by the second decoder;

[0400] The second decoder is trained based on the node information loss and the confidence loss.

[0401] Optionally, the parsing module 1504 is used to:

[0402] Performing segmentation line regression on the table through the segmentation line regression network of the table parsing model to obtain segmentation line coordinates, wherein the segmentation lines include row segmentation lines and column segmentation lines;

[0403] Based on the dividing line coordinates and the table, cells are merged through a cell merging network of the table parsing model to obtain cell information.

[0404] In summary, the embodiments of the present application optimize different stages in the chart data parsing pipeline. Among them, in the image preprocessing stage, by enhancing the image quality of the image containing the parseable chart and extracting the chart image of the parseable chart from the obtained enhanced image, the image quality of the chart image is improved, which in turn helps to improve the accuracy of the data parsed from the chart image. In the data parsing stage, by identifying the chart type of the chart contained in the chart image and performing structured data parsing on the chart in the chart image based on the chart type, the structured data of the chart is obtained, and targeted structured data extraction for different types of charts is achieved, which helps to improve the accuracy of the structured data finally extracted.

[0405] The specific process of implementing the data analysis of the chart by the above-mentioned device can be referred to the method embodiment, which will not be described in detail in this embodiment.

[0406] See also Fig.16 , Fig.16 16 is a schematic diagram of a computer device provided by an exemplary embodiment of the present application, and the computer device may be a terminal or a server. The computer device may include one or more of the following components: a processor 1610 and a memory 1620 .

[0407] Optionally, the processor 1610 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1620, and calling data stored in the memory 1620. Optionally, the processor 1610 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1610 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 1610, but may be implemented by a separate chip.

[0408] The memory 1620 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1620 includes a non-transitory computer-readable storage medium. The memory 1620 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0409] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device, and the computer device may include more (such as communication components, display components) or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0410] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the data parsing method of the chart as described in the above embodiment.

[0411] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the data analysis method of the chart as described in the above embodiment.

[0412] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0413] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data analysis method for a chart, characterized in that: The method comprises: Acquire an image to be identified, wherein the image to be identified is an image containing a parsable chart; Performing image quality enhancement on the image to be identified to obtain an enhanced image; extracting a chart image of the parseable chart from the enhanced image; Based on the chart type of the chart included in the chart image, structured data parsing is performed on the chart in the chart image to obtain structured data of the chart in the chart image.

2. The method according to claim 1, characterized in that The step of obtaining the image to be identified comprises: Performing image classification on the input image by an image classification model, wherein the image classification model is used to classify images that contain the parseable graph and images that do not contain the parseable graph; The image to be identified is determined based on the image classification result output by the image classification model.

3. The method according to claim 2, characterized in that The image classification model includes a feature extraction network and a classification network, wherein the feature extraction network is obtained through pre-training, and the classification network is obtained through fine-tuning after the feature extraction network pre-training is completed; The method comprises: During the pre-training process, the feature extraction network is used to extract features from the sample image and the sample mixed image to obtain sample image features and sample mixed image features, wherein the sample mixed image is obtained by scaling and mixing different sample images; Performing gradient backpropagation perturbation processing on the sample image features based on the classification loss to obtain sample perturbation image features, wherein the classification loss is determined based on the classification result of the sample image features by the auxiliary classification network; Determining feature extraction loss based on the sample image features, the sample mixed image features, and the sample perturbation image features; training the feature extraction network based on the feature extraction loss; When the feature extraction network is pre-trained, extracting features from the sample image through the feature extraction network to obtain features of the sample image; Based on the sample image features, classification is performed through the classification network to obtain a sample classification result of the sample image; The classification network is fine-tuned based on the sample classification result and the sample image classification label of the sample image.

4. The method according to claim 3, characterized in that The determining of feature extraction loss based on the sample image feature, the sample mixed image feature and the sample perturbation image feature includes: Determining Info NCE loss based on the sample image feature and the sample perturbation image feature; Determining a SupCon loss based on the sample image feature, the sample mixed image feature, the sample image classification label of the sample image, and the sample mixed image classification label of the sample mixed image; The Info NCE loss and the SupCon loss are determined as the feature extraction loss.

5. The method according to claim 1, characterized in that The step of enhancing the image quality of the image to be identified to obtain an enhanced image includes: At least one of image correction processing, moiré removal processing and deblurring processing is performed on the image to be recognized to obtain the enhanced image.

6. The method according to claim 5, characterized in that The performing image correction processing on the image to be identified to obtain the enhanced image includes: Performing image vertex detection on the image to be identified through a target detection network to obtain corrected image vertices in the image to be identified, wherein the corrected image vertices are vertices of the image region to be corrected in the image; Determining a perspective transformation matrix based on the vertices of the corrected image, and performing trapezoidal correction on the image to be identified based on the perspective transformation matrix; The image to be recognized after trapezoidal correction is subjected to curved surface correction to obtain the enhanced image.

7. The method according to claim 6, characterized in that The target detection network comprises a vertex output head and an image segmentation head, wherein the vertex output head is used to output the corrected image vertices, and the image segmentation head is used to output the image area to be corrected in the image; The method further comprises: Inputting the sample image into the target detection network, obtaining the sample corrected image vertices output by the vertex output head in the target detection network, and the sample image region to be corrected output by the image segmentation head in the target detection network; Determine vertex detection loss based on the sample corrected image vertices, the true values ​​of the corrected image vertices of the sample image, the sample image region to be corrected, and the true value of the image region to be corrected of the sample image; The object detection network is trained based on the vertex detection loss.

8. The method according to claim 7, characterized in that The determining of vertex detection loss based on the sample corrected image vertices, the corrected image vertex true values ​​of the sample image, the sample image region to be corrected, and the true value of the image region to be corrected of the sample image comprises: Determining vertex position loss based on the rectified image vertex and the true value of the rectified image vertex; Determine edge segmentation loss based on a sample region edge of the sample to-be-corrected image region and a region edge true value of a true value of the to-be-corrected image region; Determine a regional mask loss based on a sample regional mask of the sample image region to be corrected and a regional mask true value of a true value of the image region to be corrected; The vertex position loss, the edge segmentation loss, and the region mask loss are determined as the vertex detection loss.

9. The method according to claim 5, characterized in that The performing a moiré removal process on the image to be identified to obtain the enhanced image includes: The image to be identified is subjected to moiré removal processing by a moiré removal model to obtain the enhanced image, wherein the moiré removal model adopts a codec structure, the encoder and the decoder in the codec structure are jump-connected, and the encoder and the decoder in the codec structure are composed of basic codec modules based on transformer.

10. The method according to claim 9, characterized in that The basic encoding and decoding module includes a first feature processing branch and a second feature processing branch; The first feature processing branch is used to perform spatial dimension reduction and feature stretching processing on the input features to obtain a spatial representation; and perform spatial attention processing on the spatial representation through a transformer; Performing feature restoration on the spatial representation after the spatial attention processing to obtain a spatial attention feature; The second feature processing branch is used to perform channel dimension reduction processing and feature stretching processing on the input feature to obtain a channel representation; and perform channel attention processing on the channel representation through a transformer; Performing feature restoration on the channel representation after channel attention processing to obtain channel attention features; The basic encoding and decoding module is used to perform residual connection on the spatial attention feature, the channel attention feature and the input feature.

11. The method according to claim 10, characterized in that The basic encoding and decoding module in the encoder is also used to perform identity mapping or downsampling processing on the features obtained by residual connection; The basic encoding and decoding module in the decoder is also used to perform identity mapping or upsampling processing on the features obtained by residual connection.

12. The method according to claim 9, characterized in that The method further comprises: Inputting a sample image into the moiré removal model to obtain a multi-scale sample de-moiré image output by the moiré removal model, wherein the sample de-moiré images of different scales are generated based on different scale features output by the decoder in the moiré removal model; Determine a multi-scale spatial domain loss based on a true value of a multi-scale sample de-moiré image corresponding to the sample image and the multi-scale sample de-moiré image, wherein the multi-scale spatial domain loss is used to characterize the difference of the multi-scale images in the spatial domain; Determine a multiscale frequency domain loss based on a multiscale down-sampled image corresponding to the sample image, a true value of the multiscale sample de-moiré image, and the multiscale sample de-moiré image, wherein the multiscale frequency domain loss is used to characterize the difference of the multiscale image in the frequency domain; The moiré removal model is trained based on the multi-scale spatial domain loss and the multi-scale frequency domain loss.

13. The method according to claim 12, characterized in that The determining of the multi-scale frequency domain loss based on the multi-scale down-sampled image corresponding to the sample image, the true value of the multi-scale sample de-moiré image, and the multi-scale sample de-moiré image includes: Based on the sample image and the sample de-moiré image at the same scale, determining a frequency domain low-pass loss, wherein the frequency domain low-pass loss is used to guide learning of low-frequency features of the sample image; Based on the true value of the sample demoiré image and the sample demoiré image at the same scale, a frequency domain high-pass loss is determined, and the frequency domain high-pass loss is used to guide the learning of high-frequency features of the true value of the sample demoiré image.

14. The method according to claim 1, characterized in that The step of performing structured data parsing on the chart in the chart image based on the chart type of the chart included in the chart image to obtain structured data of the chart in the chart image includes: In the case where the chart type is a graph, performing structured data parsing on the graph in the chart image by using a graph parsing model to obtain graph structured data; When the chart type is a table, the table in the chart image is parsed for structured data using a table parsing model to obtain table structured data.

15. The method according to claim 14, characterized in that The graph parsing model adopts a codec structure; The step of performing structured data parsing on the graph in the chart image by using a graph parsing model to obtain graph structured data includes: Extracting features from the graph through an encoder of the graph parsing model to obtain graph features of the graph; Based on the graph features, node target detection and node relationship extraction are performed through the decoder of the graph parsing model to obtain node information of the nodes in the graph and node relationships between the nodes.

16. The method according to claim 15, characterized in that The extracting features of the graph by using the encoder of the graph parsing model to obtain graph features of the graph includes: Performing multi-scale feature extraction on the image by the encoder to obtain multi-scale features; Performing self-attention processing on the smallest scale feature in the multi-scale features to obtain full-image features; The full-image feature is fused with features of other scales in the multi-scale feature to obtain the image feature of the image.

17. The method according to claim 15, characterized in that The step of performing node target detection and node relationship extraction based on the graph features by using a decoder of the graph parsing model to obtain node information of nodes in the graph and node relationships between nodes includes: Performing multi-head self-attention processing on the graph feature through the first decoder to obtain a multi-head self-attention feature; performing deformable attention processing on the graph feature and the multi-head self-attention feature through the first decoder to obtain the node relationship pair; The second decoder performs multi-head self-attention processing on the graph features to obtain multi-head self-attention features; the second decoder performs deformable attention processing on the graph features and the multi-head self-attention features to obtain the node information.

18. The method according to claim 17, characterized in that The first decoder is provided with a first prediction head, the first prediction head is used to output a node relationship pair correction value and a first confidence, the node relationship pair correction value is used to correct the node relationship pair; The method further comprises: During the training process, determining the node relationship loss based on the sample node relationship pairs and the true value of the node information output by the first decoder; Determining a confidence loss based on the first confidence output by the first decoder; The first decoder is trained based on the node relationship loss and the confidence loss.

19. The method according to claim 18, characterized in that The determining the node relationship loss based on the sample node relationship pairs and the true value of the node information output by the first decoder includes: By using the Hungarian algorithm, a first node circumscribed rectangle and a second node circumscribed rectangle are respectively assigned to the first node predicted coordinates and the second node predicted coordinates represented by the sample node relationship from the true value of the node information; The node relationship loss is determined based on the first node prediction coordinates, the second node prediction coordinates, the first node circumscribed rectangle, and the second node circumscribed rectangle.

20. The method according to claim 17, characterized in that The second decoder is provided with a second prediction head, the second prediction head is used to output a node information correction value and a second confidence, the node information correction value is used to correct the node information; The method further comprises: During the training process, determining the node information loss based on the sample node information output by the second decoder and the true value of the node information; Determining a confidence loss based on the second confidence output by the second decoder; The second decoder is trained based on the node information loss and the confidence loss.

21. The method according to claim 14, characterized in that When the chart type is a table, performing structured data parsing on the table in the chart image by using a table parsing model to obtain table structured data includes: Performing segmentation line regression on the table through the segmentation line regression network of the table parsing model to obtain segmentation line coordinates, wherein the segmentation lines include row segmentation lines and column segmentation lines; Based on the dividing line coordinates and the table, cells are merged through a cell merging network of the table parsing model to obtain cell information.

22. A data analysis device for a chart, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be identified, wherein the image to be identified is an image containing a parsable chart; An image processing module, used for enhancing the image quality of the image to be identified to obtain an enhanced image; an image extraction module for extracting a chart image of the parseable chart from the enhanced image; The parsing module is used to perform structured data parsing on the chart in the chart image based on the chart type of the chart contained in the chart image to obtain structured data of the chart in the chart image.

23. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the data parsing method of the chart as described in any one of claims 1 to 21.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the data parsing method of the chart according to any one of claims 1 to 21.

25. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the data analysis method of the chart as described in any one of claims 1 to 21.

Images