Chart data extraction method and device

Abstract

The invention provides a chart data extraction method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target image corresponding to a current chart; performing content extraction on the target image to obtain a text extraction result; inputting the target image, the text extraction result and a first cue word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; and inputting the target image, the semantic classification result and a second cue word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model. According to the chart data extraction method provided by the invention, semantic classification and structured conversion are respectively performed on the current chart according to the target image, the structured table data readable by the machine is generated, and accurate data extraction of the current chart is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a chart data extraction method and device. BACKGROUND

[0002] Compared with pure text or raw data tables, charts can more efficiently reveal key information such as data distribution, trends, correlations, and outliers. In order to efficiently utilize the visual information in charts (such as data points, coordinate axes, labels, legends, titles, etc.), it is necessary to accurately and completely convert it into structured machine-readable data (such as tables, JSON, relational database records), and to achieve chart data extraction.

[0003] The methods for chart data extraction in the prior art mainly include computer vision methods (such as using image processing or optical character recognition technology, etc.), key point detection-based methods (such as using pre-trained convolutional neural networks or YOLO series target detection models, etc.), and multimodal large model (MLLMs) based methods. However, the above methods do not fully utilize the prior information such as text and layout in charts, resulting in low accuracy of chart data extraction. SUMMARY

[0004] The present application provides a chart data extraction method and device to solve the technical problem of low accuracy of chart data extraction in the prior art, and to improve the accuracy of chart data extraction.

[0005] The present application provides a chart data extraction method, comprising the following steps: obtaining a target image corresponding to a current chart; performing content extraction on the target image to obtain a text extraction result; inputting the target image, the text extraction result, and a first prompt word into a first multimodal large model to obtain a semantic classification result output by the first multimodal large model; the first multimodal large model is trained using historical charts as feature data and corresponding semantic annotation data as label data; inputting the target image, the semantic classification result, and a second prompt word into a second multimodal large model to obtain structured table data output by the second multimodal large model; the second multimodal large model is trained using the historical charts as feature data and corresponding table annotation data as label data.

[0006] According to the chart data extraction method provided by the present application, the target image corresponding to the chart is obtained, comprising: obtaining an original image corresponding to the current chart; The bounding boxes of the text regions in the original image are extracted using an optical character recognition tool; The target scaling factor is determined based on the height of the bounding box; The original image is scaled based on the target scaling factor to obtain the target image.

[0007] According to a chart data extraction method provided by the present invention, determining the target scaling factor based on the height of the bounding box includes: The target scaling factor is the ratio of a first preset threshold to the average height.

[0008] According to a chart data extraction method provided by the present invention, the step of extracting content from the target image to obtain text extraction results includes: Using an optical character recognition tool, the horizontal and vertical coordinates of each text box corresponding to each text segment in the target image are extracted; Based on the horizontal coordinates, all text boxes are grouped to obtain multiple groups of text boxes; Based on the vertical coordinates, adjacent text boxes in each group of text boxes are merged to obtain the text extraction result.

[0009] According to a chart data extraction method provided by the present invention, the step of grouping all text boxes based on the horizontal coordinate to obtain multiple groups of text boxes includes: Based on the horizontal coordinates, determine the horizontal overlap between all the text boxes; Based on the horizontal overlap and the second preset threshold, all text boxes are grouped to obtain multiple groups of text boxes.

[0010] According to a chart data extraction method provided by the present invention, the step of merging adjacent text boxes in each group of text boxes based on the vertical coordinate to obtain the text extraction result includes: Based on the vertical coordinates, each text box in each group of text boxes is sorted to obtain multiple text box sequences; Based on the vertical coordinates and the multiple text box sequences, calculate the vertical spacing between adjacent text boxes in each text box sequence; Based on the vertical spacing and the third preset threshold, the adjacent text boxes are merged to obtain multiple text blocks; The multiple text blocks are used as the text extraction results.

[0011] The present invention also provides a chart data extraction device, comprising the following modules: The acquisition module is used to acquire the target image corresponding to the current chart; An extraction module is configured to perform content extraction on the target image to obtain a text extraction result. A semantic classification module is configured to input the target image, the text extraction result and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model, wherein the first multi-modal large model is trained by using historical charts as feature data and corresponding semantic annotation data as label data. A table generation module is configured to input the target image, the semantic classification result and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model, wherein the second multi-modal large model is trained by using the historical charts as feature data and corresponding table annotation data as label data.

[0012] The present application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the chart data extraction method according to any one of the above when executing the computer program.

[0013] The present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the chart data extraction method according to any one of the above.

[0014] The present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the chart data extraction method according to any one of the above.

[0015] The chart data extraction method and apparatus provided by this invention first acquires the target image corresponding to the current chart; then, it extracts content from the target image to obtain text extraction results, thereby obtaining semantically complete text paragraphs, reducing recognition errors caused by fragmented text, and providing reliable input for subsequent semantic classification; then, it inputs the target image, text extraction results, and a first prompt word into a first multimodal large model to obtain the semantic classification results output by the first multimodal large model; wherein, the first multimodal large model is trained using a first historical chart as feature data and corresponding semantic annotation data as label data, thereby utilizing the chain reasoning of the first multimodal large model. The system can perform fine-grained semantic classification on the extracted text results, providing a structured context for table generation, avoiding the problem of chaotic logical mapping in existing methods, and improving the accuracy of chart data extraction. Finally, the target image, semantic classification results, and second prompt words are input into the second multimodal large model to obtain the structured table data output by the second multimodal large model. The second multimodal large model is trained using the second historical chart as feature data and the corresponding table annotation data as label data. Thus, the second multimodal large model generates machine-readable structured table data based on the target image and semantic classification results, realizing accurate data extraction of the current chart. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the chart data extraction method provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the chart data extraction device provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Chart, as the core carrier of Information Visualization, is widely used in Business Intelligence (BI) analysis, scientific discovery and verification, financial reporting, education dissemination, news media, and public decision support, etc. due to its ability to transform abstract data into intuitive visual patterns. Compared with pure text or raw data tables, charts can more efficiently reveal key information such as data distribution, trends, correlations, and outliers.

[0022] However, the highly structured visual characteristics of charts (such as graphical elements, spatial layout, color coding) make it difficult for machines to accurately understand, analyze, and utilize chart content automatically. In order to effectively support downstream intelligent tasks (such as chart question answering, automatic summary generation, knowledge graph construction, or data-driven decision automation), it is necessary to accurately and completely transform the visual information in charts (data points, coordinate axes, labels, legends, titles, etc.) into structured machine-readable data (such as tables, JSON, relational database records), i.e., chart data extraction (Chart2table).

[0023] Currently, chart data extraction techniques mainly include: (1) Traditional computer vision methods: This kind of method highly depends on domain knowledge and manual design. Usually based on image processing techniques (such as edge detection, Hough transform) to identify the basic structural elements of charts (coordinate axes, scale lines, data units), combined with Optical Character Recognition (OCR) technology to extract text labels and legends, and finally through heuristic rules and templates (for example, based on pixel position mapping data points) to parse charts and reconstruct data tables.

[0024] (2) Methods based on key point detection: This kind of method uses pre-trained convolutional neural networks (CNN) or object detection models (such as Hourglass Net or YOLO series models) to locate key semantic elements in charts (such as data points, data bars, coordinate axis intersection points, text regions, legend items), and get the key points of these elements (corner points of bounding boxes, inflection points of line charts, etc.). Then, combined with OCR to recognize the text information (numerical values, category labels) associated with the key points. Finally, rely on rule-based post-processing to establish the logical relationship between elements (such as the correspondence between data points and axis labels) and assemble into tables.

[0025] (3) Vision-language model based method: This kind of method usually takes the image corresponding to the whole chart as input, and uses the pre-trained MLLMs to "end-to-end" directly generate the structured text (such as CSV table, JSON format) describing the chart data.

[0026] However, the above prior art still has significant and urgent defects: 1) excessive dependence on visual input: the existing method usually only inputs the chart image into the model, and fails to fully utilize the prior information such as text and layout available in the chart, which leads to a high error rate of text recognition when the model processes visually challenging cases (such as small font, blurred text, rotated text, low contrast background, overlapping text), thereby polluting the final data extraction result; 2) disorganized structure: the model is prone to logical confusion when establishing the mapping relationship between the row title, column title and numerical value, resulting in inaccurate table structure generated; 3) lack of systematic research: the existing technology only regards chart data extraction as an auxiliary step or preprocessing link for implementing higher-level tasks such as chart question answering (ChartQA), and the model design and training target is not directly and optimally serving high-precision structured data extraction itself, resulting in low accuracy of the structured text describing the chart data generated.

[0027] Therefore, the present application provides a chart data extraction method, which effectively fuses visual and textual information to improve the accuracy of text recognition, and ensures the logical correctness of the generated table through a structured reasoning process. At the same time, this method has strong generalization ability and can handle complex and diverse charts, thereby providing high-quality structured data support for downstream intelligent applications.

[0028] The chart data extraction method and device of the present application will be described below. Figures 1 to 3 The chart data extraction method and device of the present application will be described below.

[0029] Figure 1 The chart data extraction method provided by the present application is shown in the flowchart as shown in Figure 1 The method comprises the following steps: Step 101, obtaining a target image corresponding to the current chart.

[0030] Specifically, according to the current chart, first, the corresponding original image (such as PNG or JPG format) is obtained, and then the original image can be scaled by a predetermined scale, so that the size of each original image remains consistent, obtaining the target image, thereby unifying the feature scale, alleviating the feature deviation caused by multi-scale charts, and improving the accuracy of subsequent content extraction.

[0031] Step 102, performing content extraction on the target image to obtain a text extraction result.

[0032] Specifically, all text contents in the image and their corresponding text box coordinates (including horizontal coordinates and vertical coordinates) are detected and recognized from the target image using an optical character recognition (OCR) tool, such as the PaddleOCR tool. Each text paragraph corresponds to a text box.

[0033] According to the horizontal coordinates and vertical coordinates of the text boxes, the text boxes are sequentially grouped and merged to obtain a plurality of text blocks as the text extraction result.

[0034] Specifically, first, the overlap degree between the text boxes is calculated according to the horizontal coordinates of the text boxes; then, a series of text boxes are divided into several groups according to the overlap degree, so that the text boxes in each group have significant overlap in the horizontal direction; then, each text box in each group of text boxes is sorted according to the vertical coordinates of the text boxes, to obtain a plurality of text box sequences, and the vertical spacing between adjacent text boxes in each text box sequence is calculated; finally, according to the size of the vertical spacing, it is judged whether the text contents in the adjacent text boxes belong to the same continuous text paragraph, so that the adjacent text boxes corresponding to the same continuous text paragraph are merged to obtain a text block with complete semantics. For adjacent text boxes that do not correspond to the same continuous text paragraph, a breakpoint is set, which is regarded as a blank paragraph and is not merged.

[0035] The embodiment of the present application detects and recognizes all text contents in the image and their corresponding text boxes from the target image by content extraction of the target image, and sequentially groups and merges a series of text boxes according to the coordinates of the text boxes, so that continuous text paragraphs are merged into text blocks, to obtain a text extraction result with complete semantics, avoiding subsequent model misinterpretation of fragmented text, thereby improving the accuracy of semantic classification of the subsequent model for the text extraction result. At the same time, a breakpoint is set between non-continuous text boxes to accurately detect blank paragraphs, which can avoid mismerging when processing multiple lines of text paragraphs, enhancing the generalization and reliability of content extraction of the target image.

[0036] Step 103, inputting the target image, the text extraction result, and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; the first multi-modal large model is obtained by training historical charts as feature data and corresponding semantic annotation data as label data.

[0037] Specifically, existing Multimodal Large Language Models (MLLMs) are usually based on the Transformer architecture, consisting of a visual encoder, a language model, and a cross-modal fusion module. Among them, the visual encoder (such as ViT or BEiT, etc.) is used to convert images into feature vectors, the language model (such as the Deepseek series of large language models or the Tongyi Qianwen series of large models, etc.) is used to process text information, and the cross-modal fusion module is used to establish the association between visual and text features through an attention mechanism.

[0038] In the embodiments of the present application, the first multimodal large model is obtained by supervised training based on existing MLLMs, using historical charts as feature data and corresponding semantic annotation data as label data.

[0039] The target image, text extraction result, and first prompt word are input into the first multimodal large model, and the first multimodal large model is guided by the first prompt word to perform first-stage reasoning, i.e., according to the text extraction result and its spatial position in the target image, the semantic role of each text block in the current chart is recognized, so as to classify each text block into one of the following semantic labels: chart title (chart_title), axis title (axis_title), legend label (legend_label), tick label (tick_label), value label (value_label), or other (other). The semantic classification result finally output by the first multimodal large model is a structured list, where each list element is composed of a text block and its corresponding semantic label concatenated by a delimiter.

[0040] For example, the model needs to determine that “Sales (ten thousand yuan)” is the Y-axis title axis_title, “Q1” and “Q2” are the X-axis tick labels tick_label, and “Product A” is the legend label legend_label, etc. The output of the first multimodal large model is a structured list, which assigns a semantic label to each text block, and the output is “ <sep>" as a separator, for example "sales (ten thousand yuan) <sep>axis_title".

[0041] The embodiment of the present application realizes accurate semantic classification of the text extraction result text block by guiding the first multi-modal large model to perform first stage reasoning.

[0042] Step 104, inputting the target image, the semantic classification result and the second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; the second multi-modal large model is obtained by training the historical chart as feature data and corresponding table annotation data as label data.

[0043] Specifically, the second multi-modal large model is obtained by training the existing MLLMs as the basis, the historical chart as the feature data, and the corresponding table annotation data as the label data for supervised training.

[0044] The target image, the semantic classification result and the second prompt word are input into the second multi-modal large model, the second multi-modal large model is guided by the second prompt word to perform second stage reasoning, and the following tasks are sequentially completed: 1) accurately identifying the current chart type; 2) analyzing the coordinate axis and legend information in combination with the semantic classification result; 3) inferring the value according to the visual elements (such as the column height of the column chart in the current chart and the position of the line point in the line chart); 4) converting the extracted information into a structured table; 5) ensuring the output format to be standardized to obtain structured table data, that is, the output format is a string table data separated by line feed characters and tab characters or special characters.

[0045] In some embodiments, the structured table data generated by the second multi-modal large model after the second stage reasoning can also be parsed and processed: first, the structured table data is structured into a list by a text parsing algorithm (for example, a sequence-to-sequence (Seq2Seq) model or a Transformer-based parser), where each element of the list corresponds to a row of data in the table; then the structured list is serialized into a comma-separated value (Comma-Separated Values, CSV) format file, and the comma is used as a field separator to separate the column data in the same row during the serialization process, and the line feed character is used to separate the different row data, finally generating a CSV format file, obtaining a file with the advantages of strong universality, high compatibility, easy transmission and storage, clear structure, etc., realizing accurate data extraction of the current chart.

[0046] Based on the above embodiments, in the training process of the first multi-modal large model and the second multi-modal large model, a unified prompt template containing two inference stages is designed for the feature data in each sample; if the feature data in the sample contains semantic annotation data, only the loss of the first stage output is calculated; if the feature data in the sample contains table annotation data, only the loss of the second stage output is calculated; through dynamic selection of loss calculation, integrated joint training of two incomplete annotation sample data sets is realized. The total loss function is defined as follows: wherein, denotes the total loss function, is the total number of samples , denotes the loss of the first stage output, denotes the loss of the second stage output, ∈{0,1} is a sample-level mask coefficient, satisfying: The embodiment of the application solves the defects of MLLMs in text recognition and data structure organization by combining high-precision OCR technology with the chain inference ability of MLLMs, significantly improves the accuracy and robustness of chart data extraction; through the two-stage inference framework and the stage adaptive training strategy, efficient model training is realized using incomplete annotation data, improving data utilization and model generalization ability; by setting the model in a unified training process to learn two tasks of text semantic understanding and table structure inference at the same time, not only the complementarity of the two types of annotation data is utilized, but also the logical relationship between the two inference stages is implicitly modeled, so that the natural connection and cooperation of the two-stage output can be realized in actual inference, and the comprehensive performance of the model is improved.

[0047] ​The chart data extraction method provided by the application comprises the following steps: first, acquiring a target image corresponding to a current chart; then, performing content extraction on the target image to obtain a text extraction result, so as to obtain a text paragraph with complete semantics, reduce recognition errors caused by fragmented text, and provide reliable input for subsequent semantic classification; then, inputting the target image, the text extraction result and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; wherein, the first multi-modal large model is obtained by training a first historical chart as feature data and corresponding semantic annotation data as label data, so as to perform fine-grained semantic classification on the text extraction result by using the chain reasoning capability of the first multi-modal large model, provide structured context for table generation, avoid the problem of logical mapping confusion in the prior art, and improve the accuracy of chart data extraction; finally, inputting the target image, the semantic classification result and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; wherein, the second multi-modal large model is obtained by training a second historical chart as feature data and corresponding table annotation data as label data, so as to generate machine-readable structured table data according to the target image and the semantic classification result by using the second multi-modal large model, and realize accurate data extraction of the current chart.

[0048] Further, the acquiring the target image corresponding to the chart comprises: acquiring an original image corresponding to the current chart; extracting a bounding box of a text region of the original image by using an optical character recognition tool; determining a target scaling coefficient based on a height of the bounding box; scaling the original image based on the target scaling coefficient to obtain the target image.

[0049] First, an original image (such as a PNG or JPG format) corresponding to a current chart is acquired, and then an open-source OCR tool (such as PaddleOCR) is used to detect all text regions in the original image to obtain text content t i and a bounding box (x1, y1, x2, y2) of the text region, wherein (x1, y1) represents the coordinates of the left upper corner point of the bounding box, and (x2, y2) represents the coordinates of the right lower corner point of the bounding box.

[0050] According to the above coordinates, the height of the bounding box is calculated as y2-y1, and a target scaling coefficient is determined based on the height of the bounding box.

[0051] Further, the determining the target scaling coefficient based on the height of the bounding box comprises: taking the ratio of a first preset threshold value and the mean value of the height as the target scaling coefficient.

[0052] Specifically, the average of the heights of all the bounding boxes is calculated, and a first preset threshold is set, and the target scaling factor is calculated as follows: In the formula, represents the target scaling factor, represents the first preset threshold (in the embodiment of the application, 30 pixels is determined according to experiments), represents the average of the heights of all the bounding boxes, which is used to measure the feature scale of the original image.

[0053] The embodiment of the application obtains a target image by scaling the entire original image by the target scaling factor, thereby unifying the feature scales of different resolution charts, improving the recognition rate of small font text, and alleviating the feature deviation caused by the multi-scale problem of the image, and improving the accuracy of subsequent content extraction.

[0054] Further, the content extraction on the target image is performed to obtain a text extraction result, including: extracting the horizontal coordinates and the vertical coordinates of each text box corresponding to each text paragraph in the target image by using an optical character recognition tool; grouping all the text boxes based on the horizontal coordinates to obtain a plurality of groups of text boxes; merging adjacent text boxes in each group of text boxes based on the vertical coordinates to obtain the text extraction result.

[0055] Specifically, since an OCR tool usually outputs text in a single line unit, continuous text (such as a paragraph of explanatory text containing multiple lines) is divided into multiple discrete text blocks. Therefore, the embodiment of the application proposes a text box grouping and merging strategy based on position relationship and statistical features.

[0056] All the text regions in the target image are detected by using an open source OCR tool (such as PaddleOCR) to obtain the horizontal coordinates and the vertical coordinates of each text box corresponding to each text paragraph in the target image, that is, a text box set: In the formula, represents the text box set, represents a text paragraph corresponding text box, respectively represents the top-left corner point coordinates ( , ) and the bottom-right corner point coordinates ( , ) of the text box . Among them, and are horizontal coordinates, and All are vertical coordinates.

[0057] Furthermore, based on the horizontal coordinates, all text boxes are grouped to obtain multiple groups of text boxes, including: Based on the horizontal coordinates, determine the horizontal overlap between all the text boxes; Based on the horizontal overlap and the second preset threshold, all text boxes are grouped to obtain multiple groups of text boxes.

[0058] Specifically, a series of text boxes in the text box set are divided into several groups, such that the text boxes in each group have significant overlap in the horizontal direction and maintain reasonable spacing in the vertical direction, so as to reflect their logical relationship in the layout.

[0059] The first text box in the text box collection is used as the initial member of the first group. Each subsequent text box attempts to join an existing group based on predefined grouping conditions. If no group meets the conditions, a new group is created and the text box becomes the first member of that group.

[0060] The predefined grouping conditions are as follows: Based on the horizontal coordinate, the method for calculating the horizontal spatial overlap is defined, that is, the expression for calculating the horizontal overlap between two text boxes is as follows: In the formula, Represents any ungrouped text box coordinates This refers to any grouped text box. coordinates Text box With text box The horizontal overlap between them.

[0061] If and only if the text box It satisfies the following conditions as well as all text boxes within the group: Text box Grouping ,in, The horizontal overlap threshold (i.e., the second preset threshold, which is experimentally determined to be 0.6 in this embodiment of the invention). The vertical spacing coefficient (in this embodiment of the invention, the value was determined to be 0.3 through experiments). For text boxes The height. If none of the groups meet the criteria, a new group is created, and... As the initial members of the new group.

[0062] Further, the merging of adjacent text boxes in each group of text boxes based on the vertical coordinates to obtain the text extraction result comprises: sequencing each text box in each group of text boxes based on the vertical coordinates to obtain a plurality of text box sequences; calculating the vertical spacing between adjacent text boxes in each text box sequence based on the vertical coordinates and the plurality of text box sequences; merging the adjacent text boxes based on the vertical spacing and a third preset threshold to obtain a plurality of text blocks; using the plurality of text blocks as the text extraction result.

[0063] Specifically, for each group, first, the text boxes in the group are sorted by vertical coordinates to obtain a plurality of text box sequences; then, the vertical spacing between adjacent text boxes in the text box sequence is calculated to obtain a spacing sequence composed of a plurality of vertical spacings; then, the mean and the standard deviation of the spacing sequence are calculated; then, a judgment is made based on and If ( is a preset third preset threshold), it is determined that the text content in the group of text boxes belongs to the same continuous text paragraph, and all text boxes in the group are merged into one text block; If , an adaptive breakpoint threshold is calculated: In the formula, denotes the adaptive breakpoint threshold, is a preset coefficient of variation tolerance. The spacing position greater than in the spacing sequence is regarded as a breakpoint, and the original group is divided into a plurality of subgroups according to the spacing position; finally, a merging operation is performed on each subgroup obtained by division, that is, all text boxes in the subgroup are merged into one text block.

[0064] The embodiment of the present application groups all text boxes, then merges or divides the text boxes in the group into text blocks that are more in line with semantics, and obtains text paragraphs with more complete semantics, thereby providing more accurate and reasonable data for subsequent model input, and improving the accuracy of chart data extraction.

[0065] The chart data extraction device provided by the present application is described below. The chart data extraction device described below can be correspondingly referred to the chart data extraction method described above.

[0066] Figure 2 ​is a structural schematic diagram of a chart data extraction device provided by the present application, as shown in Figure 2 The chart data extraction device provided by the embodiment of the present application comprises an acquisition module 201, an extraction module 202, a semantic classification module 203 and a table generation module 204. The acquisition module 201 is used for acquiring a target image corresponding to a current chart; the extraction module 202 is used for performing content extraction on the target image to obtain a text extraction result; the semantic classification module 203 is used for inputting the target image, the text extraction result and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; the first multi-modal large model is obtained by training historical charts as feature data and corresponding semantic annotation data as label data; and the table generation module 204 is used for inputting the target image, the semantic classification result and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; the second multi-modal large model is obtained by training the historical charts as feature data and corresponding table annotation data as label data.

[0067] The chart data extraction device provided by the present application firstly acquires a target image corresponding to a current chart; then performs content extraction on the target image to obtain a text extraction result, thereby obtaining a text paragraph with complete semantics, reducing recognition errors caused by fragmented text and providing reliable input for subsequent semantic classification; then inputting the target image, the text extraction result and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; wherein the first multi-modal large model is obtained by training a first historical chart as feature data and corresponding semantic annotation data as label data, thereby utilizing the chain reasoning capability of the first multi-modal large model to perform fine-grained semantic classification on the text extraction result, providing structured context for table generation, avoiding the problem of chaotic logic mapping in the existing method and improving the accuracy of chart data extraction; finally, inputting the target image, the semantic classification result and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; wherein the second multi-modal large model is obtained by training a second historical chart as feature data and corresponding table annotation data as label data, thereby generating machine-readable structured table data according to the target image and the semantic classification result through the second multi-modal large model, and realizing accurate data extraction of the current chart.

[0068] Figure 3 An example of an entity structure schematic diagram of an electronic device is shown in Figure 3 As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can invoke the logic instructions in the memory 330 to execute the chart data extraction method, which includes: obtaining a target image corresponding to a current chart; performing content extraction on the target image to obtain a text extraction result; inputting the target image, the text extraction result, and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; the first multi-modal large model is obtained by training historical charts as feature data and corresponding semantic annotation data as label data; inputting the target image, the semantic classification result, and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; the second multi-modal large model is obtained by training the historical charts as feature data and corresponding table annotation data as label data.

[0069] In addition, the logic instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0070] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the chart data extraction method provided by the above-mentioned methods, which includes: obtaining a target image corresponding to a current chart; performing content extraction on the target image to obtain a text extraction result; input the target image, the text extraction result, and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; the first multi-modal large model is obtained by training historical charts as feature data and corresponding semantic annotation data as label data; input the target image, the semantic classification result, and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; the second multi-modal large model is obtained by training the historical charts as feature data and corresponding table annotation data as label data.

[0071] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the chart data extraction method provided by the above method, the method comprising: obtaining a target image corresponding to a current chart; performing content extraction on the target image to obtain a text extraction result; input the target image, the text extraction result, and a first prompt word into a first multi-modal large model to obtain a semantic classification result output by the first multi-modal large model; the first multi-modal large model is obtained by training historical charts as feature data and corresponding semantic annotation data as label data; input the target image, the semantic classification result, and a second prompt word into a second multi-modal large model to obtain structured table data output by the second multi-modal large model; the second multi-modal large model is obtained by training the historical charts as feature data and corresponding table annotation data as label data.

[0072] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0073] Those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary universal hardware platform, and of course can also be implemented by hardware, through the description of the foregoing embodiments. Based on such an understanding, the technical solutions described above can essentially or in other words make contributions to the prior art, and can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0074] It should be noted that, in this document, the terms "comprise", "comprise", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted, or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0075] In the embodiments of the present application, "determining B based on A" means that A is considered as a factor when determining B. It is not limited to "determining B based on A only", but also includes "determining B based on A and C", "determining B based on A, C and E", "determining C based on A, and determining B based on C further", and the like. In addition, it can also include A as a condition for determining B, for example, "when A meets the first condition, B is determined using the first method"; for example, "when A meets the second condition, B is determined"; for example, "when A meets the third condition, B is determined based on the first parameter"; and the like. Of course, A can also be a condition for determining B, for example, "when A meets the first condition, C is determined using the first method, and B is further determined based on C".

[0076] It should be noted that the terms "target", "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally a class and do not limit the number of objects, for example, the first object can be one or more.

[0077] In the present application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0078] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions described in the foregoing examples can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.< / sep> < / sep>

Claims

1. A method for extracting chart data, characterized in that, include: Get the target image corresponding to the current chart; Content extraction is performed on the target image to obtain text extraction results; The target image, the text extraction result, and the first prompt word are input into the first multimodal large model to obtain the semantic classification result output by the first multimodal large model; the first multimodal large model is trained using historical charts as feature data and corresponding semantic annotation data as label data. The target image, the semantic classification result, and the second prompt word are input into the second multimodal large model to obtain the structured table data output by the second multimodal large model; the second multimodal large model is trained using the historical charts as feature data and the corresponding table annotation data as label data.

2. The chart data extraction method according to claim 1, characterized in that, The step of obtaining the target image corresponding to the chart includes: Obtain the original image corresponding to the current chart; The bounding boxes of the text regions in the original image are extracted using an optical character recognition tool; The target scaling factor is determined based on the height of the bounding box; The original image is scaled based on the target scaling factor to obtain the target image.

3. The chart data extraction method according to claim 2, characterized in that, Determining the target scaling factor based on the height of the bounding box includes: The target scaling factor is the ratio of a first preset threshold to the average height.

4. The chart data extraction method according to claim 1, characterized in that, The step of extracting content from the target image to obtain text extraction results includes: Using an optical character recognition tool, the horizontal and vertical coordinates of each text box corresponding to each text segment in the target image are extracted; Based on the horizontal coordinates, all text boxes are grouped to obtain multiple groups of text boxes; Based on the vertical coordinates, adjacent text boxes in each group of text boxes are merged to obtain the text extraction result.

5. The chart data extraction method according to claim 4, characterized in that, Based on the horizontal coordinates, all text boxes are grouped to obtain multiple groups of text boxes, including: Based on the horizontal coordinates, determine the horizontal overlap between all the text boxes; Based on the horizontal overlap and the second preset threshold, all text boxes are grouped to obtain multiple groups of text boxes.

6. The chart data extraction method according to claim 5, characterized in that, The step of merging adjacent text boxes in each group of text boxes based on the vertical coordinates to obtain the text extraction result includes: Based on the vertical coordinates, each text box in each group of text boxes is sorted to obtain multiple text box sequences; Based on the vertical coordinates and the multiple text box sequences, calculate the vertical spacing between adjacent text boxes in each text box sequence; Based on the vertical spacing and the third preset threshold, the adjacent text boxes are merged to obtain multiple text blocks; The multiple text blocks are used as the text extraction results.

7. A chart data extraction device, characterized in that, include: The acquisition module is used to acquire the target image corresponding to the current chart; The extraction module is used to extract content from the target image to obtain text extraction results; The semantic classification module is used to input the target image, the text extraction result, and the first prompt word into the first multimodal large model to obtain the semantic classification result output by the first multimodal large model; the first multimodal large model is trained using historical charts as feature data and corresponding semantic annotation data as label data. The table generation module is used to input the target image, the semantic classification result, and the second prompt word into the second multimodal large model to obtain the structured table data output by the second multimodal large model; the second multimodal large model is trained using the historical charts as feature data and the corresponding table annotation data as label data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the chart data extraction method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the chart data extraction method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the chart data extraction method as described in any one of claims 1 to 6.

Images