A deep learning-based scientific research chart data extraction and reconstruction method and system

Abstract

This invention discloses a method and system for extracting and reconstructing scientific chart data based on deep learning. It is applied to the interdisciplinary fields of scientific data processing and analysis and artificial intelligence. The method includes: identifying key elements in scientific charts and outputting pixel-level coordinates; using a tick value-line coordinate alignment algorithm to match the tick value text with the corresponding tick line position; and constructing two coordinate mapping models, linear and logarithmic, to map and model the true semantic values ​​and pixel coordinate values ​​of the tick lines on the x-axis and y-axis, respectively. Based on R... 2 The coefficient of determination independently selects the optimal mapping model for each axis. For scatter plots, the coordinates of the center point of the target box of the scatter plot are directly substituted into the optimal mapping model to obtain the true values ​​of each axis of the data points. For bar charts, the true values ​​are calculated based on the direction of the bar chart, the position of the baseline, and the structure type, according to the optimal mapping model. This invention realizes end-to-end conversion of scientific charts from pixel space to real data space.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of scientific research data processing and analysis and artificial intelligence, and more specifically to a method and system for extracting and reconstructing scientific research chart data based on deep learning. Background Technology

[0002] With the exponential growth of scientific research data and the rapid development of artificial intelligence technology, data-driven approaches have become an important paradigm for modern scientific research and innovation. In fields such as materials science and chemistry, scientific charts serve as the core carriers of research results, and their data extraction and reconstruction technologies are becoming crucial links in the accumulation of high-quality data under this new research paradigm. Currently, the powerful learning capabilities of deep learning technology can effectively analyze complex visual elements in charts, providing a new technical path to overcome the limitations of traditional methods.

[0003] Chart data extraction techniques are mainly divided into two categories: categorical charts, represented by bar charts, and numerical correlation charts, represented by scatter plots. In categorical chart extraction, Dai et al.'s Chart decoder system uses GoogLeNet and SVM classifiers, combined with traditional image processing methods, to achieve basic bar chart data extraction. Liu et al.'s framework based on VGG16 and Faster-RCNN further improved the recognition accuracy of bar charts and pie charts, achieving more accurate element localization by introducing RNS and CRNN techniques. Carderas et al.'s hybrid method, combining Inceptionv3 and SRCNN, achieved significant progress in automatic bar chart data extraction. In the field of numerical correlation chart processing, Poco et al.'s system based on Darknet and Tesseract achieved automated recovery of scatter plot visual encodings; Shtok et al.'s CHARTER framework uses a four-stage processing flow, achieving element detection in pie charts and scatter plots through Faster-RCNN and CenterNet. The dedicated detector proposed by Ma et al., based on Cascade R-CNN and FCN networks, not only achieves high-precision localization of coordinate points but also solves the key problem of legend matching by constructing a feature matching network. While these methods have made breakthroughs in pixel-level element detection, they still have significant shortcomings in achieving accurate conversion from pixel coordinates to real scientific data. The Scatteract system developed by Cliche et al. is specifically designed for scatter plot extraction, but its application is limited by the accuracy of coordinate axis scale positioning. Although the ChartOCR framework by Luo et al. achieves data extraction from various types of charts, its coordinate transformation method based on simple linear fitting produces significant deviations in practical applications.

[0004] Therefore, how to provide a method for extracting and reconstructing scientific chart data based on deep learning, and achieve accurate mapping from pixel coordinates to real data coordinates, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for extracting and reconstructing scientific chart data based on deep learning. This invention integrates object detection and coordinate mapping algorithms to achieve end-to-end conversion of scientific charts from pixel space to real data space, aiming to solve the problem of automated extraction of unstructured chart data in scientific literature and providing core technical support for data-driven scientific discovery and verification of scientific integrity.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for extracting and reconstructing scientific chart data based on deep learning, comprising: Step 1: Construct a dataset of labeled elements for scientific research charts and train an object detection model based on the dataset to obtain an object detection model for extracting information from scientific research charts; Step 2: Use the trained object detection model to identify key elements in the scientific charts to be extracted and output pixel-level coordinates; the object detection model includes: chart element object detection model and data element object detection model; Step 3: Employ a tick value-line coordinate alignment algorithm to match the tick value text with the corresponding tick line positions, and construct two coordinate mapping models: linear and logarithmic coordinate axes. Map the true semantic values ​​of the tick lines on the x and y axes to their pixel coordinate values ​​using this model, based on R... 2 The coefficient of determination independently selects the optimal mapping model for each axis; Step 4: For scatter plots, directly substitute the coordinates of the center point of the target box of the scatter plot into the optimal mapping model to obtain the true values ​​of each axis of the data points; for bar charts, calculate the true values ​​based on the direction of the bar chart, the position of the baseline, and the structure type according to the optimal mapping model.

[0007] Optionally, in step 2, the chart element target detection model is used to identify and locate basic components including coordinate axes, tick marks, and legends.

[0008] Optionally, in step 2, the data element target detection model is used to identify and extract the center coordinates of the scatter plot data points and the geometric features of the histogram bars.

[0009] Optionally, in step 4, the direction of the histogram is determined based on the numerical axis, specifically as follows: If the value axis is the horizontal axis, the bar chart is vertical; if the value axis is the vertical axis, the bar chart is horizontal.

[0010] Optionally, in step 4, the baseline is determined based on the statistical characteristics of the geometric distribution of the columns, specifically as follows: For vertical bar charts, count the coordinates of the top and bottom boundaries of all bars and find the common boundary as the baseline; for horizontal bar charts, count the coordinates of the left and right boundaries of all bars and find the common boundary as the baseline.

[0011] Optionally, in step 4, the structure type of the bar chart is determined based on the overlap of the center coordinates of the bars along the numerical axis, specifically as follows: If the coordinates coincide along the numerical axis, it is a stacked type; if they are independent along the numerical axis and do not overlap, it is a non-stacked type.

[0012] Optionally, in step 4, for the histogram, the true value is calculated based on the direction of the histogram, the position of the baseline, and the structure type, according to the optimal mapping model, specifically: For vertical, non-stacked columns located above the baseline, calculate the top value as the true value. For vertical, non-stacked columns located below the baseline, calculate the bottom value as the true value. For vertical, stacked columns located above the baseline, calculate the top value minus the bottom value as the true value. For vertical, stacked columns located below the baseline, calculate the bottom value minus the top value as the true value. For horizontal, non-stacked cylinders located to the right of the baseline, calculate the right-end value as the true value. For horizontal, non-stacked cylinders located to the left of the baseline, calculate the left-end value as the true value. For a horizontally stacked column located to the right of the baseline, calculate the right-end value minus the left-end value as the true value. For a horizontally stacked column located to the left of the baseline, calculate the left-end value minus the right-end value as the true value.

[0013] This invention also provides a deep learning-based scientific research chart data extraction and reconstruction system that utilizes a deep learning-based method for extracting and reconstructing scientific research chart data, comprising: Object detection model training module: used to construct a dataset of labeled elements of scientific charts and to train the object detection model based on the dataset to obtain an object detection model for extracting information from scientific charts; Key element extraction module: Used to identify key elements in the scientific charts to be extracted using a trained object detection model, and output pixel-level coordinates; the object detection model includes: chart element object detection model and data element object detection model; The mapping model determination module is used to match the tick value text with the corresponding tick line position using a tick value-line coordinate alignment algorithm, and to construct two coordinate mapping models: linear coordinate axis and logarithmic coordinate axis. It models the mapping between the true semantic values ​​and pixel coordinate values ​​of the tick lines on the x-axis and y-axis, respectively, based on R... 2 The coefficient of determination independently selects the optimal mapping model for each axis; The true value calculation module is used to directly substitute the coordinates of the center point of the target box of the scatter plot into the optimal mapping model to obtain the true value of each axis of the data point; for the bar chart, it calculates the true value based on the direction of the bar chart, the position of the baseline, and the structure type according to the optimal mapping model.

[0014] As can be seen from the above technical solution, compared with the prior art, this invention provides a method and system for extracting and reconstructing scientific research chart data based on deep learning. It proposes an innovative coordinate transformation algorithm design that includes a scale value-line coordinate alignment algorithm and a differentiated transformation strategy, achieving accurate mapping from pixel coordinates to real data coordinates. The system framework proposed in this invention can achieve high-quality extraction and reconstruction of scientific research chart data. Based on object detection and coordinate mapping algorithms, it realizes end-to-end automated extraction from chart recognition to data coordinate reconstruction, which is of significant importance for data-driven analysis and scientific discovery. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the method flow provided by the present invention.

[0017] Figure 2 A schematic diagram illustrating the process of constructing coordinate mapping relationships provided by this invention.

[0018] Figure 3 This is a flowchart illustrating the direction (horizontal or vertical), position based on the baseline, and structure type of the bar chart provided by the present invention.

[0019] Figure 4 A schematic diagram illustrating the specific implementation and application process of this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Embodiment 1 of this invention discloses a method for extracting and reconstructing scientific chart data based on deep learning, comprising: Step 1: Construct a dataset of labeled elements for scientific research charts and train an object detection model based on the dataset to obtain an object detection model for extracting information from scientific research charts; Step 2: Use the trained object detection model to identify key elements in the scientific charts to be extracted and output pixel-level coordinates to provide input for subsequent coordinate transformation; the object detection model includes: chart element object detection model and data element object detection model.

[0022] The chart element object detection model is used to identify and locate 18 basic components, including coordinate axes, tick marks, and legends.

[0023] The data element target detection model is used to identify and extract the center coordinates of scatter plot data points and the geometric features of histogram bars.

[0024] Step 3: As Figure 2 As shown, a tick value-line coordinate alignment algorithm is used to match the tick value text with the corresponding tick line position, eliminating text offset errors. Two coordinate mapping models are constructed: linear and logarithmic coordinate axes. These models map the true semantic values ​​of the tick lines on the x and y axes to their pixel coordinate values, respectively. Based on R... 2 The coefficient of determination is used to independently select the optimal mapping model for each axis.

[0025] Step 4: For scatter plots, directly substitute the coordinates of the center point of the target box of the scatter plot into the optimal mapping model to obtain the true values ​​of each axis of the data points; for bar charts, calculate the true values ​​based on the direction of the bar chart (horizontal or vertical), the position of the baseline, and the structure type, according to the optimal mapping model.

[0026] The process of obtaining the direction (horizontal or vertical), baseline-based position, and structure type of a bar chart, such as... Figure 3 As shown.

[0027] The direction of the bar chart is determined based on the numerical axis, specifically as follows: If the value axis is the horizontal axis, the bar chart is vertical; if the value axis is the vertical axis, the bar chart is horizontal.

[0028] The baseline is determined based on the statistical characteristics of the geometric distribution of the columns, specifically as follows: For vertical bar charts, count the coordinates of the upper and lower boundaries of all bars and find the common boundary (the edge with the same numerical axis coordinates) as the baseline; for horizontal bar charts, count the coordinates of the left and right boundaries of all bars and find the common boundary as the baseline.

[0029] The structure type of the bar chart is determined by the overlap of the center coordinates of the bars along the numerical axis, specifically: If the coordinates coincide along the numerical axis, it is a stacked type; if they are independent along the numerical axis and do not overlap, it is a non-stacked type.

[0030] For the bar chart, the true value is calculated based on the direction of the bar, the position of the baseline, and the structure type, according to the optimal mapping model, as shown in Table 1: For vertical, non-stacked columns located above the baseline, calculate the top value as the true value. For vertical, non-stacked columns located below the baseline, calculate the bottom value as the true value. For vertical, stacked columns located above the baseline, calculate the top value minus the bottom value as the true value. For vertical, stacked columns located below the baseline, calculate the bottom value minus the top value as the true value. For horizontal, non-stacked cylinders located to the right of the baseline, calculate the right-end value as the true value. For horizontal, non-stacked cylinders located to the left of the baseline, calculate the left-end value as the true value. For a horizontally stacked column located to the right of the baseline, calculate the right-end value minus the left-end value as the true value. For a horizontally stacked column located to the left of the baseline, calculate the left-end value minus the right-end value as the true value.

[0031] Table 1. Calculation Rules for Different Cases of Bar Charts

[0032] The above steps enable automated extraction and reconstruction of scientific research chart data. This invention realizes a complete process from locating chart elements to converting them into true semantic values, improving the efficiency and quality of scientific research chart data acquisition, and providing a practical technical solution for scientific research integrity review and data resource construction.

[0033] Example 2: Embodiment 2 of this invention discloses a specific implementation application of a method for extracting and reconstructing scientific chart data based on deep learning, as follows: The implementation of this invention is illustrated using the self-converted open-source chart datasets UB PMC 2022 and Adobe Synthetic as examples.

[0034] According to the present invention, the specific implementation process is as follows: Figure 4 As shown, specifically: (1) The example is derived from data obtained from an open-source chart dataset, and the data is transformed and processed to construct a chart dataset that meets the requirements.

[0035] (2) After extracting, transforming and processing the dataset, the dataset used contains different types of chart data, divided into chart element detection dataset and data element detection dataset. The number of chart element detection data and data element detection data is approximately 7,000 each.

[0036] (3) Based on the MMDetection framework, the element detection module trains the graph element detection dataset and the data element detection dataset respectively based on the Co-DINO model.

[0037] (4) Input the chart to be tested into the element detection module to obtain the pixel coordinates of various elements in the image, including coordinate axes, scales, data points and other element information.

[0038] (5) By using the tick value-line coordinate alignment algorithm, the semantic meaning of the tick value text is matched with the pixel coordinates of the tick line to eliminate the influence caused by the tick value offset. The center point coordinates of the target boxes of the tick value and the tick line are calculated, the Euclidean distance between the center point of each tick value text and the corresponding tick line position is calculated, and cost matrices are constructed for the x-axis and y-axis respectively. The Hungarian matching algorithm is applied to obtain accurate association, which serves as the geometric reference of the coordinate axis.

[0039] (6) Based on the alignment algorithm, establish the mapping relationship between pixel coordinates and true values ​​for the x-axis and y-axis respectively. Establish two regression models, linear and logarithmic, and substitute the pixel coordinates and semantic values ​​of the scale values ​​of the x and y axes into the regression fitting respectively. Calculate the R2 coefficient of determination for the two models to determine the choice of the mapping function for the coordinate axes.

[0040] (7) After determining the mapping functions for each coordinate axis, the data points are analyzed. For scattered data, the center pixel coordinates are calculated using the mapping functions for the x-axis and y-axis to obtain their true physical coordinates. For bar data, the direction of the chart (horizontal or vertical) is determined based on the location of the value axis. For vertical bar charts, the upper and lower boundary coordinates of the bars are counted; for horizontal bar charts, the left and right boundary coordinates of the bars are counted, with the common boundary as the baseline. The overlap of the center coordinates of the bars along the value axis is then used to determine whether the structure is stacked or not. If the coordinates overlap, it is determined to be a stacked structure; otherwise, it is a non-stacked structure. The data points are then calculated using the mapping function of the value axis according to the calculation method in Table 1 to obtain the true values ​​of the bars.

[0041] Example 3: Embodiment 3 of this invention discloses a deep learning-based scientific research chart data extraction and reconstruction system, which utilizes a deep learning-based method for extracting and reconstructing scientific research chart data, comprising: Object detection model training module: used to construct a dataset of labeled elements of scientific charts and to train the object detection model based on the dataset to obtain an object detection model for extracting information from scientific charts; Key element extraction module: Used to identify key elements in the scientific charts to be extracted using a trained object detection model, and output pixel-level coordinates; the object detection model includes: chart element object detection model and data element object detection model; The mapping model determination module is used to match the tick value text with the corresponding tick line position using a tick value-line coordinate alignment algorithm, and to construct two coordinate mapping models: linear coordinate axis and logarithmic coordinate axis. It models the mapping between the true semantic values ​​and pixel coordinate values ​​of the tick lines on the x-axis and y-axis, respectively, based on R... 2 The coefficient of determination independently selects the optimal mapping model for each axis; The true value calculation module is used to directly substitute the coordinates of the center point of the target box of the scatter plot into the optimal mapping model to obtain the true value of each axis of the data point; for the bar chart, it calculates the true value based on the direction of the bar chart, the position of the baseline, and the structure type according to the optimal mapping model.

[0042] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0043] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting and reconstructing scientific chart data based on deep learning, characterized in that, include: Step 1: Construct a dataset of labeled elements for scientific research charts and train an object detection model based on the dataset to obtain an object detection model for extracting information from scientific research charts; Step 2: Use the trained target detection model to identify key elements in the scientific charts to be extracted and output pixel-level coordinates; wherein, the target detection model includes: a chart element target detection model and a data element target detection model; Step 3: The scale value text is matched with the corresponding scale line position by using the scale value-line coordinate alignment algorithm, and linear coordinate axis and logarithmic coordinate axis two coordinate mapping models are constructed to respectively map the real semantic value and pixel coordinate value of the scale line of the x-axis and y-axis. According to R 2 The coefficient of determination independently selects the optimal mapping model of each axis; Step 4: For scatter plots, directly substitute the coordinates of the center point of the target bounding box of the scatter plots into the optimal mapping model to obtain the true values ​​of each axis of the data points; for bar charts, calculate the true values ​​based on the direction of the bar chart, the position of the baseline, and the structure type according to the optimal mapping model.

2. The method for extracting and reconstructing scientific research chart data based on deep learning according to claim 1, characterized in that, In step 2, the chart element target detection model is used to identify and locate basic components including coordinate axes, tick marks, and legends.

3. The method for extracting and reconstructing scientific research chart data based on deep learning according to claim 1, characterized in that, In step 2, the data element target detection model is used to identify and extract the center coordinates of the scatter plot data points and the geometric features of the histogram bars.

4. The method for extracting and reconstructing scientific research chart data based on deep learning according to claim 1, characterized in that, In step 4, the direction of the histogram is determined based on the numerical axis, specifically as follows: If the numerical axis is the horizontal axis, then the bar chart is in the vertical direction; if the numerical axis is the vertical axis, then the bar chart is in the horizontal direction.

5. The method for extracting and reconstructing scientific research chart data based on deep learning according to claim 1, characterized in that, In step 4, the baseline is determined based on the statistical characteristics of the geometric distribution of the columns, specifically as follows: For vertical bar charts, count the coordinates of the top and bottom boundaries of all bars and find the common boundary as the baseline; for horizontal bar charts, count the coordinates of the left and right boundaries of all bars and find the common boundary as the baseline.

6. The method for extracting and reconstructing scientific research chart data based on deep learning according to claim 1, characterized in that, In step 4, the structural type of the histogram is determined based on the overlap of the column center coordinates along the numerical axis, specifically as follows: If the coordinates coincide along the numerical axis, it is a stacked type; if they are independent along the numerical axis and do not overlap, it is a non-stacked type.

7. The method for extracting and reconstructing scientific research chart data based on deep learning according to claim 1, characterized in that, In step 4, for the histogram, based on the direction of the histogram, the position of the baseline, and the structure type, the true value is calculated according to the optimal mapping model, specifically as follows: For a vertical, non-stacked column located above the baseline, the top value is calculated as the true value. For a vertical, non-stacked column located below the baseline, the bottom value is calculated as the true value. For a vertical, stacked column located above the baseline, calculate the top value minus the bottom value as the true value. For vertical, stacked columns located below the baseline, calculate the bottom value minus the top value as the true value; For a horizontal, non-stacked column located to the right of the baseline, the right-end value is calculated as the true value. For a horizontal, non-stacked column located to the left of the baseline, the left-end value is calculated as the true value. For a horizontally stacked column located to the right of the baseline, the right-end value minus the left-end value is calculated as the true value. For a horizontally stacked column located to the left of the baseline, the left-end value minus the right-end value is calculated as the true value.

8. A deep learning-based scientific research chart data extraction and reconstruction system utilizing the deep learning-based scientific research chart data extraction and reconstruction method described in any one of claims 1-7, characterized in that, include: Object detection model training module: used to construct a dataset of labeled elements of scientific charts and to train the object detection model based on the dataset to obtain an object detection model for extracting information from scientific charts; Key element extraction module: used to identify key elements in the scientific charts to be extracted using a trained target detection model, and output pixel-level coordinates; wherein, the target detection model includes: chart element target detection model and data element target detection model; The mapping model determining module is configured to match the scale value text with the corresponding scale line position by using a scale value-line coordinate alignment algorithm, and construct two coordinate mapping models of a linear coordinate axis and a logarithmic coordinate axis, to respectively map the real semantic value and the pixel coordinate value of the scale line of the x-axis and the y-axis, and determine the optimal mapping model of each axis according to R 2 The determination coefficient independently selects the optimal mapping model of each axis. The true value calculation module is used to directly substitute the coordinates of the center point of the target box of the scatter plot into the optimal mapping model to obtain the true value of each axis of the data point; for the bar chart, it calculates the true value based on the direction of the bar chart, the position of the baseline, and the structure type according to the optimal mapping model.

Images