Geology field literature scatter diagram data extraction method, storage medium and device

Abstract

The invention belongs to the technical field of document image processing and data mining, and particularly discloses a geoscience field literature scatter diagram data extraction method, a storage medium and equipment, and the method comprises the steps: classifying pictures extracted from geoscience field literature PDF into conventional scatter diagram pictures and table type scatter diagram pictures; positioning an independent scatter diagram region in the conventional scatter diagram picture by adopting a target detection model, segmenting the independent scatter diagram region to obtain a first type of independent scatter diagrams, extracting cells in the table type scatter diagram picture by adopting a cell segmentation method based on morphological operation and line segment intersection detection, and obtaining a second type of independent scatter diagrams; the cells are spliced with the coordinate axis area of the picture to obtain a second type of independent scatter diagrams; and inputting the first type of independent scatter diagrams and the second type of independent scatter diagrams into a scatter diagram data extraction model to obtain coordinate axis scale lines, then converting pixel coordinates of scatter points into actual data values, and outputting scatter diagram data. According to the method, the scatter diagram data can be automatically and accurately extracted from the geoscience literature.

Description

Technical Field

[0001] This application belongs to the field of document image processing and data mining technology, and more specifically, relates to a method, storage medium, and device for extracting scatter plot data from geoscience documents. Background Technology

[0002] Scatter plots are an important form of visualization in scientific literature used to show relationships between variables and present experimental or observational data. In the field of geosciences, a wealth of valuable data, such as elemental concentrations, isotope ratios, and fossil distributions, are scattered across tens of thousands of academic documents in the form of scatter plots. The construction of a unified geosciences database or multimodal knowledge graph urgently requires the transformation of this unstructured graphical data into structured numerical data to support data-driven scientific discovery. However, manually extracting scatter plot data from massive amounts of literature is an extremely tedious, time-consuming, and error-prone task, severely hindering research efficiency.

[0003] Currently, the technical solutions for automatically extracting scatter plot data face several key challenges. First, the automatic location and segmentation of scatter plots in literature is extremely difficult. Most existing studies directly process individual scatter plot images without addressing how to automatically identify and separate scatter plots from complex document layouts. Scatter plots in geoscientific literature often exist as combinations of multiple subplots or as complex table structures sharing a single coordinate axis, making traditional chart detection methods difficult to segment effectively and accurately. Second, the inherent characteristics of scatter plots in the geoscientific field present significant recognition challenges. These scatter plots typically feature complex backgrounds, dense point distribution, diverse point styles across different categories, and overlapping points. Traditional methods based on pixel traversal or graphic boundary detection are prone to false positives when faced with background interference, and are prone to missed detections when dealing with dense, overlapping points, exhibiting poor robustness. Furthermore, existing deep learning-based methods have limitations. While some studies have attempted to use object detection models to locate scatter points, early models had limited performance and typically treated different categories of scatter points as a single category for detection, failing to meet the recognition requirements of multi-category scatter plots in the geosciences. Simultaneously, conventional bounding box-based detection models are not accurate enough for locating small, densely distributed targets (i.e., scatter points) with very few features, and the annotation costs required for model training are high. Finally, the automatic regression accuracy of coordinate axis tick marks is insufficient. Accurate regression of tick marks is crucial for data transformation, but existing methods either rely on manual annotation or use object detection to locate tick value regions. However, in geosciences scatter plots, tick value regions lack distinct features, leading to poor detection results. In addition, optical character recognition (OCR) technology may produce errors when recognizing tick values ​​due to image quality issues, character adhesion, etc. Conventional linear regression methods are highly sensitive to such outliers, easily causing the final regressed tick marks to deviate significantly from the true values, directly affecting the accuracy of data extraction.

[0004] In summary, existing technologies lack a complete and high-precision data extraction scheme capable of automatically extracting scatter plots from geoscientific literature PDFs and effectively handling their complex features. Therefore, a new technical method is urgently needed to address these bottlenecks and achieve automatic and accurate acquisition of scatter plot data from geoscientific literatures. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this application is to provide a method, storage medium and device for extracting scatter plot data from geoscience literature, which can realize the automatic and accurate extraction of scatter plot data from geoscience literature.

[0006] To achieve the above objectives, firstly, this application provides a method for extracting scatter plot data from geoscience literature, comprising the following steps: S10, extract image and text information from PDF documents in the field of geosciences, and use an image classification model to classify the images into regular scatter plot images and tabular scatter plot images; S20. For regular scatter plot images, an object detection model is used to locate and segment the independent scatter plot regions to obtain the first type of independent scatter plot. For tabular scatter plot images, a cell segmentation method based on morphological operations and line segment intersection detection is used to extract the cells in the image and then the segmented cells are concatenated with the coordinate axis region of the image to obtain the second type of independent scatter plot. S30, input the first type of independent scatter plot and the second type of independent scatter plot into the pre-constructed scatter plot data extraction model, obtain the category and pixel coordinates of the scatter points through the scatter point detection module in the scatter plot data extraction model, and obtain the coordinate axis tick lines through the tick value regression module; S40, based on the coordinate axis scale lines, convert the pixel coordinates of the scatter points into actual data values ​​and output the scatter plot data.

[0007] The proposed method for extracting scatter plot data from geoscience literature has the following advantages: It automatically identifies scatter plot types using an image classification model, and independently extracts scatter plots for both regular and tabular types using object detection models and cell segmentation methods involving morphological operations and line segment intersection detection, ensuring accurate separation of scatter plot regions from complex document layouts. Subsequently, the scatter plot data extraction model utilizes a collaborative processing module for scatter point detection and a scale value regression module. The scatter point detection module classifies and locates scatter points based on center point regression, effectively addressing background interference, dense scatter points, and multiple categories in geoscience scatter plots. To address the challenges of character recognition, the scale value regression module combines image scaling with optimized optical character recognition. It also uses the RANSAC algorithm to regress the coordinate axis scale lines, overcoming the sensitivity to outliers inherent in traditional linear regression methods and improving the accuracy and robustness of scale line fitting. Finally, based on the coordinate axis scale lines, the module converts the scatter pixel coordinates into actual data values, achieving high-precision automated extraction from scatter images in geoscientific literature into structured data. This solves the technical bottlenecks of existing technologies, such as difficulties in automatic scatter plot localization and segmentation, insufficient multi-category recognition capabilities, and inaccurate coordinate axis regression, enabling automatic and accurate acquisition of scatter plot data from geoscientific literature.

[0008] As a further preferred embodiment, after step S20, the following is also included: For the first type and the second type of independent scatter plots, OCR technology is used to identify their subscript characters and positions, and a matching algorithm is used to associate the identified subscripts with the text annotations extracted from the document PDF, so that each independent scatter plot is assigned corresponding annotation information.

[0009] As a further preferred embodiment, in step S20, the cell segmentation method based on morphological operations and line segment intersection detection specifically refers to: The input tabular scatter plot image is binarized, dilated, and eroded to extract horizontal and vertical lines; the intersection points of the horizontal and vertical lines are identified; a cell region is generated based on the intersection points; invalid regions are filtered based on cell size; the image is cut according to cell coordinates, and the cut sub-images are stitched together with the leftmost Y-axis region of the image.

[0010] As a further preferred embodiment, in step S30, the scatter detection module adopts a target detection network based on center point regression to classify scatter points in the independent scatter plot and locate the center point coordinates. The scatter detection module includes a point suggestion prediction unit, a one-to-one matching unit, and a joint loss function unit. The point suggestion prediction unit generates the coordinate offset and confidence score of a preset reference point based on the feature map of the input image; the one-to-one matching unit uses the Hungarian algorithm to construct a cost matrix based on the pixel distance between the predicted point and the center point of the real scatter points and the confidence score, and assigns real scatter points to the predicted point; the joint loss function unit consists of cross-entropy loss for optimizing classification and Euclidean loss for optimizing localization.

[0011] As a further preferred embodiment, the Hungarian algorithm used in the one-to-one matching unit has a cost matrix... The expression is:

[0012] in, For the set of true scatter center points, For the prediction point set, For the first A real scatter center point, For the first One prediction point, The confidence score for this prediction point. As a weighted term, and These represent the number of actual points and the number of predicted points, respectively.

[0013] As a further preferred option, the joint loss function The expression is:

[0014] in, For classification loss, cross-entropy loss is used; To determine the loss, Euclidean loss is used; The weighting coefficients are used to balance the two losses.

[0015] As a further preferred embodiment, in step S30, the scale value regression module includes an image scaling unit, an OCR recognition unit, a coordinate axis classification unit, and a scale line regression unit. The system includes an image scaling unit to scale the input independent scatter plot according to a preset ratio; an OCR recognition unit to recognize the text and its position in the scaled image and map the recognized text position back to the original image coordinates; a coordinate axis classification unit to distinguish the X-axis and Y-axis tick values ​​based on the pixel position of the recognized text through horizontal and vertical projection analysis; and a tick line regression unit to use the RANSAC algorithm to regress the X-axis and Y-axis tick values ​​respectively and fit the coordinate axis tick lines of the scatter plot.

[0016] As a further preferred embodiment, the preset ratio of the image scaling unit is calculated using the following formula: Scaled height

[0017] Scaled width

[0018] in, h , w These represent the height and width of the original image, respectively.

[0019] Secondly, this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any one of the above descriptions.

[0020] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any one of the above descriptions.

[0021] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method for extracting scatter plot data from geoscience literature provided in this application; Figure 2 This is a flowchart of the method for extracting scatter plot data from geoscience literature provided in the embodiments of this application; Figure 3 This is an architectural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] like Figure 1 As shown, this application provides a method for extracting scatter plot data from geoscience literature, including steps S10 to S40, detailed below: Step S10: Extract image and text information from PDF documents in the geoscience field, and use an image classification model to classify the images into regular scatter plot images and tabular scatter plot images.

[0025] This step can automatically identify the type of scatter plot, providing a foundation for subsequent processing. Specifically, the type of scatter plot can be distinguished through an image classification model based on a convolutional neural network, thereby avoiding the inefficiency of manual classification.

[0026] Step S20: For regular scatter plot images, an object detection model is used to locate and segment the independent scatter plot regions to obtain the first type of independent scatter plot; for tabular scatter plot images, a cell segmentation method based on morphological operations and line segment intersection detection is used to extract the cells in the image, and the segmented cells are concatenated with the coordinate axis region of the image to obtain the second type of independent scatter plot.

[0027] This step can effectively separate independent scatter plots from complex document layouts. Specifically, it can directly locate conventional scatter plot regions through object detection models, and process tabular scatter plots through morphological operations and line segment intersection detection, thus solving the problem of insufficient accuracy of traditional methods when segmenting multi-subplot combinations or tabular structures.

[0028] Step S30: Input the first type of independent scatter plot and the second type of independent scatter plot into the pre-built scatter plot data extraction model, obtain the category and pixel coordinates of the scatter points through the scatter point detection module in the scatter plot data extraction model, and obtain the coordinate axis tick lines through the tick value regression module.

[0029] This step can accurately handle scatter point positioning and coordinate axis regression. Specifically, the scatter point detection module can classify scatter points and locate the center point coordinates, and the scale value regression module can combine image scaling, OCR recognition and random sampling consistency algorithm to fit the coordinate axis scale lines, thereby addressing the challenges of complex backgrounds and dense scatter points in geoscientific scatter plots.

[0030] Step S40: Based on the coordinate axis tick marks, convert the pixel coordinates of the scatter points into actual data values ​​and output the scatter plot data.

[0031] This step enables the conversion from images to structured data. Specifically, it maps the pixel coordinates of scattered points to actual values ​​using coordinate axis ticks and outputs the final result containing coordinate data and category labels, completing the entire extraction process of scatter plot data.

[0032] The scatter plot data extraction method for geoscience literature provided in this application has the following advantages: It automatically identifies the scatter plot type using an image classification model, and independently extracts scatter plots for both regular and tabular types using object detection models and cell segmentation methods based on morphological operations and line segment intersection detection, ensuring accurate separation of scatter plot regions from complex document layouts. Subsequently, the scatter plot data extraction model utilizes a collaborative processing module for scatter point detection and a scale value regression module. The scatter point detection module classifies and locates scatter points based on center point regression, effectively addressing background interference, dense scatter points, and multiple types of scatter plots in geoscience literature. To overcome recognition difficulties, the scale value regression module combines image scaling to optimize optical character recognition performance and uses the RANSAC algorithm to regress coordinate axis scale lines. This overcomes the sensitivity of traditional linear regression methods to outliers, improving the accuracy and robustness of scale line fitting. Finally, based on the coordinate axis scale lines, the scatter pixel coordinates are converted into actual data values, achieving high-precision automated extraction from scatter images in geoscientific literature to structured data. This solves the technical bottlenecks in existing technologies, such as difficulties in automatic location and segmentation of scatter plots, insufficient multi-class recognition capabilities, and inaccurate coordinate axis regression, enabling automatic and accurate acquisition of scatter plot data from geoscientific literature.

[0033] In one embodiment, the technical solution to achieve the above objective can be as follows: The flowchart of the scatter plot data extraction method for geoscience literature in this embodiment is shown below. Figure 2 As shown, the specific steps include: S1. Extract image and text information from PDF documents in the geosciences field, and classify the images into regular scatter plot images and tabular scatter plot images using a convolutional neural network-based image classification model. Specifically, ResNet-50 can be used as the backbone network to build the classification model, trained using a manually labeled dataset containing 100 regular scatter plot images and 200 tabular scatter plot images, divided into training and test sets in an 8:2 ratio. The final recognition accuracy reached 0.93.

[0034] S2. For regular scatter plot images, the YOLOv5 object detection model is used to locate the independent scatter plot regions. In the experimental embodiment, 100 regular scatter plot images were labeled, containing 343 independent scatter plot targets. The training set, test set, and validation set were divided in a 7:1:2 ratio for model training. The located target boxes were adjusted to completely encompass the scatter plot. The adjustment criteria were that all pixels on the four sides of the target box had the same value or were image edges. The adjusted target boxes were then segmented to obtain the first type of independent scatter plot.

[0035] S3. For tabular scatter plot images, a cell segmentation method based on morphological operations and line segment intersection detection is adopted. Specifically, this includes: binarizing the input tabular scatter plot image; then extracting horizontal and vertical lines using dilation and erosion operations with specific parameters, where the erosion parameters are set to (x1, y1) = (h / 20, 1) and (x2, y2) = (1, w / 20), where h and w are the image height and width, respectively; identifying the intersection points of the horizontal and vertical lines; generating cell regions based on the intersection points; filtering invalid regions by calculating cell sizes; segmenting the image according to cell coordinates; and concatenating the segmented sub-images with the leftmost Y-axis region of the image to obtain the second type of independent scatter plot.

[0036] S4. Construct a scatter plot data extraction model, which consists of a scatter detection module and a scale value regression module.

[0037] The scatter point detection module employs a center point regression-based object detection network, using the first 13 convolutional layers of VGG-16 to extract deep features. This module includes a point proposal prediction unit, a one-to-one matching unit, and a joint loss function unit. The point proposal prediction unit generates the coordinate offset and confidence score of a preset reference point based on the feature map of the input image. The coordinate calculation formula is as follows:

[0038] in As the reference point coordinates, This is the normalization term. The one-to-one matching unit uses the Hungarian algorithm to construct a cost matrix based on the pixel distance between the predicted point and the center point of the true scatter points, and the confidence score. The cost matrix formula is:

[0039] The joint loss function unit consists of cross-entropy loss and Euclidean loss, and the total loss function is:

[0040] The scale value regression module includes an image scaling unit, an OCR recognition unit, an axis classification unit, and a scale line regression unit. The image scaling unit scales the input independent scatter plot according to a preset ratio. The scaling formula is:

[0041]

[0042] The OCR recognition unit uses the PaddleOCR model to identify text and its location in the scaled image, and maps the identified text locations back to the original image coordinates. The coordinate axis classification unit, based on the pixel positions of the identified text, distinguishes between X-axis and Y-axis tick values ​​through horizontal and vertical projection analysis. The tick regression unit uses the RANSAC algorithm to regress the X-axis and Y-axis tick values ​​respectively, fitting the coordinate axis ticks of the scatter plot.

[0043] S5. Input the first and second type of independent scatter plots into the scatter plot data extraction model. Obtain the category and pixel coordinates of the scatter points through the scatter point detection module, and obtain the coordinate axis tick marks through the tick mark regression module. Based on the coordinate axis tick marks, convert the pixel coordinates of the scatter points into actual data values, and output the final scatter plot data.

[0044] In a preferred embodiment, the method further includes step S2a: For the segmented first-class and second-class independent scatter plots, OCR technology is used to identify their superscript characters and positions. Then, a Hungarian matching algorithm is used to associate the identified superscripts with text annotations extracted from the document PDF, assigning corresponding annotation information to each independent scatter plot. The matching cost formula is:

[0045] In experimental verification, the method of this embodiment was tested on 5000 machine-generated scatter plots and 637 real scatter plots in the geosciences field. In the scatter plot detection task, the MAE of this embodiment on real scatter plots in the geosciences field was 6.37, and the MSE was 11.31. It is 78.56. The F1 score was 86.31, outperforming the comparison methods. In the scale value recognition task, the method combining image scaling and RANSAC regression achieved an F1 score of 84.81 on real scatter plots in the geosciences domain and 90.47 on machine-generated scatter plots. The final data extraction accuracy reached 84.66% on real scatter plots in the geosciences domain and 91.67% on machine-generated scatter plots.

[0046] In one exemplary embodiment, the method further includes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for extracting scatter plot data from geoscience literature.

[0047] Please see Figure 3In one exemplary embodiment, the device further includes an electronic device comprising at least one processor, at least one memory, and at least one communication bus. The memory stores a computer program, which includes computer-readable instructions. The processor invokes the computer-readable instructions stored in the memory via the communication bus to execute the aforementioned method for extracting scatter plot data from geoscience literature.

[0048] This embodiment achieves the technical effect of automatically and accurately extracting scatter plot data from geoscientific literature through complete process design, innovative model structure and optimized algorithm combination, solving a long-standing technical problem in this field.

[0049] The beneficial effects of the technical solution provided in this embodiment are: This embodiment proposes a complete method for extracting scatter plot data from geoscience literature. Through an innovative image classification and segmentation process, it solves the challenge of automatically locating and separating independent scatter plots from complex document layouts. By introducing a scatter plot detection model based on center point regression, it effectively addresses the recognition challenges of complex backgrounds, dense scatter plots, and multiple coexisting categories in geoscience scatter plots, achieving accurate scatter plot location and classification. By combining image scaling to optimize OCR recognition and employing the RANSAC algorithm to regress the scale lines, the accuracy and robustness of coordinate axis regression are significantly improved, overcoming the sensitivity to outliers inherent in traditional methods. Ultimately, this embodiment achieves high-precision, automated extraction of structured scatter plot data from geoscience literature PDFs, providing crucial technical support for constructing geoscience databases and knowledge graphs, and greatly improving the efficiency and accuracy of scientific research data acquisition.

[0050] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for extracting scatter plot data from geoscience literature, characterized in that, Includes the following steps: S10, extract image and text information from PDF documents in the field of geosciences, and use an image classification model to classify the images into regular scatter plot images and tabular scatter plot images; S20. For regular scatter plot images, an object detection model is used to locate and segment the independent scatter plot regions to obtain the first type of independent scatter plot. For tabular scatter plot images, a cell segmentation method based on morphological operations and line segment intersection detection is used to extract the cells in the image and then the segmented cells are concatenated with the coordinate axis region of the image to obtain the second type of independent scatter plot. S30, input the first type of independent scatter plot and the second type of independent scatter plot into the pre-constructed scatter plot data extraction model, obtain the category and pixel coordinates of the scatter points through the scatter point detection module in the scatter plot data extraction model, and obtain the coordinate axis tick lines through the tick value regression module; S40, based on the coordinate axis scale lines, convert the pixel coordinates of the scatter points into actual data values ​​and output the scatter plot data.

2. The method for extracting scatter plot data from geoscience literature as described in claim 1, characterized in that, After step S20, the following is also included: For the first type and the second type of independent scatter plots, OCR technology is used to identify their subscript characters and positions, and a matching algorithm is used to associate the identified subscripts with the text annotations extracted from the document PDF, so that each independent scatter plot is assigned corresponding annotation information.

3. The method for extracting scatter plot data from geoscience literature as described in claim 1, characterized in that, In step S20, the cell segmentation method based on morphological operations and line segment intersection detection specifically includes: The input tabular scatter plot image is binarized, dilated, and eroded to extract horizontal and vertical lines; the intersection points of the horizontal and vertical lines are identified; a cell region is generated based on the intersection points; invalid regions are filtered based on cell size; the image is cut according to cell coordinates, and the cut sub-images are stitched together with the leftmost Y-axis region of the image.

4. The method for extracting scatter plot data from geoscience literature as described in claim 1, characterized in that, In step S30, the scatter detection module uses a target detection network based on center point regression to classify scatter points in the independent scatter plot and locate the center point coordinates. The scatter detection module includes a point suggestion prediction unit, a one-to-one matching unit, and a joint loss function unit. The point suggestion prediction unit generates the coordinate offset and confidence score of a preset reference point based on the feature map of the input image; the one-to-one matching unit uses the Hungarian algorithm to construct a cost matrix based on the pixel distance between the predicted point and the center point of the real scatter points and the confidence score, and assigns real scatter points to the predicted point; the joint loss function unit consists of cross-entropy loss for optimizing classification and Euclidean loss for optimizing localization.

5. The method for extracting scatter plot data from geoscience literature as described in claim 4, characterized in that, The Hungarian algorithm used in the one-to-one matching unit has a cost matrix The expression is: in, For the set of true scatter center points, For the prediction point set, For the first A real scatter center point, For the first One prediction point, The confidence score for this prediction point. As a weighted term, and These represent the number of actual points and the number of predicted points, respectively.

6. The method for extracting scatter plot data from geoscience literature as described in claim 4, characterized in that, The joint loss function The expression is: in, For classification loss, cross-entropy loss is used; To determine the loss, Euclidean loss is used; The weighting coefficients are used to balance the two losses.

7. The method for extracting scatter plot data from geoscience literature as described in claim 1, characterized in that, In step S30, the scale value regression module includes an image scaling unit, an OCR recognition unit, a coordinate axis classification unit, and a scale line regression unit; The system includes an image scaling unit to scale the input independent scatter plot according to a preset ratio; an OCR recognition unit to recognize the text and its position in the scaled image and map the recognized text position back to the original image coordinates; a coordinate axis classification unit to distinguish the X-axis and Y-axis tick values ​​based on the pixel position of the recognized text through horizontal and vertical projection analysis; and a tick line regression unit to use the RANSAC algorithm to regress the X-axis and Y-axis tick values ​​respectively and fit the coordinate axis tick lines of the scatter plot.

8. The method for extracting scatter plot data from geoscience literature as described in claim 7, characterized in that, The preset ratio of the image scaling unit is calculated using the following formula: Scaled height Scaled width in, h , w These represent the height and width of the original image, respectively.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

Images