A deep learning-based material curve image data automatic extraction system

Abstract

The application provides a kind of material curve image data automatic extraction system based on deep learning, establishes the mapping relationship of pixel space to data space by recognizing coordinate system in image through automatic coordinate axis detection and scale identification module, and uses multi-curve instance segmentation module to identify each data curve in the graph as an independent object instance to generate independent pixel-level mask of each curve, then uses sub-pixel level curve vectorization module to convert pixel mask into mathematical curve representation and automatically extracts curve data based on the mapping relationship of pixel space to data space, finally provides an interactive verification interface for users to quickly review and high-level correction of the fully automatic extraction result. The application decomposes chart data extraction task into a series of intelligent modules that cooperate with each other, realizes end-to-end automation through synergistic effect, fundamentally solves the problem of curve overlap and intersection, and improves the efficiency and convenience of user correction of extracted data.

Description

Technical Field

[0001] This invention relates to the field of data extraction technology, and in particular to an automatic extraction system for material curve image data based on deep learning. Background Technology

[0002] In materials science research and engineering practice, a large amount of technical data is presented in the form of two-dimensional graphs. However, in many cases, especially when consulting literature or third-party reports, researchers can only obtain the images of these graphs, but not their original numerical data tables. In order to reanalyze, verify, or model this data, it is necessary to first digitize the graph images, that is, extract the precise (x, y) coordinate data points from the images. This process is called graph reverse engineering or data extraction.

[0003] Currently, there are several software tools available for this purpose, with WebPlotDigitizer being a representative example. These tools typically follow a "computer-aided" workflow, which, while digitized to some extent, still heavily relies on manual operation and judgment by the user, and thus has many technical limitations.

[0004] A typical existing technology workflow includes the following steps:

[0005] (1) The user manually loads the chart image file;

[0006] (2) Manually calibrate the coordinate system by manually clicking two known coordinate points (a total of four points, such as X1, X2, Y1, Y2) on the X-axis and Y-axis respectively;

[0007] (3) Use color-based semi-automatic algorithms (such as color picking and threshold filtering) to initially extract data points;

[0008] (4) Perform extensive manual corrections on the automatically extracted results, including adding, deleting, or fine-tuning the positions point by point;

[0009] (5) Export the (x, y) data points to Excel, etc.

[0010] Other similar tools, such as Engauge Digitizer and PlotDigitizer, also commonly adopt this working mode centered on manual calibration and color / heuristic rule-based extraction, reflecting the current common practice in the field.

[0011] The following is the process of extracting data from a curve using WebPlotDigitizer:

[0012] 1. Open the "Curve Data Acquisition" function page;

[0013] 2. Open the curve image (select a graph with uniform x and y axis scales and the origin coinciding), preview the image on the left, display the tools on the right, and select the plot type: 2D (XY) Plot;

[0014] 3. Perform coordinate calibration;

[0015] A. Click the Align Axes button to confirm, then click the Proceed button to confirm. In the tools, click the four known points on the coordinate axes: two points on the X-axis (X1, X2) and two points on the Y-axis (Y1, Y2). Click the Complete button to confirm. Figure 1 ;

[0016] B. Enter the X values ​​of the two points clicked on the X-axis and the Y values ​​of the two points clicked on the Y-axis, then click the OK button to confirm. Figure 2 ;

[0017] 4. Data sampling:

[0018] a. Use the Pen tool to automatically extract the entire image along its curves, such as... Figure 3 ;

[0019] b. Select the curve color to identify. Click the small color swatch to the right of "Color" to open a window. Click "Color Picker" to sample the curve color from the image. Select the same color as the curve and click "Done". Figure 4 ;

[0020] c. Set the precision algorithm, select X Step, enter the Units value for ΔX Step (enter according to the precision of the actual data, for example: 0.01), enter the Line width value (enter according to the actual data, for example: 2), and click Run. Figure 5 ;

[0021] (Note: The Units and Line width values ​​of ΔX Step mainly set the number of plotting points: you can check the Data Points in the lower left corner and adjust the settings to keep them close to the number of points in the original data. For example, if the original data has 100 points, keep the plotting points between 120 and 200.)

[0022] d. If the plotted point appears outside the line or is inaccurately positioned, select Delete Point (D) in Manual Extraction, move the mouse to the location of the extra or inaccurate point, and click to delete it. After deletion, click View Data. Figure 6 ;

[0023] e. Click "View Data" to get the plotting results, and click "Download.CSV" to export the plotting results, such as... Figure 7 .

[0024] In summary, the shortcomings of existing software tools for extracting material curve image data and the reasons for these shortcomings are as follows:

[0025] 1. Relies on manual operation and judgment by users: Manual operation involves many steps, including curve coordinate axis calibration, curve area selection, curve color selection, and curve data point quantity setting, which has low automation, is time-consuming, and prone to errors.

[0026] 2. Color-based heuristic algorithms are fragile: Existing tools rely on the color difference between the data curve (foreground) and the chart background for extraction logic. While this method works under ideal conditions, it is very fragile in real-world applications. For example, color segmentation algorithms struggle to accurately separate data curves from black-and-white scanned document images, charts with complex backgrounds or grid lines, smoothed lines after anti-aliasing, or low-contrast images. Users need to repeatedly adjust parameters such as color distance and threshold, but the results are often still unsatisfactory, with extracted data points containing significant noise or missing important parts.

[0027] 3. Inability to automatically handle multiple overlapping or intersecting curves: When a chart contains multiple curves of the same or similar color (such as in a black and white document), and these curves overlap or intersect, color-based extraction algorithms fail. They cannot distinguish which pixels belong to which curve, causing all pixels of the same color to be incorrectly grouped into the same dataset. In such cases, users need to manually carefully draw the curve area or manually trace each point, repeating the entire extraction process for each curve, resulting in low automation.

[0028] 4. Inefficient Data Correction Mechanisms: After automatic extraction, users typically need to spend a significant amount of time on correction. Existing tools offer point-by-point correction methods, requiring manual addition of missing points, deletion of noise, or "micro-moving" of individual points using keyboard arrow keys. For smooth curves containing hundreds or thousands of data points, this correction method is extremely inefficient. For example, correcting an error point in the middle of the curve may require a chain reaction of adjustments to the positions of all subsequent points, making the operation very tedious. Summary of the Invention

[0029] In view of the above problems, the present invention is proposed to provide an automatic material curve image data extraction system based on deep learning that overcomes or at least partially solves the above problems.

[0030] This invention provides an automatic material curve image data extraction system based on deep learning, comprising:

[0031] An automated coordinate axis detection and scale recognition module is used to identify the coordinate system in a material curve image and establish a mapping relationship from pixel space to data space.

[0032] The multi-curve instance segmentation module is used to identify all data curves in the material curve image and segment each data curve as an independent object instance. For each identified and segmented data curve instance, a pixel-level mask is generated.

[0033] The subpixel-level curve vectorization module is used to convert the pixel-level mask of each data curve into a mathematical curve to obtain the vectorized data curve. Based on the mapping relationship from pixel space to data space, it converts each vectorized data curve from pixel space to data space to generate the initial data of each vectorized data curve.

[0034] The interactive calibration and verification interface is used to display each vectorized data curve and its corresponding initial data to the user, and supports the user to interact with each vectorized data curve as a whole, as well as automatically update the corresponding initial data based on the overall interaction.

[0035] Optionally, the automated coordinate axis detection and scale recognition module is also used to locate the coordinate axes in the material curve image, identify the scale labels on the coordinate axes, record the pixel coordinates of each scale label and the scale value label in the material curve image, use a robust regression algorithm to iteratively fit the pixel coordinates of the scale labels and the scale value labels in the material curve image, remove the deviated scale labels and the scale value labels in the material curve image, and calculate the coordinate transformation function based on the retained scale labels and the scale value labels in the material curve image.

[0036] Optionally, the automated coordinate axis detection and scale recognition module is also used for:

[0037] The material curve image is converted into a grayscale image, and the converted grayscale image is subjected to adaptive threshold binarization to obtain the preprocessed material curve image.

[0038] The Hough transform algorithm is used to filter the coordinate axes from the preprocessed material curve image;

[0039] A region of interest is defined along the coordinate axis. An optical character recognition engine is used to detect and recognize the scale labels on the coordinate axis within the region of interest. The pixel coordinates of each scale label and the scale label in the material curve image are recorded.

[0040] Optionally, the multi-curve instance segmentation module is also used to identify all data curves in the material curve image using a trained deep learning instance segmentation model, and to segment each data curve as an independent object instance based on the curve's shape, continuity, and context information, and to generate a pixel-level mask for each identified and segmented data curve instance.

[0041] Optionally, the deep learning instance segmentation model is trained through the following steps:

[0042] Choose a model architecture;

[0043] A parameter space containing various attributes of the chart is created. When generating each chart image, a set of parameters is randomly sampled from the parameter space and the underlying real data of the curve is generated through a random mathematical function. Based on the underlying real data and the sampled parameters, the chart image is rendered on the canvas and a pixel-level mask corresponding to each curve is generated synchronously. Data augmentation processing simulating real image degradation is applied to the chart image to obtain a synthetic dataset for training the model architecture.

[0044] The model architecture is trained using the synthetic dataset to obtain the deep learning instance segmentation model.

[0045] Optionally, the model architecture can be selected based on the different speed and accuracy requirements of the application scenario.

[0046] Optionally, the subpixel-level curve vectorization module is also used to convert the pixel-level mask of each data curve into an ordered sequence of pixel coordinate points, fit the ordered sequence of pixel coordinate points with a parametric curve and perform subpixel-level processing to obtain the vectorized data curve.

[0047] Optionally, the subpixel-level curve vectorization module is also used for:

[0048] Using morphological thinning or skeletonization algorithms, the pixel-level mask of each data curve is converted into a center line with a single pixel width that preserves the original topology, and an ordered sequence of pixel coordinate points is extracted along the center line.

[0049] The ordered sequence of pixel coordinates is fitted with a parametric curve, and the control points of the parametric curve are determined with minimizing the energy function as the optimization objective during the fitting process, so as to obtain a sub-pixel-level vectorized data curve; the energy function includes the distance term between the fitted curve and all pixels in the pixel-level mask, and the curve smoothness term.

[0050] Optionally, the interactive calibration and verification interface is also used for:

[0051] The vectorized data curve is overlaid on the original material curve image as a background layer, and a real-time data table linked with the vectorized data curve is displayed simultaneously.

[0052] Users can adjust the control points of the vectorized data curve through interactive operations to adjust the position of the curve segment as a whole, while maintaining the continuity and smoothness of the curve.

[0053] When a user adjusts the vectorized data curve, the real-time data table updates the corresponding data value in real time; when a user modifies the data value in the real-time data table, the corresponding point on the vectorized data curve moves in real time.

[0054] Optionally, the interactive calibration and verification interface is also used to visualize the confidence information of the deep learning instance segmentation model, highlighting the curve areas with low model confidence using different colors to guide users to check first.

[0055] This invention has the following advantages:

[0056] This invention presents a deep learning-based automatic material curve image data extraction system. It uses an automated coordinate axis detection and scaling module to identify the coordinate system in the image and establish a mapping relationship from pixel space to data space. A deep learning-based multi-curve instance segmentation module identifies each data curve in the image as an independent object instance, generating an independent pixel-level mask for each curve, thus fundamentally solving the problem of curve overlap and intersection. A sub-pixel-level curve vectorization module converts the segmented pixel masks into a high-precision mathematical curve representation and automatically extracts the curve data based on the pixel-to-data space mapping relationship. Finally, an efficient interactive verification interface is provided, allowing users to quickly review and make high-level corrections to the fully automated extraction results. This invention decomposes the chart data extraction task into a series of collaborative intelligent modules, achieving end-to-end automation through synergy, fundamentally solving the problem of curve overlap and intersection, and improving the efficiency and convenience of users correcting the extracted data. Attached Figure Description

[0057] Figure 1 This is a diagram illustrating the selection of coordinate points for extracting curve data using WebPlotDigitizer.

[0058] Figure 2 This is a schematic diagram illustrating the input of coordinate point values ​​for extracting curve data using WebPlotDigitizer;

[0059] Figure 3 This is a schematic diagram showing the complete coverage selection of curves extracted using WebPlotDigitizer;

[0060] Figure 4 This is a schematic diagram illustrating curve color recognition using WebPlotDigitizer to extract curve data.

[0061] Figure 5 This is a schematic diagram illustrating the precision values ​​of curve data extracted using WebPlotDigitizer.

[0062] Figure 6 This is a schematic diagram illustrating the plotting point correction for curve data extracted using WebPlotDigitizer.

[0063] Figure 7 This is a schematic diagram of the exported plotting results of curve data extracted using WebPlotDigitizer;

[0064] Figure 8 This is a structural block diagram of an automatic material curve image data extraction system based on deep learning provided in an embodiment of the present invention;

[0065] Figure 9 This is a flowchart of the automated coordinate axis detection and scale recognition module provided in an embodiment of the present invention;

[0066] Figure 10 This is a schematic diagram of the output result of the multi-curve instance segmentation module provided in an embodiment of the present invention;

[0067] Figure 11 This is an enlarged view of the rasterized curve (pixelated step-like) generated by the comparison instance mask provided in the embodiments of the present invention and the smooth and continuous B-spline curve output after processing by the sub-pixel level vectorization module.

[0068] Figure 12 This is a user interface (UI) design model diagram of the interactive calibration and verification interface provided in the embodiments of the present invention. Detailed Implementation

[0069] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] Reference Figure 8 The diagram illustrates a structural block diagram of an automatic material curve image data extraction system based on deep learning, provided in an embodiment of the present invention. Specifically, it may include the following modules:

[0071] An automated coordinate axis detection and scale recognition module is used to identify the coordinate system in a material curve image and establish a mapping relationship from pixel space to data space.

[0072] The multi-curve instance segmentation module is used to identify all data curves in the material curve image and segment each data curve as an independent object instance. For each identified and segmented data curve instance, a pixel-level mask is generated.

[0073] The subpixel-level curve vectorization module is used to convert the pixel-level mask of each data curve into a mathematical curve to obtain the vectorized data curve. Based on the mapping relationship from pixel space to data space, it converts each vectorized data curve from pixel space to data space to generate the initial data of each vectorized data curve.

[0074] The interactive calibration and verification interface is used to display each vectorized data curve and its corresponding initial data to the user, and supports the user to interact with each vectorized data curve as a whole, as well as automatically update the corresponding initial data based on the overall interaction.

[0075] like Figure 8 As shown, the deep learning-based automatic material curve image data extraction system of the present invention is an automated processing pipeline composed of multiple intelligent modules. The entire workflow begins with the user inputting a chart image, such as a PNG, JPEG, or TIFF file obtained from a scanned document.

[0076] The input image is first fed into two parallel preprocessing modules: an automated coordinate axis detection and scale recognition module (Module 1) and a multi-curve instance segmentation module (Module 2). Module 1 is responsible for identifying the coordinate system of the chart, including the position, direction, type (linear or logarithmic), and scale values ​​of the X and Y axes, and finally calculating an accurate coordinate transformation function to map the pixel coordinates of the image to the actual data coordinates of the chart. Meanwhile, Module 2 is responsible for identifying all target data curves in the image and generating an independent, pixel-level segmentation mask for each curve.

[0077] The outputs of Modules 1 and 2 are then fed into the subpixel-level curve vectorization module (Module 3). This module receives the pixel mask of each curve from Module 2, converts it into a smooth mathematical curve representation, such as a B-spline curve, and uses the coordinate transformation function provided by Module 1 to transform these curves from pixel space to data space, thereby generating the initial numerical data.

[0078] Finally, the vectorized curves and corresponding numerical data are fed into the interactive calibration and verification interface (Module 4). Users can visually check the accuracy of the automatically extracted results on this interface and perform efficient, high-level corrections. After user confirmation, the final accurate numerical data can be exported as a standard format file, such as CSV, for subsequent scientific calculations and analysis.

[0079] In an optional embodiment of the present invention, the automated coordinate axis detection and scale recognition module is further configured to locate the coordinate axes in the material curve image, identify the scale value labels on the coordinate axes, record the pixel coordinates of each scale value label and the scale value label in the material curve image, use a robust regression algorithm to iteratively fit the pixel coordinates of the scale value labels and the scale value labels in the material curve image, remove the deviated pixel coordinates of the scale value labels and the scale value labels in the material curve image, and calculate the coordinate transformation function based on the retained pixel coordinates of the scale value labels and the scale value labels in the material curve image.

[0080] In an optional embodiment of the present invention, the automated coordinate axis detection and scale recognition module is further configured to:

[0081] The material curve image is converted into a grayscale image, and the converted grayscale image is subjected to adaptive threshold binarization to obtain the preprocessed material curve image.

[0082] The Hough transform algorithm is used to filter the coordinate axes from the preprocessed material curve image;

[0083] A region of interest is defined along the coordinate axis. An optical character recognition engine is used to detect and recognize the scale labels on the coordinate axis within the region of interest. The pixel coordinates of each scale label and the scale label in the material curve image are recorded.

[0084] The automated coordinate axis detection and scale recognition module aims to fully automate the tedious and error-prone coordinate axis calibration steps in existing technologies. Its core idea is to combine multiple computer vision techniques and integrate the results of these techniques through robust statistical methods, thereby achieving reliable calibration across images of varying quality. Detailed workflow is as follows: Figure 9 As shown:

[0085] 1. Image Preprocessing: The input image is first converted to grayscale and then subjected to adaptive threshold binarization. This step effectively enhances high-frequency information such as lines and text in the image while suppressing background noise, laying the foundation for subsequent line and text detection.

[0086] 2. Axis Detection: The Hough Transform algorithm is applied to the preprocessed image. The Hough Transform is a powerful feature extraction technique, adept at detecting geometric shapes such as straight lines in images, even if the lines are broken or disturbed by noise. By voting in the parameter space, the algorithm can identify all salient straight lines in the image. From the detected set of lines, the system selects the two longest and nearly perpendicular lines, identifying them as candidate lines for the X and Y axes, respectively. This method is robust to slight image rotations.

[0087] 3. Tick Label Recognition: After determining the positions of the X and Y axes, the system defines a series of Regions of Interest (ROIs) along these axes. Then, within these ROIs, an Optical Character Recognition (OCR) engine, such as a finely tuned PaddleOCR model, is used. The OCR engine's task is to detect and recognize the tick labels on the coordinate axes. For each successfully recognized number, the system records its value and its pixel coordinates in the image (usually the center point coordinates of a text box).

[0088] 4. Robust Scale Calibration: This is a key innovative step in this module. Considering the potential for errors in the OCR recognition process (e.g., misidentifying the digits "3" and "5" as "8," or failing to recognize certain labels), the system does not directly use all OCR results. Instead, it treats all recognized (pixel coordinates, numerical values) pairs as a dataset and uses a robust regression algorithm, such as RANSAC (Random Sample Consensus), to fit these data points. The RANSAC algorithm estimates model parameters (e.g., linear or logarithmic mappings) by iteratively sampling small subsets from the dataset. It effectively identifies "inliers" (correctly recognized tick marks) that conform to the mainstream trend and automatically removes "outliers" (labels incorrectly recognized by the OCR). Finally, the system recalculates a high-precision coordinate transformation function using all inliers. The coordinate transformation function defines an accurate mapping from arbitrary pixel coordinates in the image to the actual data coordinates in the chart. This method significantly improves the automation and reliability of axis calibration.

[0089] In an optional embodiment of the present invention, the multi-curve instance segmentation module is further configured to identify all data curves in the material curve image using a trained deep learning instance segmentation model, and to segment each data curve as an independent object instance based on the shape, continuity and context information of the curve, and to generate a pixel-level mask for each identified and segmented data curve instance.

[0090] In an optional embodiment of the present invention, the deep learning instance segmentation model is trained through the following steps:

[0091] Choose a model architecture;

[0092] A parameter space containing various attributes of the chart is created. When generating each chart image, a set of parameters is randomly sampled from the parameter space and the underlying real data of the curve is generated through a random mathematical function. Based on the underlying real data and the sampled parameters, the chart image is rendered on the canvas and a pixel-level mask corresponding to each curve is generated synchronously. Data augmentation processing simulating real image degradation is applied to the chart image to obtain a synthetic dataset for training the model architecture.

[0093] The model architecture is trained using the synthetic dataset to obtain the deep learning instance segmentation model.

[0094] In one optional embodiment of the present invention, the model architecture is selected according to the different requirements of speed and accuracy in the application scenario.

[0095] The multi-curve instance segmentation module of this invention solves the problem of existing technologies being unable to handle multiple intersecting curves with the same or similar colors. The core idea is to define the curve separation problem as an instance segmentation task, rather than a simple color matching task. The goal of instance segmentation is to detect all objects in an image and generate a pixel-level mask for each independent instance of each object.

[0096] 1. Model Architecture: The system employs a deep convolutional neural network designed for instance segmentation tasks. Depending on the application scenario's requirements for speed and accuracy, various model architectures can be used as implementation examples. For instance, to pursue ultimate real-time performance, a lightweight single-stage model, such as YOLACT or YOLOv8-seg, can be chosen. If the application scenario has extremely high accuracy requirements, a more powerful two-stage model, such as Mask R-CNN, can be selected. Regardless of the architecture used, the model is specifically trained to identify a single category: "data curves." The model's output is a set of independent binary masks, where each mask precisely outlines the contour of a complete and unique curve in the graph, such as... Figure 10As shown. The method of this invention solves the curve intersection problem because the model learns the shape, continuity, and contextual information of the curve, not just its color.

[0097] 2. Generation of Synthetic Datasets for Training: Training a powerful deep learning model requires a large amount of training data with precise annotations. However, collecting and manually annotating thousands of diverse scientific charts in the real world is very costly. To address this data bottleneck, this invention includes a key step in procedurally generating synthetic datasets. The process is as follows:

[0098] 1. Define the parameter space: Create a parameter space that contains various chart attributes, such as image size, axis type (linear / logarithmic), number of curves, line style (solid, dashed, dotted), color, thickness, noise level, background complexity, etc.

[0099] 2. Random Sampling and Data Generation: For each composite chart to be generated, a set of parameters is randomly selected from the parameter space described above. Then, the underlying real data of the curve is generated by defining random mathematical functions (such as polynomials, sine functions, exponential decay functions, etc.).

[0100] 3. Image Rendering and Ground Truth Mask Generation: On a blank canvas, based on the generated real data and sampling parameters, a complete chart image is rendered, including axes, grid lines, tick labels, and data curves. Simultaneously, the system generates pixel-perfect instance masks corresponding to each curve. Since the curves are generated by known mathematical functions, absolutely accurate annotation information (ground truth) can be obtained at zero cost.

[0101] 4. Data Augmentation: To improve the model's generalization ability and robustness on real-world images, a series of data augmentation techniques are applied to the generated "perfect" chart images. These techniques simulate various image degradations that may occur in real scanned documents, such as adding Gaussian noise, motion blur, adjusting contrast and brightness, and simulating scanning artifacts (such as stripes and spots).

[0102] This automated process can cost-effectively generate hundreds of thousands or even millions of diverse training samples, enabling the training of an instance segmentation model that is highly robust to various chart styles, image quality, and complex situations.

[0103] The table below provides a specific configuration and training parameters for an instance segmentation model in a preferred example.

[0104] Table 1. Examples of Optimal Instance Segmentation Model Architecture and Training Parameters

[0105]

[0106] In an optional embodiment of the present invention, the subpixel-level curve vectorization module is further configured to convert the pixel-level mask of each data curve into an ordered sequence of pixel coordinate points, fit the ordered sequence of pixel coordinate points with a parametric curve and perform subpixel-level processing to obtain the vectorized data curve.

[0107] In an optional embodiment of the present invention, the sub-pixel level curve vectorization module is further configured to:

[0108] Using morphological thinning or skeletonization algorithms, the pixel-level mask of each data curve is converted into a center line with a single pixel width that preserves the original topology, and an ordered sequence of pixel coordinate points is extracted along the center line.

[0109] The ordered sequence of pixel coordinates is fitted with a parametric curve, and the control points of the parametric curve are determined with minimizing the energy function as the optimization objective during the fitting process, so as to obtain a sub-pixel-level vectorized data curve; the energy function includes the distance term between the fitted curve and all pixels in the pixel-level mask, and the curve smoothness term.

[0110] The subpixel-level curve vectorization module aims to address the precision limitations imposed by the discrete pixel grid of an image. By converting the rasterized mask output by the instance segmentation module into a continuous vector representation, the system can extract more precise data than the image itself, significantly improving the accuracy of the extracted data.

[0111] 1. Mask to Point Sequence Conversion: For each binary mask received from the instance segmentation module, a morphological thinning or skeletonization algorithm is first applied. These algorithms can erode a curve mask with a certain width into a center line with only a single pixel width, while preserving its original topology. Then, along this single-pixel-wide line, an ordered sequence of pixel coordinates is extracted.

[0112] 2. Spline Fitting and Sub-pixel Processing: Next, a parametric curve, such as a B-spline, is used to fit the ordered sequence of points obtained in the previous step. The key innovation here is that the control points of the spline curve are determined through an optimization process. This optimization process aims to minimize an energy function that takes into account two factors: (a) the distance between the fitted curve and all pixels in the original mask; and (b) the smoothness of the curve itself (e.g., by minimizing the integral of its curvature). Through this optimization process, the control points of the finally fitted B-spline curve can be located at arbitrary floating-point positions between pixels. This means that the generated vector curve can smoothly pass through the centers of the pixel grid, thus representing the true path of the original data curve with sub-pixel precision. Figure 11 As shown, this vectorization eliminates the staircase effect caused by pixelation, and its accuracy is no longer limited by the resolution of the input image, but is determined by the fitting algorithm and the signal-to-noise ratio of the original curve.

[0113] In an optional embodiment of the present invention, the interactive calibration and verification interface is further used for:

[0114] The vectorized data curve is overlaid on the original material curve image as a background layer, and a real-time data table linked with the vectorized data curve is displayed simultaneously.

[0115] Users can adjust the control points of the vectorized data curve through interactive operations to adjust the position of the curve segment as a whole, while maintaining the continuity and smoothness of the curve.

[0116] When a user adjusts the vectorized data curve, the real-time data table updates the corresponding data value in real time; when a user modifies the data value in the real-time data table, the corresponding point on the vectorized data curve moves in real time.

[0117] In an optional embodiment of the present invention, the interactive calibration and verification interface is also used to visualize the confidence information of the deep learning instance segmentation model, highlighting the curve areas with low model confidence using different colors to guide the user to check first.

[0118] The interactive verification and validation interface embodies a paradigm shift from users acting as "operators" to users acting as "verifiers." It leverages modern user experience (UX) design principles to make the manual review process fast, intuitive, and efficient.

[0119] 1. Interface design: such as Figure 12As shown, the main view of this interface uses the original graphic image as a background layer. Above this background, the data curves, automatically extracted and vectorized by the system, are clearly overlaid. A real-time data table linked to the chart is displayed on one side of the interface.

[0120] 2. Interactive Curve Correction: Unlike the tedious point-by-point editing in existing technologies, this invention provides a high-level, curve-centric interactive method. Users can click and drag with the mouse to adjust the control points of the underlying B-spline curve, allowing curve segments to move smoothly to new positions while maintaining overall continuity and smoothness. This method allows users to quickly correct deviations in the entire curve, rather than fine-tuning hundreds or thousands of data points one by one.

[0121] 3. Confidence Visualization: To proactively guide user attention, the system visualizes the confidence information of the instance segmentation model. For example, for curve portions where the model is less certain (such as near curve intersections or areas with severe image noise), different colors (such as yellow or red) are highlighted on the interface. This allows users to prioritize checking the areas most likely to contain errors, thus greatly improving verification efficiency.

[0122] 4. Real-time Data Updates: The curves and data tables in the interface are linked. When a user drags and adjusts a curve on the chart, the adjacent data table updates in real time to reflect the new coordinate values. Conversely, if a user directly modifies a value in the table, the corresponding point on the chart immediately moves to the new position. This instant feedback provides users with a more intuitive editing experience.

[0123] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0124] The above provides a detailed description of the automatic material curve image data extraction system based on deep learning provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application. The above embodiments are merely preferred embodiments given to fully illustrate this invention, and the protection scope of this invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this invention are all within the protection scope of this invention.

Claims

1. A deep learning-based automatic material curve image data extraction system, characterized in that, include: An automated coordinate axis detection and scale recognition module is used to identify the coordinate system in a material curve image and establish a mapping relationship from pixel space to data space. The multi-curve instance segmentation module is used to identify all data curves in the material curve image and segment each data curve as an independent object instance. For each identified and segmented data curve instance, a pixel-level mask is generated. The subpixel-level curve vectorization module is used to convert the pixel-level mask of each data curve into a mathematical curve to obtain the vectorized data curve. Based on the mapping relationship from pixel space to data space, it converts each vectorized data curve from pixel space to data space to generate the initial data of each vectorized data curve. An interactive calibration and verification interface is used to display each vectorized data curve and its corresponding initial data to the user, and supports the user to interact with each vectorized data curve as a whole, as well as automatically update the corresponding initial data based on the overall interaction. The automated coordinate axis detection and scale recognition module is also used to locate the coordinate axes in the material curve image and identify the scale labels on the coordinate axes. It records the pixel coordinates of each scale label and the scale value label in the material curve image, uses a robust regression algorithm to iteratively fit the pixel coordinates of the scale labels and the scale value labels in the material curve image, removes the deviated scale labels and the scale value labels in the material curve image, and calculates the coordinate transformation function based on the retained scale labels and the scale value labels in the material curve image.

2. The automatic material curve image data extraction system based on deep learning according to claim 1, characterized in that, The automated coordinate axis detection and scale recognition module is also used for: The material curve image is converted into a grayscale image, and the converted grayscale image is subjected to adaptive threshold binarization to obtain the preprocessed material curve image. The Hough transform algorithm is used to filter the coordinate axes from the preprocessed material curve image; A region of interest is defined along the coordinate axis. An optical character recognition engine is used to detect and recognize the scale labels on the coordinate axis within the region of interest. The pixel coordinates of each scale label and the scale label in the material curve image are recorded.

3. The automatic material curve image data extraction system based on deep learning according to claim 1, characterized in that, The multi-curve instance segmentation module is also used to identify all data curves in the material curve image using a trained deep learning instance segmentation model, and to segment each data curve as an independent object instance based on the curve's shape, continuity, and context information, generating a pixel-level mask for each identified and segmented data curve instance.

4. The automatic material curve image data extraction system based on deep learning according to claim 3, characterized in that, The deep learning instance segmentation model is trained through the following steps: Choose a model architecture; A parameter space containing various attributes of the chart is created. When generating each chart image, a set of parameters is randomly sampled from the parameter space and the underlying real data of the curve is generated through a random mathematical function. Based on the underlying real data and the sampled parameters, the chart image is rendered on the canvas and a pixel-level mask corresponding to each curve is generated synchronously. Data augmentation processing simulating real image degradation is applied to the chart image to obtain a synthetic dataset for training the model architecture. The model architecture is trained using the synthetic dataset to obtain the deep learning instance segmentation model.

5. The automatic material curve image data extraction system based on deep learning according to claim 4, characterized in that, The model architecture is selected based on the different speed and accuracy requirements of the application scenario.

6. The automatic material curve image data extraction system based on deep learning according to claim 1, characterized in that, The subpixel-level curve vectorization module is also used to convert the pixel-level mask of each data curve into an ordered sequence of pixel coordinate points, fit the ordered sequence of pixel coordinate points with a parametric curve and perform subpixel-level processing to obtain the vectorized data curve.

7. The automatic material curve image data extraction system based on deep learning according to claim 6, characterized in that, The subpixel-level curve vectorization module is also used for: Using morphological thinning or skeletonization algorithms, the pixel-level mask of each data curve is converted into a center line with a single pixel width that preserves the original topology, and an ordered sequence of pixel coordinate points is extracted along the center line. The ordered sequence of pixel coordinates is fitted with a parametric curve, and the control points of the parametric curve are determined with minimizing the energy function as the optimization objective during the fitting process, so as to obtain a sub-pixel-level vectorized data curve; the energy function includes the distance term between the fitted curve and all pixels in the pixel-level mask, and the curve smoothness term.

8. The automatic material curve image data extraction system based on deep learning according to claim 1, characterized in that, The interactive calibration and verification interface is also used for: The vectorized data curve is overlaid on the original material curve image as a background layer, and a real-time data table linked with the vectorized data curve is displayed simultaneously. Users can adjust the control points of the vectorized data curve through interactive operations to adjust the position of the curve segment as a whole, while maintaining the continuity and smoothness of the curve. When a user adjusts the vectorized data curve, the real-time data table updates the corresponding data value in real time; when a user modifies the data value in the real-time data table, the corresponding point on the vectorized data curve moves in real time.

9. The automatic material curve image data extraction system based on deep learning according to claim 1, characterized in that, The interactive calibration and verification interface is also used to visualize the confidence information of the deep learning instance segmentation model. Different colors are used to highlight the curve areas with low model confidence to guide users to check them first.

Images