A curve extraction method and device based on instance segmentation

Abstract

The application discloses a curve extraction method and device based on instance segmentation, and relates to the technical field of data processing. The method comprises the following steps: constructing a general image segmentation model based on a query; inputting an obtained line graph image into the model, training the model by using a multilayer curve mask guided training method, and outputting a predicted curve instance mask; identifying graph element information by using a framework of a pre-trained PaddleOCR; extracting central data points and start and end positions of each curve instance mask by using a median filter algorithm based on the predicted curve instance mask; inputting the predicted curve instance mask, a legend symbol and a legend text into a pre-trained twin neural network for matching, so as to obtain a result of matching of the curve instance and the legend; and calibrating and numerically mapping a curve coordinate axis by using a data conversion algorithm based on the result of matching of the curve instance and the legend, so as to obtain a reconstructed line graph. The application can improve the accuracy of data extraction in the line graph.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a curve extraction method and apparatus based on instance segmentation. Background Technology

[0002] In recent years, data visualization has played a crucial role in scientific research and business analysis. Bar charts, scatter plots, and pie charts, containing a wealth of key data, have become fundamental tools for research and decision-making across various fields. Line charts, as an important form of presenting experimental results, are widely used in literature and reports in materials science, physics, and biomedicine. While these charts provide rich experimental data, they are often presented in an unstructured format. Researchers must manually extract numerical information from these charts, which is not only time-consuming and laborious but also hinders the efficiency of large-scale data analysis. With the rise of data-driven approaches, the need for automated chart data extraction is becoming increasingly urgent.

[0003] WebPlotDigitizer is one of the most popular chart data extraction tools available, allowing users to manually align axes, enter tick values, and select target colors to extract data. However, this process is cumbersome and inefficient, especially when dealing with large datasets. While existing tools have made some progress in extracting data from simple graphical representations, the complexity of manual operations still limits their applicability to complex charts.

[0004] Methods based on bounding boxes and point detectors can achieve reasonable accuracy in handling charts with relatively simple structures and visual variations, such as bar charts and scatter plots. However, the complexity of line charts presents additional challenges. Line charts typically contain multiple line types, colors, and markers, and the presence of background grid lines further complicates data extraction. Furthermore, line intersections, occlusions, and dense arrangements cause existing algorithms to perform poorly when processing line charts. Although there are specific solutions for line charts, they still fail to effectively address visual complexity and overlap issues. For example, keypoint-based line extraction algorithms often struggle to accurately identify keypoints in overlapping and occluded areas, and attempts to extract features from local image fragments may result in misgrouping of keypoints due to occlusion effects. Traditional methods remain insufficient for extracting data from complex line charts. In addition, keypoint-based line extraction algorithms generally suffer from two major problems: (a) they cannot accurately predict the different keypoints of each line segment in overlapping and occluded areas; and (b) they cannot effectively handle multi-layered line structures. Summary of the Invention

[0005] To address the technical problem in existing technologies where multiple line segments in a chart easily intersect, overlap, and are densely packed, making it difficult for algorithms based on bounding boxes, point detection, or keypoint prediction to accurately track and separate each independent line segment, thus reducing the accuracy of data extraction from line charts, this invention provides a curve extraction method and apparatus based on instance segmentation. The technical solution is as follows:

[0006] On the one hand, an instance-segmentation-based curve extraction method is provided, which is implemented by an instance-segmentation-based curve extraction device. The method includes: S1. Construct a query-based general image segmentation model; the model includes: an image encoder, a contextual semantic fusion module, and an improved Transformer decoder; S2. Input the line graph image from the curve instance segmentation task into the model, train the model using a multi-layer curve mask-guided training method, and output the predicted curve instance mask. S3. The text detection network of PaddleOCR, which is pre-trained, is used to locate the text region and the chart element region in the line chart image to obtain the text region and the chart element position region. Based on the text region and the chart element position region, the text recognition module of PaddleOCR is used to recognize the specific content in the text region and the chart element region to obtain the chart element information. The chart element information includes: legend symbol and legend text. S4. Based on the predicted curve instance mask, the median filtering algorithm is used to extract the start and end positions of each curve instance mask, locate the center point of the curve in the vertical direction of the non-zero position, and obtain the data value of each complete curve. S5. Input the predicted curve instance mask, legend symbol, and legend text into the pre-trained Siamese neural network for matching to obtain the matching results of curve instances and legends; S6. Based on the results of matching curve instances with legends and the data values ​​of each complete curve, the curve coordinate axes are calibrated and numerically mapped using a data conversion algorithm to obtain the reconstructed line graph.

[0007] On the other hand, an instance-segmentation-based curve extraction apparatus is provided, which is applied to an instance-segmentation-based curve extraction method. The apparatus includes: A building unit is used to construct a query-based general image segmentation model; the model includes: an image encoder, a contextual semantic fusion module, and an improved Transformer decoder; The output unit is used to input the polyline image from the curve instance segmentation task into the model, train the model through a multi-layer curve mask-guided training method, and output the predicted curve instance mask. The recognition and localization unit is used to locate the text regions and chart element regions in the line chart image using a pre-trained PaddleOCR text detection network, and obtain the text regions and chart element location regions; based on the text regions and chart element location regions, the text recognition module of PaddleOCR is used to recognize the specific content within the text regions and chart element regions to obtain chart element information; wherein, the chart element information includes: legend symbols and legend text; The extraction unit is used to extract the start and end positions of each curve instance mask based on the predicted curve instance mask, locate the center point of the curve in the vertical direction of the non-zero position, and obtain the data value of each complete curve. The matching unit is used to input the predicted curve instance mask, legend symbol and legend text into the pre-trained Siamese neural network for matching, and obtain the matching result of the curve instance and legend; The reconstruction unit is used to calibrate and numerically map the curve coordinate axes based on the matching results of curve instances and legends, as well as the data values ​​of each complete curve, to obtain the reconstructed line graph.

[0008] On the other hand, an instance-segmentation-based curve extraction device is provided, the instance-segmentation-based curve extraction device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the above-described instance-segmentation-based curve extraction methods is implemented.

[0009] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for curve extraction based on instance segmentation.

[0010] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention addresses the challenges of data extraction from complex line chart structures by proposing a query-based curve instance segmentation method. A median filtering algorithm effectively solves the problem of curve discontinuities and extracts the starting points of all curves. Furthermore, a large-scale line chart text recognition dataset is constructed to overcome the accuracy challenge of coordinate axis scale recognition. Finally, a numerical conversion algorithm is used to convert the line chart to the original data and reconstruct the line chart. The proposed method demonstrates excellent performance on both synthetic and real chart datasets, showcasing its practical application potential and providing researchers with an effective tool for efficiently extracting line chart data from papers and understanding document content. This invention also solves the problem of line intersections, occlusions, and dense arrangement when multiple lines exist in a chart, making it difficult for bounding box, point detection, or keypoint prediction algorithms to accurately track and separate each independent line segment, thus improving the accuracy of data extraction from line charts. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a curve extraction method based on instance segmentation provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a query-based general image segmentation model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an improved Transformer decoder provided in an embodiment of the present invention; Figure 4 This is a block diagram of a curve extraction device based on instance segmentation provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a curve extraction device based on instance segmentation provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0014] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0015] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0016] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0017] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0018] This invention provides a curve extraction method based on instance segmentation. This method can be implemented by an instance segmentation-based curve extraction device, which can be a terminal or a server. Figure 1 The flowchart shown is for a curve extraction method based on instance segmentation. The processing flow of this method may include the following steps:

[0019] S1. Construct a query-based general image segmentation model; the model includes: an image encoder, a contextual semantic fusion module, and an improved Transformer decoder.

[0020] Among them, such as Figure 2 The diagram shown is a schematic representation of a query-based general image segmentation model provided in an embodiment of the present invention.

[0021] Optionally, the improvement process of the improved Transformer decoder includes: The execution order of the self-attention module and cross-attention module in the traditional Transformer decoder layer is swapped, and the query features are set as learnable parameters for supervised learning before predicting the curve mask. An improved Transformer decoder is obtained by adding the Mamba module and replacing the standard feedforward network of the traditional Transformer decoder.

[0022] Among them, such as Figure 3 The diagram shows a structural schematic of an improved Transformer decoder provided in an embodiment of the present invention. In one feasible approach, the improved Transformer decoder is implemented by: inputting a pixel feature map and a randomly generated query vector into the improved Transformer decoder; processing the vector using a cross-attention mechanism and residual connections and normalization methods to obtain the processed query vector, key, and value; further processing the vector using a self-attention mechanism and residual connections and normalization methods; and adding components via Mamba to output the original query vector.

[0023] In one feasible implementation, to address the issues of discontinuity in curve prediction and insufficient generalization ability of LineFormer, this embodiment improves the Transformer decoder, achieving efficient computation without additional calculations during the inference phase and maintaining extremely low computational overhead during training. A traditional Transformer decoder layer contains three modules for processing query features, in the following order: self-attention module, cross-attention module, and standard feedforward network. The query features are initialized to zero before being input into the traditional Transformer decoder and associated with learnable position embeddings. To optimize the Transformer decoder design, this invention makes the following improvements. First, since the query features of the first layer of self-attention in the traditional Transformer do not yet rely on image features, applying self-attention will not generate any meaningful features. Therefore, this embodiment swaps the order of the self-attention module and the cross-attention module to improve computational efficiency. The query features are set as learnable parameters and supervised learning is performed directly before being used to predict the curve mask. Traditional Transformer decoders rely on standard feedforward networks, often processing each point independently, which frequently leads to fragmented predictions. Due to the special nature of piecewise linear prediction, mask region prediction must consider adjacent curve points to maintain continuity. Therefore, this invention proposes a Mamba-enhanced Transformer decoder, which introduces recursive dependencies between predictions into the Mamba module to ensure that the output of one step affects subsequent steps. The standard feedforward network structure in the traditional Transformer decoder is represented by the following formula (1):

[0024] (1) in, This represents a position-by-position feedforward nonlinear transformation of the input feature vector x; This represents the weight matrix of the second fully connected layer; This represents the weight matrix of the first fully connected layer; This represents the input vector in the decoder; This represents the bias term of the first-level linear transformation; This represents the bias term of the second-level linear transformation; This represents a non-linear activation function.

[0025] In one feasible implementation, the present invention uses a Mamba module to replace the standard feedforward network, and its structure is represented by the following formula (2): (2) in, It is the hidden state at time step t, determined by the current input. and the previous hidden state Joint decision; Represents a non-linear activation function; The weight matrix represents the weights from hidden state to hidden state, used to adjust the hidden state from the previous time step. Perform a linear transformation. The weight matrix represents the input to the hidden state, used to process the input features at the current time step. Perform a linear mapping.

[0026] Among them, the improved loop structure enables the model to make more consistent predictions on occluded and masked curve segments by utilizing the contextual information of previously predicted points, thereby solving the discontinuity problem that often occurs in traditional methods based on standard feedforward networks.

[0027] S2. Input the line graph image from the curve instance segmentation task into the model, train the model using a multi-layer curve mask-guided training method, and output the predicted curve instance mask.

[0028] Optionally, the specific implementation process of S2 includes S21-S24: S21. Input the line graph image from the curve instance segmentation task into the model, extract features through the image encoder to obtain multi-layer feature maps; input the multi-layer feature maps into the context semantic fusion module for processing to obtain pixel feature maps with context semantic information; input the pixel feature maps into the improved Transformer decoder to generate the original query vector; perform a dot product between the original query vector and the pixel feature maps to generate prediction results for multiple image masks; wherein, the prediction results for the image masks include the prediction mask and the prediction category; Among them, multi-layer feature maps are extracted through a small-scale hierarchical sliding window Transformer model (Swin Transformer-S model) in the backbone network of the image encoder.

[0029] S22. Train the model using an additional set of query vectors and additional masks; where the additional query vectors are GT (ground truth) class embeddings and the additional masks are GT instance masks, used to calculate the model loss function. S23. The training process is divided into two branches: a mask-guided training branch and a binary matching training branch. The mask-guided training branch aligns the prediction result of the i-th image mask with the mask of the i-th ground truth instance through direct mapping to obtain a set of alignment results between the image mask prediction results and the ground truth instance mask. The loss function of the mask-guided training branch is calculated based on the alignment results. Among them, the alignment method of mask-guided training, for example, image prediction result 1 corresponds to the category + mask of GT1, and image prediction result 2 corresponds to the category + mask of GT2.

[0030] The binary matching training branch calculates the combined cost of class difference and mask overlap for each image prediction result and each GT instance mask. Based on the combined cost, the optimal matching result between the image mask prediction result and the GT instance mask is calculated using the Hungarian algorithm. Based on the optimal matching result, the loss function of the binary matching training branch is calculated. The loss function of the mask-guided training branch is consistent with the loss function of the binary matching training branch. The alignment method for binary matching training is as follows: image prediction result 3 corresponds to GT1, image mask prediction result 7 corresponds to GT2, and unmatched results are considered as having no target.

[0031] S24. The loss function of the mask-guided training branch and the loss function of the binary matching training branch are weighted and summed to construct the total loss function; based on the total loss function, the parameters of all models are updated through gradient descent backpropagation to obtain the trained query-based general image segmentation model; based on the model, the predicted curve instance mask is obtained.

[0032] The loss functions of the mask-guided training branch and the binary matching training branch are both expressed by the following formula (3): (3) in, This represents the total loss function value; This represents the weight coefficient corresponding to the i-th loss term; Represents the sub-loss function term; This represents the prediction result output by the model in the i-th training branch; This represents the actual labeled value corresponding to the prediction result; Indicates the index number of the loss item.

[0033] In one feasible implementation, the total loss function is expressed by the following formula (4): (4) in, Indicates the total loss; Indicates the first loss weight; This represents the loss of the training branch guided by the mask; Indicates the second loss weight; This represents the loss of the binary matching training branch.

[0034] Among them, by predicting the mask of curve instances, the robustness and accuracy problems caused by curve overlap, occlusion and visual complexity are effectively solved, and the accuracy of curve tracking is significantly improved.

[0035] S3. The text detection network of PaddleOCR, which is pre-trained, is used to locate the text region and the chart element location region in the line chart image to obtain the text region and the chart element region. Based on the text region and the chart element location region, the text recognition module of PaddleOCR is used to recognize the specific content in the text region and the chart element region to obtain the chart element information. The chart element information includes: legend symbol and legend text.

[0036] In one feasible implementation, this invention models the problem of recognizing chart elements and their associated text information in a line chart as an end-to-end visual perception task integrating text detection and text recognition, and completes the overall implementation based on the PaddleOCR framework. The text detection network in PaddleOCR is used to automatically locate text and key chart element regions in the image, obtaining corresponding candidate region bounding boxes. Subsequently, for the detected regions, the text recognition module of PaddleOCR is used to recognize the text content within the regions, obtaining the spatial location information and semantic label information of various elements in the chart.

[0037] In this embodiment of the invention, chart elements are defined as targets and divided into 18 categories. The basic elements mainly include 14 types of text information, including: x-axis title text, y-axis title text, x-axis tick text, y-axis tick text, and legend text, etc.; while the structural information includes 4 types of information mainly related to the drawing and coordinate areas, such as the x-axis area, y-axis area, and drawing area, etc.

[0038] In one feasible implementation, this embodiment of the invention uses labeled data from the ICPR22 dataset as training data to train the text detection network in PaddleOCR. To address the problem of uneven distribution of chart elements, a weighted cross-entropy loss is used as the classification loss function, and balance is achieved by calculating and allocating weights for each category. The classification loss function is expressed by the following formulas (5)-(6):

[0039] (5) (6) in, This represents the complete list of labels in the training set; bincount counts the number of each category, where c represents the category. This represents the category weight coefficient corresponding to the c-th class sample; Represents the classification loss function; This represents the model's predicted score for the c-th class of output; denoted by , represents the model's predicted score for the i-th class; C represents the total number of classes in the classification task.

[0040] In one feasible implementation, based on the actual needs of line chart data parsing and reconstruction, this embodiment of the invention performs unified modeling and recognition of multiple types of chart text elements involved in line charts, including 18 key structural and textual information categories such as axis ticks, axis labels, legend text, and title text. To improve the accuracy and robustness of text recognition in scientific paper charts, this embodiment of the invention performs targeted optimization on the text recognition module of PaddleOCR. Text image data containing special symbols, subscripts, superscripts, and unit expressions are collected from a large number of scientific paper charts to construct a chart text recognition dataset of 100,000 images. This embodiment of the invention uses the average category accuracy as a comprehensive evaluation index for chart text and element recognition, achieving 99% and 95% accuracy in x-axis label and y-axis label text recognition tasks, 96% and 99% accuracy in legend region and legend text recognition tasks, and 97% and 99% accuracy in x-axis title and y-axis title text recognition tasks, respectively. The PaddleOCR-based overall recognition solution can stably and efficiently complete the automatic recognition of chart elements and their text information in line charts, providing a reliable foundation for the subsequent structured analysis and accurate recovery of line chart data.

[0041] S4. Based on the predicted curve instance mask, the median filtering algorithm is used to extract the start and end positions of each curve instance mask, locate the center point of the curve in the vertical direction of the non-zero position, and obtain the data value of each complete curve.

[0042] Optionally, the specific implementation process of S4 includes S41-S46: S41. Filter the predicted curve instance masks to obtain the classification confidence of each curve instance, remove curve instances with confidence scores lower than the preset threshold of 0.3, and retain the masks corresponding to the valid curve instances. S42. Construct a one-dimensional sequence by projecting along the column direction. For each valid curve instance mask, accumulate the mask pixels along the image height direction to generate a one-dimensional column projection sequence with the same width as the image. In the sequence, non-zero values ​​indicate that there are curve pixels in the corresponding column. S43. The one-dimensional column projection sequence is smoothed by a median filter with a preset window of 5 to eliminate abnormal fluctuations caused by curve prediction noise or local breaks, and the processed one-dimensional column projection sequence is obtained. S44. Based on the processed one-dimensional column projection sequence, determine the horizontal position interval corresponding to the continuous non-zero values, and determine the start and end points of the curve based on the start and end positions of the interval, respectively. S45. Extract the center data point of the curve based on the start and end points. Sampling is performed in the horizontal direction between the start and end points according to a preset step size. For each sampling position, all non-zero pixels are extracted in the corresponding column, and their center position in the vertical direction is calculated as the curve data point at that position. If the vertical positions of two points differ by no more than two pixels, they are defined as belonging to the same curve. S46. Connect the curve data points obtained at each sampling position in sequence to form a sequence of discrete data points corresponding to the curve.

[0043] Based on the above steps, stable extraction of the start and end positions of the line graph curve and its center data point is achieved even in the presence of curve overlap, occlusion, or prediction noise.

[0044] S5. Input the predicted curve instance mask, legend symbol, and legend text into the pre-trained Siamese neural network for matching to obtain the matching results of curve instances and legend.

[0045] In this embodiment of the invention, a graph and curve matching method based on Siamese neural network is used to achieve accurate association between curve instances and graph semantics.

[0046] Optionally, the Siamese neural network consists of two feature extraction branches with shared parameters, which are used to encode the features of the input chart elements and curve instances, and calculate the similarity between the chart elements and curve instances in the embedding space through metric learning to complete the matching; the chart element information also includes: x-axis label text, y-axis label text, x-axis tick text, y-axis tick text, and information about the plotting and coordinate areas.

[0047] In this method, the acquired positive and negative samples are used to supervise learning based on the Siamese neural network, which makes semantically consistent legends and curves closer in the feature space, while unrelated combinations are farther apart.

[0048] Optionally, S5 inputs the predicted curve instance mask, legend symbol, and legend text into a pre-trained Siamese neural network for matching, obtaining the result of curve instance and legend matching, including: When legend text and legend symbols are displayed side by side, the legend symbols and legend text are initially paired based on spatial proximity. Candidate symbol-text combinations are determined by calculating the minimum Euclidean distance. The candidate symbol-text combinations and curve instances are then input into a pre-trained Siamese neural network for feature extraction. By calculating the similarity between the curve instances and the symbol-text combinations, the symbol-text combination with the highest similarity is paired with the curve instance, assigning only one unique symbol-text combination to each curve. When only legend text is included and no legend symbol is present, the images corresponding to the curve pixel region and the legend text are respectively input into a pre-trained Siamese neural network. By learning spatial location, local visual features and semantic relevance, the similarity between the curve instance and the legend text is calculated. The spatial distance between the curve instance and the legend text is calculated and used as a constraint condition. The text with the highest comprehensive similarity is selected as the semantic label of the curve instance. Each curve is assigned only one unique legend text.

[0049] S6. Based on the results of matching curve instances with legends and the data values ​​of each complete curve, the curve coordinate axes are calibrated and numerically mapped using a data conversion algorithm to obtain the reconstructed line graph.

[0050] Optionally, S6 uses the results of matching curve instances with legends and the data values ​​of each complete curve to calibrate and numerically map the curve coordinate axes through a data transformation algorithm, including: S61. Based on two adjacent positions of the x-axis label and the y-axis label, the center point of each label is used to represent the corresponding scale value; S62. Assume the scale lines are uniformly distributed, the Y-axis data decreases from top to bottom (maximum value at the top left), and the X-axis data increases from left to right (minimum value at the top left). To obtain accurate data, the pixel coordinates are mapped to the actual image values. The top-left corner coordinate of the image's pixel coordinates is defined as (…). , The actual image coordinates of the Y-axis are represented by the following formulas (7)-(8): (7) (8) in, This indicates the proportional relationship between the pixel coordinates on the y-axis and the actual image values. and This represents the actual image values ​​taken at two points on the y-axis. and This represents the pixel coordinates of two points on the y-axis; This represents the actual image value corresponding to the maximum value on the y-axis; In one feasible implementation, the X-axis has two cases: non-numerical scales (such as time and text descriptions) and numerical scales. For non-numerical scales, pixel coordinates can be used directly. For numerical scales, the X-axis coordinates are calculated using the same method as calculating the Y-axis coordinates.

[0051] The X-axis coordinate is represented by the following formulas (9)-(10): (9) (10) in, This represents the proportional relationship between the pixel coordinates on the x-axis and the actual image values. and This represents the actual image values ​​taken at two points on the x-axis. and This represents the pixel coordinates of two points on the x-axis; This represents the actual image value corresponding to the minimum value on the x-axis; In one feasible implementation, any point on the curve can be calculated based on the coordinates of the top-left corner of the image. The value of is expressed by the following formula (11): (11) in, This represents the actual image value corresponding to the minimum value on the x-axis; This represents the proportional relationship between the pixel coordinates on the x-axis and the actual image values. This represents the pixel coordinate value of the i-th point on the x-axis; This represents the actual image value corresponding to the maximum value on the y-axis; This indicates the proportional relationship between the pixel coordinates on the y-axis and the actual image values. This represents the pixel coordinate value of the i-th point on the y-axis.

[0052] This invention addresses the challenges of data extraction from complex line chart structures by proposing a query-based curve instance segmentation method. A median filtering algorithm effectively solves the problem of curve discontinuities and extracts the starting points of all curves. Furthermore, a large-scale line chart text recognition dataset is constructed to overcome the accuracy challenge of coordinate axis scale recognition. Finally, a numerical conversion algorithm is used to convert the line chart to the original data and reconstruct the line chart. The proposed method demonstrates excellent performance on both synthetic and real chart datasets, showcasing its practical application potential and providing researchers with an effective tool for efficiently extracting line chart data from papers and understanding document content. This invention also solves the problem of line intersections, occlusions, and dense arrangement when multiple lines exist in a chart, making it difficult for bounding box, point detection, or keypoint prediction algorithms to accurately track and separate each independent line segment, thus improving the accuracy of data extraction from line charts.

[0053] Figure 4 This is a block diagram of a curve extraction device based on instance segmentation provided in an embodiment of the present invention. This device is used for a curve extraction method based on instance segmentation. (Refer to...) Figure 4 The device includes a construction unit 410, a training and output unit 420, a recognition and localization unit 430, an extraction unit 440, a matching unit 450, and a reconstruction unit 460. Wherein:

[0054] The construction unit 410 is used to construct a query-based general image segmentation model; the model includes: an image encoder, a context semantic fusion module, and an improved Transformer decoder; The training and output unit 420 is used to input the line graph image in the curve instance segmentation task into the model, train the model through the multi-layer curve mask-guided training method, and output the predicted curve instance mask. The recognition and localization unit 430 is used to locate the text region and chart element region in the line chart image using a pre-trained PaddleOCR text detection network to obtain the text region and chart element position region; based on the text region and chart element position region, the text recognition module of PaddleOCR is used to recognize the specific content within the text region and chart element region to obtain chart element information; wherein, the chart element information includes: legend symbol and legend text; Extraction unit 440 is used to extract the start and end positions of each curve instance mask based on the predicted curve instance mask, locate the center point of the curve in the vertical direction of the non-zero position, and obtain the data value of each complete curve. The matching unit 450 is used to input the predicted curve instance mask, legend symbol and legend text into the pre-trained Siamese neural network for matching, and obtain the result of matching the curve instance with the legend; Reconstruction unit 460 is used to calibrate and numerically map the curve coordinate axes based on the matching results of curve instances and legends and the data values ​​of each complete curve, so as to obtain the reconstructed line graph.

[0055] Optionally, the improvement process of the improved Transformer decoder includes: The execution order of the self-attention module and cross-attention module in the traditional Transformer decoder layer is swapped, and the query features are set as learnable parameters for supervised learning before predicting the curve mask. An improved Transformer decoder is obtained by adding the Mamba module and replacing the standard feedforward network of the traditional Transformer decoder.

[0056] Optionally, the training and output unit 420 is used for: The line graph image from the curve instance segmentation task is input into the model, and features are extracted by an image encoder to obtain multi-layer feature maps. The multi-layer feature maps are then processed by a contextual semantic fusion module to obtain pixel feature maps with contextual semantic information. Based on the pixel feature maps, the improved Transformer decoder is input to generate an original query vector. The original query vector is multiplied by the pixel feature maps to generate prediction results for multiple image masks. The prediction results for the image masks include a prediction mask and a prediction category. The model is trained using an additional set of query vectors and additional masks; the additional query vectors are ground truth class embeddings, and the additional masks are ground truth instance masks, which are used to calculate the model loss function. The training process is divided into two branches: a mask-guided training branch and a binary matching training branch. The mask-guided training branch aligns the prediction result of the i-th image mask with the mask of the i-th ground truth instance through direct mapping, obtaining a set of alignment results between the image mask prediction results and the ground truth instance masks. The loss function of the mask-guided training branch is calculated based on the alignment results. The binary matching training branch calculates the combined cost of class difference and mask overlap for each image mask prediction result and each GT instance mask. Based on the combined cost, the optimal matching result between the image mask prediction result and the GT instance mask is calculated using the Hungarian algorithm. Based on the optimal matching result, the loss function of the binary matching training branch is calculated. The loss function of the mask-guided training branch is consistent with the loss function of the binary matching training branch. The loss function of the mask-guided training branch and the loss function of the binary matching training branch are weighted and summed to construct the total loss function. Based on the total loss function, the parameters of all models are updated through gradient descent backpropagation to obtain a trained query-based general image segmentation model. Based on the model, the predicted curve instance mask is obtained.

[0057] Optionally, the extraction unit 440 is used for: The predicted curve instance masks are filtered to obtain the classification confidence of each curve instance. Curve instances with confidence scores below a preset threshold of 0.3 are removed, and the masks corresponding to the valid curve instances are retained. A one-dimensional sequence is constructed by projecting along the column direction. For each valid curve instance mask, the mask pixels are accumulated along the image height direction to generate a one-dimensional column projection sequence with the same width as the image. Non-zero values ​​in the sequence indicate that there are curve pixels in the corresponding column. The one-dimensional column projection sequence is smoothed by a median filter with a preset window of 5 to eliminate abnormal fluctuations caused by curve prediction noise or local breaks, and the processed one-dimensional column projection sequence is obtained. Based on the processed one-dimensional column projection sequence, the horizontal position intervals corresponding to continuous non-zero values ​​are determined, and the start and end positions of the interval are determined as the start and end points of the curve, respectively. Based on the start and end points, the center data points of the curve are extracted. Between the start and end points, sampling is performed along the horizontal direction according to a preset step size. For each sampling position, all non-zero pixels are extracted in the corresponding column, and their center position in the vertical direction is calculated as the curve data point at that position. If the vertical positions of two points differ by no more than two pixels, they are defined as belonging to the same curve. The curve data points obtained from each sampling position are sequentially connected to form a sequence of discrete data points corresponding to the curve.

[0058] Optionally, the Siamese neural network consists of two feature extraction branches with shared parameters, used to encode the features of the input chart elements and curve instances, and to calculate the similarity between the chart elements and curve instances in the embedding space through metric learning to complete the matching; the chart element information also includes: x-axis label text, y-axis label text, x-axis tick text, y-axis tick text, and information about the plotting and coordinate areas.

[0059] Optionally, the step of inputting the predicted curve instance mask, legend symbol, and legend text into a pre-trained Siamese neural network for matching to obtain the matching result includes: When legend text and legend symbols are displayed side by side, the legend symbols and legend text are initially paired based on spatial proximity. Candidate symbol-text combinations are determined by calculating the minimum Euclidean distance. The candidate symbol-text combinations and curve instances are then input into a pre-trained Siamese neural network for feature extraction. By calculating the similarity between curve instances and symbol-text combinations, the symbol-text combination with the highest similarity is paired with the curve instance, assigning only one unique symbol-text combination to each curve. When only legend text is included and no legend symbol is present, the images corresponding to the curve pixel region and the legend text are respectively input into a pre-trained Siamese neural network. By learning spatial location, local visual features and semantic relevance, the similarity between the curve instance and the legend text is calculated. The spatial distance between the curve instance and the legend text is calculated and used as a constraint condition. The text with the highest comprehensive similarity is selected as the semantic label of the curve instance. Each curve is assigned only one unique legend text.

[0060] Optionally, the calibration and numerical mapping of the curve coordinate axes based on the results of matching curve instances with legends and the data values ​​of each complete curve, using a data conversion algorithm, includes: Based on two adjacent positions of the x-axis and y-axis labels, the center point of each label is used to represent the corresponding scale value; Assuming the scale lines are uniformly distributed, the Y-axis data decreases from top to bottom (maximum value at the top left corner), and the X-axis data increases from left to right (minimum value at the top left corner); to obtain accurate data, a mapping transformation is performed between pixel coordinates and the actual image numerical values; the top left corner coordinate of the image's pixel coordinates is defined as (…). , The actual image coordinates of the Y-axis are represented by the following formulas (1) and (2): (1) (2) in, This indicates the proportional relationship between the pixel coordinates on the y-axis and the actual image values. and This represents the actual image values ​​taken at two points on the y-axis. This represents the pixel coordinates of two points on the y-axis; This represents the actual image value corresponding to the maximum value on the y-axis; The X-axis coordinate is represented by the following formulas (3)-(4): (3) (4) in, This represents the proportional relationship between the pixel coordinates on the x-axis and the actual image values. and This represents the actual image values ​​taken at two points on the x-axis. and This represents the pixel coordinates of two points on the x-axis; This represents the actual image value corresponding to the minimum value on the x-axis.

[0061] This invention addresses the challenges of data extraction from complex line chart structures by proposing a query-based curve instance segmentation method. A median filtering algorithm effectively solves the problem of curve discontinuities and extracts the starting points of all curves. Furthermore, a large-scale line chart text recognition dataset is constructed to overcome the accuracy challenge of coordinate axis scale recognition. Finally, a numerical conversion algorithm is used to convert the line chart to the original data and reconstruct the line chart. The proposed method demonstrates excellent performance on both synthetic and real chart datasets, showcasing its practical application potential and providing researchers with an effective tool for efficiently extracting line chart data from papers and understanding document content. This invention also solves the problem of line intersections, occlusions, and dense arrangement when multiple lines exist in a chart, making it difficult for bounding box, point detection, or keypoint prediction algorithms to accurately track and separate each independent line segment, thus improving the accuracy of data extraction from line charts.

[0062] Figure 5 This is a schematic diagram of the structure of a curve extraction device based on instance segmentation provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the curve extraction device based on instance segmentation may include the above-mentioned Figure 4 The curve extraction apparatus based on instance segmentation is shown. Optionally, the curve extraction apparatus 510 based on instance segmentation may include a first processor 2001.

[0063] Optionally, the instance-segmentation-based curve extraction device 510 may also include a memory 2002 and a transceiver 2003.

[0064] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0065] The following is combined with Figure 5 The individual components of the instance-segmentation-based curve extraction device 510 are described in detail below: The first processor 2001 is the control center of the instance-based curve extraction device 510. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0066] Optionally, the first processor 2001 can perform various functions of the instance-segmentation-based curve extraction device 510 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0067] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0068] In a specific implementation, as one example, the instance-segmentation-based curve extraction device 510 may also include multiple processors, for example... Figure 5 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0069] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0070] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the instance-based segmentation curve extraction device 510. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0071] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0072] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0073] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the instance-segmentation-based curve extraction device 510. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0074] It should be noted that, Figure 5 The structure of the instance-based curve extraction device 510 shown in the diagram does not constitute a limitation on the router. Actual instance-based curve extraction devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0075] Furthermore, the technical effects of the instance-segmentation-based curve extraction device 510 can be referred to the technical effects of the instance-segmentation-based curve extraction method described in the above method embodiments, and will not be repeated here.

[0076] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0077] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0078] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0079] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0080] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0081] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A curve extraction method based on instance segmentation, characterized in that, The method includes: S1. Construct a query-based general image segmentation model; the model includes: an image encoder, a contextual semantic fusion module, and an improved Transformer decoder; S2. Input the line graph image from the curve instance segmentation task into the model, train the model using a multi-layer curve mask-guided training method, and output the predicted curve instance mask. S3. The text detection network of PaddleOCR, which is pre-trained, is used to locate the text region and the chart element region in the line chart image to obtain the text region and the chart element position region. Based on the text region and the chart element position region, the text recognition module of PaddleOCR is used to recognize the specific content in the text region and the chart element region to obtain the chart element information. The chart element information includes: legend symbol and legend text. S4. Based on the predicted curve instance mask, the median filtering algorithm is used to extract the start and end positions of each curve instance mask, locate the center point of the curve in the vertical direction of the non-zero position, and obtain the data value of each complete curve. S5. Input the predicted curve instance mask, legend symbol, and legend text into the pre-trained Siamese neural network for matching to obtain the matching results of curve instances and legends; S6. Based on the results of matching curve instances with legends and the data values ​​of each complete curve, the curve coordinate axes are calibrated and numerically mapped using a data conversion algorithm to obtain the reconstructed line graph.

2. The curve extraction method based on instance segmentation according to claim 1, characterized in that, The improvement process of the improved Transformer decoder includes: The execution order of the self-attention module and cross-attention module in the traditional Transformer decoder layer is swapped, and the query features are set as learnable parameters for supervised learning before predicting the curve mask. An improved Transformer decoder is obtained by adding the Mamba module and replacing the standard feedforward network of the traditional Transformer decoder.

3. The curve extraction method based on instance segmentation according to claim 1, characterized in that, The step S2 involves inputting the line graph image from the curve instance segmentation task into the model, training the model using a multi-layer curve mask-guided training method, and outputting a predicted curve instance mask, including: S21. Input the line graph image from the curve instance segmentation task into the model, extract features through an image encoder to obtain a multi-layer feature map; input the multi-layer feature map into the context semantic fusion module for processing to obtain a pixel feature map with context semantic information; input the pixel feature map into an improved Transformer decoder to generate an original query vector; perform a dot product between the original query vector and the pixel feature map to generate prediction results for multiple image masks; wherein, the prediction results for the image masks include a prediction mask and a prediction category; S22. The model is trained using an additional set of query vectors and additional masks; where the additional query vectors are GT class embeddings and the additional masks are GT instance masks, which are used to calculate the model loss function. S23. The training process is divided into two branches: a mask-guided training branch and a binary matching training branch. The mask-guided training branch aligns the prediction result of the i-th image mask with the mask of the i-th ground truth instance through direct mapping to obtain a set of alignment results between the image mask prediction results and the ground truth instance mask. The loss function of the mask-guided training branch is calculated based on the alignment results. The binary matching training branch calculates the combined cost of class difference and mask overlap for each image mask prediction result and each GT instance mask. Based on the combined cost, the optimal matching result between the image mask prediction result and the GT instance mask is calculated using the Hungarian algorithm. Based on the optimal matching result, the loss function of the binary matching training branch is calculated. The loss function of the mask-guided training branch is consistent with the loss function of the binary matching training branch. S24. The loss function of the mask-guided training branch and the loss function of the binary matching training branch are weighted and summed to construct the total loss function; based on the total loss function, the parameters of all models are updated through gradient descent backpropagation to obtain the trained query-based general image segmentation model; based on the model, the predicted curve instance mask is obtained.

4. The curve extraction method based on instance segmentation according to claim 1, characterized in that, The S4-based prediction-based curve instance mask uses a median filtering algorithm to extract the start and end positions of each curve instance mask, locates the center point of the curve in the vertical direction of the non-zero position, and obtains the data value of each complete curve, including: S41. Filter the predicted curve instance masks to obtain the classification confidence of each curve instance, remove curve instances with confidence scores lower than the preset threshold of 0.3, and retain the masks corresponding to the valid curve instances. S42. Construct a one-dimensional sequence by projecting along the column direction. For each valid curve instance mask, accumulate the mask pixels along the image height direction to generate a one-dimensional column projection sequence with the same width as the image. In the sequence, non-zero values ​​indicate that there are curve pixels in the corresponding column. S43. The one-dimensional column projection sequence is smoothed by a median filter with a preset window of 5 to eliminate abnormal fluctuations caused by curve prediction noise or local breaks, and the processed one-dimensional column projection sequence is obtained. S44. Based on the processed one-dimensional column projection sequence, determine the horizontal position interval corresponding to the continuous non-zero values, and determine the start and end points of the curve based on the start and end positions of the interval, respectively. S45. Extract the center data point of the curve based on the start and end points. Sampling is performed in the horizontal direction between the start and end points according to a preset step size. For each sampling position, all non-zero pixels are extracted in the corresponding column, and their center position in the vertical direction is calculated as the curve data point at that position. If the vertical positions of two points differ by no more than two pixels, they are defined as belonging to the same curve. S46. Connect the curve data points obtained at each sampling position in sequence to form a sequence of discrete data points corresponding to the curve.

5. The curve extraction method based on instance segmentation according to claim 1, characterized in that, The Siamese neural network consists of two feature extraction branches with shared parameters, which are used to encode the features of the input chart elements and curve instances, and calculate the similarity between the chart elements and curve instances in the embedding space through metric learning to complete the matching; the chart element information also includes: x-axis label text, y-axis label text, x-axis tick text, y-axis tick text, and information about the plotting and coordinate areas.

6. The curve extraction method based on instance segmentation according to claim 1, characterized in that, The step S5 involves inputting the predicted curve instance mask, legend symbol, and legend text into a pre-trained Siamese neural network for matching, obtaining the result of matching the curve instance with the legend, including: When legend text and legend symbols are displayed side by side, the legend symbols and legend text are initially paired based on spatial proximity. Candidate symbol-text combinations are determined by calculating the minimum Euclidean distance. The candidate symbol-text combinations and curve instances are then input into a pre-trained Siamese neural network for feature extraction. By calculating the similarity between curve instances and symbol-text combinations, the symbol-text combination with the highest similarity is paired with the curve instance, assigning only one unique symbol-text combination to each curve. When only legend text is included and no legend symbol is present, the images corresponding to the curve pixel region and the legend text are respectively input into a pre-trained Siamese neural network. By learning spatial location, local visual features and semantic relevance, the similarity between the curve instance and the legend text is calculated. The spatial distance between the curve instance and the legend text is calculated and used as a constraint condition. The text with the highest comprehensive similarity is selected as the semantic label of the curve instance. Each curve is assigned only one unique legend text.

7. The curve extraction method based on instance segmentation according to claim 1, characterized in that, The S6 method, based on the results of matching curve instances with legends and the data values ​​of each complete curve, uses a data conversion algorithm to calibrate and numerically map the curve coordinate axes, including: S61. Based on two adjacent positions of the x-axis label and the y-axis label, the center point of each label is used to represent the corresponding scale value; S62. Assume the scale lines are uniformly distributed, the Y-axis data decreases from top to bottom (maximum value at the top left), and the X-axis data increases from left to right (minimum value at the top left). To obtain accurate data, the pixel coordinates are mapped to the actual image values. The top-left corner coordinate of the image's pixel coordinates is defined as (…). , The actual image coordinates of the Y-axis are represented by the following formulas (1) and (2): （1） （2） in, This indicates the proportional relationship between the pixel coordinates on the y-axis and the actual image values. and This represents the actual image values ​​taken at two points on the y-axis; and This represents the pixel coordinates of two points on the y-axis; This represents the actual image value corresponding to the maximum value on the y-axis; The X-axis coordinate is represented by the following formulas (3)-(4): （3） （4） in, This represents the proportional relationship between the pixel coordinates on the x-axis and the actual image values. and This represents the actual image values ​​taken at two points on the x-axis. and This represents the pixel coordinates of two points on the x-axis; This represents the actual image value corresponding to the minimum value on the x-axis.

8. A curve extraction device based on instance segmentation, wherein the curve extraction device based on instance segmentation is used to implement the curve extraction method based on instance segmentation as described in any one of claims 1-7, characterized in that, The device includes: A building unit is used to construct a query-based general image segmentation model; the model includes: an image encoder, a contextual semantic fusion module, and an improved Transformer decoder; The output unit is used to input the polyline image from the curve instance segmentation task into the model, train the model through a multi-layer curve mask-guided training method, and output the predicted curve instance mask. The recognition and localization unit is used to locate the text regions and chart element regions in the line chart image using a pre-trained PaddleOCR text detection network, and obtain the text regions and chart element location regions; based on the text regions and chart element location regions, the text recognition module of PaddleOCR is used to recognize the specific content within the text regions and chart element regions to obtain chart element information; wherein, the chart element information includes: legend symbols and legend text; The extraction unit is used to extract the start and end positions of each curve instance mask based on the predicted curve instance mask, locate the center point of the curve in the vertical direction of the non-zero position, and obtain the data value of each complete curve. The matching unit is used to input the predicted curve instance mask, legend symbol, and legend text into the pre-trained Siamese neural network for matching, and obtain the matching result between the curve instance and the legend; The reconstruction unit is used to calibrate and numerically map the curve coordinate axes based on the matching results of curve instances and legends, as well as the data values ​​of each complete curve, to obtain the reconstructed line graph.

9. A curve extraction device based on instance segmentation, characterized in that, The instance-segmentation-based curve extraction device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images