A system and method for extracting polyline / curve data based on image processing

Abstract

The present invention discloses a system and method for extracting broken line / curve data based on image processing. Steps of the present invention: 1. Searching for the border of the curve / broken line graph; 2. Preprocessing the curve / broken line picture; 3. Searching for the digital contours of the coordinate axes; 4. Recognizing the numbers on the x and y coordinate axes; 5. Calculating the ratios of the x and y coordinate axes; 6. Searching for and obtaining the y values. The present invention can handle more special situations. The present invention can reconstruct and sample complex curve / broken line graphs with diverse background colors, grids, and large curve slopes, and obtain the coordinate information of the pictures, so as to realize the reuse of the data in the pictures. The present invention does not require input of parameters such as the coordinates of the origin, only the picture needs to be input, reducing manual operations, and compared with traditional methods, the present invention can handle more special situations.

Description

Technical Field

[0001] The present invention relates to the field of digital image information extraction (Image Information Acquisition), and particularly to the extraction of curve / polygon picture information, for reusing and reconstructing the trends and values of chart information. A polygon / curve data extraction system and method based on image processing are provided. Background Art

[0002] With the development of basic science +, the process information involved in various scientific research activities and production and living activities tends to be complex. In industries such as the Internet, healthcare, biology, chemical engineering, and transportation, a single project often covers a data volume of over ten thousand. To present the data to the audience in a clearer way, visualization methods have become an indispensable part. Curve and polygon charts stand out from a multitude of data charts by virtue of their ability to represent data trends and simple and easy-to-understand presentation methods. Nowadays, curve and polygon charts are widely used in scenarios such as stocks, electrocardiograms, simulation software, and academic articles.

[0003] At the same time, the exponentially growing data volume has brought great difficulties to fine-grained data analysis operations such as process reproduction and data mining. For some research results with a large data volume or a long time span, it is obviously unrealistic to obtain detailed research data. If data charts can be used as data carriers to accurately record data, it will undoubtedly reduce the difficulty of data storage and acquisition. At the same time, this also requires a complete set of chart data extraction algorithms as support.

[0004] Currently, there are many common tools for extracting data points from curve and polygon charts. Taking functions as an example, there is ginput in Matlab; push_back in opencv, etc.; taking software as an example, there are WebPlotDigitizer and Graphixy, etc. They all have good effects in the work of extracting curve chart data. However, these aforementioned tools all have some problems that are difficult to solve: such as the need to manually input the starting point of the image and the maximum values of the x and y axes, etc.

[0005] Due to the complex and diverse formats of the obtained pictures, and the fact that the forms of the coordinate axes on the pictures are not determined, it makes it difficult to obtain data information from curve / polygon charts in the following aspects:

[0006] (1) Lack of batch processing means. Among the existing methods, it is necessary to manually input the position of the origin of the curve chart in the picture and give the maximum values of the x and y axes. Therefore, if the above methods are used, it requires a large amount of manpower to obtain the horizontal and vertical coordinates of the points from the polygon / curve chart, and it has mechanical repeatability.

[0007] (2) Low precision and weak universality. In daily life, the pictures we obtain are often of various types, including problems such as different background colors, thicknesses of curves, and boundaries of borders. The existing methods have poor processing effects on these pictures.

[0008] Based on the above problems, the present invention proposes a method for extracting broken line / curve data based on image processing and digital recognition to solve the above-mentioned difficulties.

[0009] 1. A method for automatically identifying the border of a curve graph and the maximum coordinate value is proposed;

[0010] 2. A method for preprocessing images is proposed, which can exclude the interference of the background of the curve on recognition;

[0011] 3. A method for reducing the error of curves with large slopes based on the trend of the graph line is proposed. Summary of the Invention

[0012] The purpose of the present invention is to provide a broken line / curve data extraction system and method based on image processing in view of the deficiencies of the prior art.

[0013] The solutions adopted by the present invention to solve its technical problems are as follows:

[0014] A broken line / curve data extraction system based on image processing includes a search module for the border of a curve / broken line graph, an image preprocessing module, a search module for the digital contour of the coordinate axis, an identification module for the coordinate axis numbers, a calculation module for the coordinate axis ratio, and a y-value search module;

[0015] Search module for the border of a curve / broken line graph: Search for the edge contour of the original image, and select the required border of the curve / broken line graph from the results returned by the edge contour search.

[0016] Image preprocessing module: Preprocess a given curve / broken line picture img. First, perform grayscale transformation on the picture, and then perform adaptive thresholding on the grayscale picture to improve the accuracy of the subsequent digital contour search.

[0017] Search module for the digital contour of the coordinate axis: Find the range where the numbers on the coordinate axis are located in the curve / broken line picture img. The input is the preprocessed picture. Find all the contours on the picture through the contour search function, screen out the contours that actually contain numbers, and record the corresponding coordinates of the digital area in a list to be the input for the next step.

[0018] Axis number recognition module: The left and right boundaries and the upper and lower boundaries of the curve area are obtained in the curve / polygon graph border search module. The numbers on the X and Y axes are recognized separately. The number position list in the axis number contour search module is used as the input, and the numbers in the specified area are recognized. The recognized numbers are used as the fifth-dimensional elements of the list and are also the inputs to the axis ratio calculation module.

[0019] Axis ratio calculation module: According to the number positions and recognition results obtained from the axis number contour search module and the axis number recognition module, the corresponding ratios in the horizontal and vertical directions are obtained;

[0020] y-value search module: According to the x-value to be searched, the pixel coordinates on the corresponding curve are found, and the true value of y is calculated by converting through the axis ratio obtained by the axis ratio calculation module.

[0021] Furthermore, the curve / polygon graph border search module is implemented as follows: Apply canny edge detection to the input image to obtain a two-dimensional pixel vector v1 of size h*w containing the clear curve contour of the input image and the edge shape of the input image; Transpose the two-dimensional pixel vector v1 to obtain the transposed vector v2, and perform row search and column search on the two-dimensional pixel vector v1 and the transposed vector v2 respectively. Since the pixel point cumulative sum will mutate on the left and right of the starting and ending points of the polygon after binarization, accurate border positioning of the original polygon / curve image is achieved based on this, and x_boundary_min, x_boundary_max, y_boundary_min, and y_boundary_max are obtained;

[0022] Image preprocessing module: Perform grayscale processing on the input image;

[0023] Axis number contour search module: Perform contour search on the preprocessed image, find all the contours and store them in a list, traverse and return the information of the smallest upright rectangle containing the input information, that is, store the upper left corner coordinates of the rectangle frame of the smallest upright rectangle and the width and height of the rectangle in the list; Screen all the smallest upright rectangles in the list and store the screening results in a new list a; After merging and processing the new list a, obtain the x list and the y list;

[0024] Perform character recognition on the x list and the y list respectively using pytesserac, perform re-screening processing on the recognition results, store the screened digital results in the fifth dimension of the x list and the y list respectively, and finally sort the list in ascending order with the fifth dimension as the keyword to obtain two new lists: result_x, result_y;

[0025] Axis ratio calculation module: Calculate the axis ratios for adjacent elements in the result_x and result_y lists respectively;

[0026] y - value lookup module: Input the x - value to be looked up and calculate its true value in the pixel coordinate system.

[0027] A method for extracting polyline / curve data based on image processing, comprising the following steps:

[0028] Step (1), find the border of the curve / polyline graph;

[0029] Step (2), preprocess the image;

[0030] Step (3), find the digital contour of the coordinate axis;

[0031] Step (4), recognize the numbers on the coordinate axis;

[0032] Step (5), calculate the scale of the coordinate axis;

[0033] Step (6), through the input x - value, find the true y - value of this x - value in the coordinate system.

[0034] Further, the specific implementation of step (1) is as follows:

[0035] Apply canny edge detection to the input image to obtain a two - dimensional pixel vector v1 of size h*w containing the clear curve contour of the input image and the edge shape of the input image;

[0036] Transpose the two - dimensional pixel vector v1 to obtain the transposed vector v2;

[0037] Perform row search and column search on the two - dimensional pixel vector v1 and the transposed vector v2 respectively;

[0038] Since the pixel point cumulative sum will mutate on the left and right of the starting point and ending point of the polyline after binarization, based on this, precise border positioning of the original polyline / curve image is achieved to obtain x_boundary_min, x_boundary_max, y_boundary_min, and y_boundary_max.

[0039] Further, the specific implementation of step (2) is as follows:

[0040] Perform grayscale processing on the input image: Make the values of the three channels of the input image the same, and the value size is the sum of the three - channel pixels divided by 3;

[0041] Perform adaptive binarization on the grayscale - processed image. For the image with a value greater than the adaptive threshold, set the single - channel pixel value of its corresponding grayscale image to 255 to obtain the pre - processed image.

[0042] Further, the specific implementation of step (3) is as follows:

[0043] Perform contour finding on the preprocessed image obtained in step (2), find all contours and store them in a list, traverse and return the information of the smallest upright rectangle containing the input information, that is, store the upper left corner coordinates of the rectangle frame of the smallest upright rectangle and the width and height of the rectangle in the list; screen all the smallest upright rectangles in the list and store the screening results in a new list a.

[0044] Further, there may be two-digit numbers in the new list a that are split into two numbers due to a long segmentation distance. Therefore, it is necessary to further judge the new list a; if the distance between two rectangle frames is less than the set threshold, the two rectangle frames will be merged to obtain the final required list x and list y; the specific merging formula is:

[0045] a[i] = [a[x1][0], a[x1][1], a[x2][0] - a[x1][0], max(a[x1][3], a[x2][3])] (1)

[0046] Among them, a[x1][0] represents the upper left corner x, a[x1][1] represents the upper left corner y, a[x2][0] - a[x1][0] represents the length of the rectangle, and max(a[x1][3], a[x2][3]) represents the width of the rectangle;

[0047] Specifically, according to x_boundary_min and y_boundary_max obtained in step (1), store the rectangle frame a[i][0] < x_boundary_min in list a into list y, which are the numbers on the Y axis; store a[i][1] > y_boundary_max in list a into list y, which are the numbers found on the X axis.

[0048] Further, the specific implementation of step (4) is as follows:

[0049] Perform character recognition on the two merged lists x and y respectively using pytesserac;

[0050] Perform screening processing on the recognition results, and store the obtained digital results into the fifth dimension of lists x and y respectively. The format of the list is as follows:

[0051] list = [x1, y1, x2, y2, num] (2)

[0052] Finally, sort the list in ascending order with the fifth dimension as the keyword to obtain two new lists: result_x, result_y.

[0053] Further, the screening process is as follows: Since character recognition may identify spaces, line breaks, and incorrect characters, each digital rectangular box needs to be screened to extract the numbers and merge them. Then, the character-form data is converted into integer-type data and stored in the fifth dimension of the list result_x and the list result_y respectively.

[0054] Further, the specific implementation process of step (5) is as follows:

[0055] Calculate the axis ratio for adjacent elements in the lists result_x and result_y respectively; for the axis ratio calculation of the result_x list, subtract the fifth-dimensional elements of two adjacent elements in the result_x list in sequence and traverse the elements:

[0056] d = result_x[i + 1][4] - result_x[i][4] (3)

[0057] where i is greater than or equal to 0, representing the i-th rectangular box in the result_x list; use the difference d as the key of the dictionary, and set the values of the dictionary as a list to record which rectangular box it is, which is i in this example; perform the same operation on the result_y list, and finally obtain the dictionaries my_dict_x and my_dict_y; in the two dictionaries, find the number of elements in their lists according to the key, that is, the possible d value; accumulate the d values of the corresponding elements in my_dict_x[Delta_x] to obtain Delta_sum_x. Finally, the axis ratio formula is:

[0058] x_ratio = Delta_sum_x / len(my_dictx_x[Delta_x]) (4)

[0059] where Delta_x represents the key corresponding to the list with the largest length of the value in the dictionary my_dict_x; x_ratio represents how many pixels the actual length on the axis is in the pixel image; Delta_sum_x represents the accumulated value of the differences between the left boundaries of adjacent rectangular boxes with the key Delta_x in the dictionary.

[0060] Further, the specific implementation process of step (6) is as follows:

[0061] Input the x value to be searched. Assume there is a reference point on the X-axis. Since the number needs to be correctly recognized, an index m is taken from my_dict_x[Delta_x], and its coordinates and the truly recognized number are taken from result_x[m]. The abscissa x of the pixel point symbol =(result_x[m][0] + result_x[m][2]) / 2, and the true value of the reference point standard_num_x = result[m][4]; so for the input x value, its abscissa in the pixel coordinate system is:

[0062] x′ = x symbol +(x - standard_num_x)*x_ratio (5)

[0063] Since x′ may be a floating-point number, the integer part and the decimal part need to be calculated separately; by traversing the pixel points in the image, the background color in the image is removed, and its grayscale pixel value is set to 255, that is, white;

[0064] For the abscissa x′, x_int = int(x′), x_float = x′ - x_int. For finding the corresponding y value of the abscissa x′, the color needs to be searched first through the color search function to obtain the main color of the curve: line_color. Then, the method of traversing the pixels in the x_int column is adopted. When traversing from top to bottom to the first pixel point whose value is line_color, this is the upper boundary of y to be searched. Similarly, when traversing from bottom to top to the first pixel point whose value is line_color, this is the lower boundary of y to be searched. Taking the median value is the y value corresponding to x_int to be found; for the x_float part, the slope is searched, the slope adjacent to x_int is found, and then x_float is multiplied by the slope to make up for it, forming the ordinate of the corresponding pixel point on the final y-axis; similarly, a reference point is also needed for the Y-axis. It needs to be correctly recognized as a number. An index m is taken from my_dict_y[Delta_y], and its coordinates and the truly recognized number are taken from result_y[m]. The ordinate of the pixel point

[0065]

[0066] True value of the reference point:

[0067] standard_num_y = result_y[m][4](7)

[0068] So for the input x value, its true value in the pixel coordinate system is:

[0069] x result = standard_num_y + (y symbol - (k x * x_float)) / y_ratio (8)

[0070] x result is the true value of the y-axis to be obtained.

[0071] The beneficial effects of the present invention are as follows:

[0072] The present invention proposes a method for extracting polyline / curve picture data based on image processing and digital recognition. Although existing open-source software can achieve the extraction of coordinate values, it requires manual input of parameters such as the coordinates of the origin and cannot perform batch processing. Therefore, through modules such as image processing and digital recognition, combined with the image batch processing function, the present invention solves the problems of the need for a large amount of manpower and mechanical repetition in the extraction of daily graphic coordinate information. At the same time, traditional software has relatively high requirements for pictures and lacks universality, and it is unable to achieve good result extraction for pictures with grids, background colors, and large curve slopes, or even unable to extract the corresponding coordinate information. Therefore, the present invention performs grayscale transformation on the input pictures and then performs adaptive thresholding on the grayscale pictures to eliminate the interference of factors such as backgrounds and grids. Combined with the axis number contour search module and the final module for optimizing the results according to the graph line trend gradient, the coordinate extraction of complex polyline / curve graphs with diverse background colors, grids, and large curve slopes is achieved. Combining the two, a data extraction method that can achieve batch processing, does not require manpower, and can handle diverse complex curves / polylines is realized. Compared with traditional methods and software, the simplicity, convenience, accuracy, and universality have been greatly improved. Brief Description of the Drawings

[0073] Figure 1 is the result graph of image border search;

[0074] Figure 2 is the result after image preprocessing;

[0075] Figure 3 is the complete flowchart of the present invention; Detailed Embodiment

[0076] The following further specifically describes the detailed parameters of the present invention in conjunction with the drawings and embodiments.

[0077] The complete flowchart of the present invention is as Figure 3 shown. The present invention provides a method for extracting polyline / curve data based on image processing and digital recognition.

[0078] Step 1: Search for the border of the curve / polyline graph

[0079] 1-1. Apply Canny edge detection to the original image. The main steps involved are as follows: 1. Grayscale the image; 2. Denoise; 3. Use the Sobel operator to solve for the gradient magnitude and direction; 4. Non-maximum suppression; 5. Detect and connect edges using the double-threshold algorithm. By adjusting the range of the double thresholds to appropriate values, a two-dimensional pixel vector v (binary result) containing clear curve contours and edge shapes is obtained.

[0080] 1-2. Image border finding. Given a vector v of size h*w obtained through Canny edge detection, transpose v to get v2. Perform row search and column search on v1 and v2 respectively. Since there are mutations in the pixel point accumulation sum (the number of 255 values) on the left and right of the starting and ending points of the binary image, based on this idea, accurate border positioning of the original broken line / curve image is achieved. The results are as Figure 1 shown.

[0081] Step 2: Preprocess a given curve / broken line picture img

[0082] 2-1. Use the cvtColor function in the OpenCV library to grayscale the picture, making the values of the three channels of the picture the same. The value size is the sum of the three-channel pixels divided by 3.

[0083] 2-2. Use the adaptiveThreshold() function in the OpenCV library to perform adaptive binarization on the image. For pixels greater than the adaptive threshold, set the single-channel pixel value of its grayscale image to 255. Set the adaptiveMethod parameter to ADAPTIVE_THRESH_GAUSSIAN_C, the blockSize parameter to 15, and C to 4. The finally achieved adaptive binarization has the best effect. The final processing effect is as Figure 2 shown.

[0084] Step 3: Find the contours of the axis numbers

[0085] 3-1. Use the findContours function in the OpenCV library to find the contours of the preprocessed image generated in Step 2, and store all the contours in the contours array.

[0086] 3-2. Use the boundingRect function in the OpenCV library to traverse and return the smallest upright rectangle containing the input information, and store the coordinates of the upper left corner of its rectangle frame, as well as the width and height of the rectangle in the bounding_boxex list. The format of the list is: [x, y, w, h], where x and y are the horizontal and vertical coordinates of the upper left corner of the border, and w and h are the width and height of the rectangle frame respectively.

[0087] 3-3. Since the found rectangles may not be the ones we want, we need to filter them. We require that for the rectangle frames, x > x_boundary_min, y < y_boundary_max, and the values of w and h should not be greater than 20 pixels nor less than 8 pixels. Then, we store the [x, y, w, h] that meet the conditions in a new list a.

[0088] 3-4. For the list a, some two-digit numbers may be split into two numbers due to a relatively large separation distance during segmentation. Therefore, we need to make a judgment. If the distance between two rectangle frames is less than the set threshold, the two rectangle frames will be merged. The specific merging formula is: a[i] = [a[x1][0], a[x1][1], a[x2][0] - a[x1][0], max(a[x1][3], a[x2][3])] (Formula 1)

[0089] Step Four: Recognition of Axis Numbers

[0090] 4-1. In the list a obtained in the previous step, the numbers on the X and Y axes are stored in one list. Here, we separate and store the rectangle frames on the X and Y axes according to x_boundary_min and y_boundary_max. The elements in list a where x < x_boundary_min are stored in boxx1, and the elements in list a where y > y_boundary_max are stored in boxx2, which facilitates the subsequent recognition of the numbers on the X and Y axes respectively.

[0091] 4-2. For number recognition, for the pictures that determine the specific positions and contours of the numbers, we use the pytesseract.image_to_string function for the coordinates on the X and Y axes respectively.

[0092] 4-3. Processing of recognition results: Through step 4-2, we obtain a list l containing the recognition results. Each result in l is extracted and stored in list temp. We traverse temp. When encountering non-character variables such as spaces and tab characters, we discard them and splice adjacent character variables. Finally, the obtained strings are the recognized numerical values on the X and Y axes. After type conversion, the recognized numbers are respectively stored in the fifth dimension of boxx1 and boxx2. Now, the formats of boxx1 and boxx2 are as follows:

[0093] boxx = [x1, y1, x2, y2, num]

[0094] where x1, y1 and x2, y2 are the coordinate positions of the upper left and lower right corners of the rectangle frame respectively, and num is the number recognized in this step;

[0095] 4-4. The list obtained in step 4-3 is sorted in ascending order with the fifth dimension as the keyword, resulting in two new lists: result_x and result_y, which facilitate the calculation of the axis ratios in step five.

[0096] Step Five: Axis Ratio Calculation

[0097] 5-1. Subtract the fifth-dimensional elements of adjacent elements in result_x and result_y (recognized recognition). Taking result_x as an example, traverse the elements in it.

[0098] d = result_x[i + 1][4] - result_x[i][4] (Formula 2)

[0099] Use the difference d as the keyword of the dictionary, and the values of the dictionary are set as lists to record which rectangle it is. In this example, it is i, obtaining two dictionaries my_dict_x and my_dict_y;

[0100] 5-2. According to the keyword, that is, the possible d values, find the number of elements in the lists of the two dictionaries obtained in 5-1. Since recognition may sometimes be inevitably incorrect, using this method can avoid the problem that a small number of numbers are not recognized. Taking the x-axis as an example, the difference between the numbers on its coordinate is the key value corresponding to the longest values list in my_dict_x, denoted as Delta_x.

[0101] 5-3. On the basis of 5-2, taking the x-axis as an example, accumulate the d values of the corresponding elements in my_dict_x[Delta_x] to obtain Delta_sum_x. Finally, the axis ratio formula is:

[0102] x_ratio = Delta_sum_x / len(my_dictx_x[Delta_x]) (Formula 3)

[0103] Step Six: y-value Search

[0104] 6-1. Input the x value to be searched. Assume there is a reference point on the x-axis that needs to be correctly recognized with numbers. Therefore, an index m needs to be taken out from my_dict_x[Delta_x], and its coordinates and the actually recognized number are taken out from result_x[m]. The abscissa x of its pixel point symbol = (result_x[m][0] + result_x[m][2]) / 2, and the true value of the reference point standard_num_x = result[m][4]. So for the input x, its abscissa in the pixel coordinate system is

[0105] x' = x symbol +(x - standard_num_x)*x_ratio (Formula 4)

[0106] Since x' may be a floating - point number, in order to improve the accuracy, it is necessary to calculate the integer part and the decimal part separately

[0107] 6 - 2. By traversing the pixel points in the image, the background color in the picture is removed, and its grayscale pixel value is set to 255, that is, white. For x', x_int = int(x'), x_float = x' - x_int. For finding the corresponding y value of x', we adopt the method of traversing the pixels in this column of x_int to find the interval with the longest pixel length not equal to 255, which must be the curve interval we finally want to find. Taking the median value is the y value corresponding to the x_int we want to find

[0108] 6 - 3 For the x_float part, we search for the slope, find the slope near x_int, and then multiply x_float by the slope to make it up, forming the corresponding pixel point ordinate on the final y - axis

[0109] 6 - 4 Similarly, for the y - axis, a reference point is also needed. It needs to correctly identify the number. We need to take a subscript m from my_dict_y[Detla_y], and take its coordinates and the truly recognized number from result_y[m], which is the ordinate of the pixel point

[0110] y_symbol=(result_y[m][0]+result_y[m][2]) / 2(Formula 5)

[0111] True value of the reference point

[0112] standard_num_y = result_y[m][4](Formula 6)

[0113] So for the input y, its true value in the pixel coordinate system is:

[0114] x result = standard_num_y+(y symbol -(k x *x_float)) / y_ratio (Formula 7)

[0115] x result is the true value of the y - axis that we finally need to obtain

[0116] Example 1:

[0117] Step 1: Finding the border of the curve / polygon graph

[0118] Apply Canny edge detection to the original image p, set the parameters threshold1 and threshold2 to 50 and 50 respectively, and obtain a two-dimensional pixel vector v1 (binary result) with a clear curve contour and edge shape h*w. Transpose v1 to get v2, and perform row search and column search on v1 and v2 respectively. Since the pixel point accumulation sum (the number of 255 values) will mutate on the left and right of the start and end points of the binary image, based on this idea, accurate border positioning of the original line / curve image is achieved, and x_boundary_min, x_boundary_max, y_boundary_min, and y_boundary_max are obtained.

[0119] Example 2:

[0120] Step 2: Image preprocessing module

[0121] For the input image p, use the cvtColor function in the opencv library to perform grayscale processing on the image, making the values of the three channels of the image the same, and the value size is the sum of the three-channel pixels divided by 3. For the grayscale processed image, use the adaptiveThreshold() function in the opencv library to perform adaptive binarization. For values greater than the adaptive threshold, set the single-channel pixel value of its grayscale image to 255. Set the adaptiveMethod parameter to ADAPTIVE_THRESH_GAUSSIAN_C, the blockSize parameter to 15, and C to 4 to obtain the preprocessed image.

[0122] Example 3:

[0123] Step 3: Axis number contour search module

[0124] Use the findContours function in the opencv library to find the contours of the preprocessed image generated in Step 2, store all the contours in the contours array, use the boundingRect function in the opencv library to traverse and return the smallest upright rectangle containing the input information, and store the upper left corner coordinates of the rectangle frame, as well as the width and height of the rectangle in the bounding_boxex list, and perform screening on it, storing the screening results in the new list a. For the list a, some two-digit numbers may be split into two numbers due to a relatively large separation distance, so it is necessary to make a judgment. If the distance between the two rectangle frames is less than the set threshold, the two rectangle frames will be merged.

[0125] Example 4:

[0126] Step 4: Axis Number Recognition Module

[0127] For the result obtained in Step 3, store x and y separately. For the x and y coordinates, use the pytesseract.image_to_string function respectively, and set the config parameter to '--psm 6 --oem 3 - ctessedit_char_whitelist=0123456789'. The parameter 6 of psm assumes the image as a unified text block, which greatly improves the accuracy of number recognition. And process the recognition results, and store the obtained digital results into the fifth dimension of boxx1 and boxx2 respectively. Now the formats of boxx1 and boxx2 are as follows:

[0128] boxx = [x1, y1, x2, y2, num]

[0129] Finally, sort boxx1 and boxx2 in ascending order with the fifth dimension as the keyword to obtain two new lists: result_x and result_y.

[0130] Example 5:

[0131] Calculate the standardized Euclidean distance between the y-axis coordinate values recognized by the algorithm and the actual y-axis coordinate values. Let the true label of the data be y i ′. Since the range of the vertical axis coordinates is uncertain, the label and the calculation result are scaled before calculation, and the coefficient is k, which is the maximum value of the y-axis value of the line data, that is:

[0132]

[0133] Table 1 shows the results obtained by the method described in the text on line charts / curve charts

[0134] Line graph Curve graph Line graph / Curve graph d value 0.0952 0.0944 0.0966 .

Claims

1. A polyline / curve data extraction system based on image processing, characterized in that it includes a search module for the border of the curve / polyline graph, an image preprocessing module, an axis number contour search module, an axis number recognition module, an axis ratio calculation module, and a y-value search module; The search module for the border of the curve / polyline graph is implemented as follows: Apply canny edge detection to the input image to obtain a two-dimensional pixel vector v1 containing the clear curve contour of the input image and the edge shape h*w of the input image; Transpose the two-dimensional pixel vector v1 to obtain a transposed vector v2, and perform row search and column search on the two-dimensional pixel vector v1 and the transposed vector v2 respectively. Since the pixel point cumulative sum will mutate on the left and right of the start point and end point of the polyline after binarization, based on this, precise border positioning of the original polyline / curve image is realized to obtain x_boundary_min, x_boundary_max, y_boundary_min, and y_boundary_max; The preprocessing module of the image: Perform grayscale processing on the input image; The axis number contour search module: Perform contour search on the preprocessed image, find all the contours and store them in a list, traverse and return the information of the smallest positive rectangle containing the input information, that is, store the upper left corner coordinates of the rectangle frame of the smallest positive rectangle and the width and height of the rectangle in the list; Screen all the smallest positive rectangles in the list, and store the screening results in a new list a; After merging the new list a, obtain an x list and a y list; Use pytesserac to perform character recognition on the x list and the y list respectively, perform re-screening processing on the recognition results, store the screened digital results in the fifth dimension of the x list and the y list respectively, and finally sort the list in ascending order with the fifth dimension as the keyword to obtain two new lists: result_x, result_y; The axis ratio calculation module: Calculate the axis ratio for adjacent elements of the result_x and result_y lists respectively; The y-value search module: Input the x value to be searched, and find its true value in the pixel coordinate system.

2. A polyline / curve data extraction method based on image processing, which is applied to a polyline / curve data extraction system based on image processing as described in claim 1, characterized in that it includes the following steps: Step (1), search for the border of the curve / polyline graph; Step (2), preprocess the image; Step (3), search for the axis number contour; Step (4), recognize the axis number; Step (5), calculate the axis ratio; Step (6), through the input x value, search for the true value of this x value in the coordinate system.

3. According to the polyline / curve data extraction method based on image processing described in claim 2, characterized in that Step (1) is specifically implemented as follows: Apply canny edge detection to the input image to obtain a two-dimensional pixel vector v1 containing the clear curve contour of the input image and the edge shape h*w of the input image; Transpose the two-dimensional pixel vector v1 to obtain a transposed vector v2; Perform row search and column search on the two-dimensional pixel vector v1 and the transposed vector v2 respectively; Since there are sudden changes in the cumulative sum of pixel points on the left and right of the starting point and the ending point of the broken line in the binarized image, based on this, accurate border positioning is realized for the original broken line / curve image, and x_boundary_min, x_boundary_max, y_boundary_min, and y_boundary_max are obtained.

4. A method for extracting broken line / curve data based on image processing according to claim 3, characterized in that Step (2) is specifically implemented as follows: Perform grayscale processing on the input image: change the values of the three channels of the input image to the same value, and the value size is the sum of the three-channel pixels divided by 3; Perform adaptive binarization on the grayscale-processed image. For the image greater than the adaptive threshold, set the single-channel pixel value of its corresponding grayscale image to 255 to obtain the preprocessed image.

5. A method for extracting broken line / curve data based on image processing according to claim 4, characterized in that Step (3) is specifically implemented as follows: Perform contour search on the preprocessed image obtained in step (2), find all contours and store them in a list, traverse and return the information of the smallest upright rectangle containing the input information, that is, store the upper left corner coordinates of the rectangle frame of the smallest upright rectangle and the width and height of the rectangle in the list; Screen all the smallest upright rectangles in the list and store the screening results in a new list a.

6. A method for extracting broken line / curve data based on image processing according to claim 5, characterized in that There may be two-digit numbers in the new list a that are split into two numbers due to the long segmentation distance. Therefore, it is necessary to further judge the new list a; if the distance between two rectangle frames is less than the set threshold, the two rectangle frames will be merged to obtain the final required list x and list y; the specific merging formula is: a[i] = [a[x1][0], a[x1][1], a[x2][0] - a[x1][0], max(a[x1][3], a[x2][3])] (1) Among them, a[x1][0] represents the upper left corner x, a[x1][1] represents the upper left corner y, a[x2][0] - a[x1][0] represents the length of the rectangle, and max(a[x1][3], a[x2][3]) represents the width of the rectangle; Specifically, according to x_boundary_min and y_boundary_max obtained in step (1), store the rectangle frame a[i][0] < x_boundary_min in list a into list y, which are the numbers on the Y axis; store a[i][1] > y_boundary_max in list a into list y, which are the numbers found on the X axis.

7. A method for extracting broken line / curve data based on image processing according to claim 6, characterized in that Step (4) is specifically implemented as follows: Perform character recognition on the two merged lists x and y using pytesserac respectively; The recognition results are screened. The digital results obtained after screening are respectively stored in the fifth dimension of list x and list y. The format of the list is as follows: list=[x1,y1,x2,y2,num] (2) Finally, the list is sorted in ascending order with the fifth dimension as the keyword to obtain two new lists: result_x and result_y.

8. A method for extracting polyline / curve data based on image processing according to claim 7, characterized in that The screening process is as follows: Since character recognition may recognize spaces, line breaks, and incorrect characters, it is necessary to screen each digital rectangle, select the numbers therein and perform merging processing, and convert the character-form data into integer-type data, which are respectively stored in the fifth dimension of list result_x and list result_y.

9. A method for extracting polyline / curve data based on image processing according to claim 7 or 8, characterized in that The specific implementation process of step (5) is as follows: Calculate the coordinate axis ratio for adjacent elements in list result_x and list result_y respectively; The coordinate axis ratio calculation of the result_x list is to subtract the fifth-dimensional elements of two adjacent elements in the result_x list in turn, and traverse the elements therein: d=result_x[i+1][4]-result_x[i][4] (3) where i is greater than or equal to 0, representing the i-th rectangle in the result_x list; The difference d is used as the keyword of the dictionary, and the values of the dictionary are set as lists, recording which rectangle it is. In this example, it is i; Similarly, the same operation is performed on the result_y list to finally obtain dictionaries my_dict_x and my_dict_y; In the two dictionaries, find the number of elements in their lists according to the keyword, that is, the possible d value; Accumulate the d values of the corresponding elements in my_dict_x[Delta_x] to obtain Delta_sum_x. Finally, the coordinate axis ratio formula is: x_ratio=Delta_sum_x / len(my_dictx_x[Delta_x]) (4) where Delta_x represents the keyword with the largest length of the list corresponding to the value in dictionary my_dict_x; x_ratio represents how many pixels the real length on the coordinate axis is in the pixel image; Delta_sum_x represents the accumulated value of the difference between the left boundaries of the corresponding adjacent rectangles with the keyword Delta_x in the dictionary.

10. A method for extracting polyline / curve data based on image processing according to claim 9, characterized in that The specific implementation process of step (6) is as follows: Input the x value to be searched. Assume there is a reference point on the X-axis. Since the number needs to be correctly identified, an index m is taken from my_dict_x[Delta_x], and its coordinates and the truly identified number are taken from result_x[m]. The abscissa x of the pixel point symbol =(result_x[m][0] + result_x[m][2]) / 2, and the true value standard_num_x of the reference point = result[m][4]; therefore, for the input x value, its abscissa in the pixel coordinate system is: x' = x symbol +(x - standard_num_x)*x_ratio (5) Since x' may be a floating point number, it is necessary to calculate the integer part and the decimal part separately; By traversing the pixel points in the image, the background color in the image is removed, and its grayscale pixel value is set to 255, that is, white. For the abscissa x', x_int = int(x'), x_float = x' - x_int. To find the corresponding y value for the abscissa x', it is necessary to first perform a color search through the color search function to obtain the main color of the curve: line_color. Then, the method of traversing the pixels in the x_int column is adopted. When traversing from top to bottom to the first pixel whose value is line_color, this is the upper boundary of y to be searched. Similarly, when traversing from bottom to top to the first pixel whose value is line_color, this is the lower boundary of y to be searched. Taking the median value is the y value corresponding to x_int to be found; for the x_float part, the slope is searched, the slope adjacent to x_int is found, and then x_float is multiplied by the slope and added back to form the ordinate of the corresponding pixel point on the final y-axis; similarly, a reference point is also required for the y-axis. It needs to correctly identify the number. It is necessary to take a subscript m from my_dict_y[Delta_y], and take its coordinates and the truly recognized number from result_y[m], and the ordinate on its pixel point True value of the reference point: standard_num_y = result_y[m][4](7) Therefore, for the input x value, its true value in the coordinate system is: x result = standard_num_y+(y symbol -(k x *x_float)) / y_ratio(8) x result is the true value of the y-axis to be obtained.

Images