Table or text line processing method, bill processing method, device and storage medium

Abstract

The present invention belongs to the field of document recognition technology, and specifically relates to a table or text line processing method, bill processing method, device and storage medium. It mainly includes: obtaining points on two relative boundaries of the object to be processed, and counting them into a first boundary point set and a second boundary point set respectively; obtaining a first boundary curve according to the first boundary point set, and obtaining a second boundary curve according to the second boundary point set; extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively; aligning each source control point to obtain a corresponding target control point; obtaining an interpolation function according to each source control point, target control point and minimum energy function; applying the interpolation function to process the text line area corresponding to the object to be processed. The thin plate spline interpolation technology is used in the processing of curved text lines or curved tables, which effectively solves the problem in the prior art that when using machine learning methods to correct curved text, the correction effect is limited by the training samples.

Description

Technical Field

[0001] The present invention belongs to the technical field of document recognition, and in particular relates to a table or text line processing method, a bill processing method, a device and a storage medium. Background Art

[0002] Over the past few years, computer vision research has focused on convolutional neural networks (often abbreviated as ConvNet or CNN). Because convolutional neural networks can use local operations to perform hierarchical abstraction of representations, convolutional neural networks have achieved good performance on a wide range of classification and regression tasks. The success of convolutional architectures in computer vision is mainly driven by two key design ideas: First, CNNs exploit the 2D structure of images, and since pixels in adjacent areas are usually highly correlated, CNNs do not need to use one-to-one connections between all pixel units (which most neural networks do), but can use grouped local connections. Second, the CNN architecture relies on feature sharing, so each channel (i.e., output feature map) is generated by convolution with the same filter at all locations.

[0003] Spatial Transformer Networks (STNs) are a convolutional neural network architecture model proposed by Jaderberg et al. They consist of three components: a localization network, a grid generator, and a sampler. STNs can be used in the input layer or inserted after a convolutional layer or other layers without changing the internal structure of the original CNN model. For an input image, STNs first use a localization network to predict the required transformation. After several consecutive layers of calculations (including convolutions and fully connected computations), the grid generator and sampler transform the image. The transformed image is then fed into a CNN for classification. The network generator uses bilinear interpolation to generate the network, while the sampler employs a formally differentiable image sampling method, allowing the entire network to maintain end-to-end backpropagation (BP) training. The trained network can correct curved networks.

[0004] Existing deep learning correction algorithms based on STNs can address the problem of text curvature in real-world applications to a certain extent. However, these algorithms often use CNNs as a black box. While this approach can produce excellent results, the underlying mechanism of its effectiveness is unclear, resulting in incomplete image correction and distortion, which seriously fails to meet the requirements of scientific research. Furthermore, training a usable deep learning network requires a vast amount of data. However, in real-world applications, text types vary greatly, and the number of examples of each type of curvature is relatively small. Collecting the required data for training a network is extremely difficult, and the annotation cost is also very high. Summary of the Invention

[0005] The present invention proposes a table or text line processing method, a bill processing method, a device and a storage medium, and applies thin plate spline interpolation technology to the processing of curved text lines or curved tables, effectively solving the problem in the prior art of using machine learning methods to correct curved text, in which the correction effect is limited by training samples.

[0006] In the first aspect of the present invention, the present application provides a method for processing a table or text line, comprising the following steps: obtaining points on two relative boundaries of the object to be processed, and counting them into a first boundary point set and a second boundary point set respectively; obtaining a first boundary curve based on the first boundary point set, and obtaining a second boundary curve based on the second boundary point set; extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively; aligning each source control point to obtain a corresponding target control point; obtaining an interpolation function based on each source control point, target control point and minimum energy function; and applying the interpolation function to process the text line area corresponding to the object to be processed.

[0007] In the present invention, the points on the two relative boundaries of the object to be processed are obtained and counted into the first boundary point set and the second boundary point set respectively. One implementation method includes the following steps: traversing the text row area of ​​the object to be processed in a first preset order, the first preset order is from left to right for each column and from top to bottom for each column; counting the first pixel point after the change in the brightness indication value of the pixel points in each column reaches the preset difference for the first time into the first boundary point set, and counting the last pixel point before reaching the preset difference for the last time into the second boundary point set; or traversing the text row area of ​​the object to be processed in a second preset order, the second preset order is from top to bottom for each row and from left to right for each row; counting the first pixel point after the change in the brightness indication value of the pixel points in each row reaches the preset difference for the first time into the first boundary point set, and counting the last pixel point before reaching the preset difference for the last time into the second boundary point set.

[0008] Furthermore, in the table or text line processing method provided by the present invention, the points on the two boundaries of the object to be processed are obtained and respectively counted into the first boundary point set and the second boundary point set, and also includes: judging the direction of the object to be processed; when the object to be processed is horizontal, traversing the text line area in a first preset order; when the object to be processed is vertical, traversing the text line area in a second preset order.

[0009] Furthermore, in the table or text line processing method provided by the present invention, the light and dark indication value is one of a grayscale value, a red component value, a blue component value and a green component value; or, when the text line area is a binary text line area, the light and dark indication value is 1 or 0, and at this time, the preset difference value is 1.

[0010] Furthermore, in the table or text line processing method provided by the present invention, before executing the step of obtaining the points on the two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively, it also includes: performing progressive expansion network processing on the text line area corresponding to the object to be processed to obtain the object to be processed.

[0011] Furthermore, in the table or text line processing method provided by the present invention, before executing the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set, it also includes: fitting the points in the first boundary point set with the regression line equation to obtain the corresponding first regression line equation; or fitting the points in the second boundary point set with the regression line equation to obtain the corresponding second regression line equation; judging whether the goodness of fit of the obtained first regression line equation or the second regression line equation meets a first goodness threshold; when the first goodness threshold is met, directly outputting the text line area of ​​the object to be processed; when the first goodness threshold is not met, continuing to execute the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set;

[0012] Alternatively, the regression line equation is used to fit the points in the first boundary point set and the points in the second boundary point set respectively to obtain the corresponding third regression line equation and fourth regression line equation; it is determined whether the goodness of fit of the third regression line equation and the fourth regression line equation both meet the second goodness threshold; when the goodness of fit of the third regression line equation and the goodness of fit of the fourth regression line equation both meet the second goodness threshold, the text line area of ​​the object to be processed is directly output; when the goodness of fit of the third regression line equation and / or the goodness of fit of the fourth regression line equation does not meet the second goodness threshold, the method of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set is continued.

[0013] The regression line equation is: y=φ(x)=a0+a1x.

[0014] Furthermore, in the table or text line processing method provided by the present invention, the first boundary curve is obtained according to the first boundary point set, and the second boundary curve is obtained according to the second boundary point set, including: using a method of successively increasing the highest order of the polynomial, using polynomials to fit the points in the first boundary point set and the points in the second boundary point set, respectively, to obtain the first boundary curve and the second boundary curve whose fitting goodness meets the third goodness threshold, specifically including: using polynomials to fit the points in the first boundary point set to obtain the corresponding polynomial boundary equation; judging whether the fitting goodness of the obtained polynomial boundary equation meets the third goodness threshold; when the fitting goodness of the obtained polynomial boundary equation meets the third goodness threshold, determining that the polynomial boundary equation is the first boundary curve, and using the polynomial to fit the points in the second boundary point set to obtain the second boundary curve; when the fitting goodness of the obtained polynomial boundary equation does not meet the third goodness threshold, switching to a higher order polynomial and repeating the above steps.

[0015] Furthermore, in the table or text line processing method provided by the present invention, the first goodness threshold, the second goodness threshold, and the third goodness threshold are all within any range of 0.7-1;

[0016] Goodness of fit (G 2 ) is calculated as:

[0017]

[0018] Among them, SSR is the regression sum of squares, which refers to the impact of changes in the independent variable x on the changes in the value of the dependent variable y; SST is the total sum of squares, which refers to the total deviation of the n observed values ​​of the dependent variable from its mean; SSE is the residual sum of squares, which reflects the impact of other factors other than x on the value of y, SSR = SST + SSE; represents the fitted value calculated using the formula, is the mean of the fitted points.

[0019] Furthermore, in the table or text line processing method provided by the present invention, when using a regression direct equation or polynomial to fit the points within the first boundary point set and the points within the second boundary point set, the two most boundary points in the first boundary point set and the two most boundary points in the second boundary point set do not participate in the calculation.

[0020] Furthermore, in the table or text line processing method provided by the present invention, the extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively includes: extracting a preset number of first source control points and second source control points from the first boundary curve and the second boundary curve respectively, each first source control point corresponding to each second source control point one by one; wherein the second source control point is the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve; or, the first source control point is the intersection of the normal of the first boundary curve passing through the second source control point and the first boundary curve.

[0021] Furthermore, in the table or text line processing method provided by the present invention, a preset number of first source control points and second source control points are respectively extracted from the first boundary curve and the second boundary curve, and each first source control point corresponds to each second source control point one-to-one, including: judging the bending direction of the object to be processed; when the bending direction is bending toward the second boundary point set direction, extracting a preset number of second source control points from the second boundary curve; extracting first source control points corresponding to each second source control point on the first boundary curve, the first source control point being the intersection of the normal of the first boundary curve passing through the second source control point and the first boundary curve; when the bending direction is bending toward the first boundary point set direction, extracting a preset number of first source control points from the first boundary curve; extracting second source control points corresponding to each first source control point on the second boundary curve, the second source control point being the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve.

[0022] Furthermore, in the table or text line processing method provided by the present invention, the determining of the bending direction of the object to be processed includes: calculating the distance between the first boundary point counted in the first boundary point set and the boundary of the text line region, and the distance between the first second boundary point counted in the second boundary point set and the boundary of the text line region, and comparing them; when the distance between the first boundary point and the boundary of the text line region is closer, determining that the object to be processed is curved in the direction of the second boundary point set; when the distance between the second boundary point and the boundary of the text line region is closer, determining that the object to be processed is curved in the direction of the first boundary point set;

[0023] Alternatively, the time of the first first boundary point included in the first boundary point set is compared with the time of the first second boundary point included in the second boundary point set. When the time of the first first boundary point is earlier than the time of the first second boundary point, it is determined that the object to be processed is curved toward the second boundary point set; when the time of the first second boundary point is earlier than the time of the first first boundary point, it is determined that the object to be processed is curved toward the first boundary point set.

[0024] Alternatively, the distances between each first boundary point in the first boundary point set are calculated, and the maximum distance therebetween is extracted and recorded as the first distance; the distances between each second boundary point in the second boundary point set are calculated, and the maximum distance therebetween is extracted and recorded as the second distance; the first distances are compared with the second distances; when the first distance is greater than the second distance, it is determined that the object to be processed is curved toward the second boundary point set; when the second distance is greater than the first distance, it is determined that the object to be processed is curved toward the first boundary point set;

[0025] Alternatively, the direction of the object to be processed is determined. When the object to be processed is horizontal, the leftmost point in the first boundary point set and the second boundary point set is extracted and recorded as the text line edge point; when the object to be processed is vertical, the topmost point in the first boundary point set and the second boundary point set is extracted and recorded as the text line edge point; the distance between the text line edge point and the first boundary curve and the second boundary curve is calculated and compared. When the distance between the text line edge point and the first boundary curve is greater than the distance between the text line edge point and the second boundary curve, it is determined that the object to be processed is curved in the direction of the first boundary point set; when the distance between the text line edge point and the second boundary curve is greater than the distance between the text line edge point and the first boundary curve, it is determined that the object to be processed is curved in the direction of the second boundary point set.

[0026] Furthermore, in the table or text line processing method provided by the present invention, the aligning of the source control points to obtain the corresponding target control points includes: aligning the first coordinates of the first source control points to obtain the first coordinates of the target control points corresponding to the first source control points; aligning the first coordinates of the second source control points to obtain the first coordinates of the target control points corresponding to the second source control points; and aligning the second coordinates of the corresponding groups of first source control points and second source control points one by one to obtain the second coordinates of the target control points corresponding to the groups of first source control points and second source control points.

[0027] Furthermore, in the table or text row processing method provided by the present invention, the first preset order is from left to right for each column and from top to bottom for each column; the second preset order is from top to bottom for each row and from left to right for each row; when the text row area of ​​the object to be processed is traversed in the first preset order to obtain the first boundary point set and the second boundary point set, the first coordinate is the horizontal coordinate and the second coordinate is the vertical coordinate; when the text row area of ​​the object to be processed is traversed in the second preset order to obtain the first boundary point set and the second boundary point set, the first coordinate is the vertical coordinate and the second coordinate is the horizontal coordinate.

[0028] Furthermore, in the table or text line processing method provided by the present invention, obtaining the interpolation function according to each source control point, target control point and minimum energy function includes:

[0029] Assume that in the space region R 2 There are q known points in the internal distribution P i , i=1,2,…,q, its coordinates are expressed as: (x i ,y i ,z(x i ,y i )), if z(x i ,y i ) has a second-order continuous derivative, then the energy function is expressed as:

[0030]

[0031] For TPS, it can be obtained by minimizing the energy function:

[0032] Z tps =argminE

[0033] Since the data points are expressed in discrete table form during interpolation, the function to be minimized in formula (1) is expressed as:

[0034]

[0035] In formula (2), ||·|| represents the Euclidean norm, c i is the coefficient , is the kernel function of the thin plate interpolation along the strip, and its value is: Among them, r i =(xx i ) 2 +(yy i ) 2 .

[0036] Furthermore, in the table or text line processing method provided by the present invention, the application of the interpolation function to process the text line area corresponding to the object to be processed includes: determining the correspondence between each pixel point in the target image and the pixel points of the text line area of ​​the object to be processed based on the interpolation function; and filling the pixel values ​​of the pixel points of the text line area of ​​the object to be processed into each pixel point of the target image according to the correspondence.

[0037] In a second aspect, the present invention provides a bill processing method, which processes bills using the above table or text line processing method.

[0038] Furthermore, the bill processing method provided by the present invention also includes identifying the processed text lines to obtain the face content of the bill.

[0039] In a third aspect, the present invention provides a device for processing a table or a text line, the device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and the device implementing the steps of the text line processing method when the computer instructions are executed by the processor.

[0040] In a fourth aspect, the present invention provides a computer storage medium having a computer program stored thereon, which implements the steps of the text line processing method when the computer program is executed by a processor.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. The present invention provides a method for processing a table or text line, which uses a method of obtaining boundary curves of two relative boundaries of an object to be processed, selecting a certain number of source control points from the boundary curves, and obtaining corresponding target control points. An interpolation function is then obtained based on the minimum energy function of the source control points, the target control points, and thin plate spline interpolation, and the interpolation function is then applied to process the text line area corresponding to the object to be processed. Utilizing the characteristic that the degree of curvature of a text line is greatest at its relative boundaries, a series of methods are used to determine the relative boundaries of the object to be processed, and control points are then selected on the boundaries to obtain an interpolation function for the object to be processed. After the object to be processed is processed by the interpolation function, a curved text line can be corrected into a non-curved text line, and the corrected text line can be ensured to be undistorted, without affecting subsequent operations such as text line recognition, thereby achieving effective correction of the text line. Moreover, the acquisition of this interpolation function does not rely on training, which effectively solves the problem in the existing technology that before using machine learning methods to correct curved text lines, a large number of samples need to be used to train the correction function. However, there are many types of text curvature, and samples of various types are difficult to obtain, which makes the obtained correction function easily fall into the local minimum during the iteration process, resulting in poor correction effect on the text.

[0043] 2. The table or text line processing method provided by the present invention adopts the method of obtaining the points on the two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively. Based on the characteristic that the brightness indication value of the text line and the brightness indication value of the background are relatively different, the first and last pixel points in each column (or each row) whose brightness indication values ​​meet the text characteristics are extracted as the two boundary points of the text line on the column (or row). According to this method, the points on the corresponding boundaries of the object to be processed can be accurately found by traversing the entire text line area. In addition, the method of determining the boundary points based on the change of the brightness indication value does not need to determine in advance which of the brightness indication values ​​between the text line and the background is "large" and which is "small", making the method of obtaining the points on the two relative boundaries of the object to be processed more widely applicable.

[0044] 3. The table or text line processing method provided by the present invention, when using the first boundary point set and the second boundary point set obtained, also includes first determining the direction of the object to be processed, and then selecting the traversal order to be adopted when obtaining the first boundary point set and the second boundary point set according to the direction. Taking advantage of the fact that the lengths of the upper and lower boundaries of the horizontal text line are greater than the lengths of the left and right boundaries, the curvature of the text line is more obvious at the upper and lower boundaries; and the fact that the lengths of the left and right boundaries of the vertical text line are greater than the lengths of the upper and lower boundaries, the curvature of the text line is more obvious at the left and right boundaries. By first determining the direction of the object to be processed, and then selecting and obtaining a boundary point set with a more obvious curvature, the obtained interpolation function has a better correction effect on the object to be processed.

[0045] 4. The table or text line processing method provided by the present invention adopts the brightness indication value as one of the grayscale value, red component value, blue component value and green component value; or, when the text line area is a binary text line area, the brightness indication value is 1 or 0, in which case the preset difference value is designed to be 1; the brightness indication value can be selected based on the characteristics of the pixel points of the acquired object image to be processed. When only the text image to be processed having one or more elements of the grayscale value, red component value, blue component value and green component value can be obtained, the judgment can be made based on the obtained brightness indication value; there is no need to convert the brightness indication value into other types of brightness indication values. Of course, the obtained brightness indication value can also be converted, such as converting the grayscale value to obtain a binary value represented by 1 and 0, as long as the obtained brightness indication value can express the difference between the text line and the corresponding background, so that the text line processing method of the present invention has greater applicability. When the text is dark and the background is light, the binary image corresponds to 1 for the text and 0 for the background. When the text is light and the background is dark, the binary image corresponds to 0 for the text and 1 for the background. Regardless of the lightness of the text or background, the corresponding pixel values ​​can be selected to determine which pixels belong to the first and second boundary point sets. Furthermore, the difference between the text lines and the background in the binarized image is larger, making it easier to accurately select the boundary pixels of the text lines.

[0046] 5. The method for processing tables or text lines provided by the present invention, before executing the step of obtaining points on the two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively, also includes performing progressive expansion network processing on the text line area corresponding to the object to be processed to obtain the design of the object to be processed. After the processing, the text line becomes an arc-shaped frame that bends along the bending direction of the text line. The edge of the arc-shaped frame is close to the edge of the text line. By obtaining the set of edge points of the arc-shaped frame (the first boundary point set and the second boundary point set), the set of edge lines where the upper and lower edges of the original text line are located can be obtained. No matter how much the original text line is bent, the first boundary point set and the second boundary point set can accurately and comprehensively represent the degree of bending of the text line.

[0047] 6. The method for processing a table or text line provided by the present invention adopts a method of fitting the points within the first boundary point set and / or the points within the second boundary point set using a regression line equation to obtain the corresponding regression line equation before executing the steps of obtaining the first boundary curve based on the first boundary point set and obtaining the second boundary curve based on the second boundary point set; determining whether the goodness of fit of the obtained regression line equation meets the corresponding goodness threshold, and directly outputting the text line area of ​​the object to be processed when the goodness of fit meets the corresponding goodness threshold; that is, before correcting the object to be processed, determining whether the text line is straight; if the object to be processed is straight, no correction processing is performed. This method realizes automatic determination of whether the object to be processed is a non-curved text line, eliminating the need for manual determination of whether the object to be processed is curved; for non-curved text lines, the correction step is omitted and the text line area is directly output, thereby increasing the processing speed of the object to be processed and also saving computational effort.

[0048] 7. The method for processing tables or text lines provided by the present invention adopts the method of obtaining the first boundary curve based on the first boundary point set and obtaining the second boundary curve based on the second boundary point set, including using a method of successively increasing the highest order of the polynomial to fit the points in the first boundary point set and the points in the second boundary point set with polynomials, respectively, to obtain the design of the first boundary curve and the second boundary curve whose fitting goodness meets the third goodness threshold; so that the obtained boundary curve of the object to be processed best conforms to the boundary condition of the text line to ensure the correction effect on the text line.

[0049] 8. The method for processing a table or text line provided by the present invention adopts the method of extracting a corresponding number of source control points on the first boundary curve and the second boundary curve, including: extracting a preset number of first source control points and second source control points from the first boundary curve and the second boundary curve, respectively, with each first source control point corresponding to each second source control point; wherein the second source control point is the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve; or, the first source control point is the intersection of the normal of the first boundary curve passing through the second source control point and the first boundary curve; based on the degree of bending of the two relative boundary lines and the change in the position of each point on the boundary line when the text line is bent, the source control points are determined in a tangent manner to further ensure the correction effect of the text line and effectively improve the accuracy of the correction of curved text.

[0050] 9. The table or text line processing method and device provided by the present invention, after determining the boundaries of a curved image, utilizes a thin plate spline interpolation method to determine source and target control points. The interpolation function is then derived from the source and target control points. The interpolation function is then applied to automatically correct the text, resulting in a complete, clear, corrected image. The present invention does not require training and can directly correct curved text or images, overcoming the drawbacks of a small curved text database and the difficulty in obtaining training data. This method is simple and easy to use, resulting in complete image correction. Furthermore, the method is unaffected by the degree or direction of text curvature and has a wide range of applications, particularly in the invoice industry. Applying the text line processing method to the bill industry can effectively improve the image clarity of the corrected bill.

[0051] 10. Each step of the present invention is described and explained using a corresponding mathematical model (mathematical formula). The process is clear and understandable, with better interpretability, making it more suitable for the field of theoretical research and providing a deep theoretical explanation for the study of the correction of curved text. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is an exemplary flow chart of a method for processing a table or text line in embodiment 1 of the present invention;

[0053] Figure 2 is an exemplary flow chart of a method for processing a table or text line in embodiment 2 of the present invention;

[0054] Figure 3 is a schematic diagram of a text line area of ​​an object to be processed before correction in the method for processing a table or text line in Example 2 of the present invention;

[0055] Figure 4 is a schematic diagram of an object to be processed obtained through progressive expansion network processing in the method for processing a table or text line in embodiment 2 of the present invention;

[0056] Figure 5 is a schematic diagram of a text line area of ​​an object to be processed after correction in the table or text line processing method in Example 2 of the present invention;

[0057] Figure 6 It is a structural diagram of the table or text line processing device in the present invention. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined, will not be interpreted in an idealized or overly formal sense. It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this application may be combined with each other.

[0060] Example 1

[0061] like Figure 1 As shown, embodiment 1 of the present invention provides a method for processing a table or a text line, comprising the following steps:

[0062] S11: Obtain points on two opposite boundaries of the object to be processed, and count them into the first boundary point set and the second boundary point set respectively;

[0063] S12: obtaining a first boundary curve according to the first boundary point set, and obtaining a second boundary curve according to the second boundary point set;

[0064] S13: extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively;

[0065] S14: Align each source control point to obtain the corresponding target control point;

[0066] S15: obtaining an interpolation function according to each source control point, target control point and minimum energy function;

[0067] S16: Applying an interpolation function to process the text line area corresponding to the object to be processed.

[0068] In the present invention, the points on the two relative boundaries of the object to be processed are obtained and counted into the first boundary point set and the second boundary point set respectively. One implementation method includes the following steps:

[0069] Traverse the text line area of ​​the object to be processed in a preset order;

[0070] The first pixel in each column after the change in the brightness indicator value reaches a preset difference for the first time is counted as the first boundary point set, and the last pixel before the change reaches the preset difference for the last time is counted as the second boundary point set; alternatively, the first pixel in each row after the change in the brightness indicator value reaches a preset difference for the first time is counted as the first boundary point set, and the last pixel before the change reaches the preset difference for the last time is counted as the second boundary point set. The change in the brightness indicator value of a pixel refers to the absolute value of the difference between the brightness indicator values ​​of the two pixels being compared.

[0071] In the present invention, traversing the text line area of ​​the object to be processed in a preset order includes: traversing the text line area in a first preset order, the first preset order is from left to right for each column, from top to bottom for each column, traversing one column and then traversing the next column or multiple columns at the same time; or traversing the text line area in a second preset order, the second preset order is from top to bottom for each row, from left to right for each row, traversing one row and then traversing the next row or multiple rows at the same time. Either of the first preset order and the second preset order can find the corresponding boundary points, the first preset order obtains the upper and lower boundary points of the text line, and the second preset order obtains the left and right boundary points of the text line.

[0072] In this embodiment, obtaining points on two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively includes: traversing the text line area of ​​the object to be processed in a first preset order, where the first preset order is from left to right for each column and from top to bottom for each column; counting the first pixel point after the change in the brightness indicator value of the pixel points in each column reaches a preset difference for the first time into the first boundary point set, and counting the last pixel point before the change reaches the preset difference for the last time into the second boundary point set;

[0073] Alternatively, the text line area of ​​the object to be processed is traversed in a second preset order, where the second preset order is from top to bottom for each line and from left to right for each line; the first pixel point after the change in the brightness indicator value of the pixel point in each line reaches the preset difference for the first time is counted into the first boundary point set, and the last pixel point before the change in the preset difference for the last time is counted into the second boundary point set.

[0074] During specific implementation, the points on the two boundaries of the object to be processed are obtained and counted into the first boundary point set and the second boundary point set respectively, and also include: judging the direction of the object to be processed; when the object to be processed is horizontal, traversing the text line area in the first preset order; when the object to be processed is vertical, traversing the text line area in the second preset order. Judging the direction of text lines in images with text lines is a prior art. For example, the width and height of the text lines can be compared. When the width is greater than the height, the text line is usually horizontal, and when the height is greater than the width, the direction of the text line is usually vertical. Of course, the width and height are relative to the current placement of the image where the text line is located. Those skilled in the art can choose a method for judging the direction of the object to be processed according to actual needs, and will not be elaborated here.

[0075] In a specific implementation, the brightness indication value is one of a grayscale value, a red component value, a blue component value, and a green component value; or, when the text line area is a binary text line area, the brightness indication value is 1 or 0, and in this case, the preset difference value is 1. The brightness indication value can be selected based on the characteristics of the pixel points of the obtained image of the object to be processed. When the image of the object to be processed is a color image, one of the red component value, the blue component value, and the green component value can be used. There is no need to convert the color image into other types of images. In this case, the preset difference value can be selected from any value between 10-251. Of course, the obtained image can also be converted, such as converting the color image into a grayscale image. In this case, the preset difference value can be selected from any value between 10-251; or, the grayscale value can be converted to obtain a binary image represented by 1 and 0. As long as the brightness indication value obtained can express the difference between the text line and the background, of course, the greater the difference between the text line and the background, the better.

[0076] When the text line is dark and the background is light, the text line in the binary image corresponds to 1 and the background is 0; when the text line is light and the background is dark, the text line in the binary image corresponds to 0 and the background is 1. Regardless of the color difference between the text line and the background, the corresponding pixel values ​​can be selected to determine the pixel points belonging to the first boundary point set and the second boundary point set. Moreover, after the binary processing, the difference between the text line and the background is larger, which is more conducive to accurately selecting the boundary pixel points of the text line. Therefore, when the image corresponding to the area of ​​the object to be processed in the binary form with a white background and black text line is obtained, the points on the two relative boundaries of the object to be processed are respectively counted into the first boundary point set and the second boundary point set. This can also be achieved by the following steps: for each column, from left to right, for each column, from top to bottom, each pixel point in the text line area is traversed sequentially; when the pixel value in each column changes from 0 to 1 for the first time, the pixel point corresponding to 1 is counted into the first boundary point set; when the pixel value changes from 1 to 0 for the last time, the pixel point corresponding to 1 is counted into the second boundary point set. Alternatively, when an image corresponding to the area of ​​the object to be processed is obtained in a binary form with a black background and white text lines, the specific method for obtaining the points on the two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively is: when the pixel value in each column changes from 1 to 0 for the first time, the pixel point corresponding to 0 is counted into the first boundary point set; when the pixel value changes from 0 to 1 for the last time, the pixel point corresponding to 0 is counted into the second boundary point set.

[0077] The object to be processed in this application can be a table image such as an invoice, or a text line image. The text line area to be processed is the text line and its adjacent background area, or the table and its adjacent background area. Typically, the text line area corresponding to the object to be processed is the original image, but it can also be a grayscale image or a single-channel RGB image. This application uses pixels, the smallest unit representing an area in an image, as the processing object, to accurately find the edges of characters in a text line in the image. Of course, a fixed number of pixels can also be used as a processing unit to reduce the amount of calculation.

[0078] The present invention can be used to process text lines and can also be used to process tables. When processing text lines, if the degree of curvature of the text line is too large or the characters in the text line are not characters in the form of a mouth-shaped frame, then the boundary points of the text line obtained by directly adopting the above-mentioned method of traversing the image are usually not the boundary points of the text line. Therefore, in a specific implementation, before executing the step of obtaining the points on the two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively, it also includes: performing progressive expansion network processing on the text line area corresponding to the object to be processed to obtain the object to be processed. Progressive expansion network processing (full name in English: Shape Robust Text Detection with Progressive ScaleExpansion Network, English abbreviation PSENet) is an existing technology. This processing can accurately locate curved text, and the segmented curved text covers the edge of the text line, which is equivalent to putting an arc frame on the entire object to be processed. The boundary points of the text line can be obtained by executing the above steps to extract boundary points on the basis of the arc frame.

[0079] The object to be processed may be curved or uncurved. Curved text lines are not conducive to subsequent processing such as recognition, and the greater the degree of curvature, the greater the impact. Straight text lines basically do not affect subsequent processing such as recognition. Therefore, after determining the position of the text line, it is best to first determine whether the text line is straight or curved. If it is curved, then extract the source control points and target control points and use an interpolation function based on thin plate spline interpolation to process the text line area to correct the text line. Based on the characteristic that the two relative boundaries of the object to be processed are also curved when it is curved, it is possible to choose to determine whether the line containing the points in the first boundary point set and / or the second boundary point set is straight or curved. The following introduces several feasible methods for determining whether the object to be processed is curved.

[0080] In a specific implementation, before executing the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set, it also includes: fitting the points in the first boundary point set with the regression line equation to obtain the corresponding first regression line equation; judging whether the goodness of fit of the obtained first regression line equation meets the first goodness threshold; when it meets the first goodness threshold, the object to be processed is not curved, and the text line area of ​​the object to be processed can be directly output; when it does not meet the first goodness threshold, the object to be processed is curved, and it is necessary to continue to execute the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set to correct the object to be processed;

[0081] or,

[0082] Fitting the points within the second boundary point set with the regression line equation to obtain a corresponding second regression line equation; determining whether the goodness of fit of the obtained second regression line equation meets a first goodness threshold; when the first goodness threshold is met, the object to be processed is not curved, and the text line region of the object to be processed can be directly output; when the first goodness threshold is not met, the object to be processed is curved, and it is necessary to continue performing the steps of obtaining a first boundary curve based on the first boundary point set, obtaining a second boundary curve based on the second boundary point set, and so on, to correct the object to be processed;

[0083] When one boundary of the object to be processed is straight, the credibility that the object to be processed is straight is low. In order to improve the accuracy and credibility of the judgment result, it is possible to judge whether the two relative boundaries of the object to be processed are used. At this time, the method for judging whether the object to be processed is curved is: use the regression line equation to fit the points in the first boundary point set and the points in the second boundary point set respectively to obtain the corresponding third regression line equation and fourth regression line equation; judge whether the fitting goodness of the obtained third regression line equation and the fourth regression line equation both meet the second goodness threshold; when the fitting goodness of the third regression line equation and the fitting goodness of the fourth regression line equation both meet the second goodness threshold, the object to be processed is not curved, and the text line area of ​​the object to be processed can be directly output; when the fitting goodness of the third regression line equation and / or the fitting goodness of the fourth regression line equation do not meet the second goodness threshold, the object to be processed is curved, and it is necessary to continue to execute the method of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set.

[0084] In a specific implementation, the regression line equation is: y=φ(x)=a0+a1x.

[0085] The specific operation method is as follows: first, select a point in the first boundary point set and fit it with the straight line equation y = φ (x) = a0 + a1x, then use Gaussian elimination method to find the parameters of the straight line equation; then determine the deviation δi = φ (x) between the point pi (xi, yi) in the first boundary point set and the obtained straight line equation i )-y, the linear equation with the minimum deviation is the linear regression equation corresponding to the first boundary point set. When fitting the points in the first boundary point set with the linear equation, a portion of the points in the set can be selected, or all of the points can be selected; preferably, in this embodiment, all points except the two boundary points at the ends are selected. The "deviation" in "obtaining the linear equation with the minimum deviation" is the sum of the deviations (δi) between the multiple points in the first boundary point set and the linear equation; preferably, in this embodiment, it is the sum of the deviations (δi) between all points except the two boundary points at the ends and the linear equation.

[0086] In specific implementation, the first boundary curve is obtained based on the first boundary point set, and the second boundary curve is obtained based on the second boundary point set, including: using a method of successively increasing the highest order of the polynomial, using polynomials to fit the points in the first boundary point set and the points in the second boundary point set respectively, to obtain the first boundary curve and the second boundary curve whose fitting goodness meets the third goodness threshold.

[0087] In a specific implementation, a polynomial is used to fit the points within the first boundary point set to obtain a corresponding polynomial boundary equation; the goodness of fit of the obtained polynomial boundary equation is determined to be whether it meets a third goodness threshold; when the goodness of fit of the obtained polynomial boundary equation meets the third goodness threshold, the polynomial boundary equation is determined to be a first boundary curve, and the polynomial is used to fit the points within the second boundary point set to obtain a second boundary curve; when the goodness of fit of the obtained polynomial boundary equation does not meet the third goodness threshold, a higher-order polynomial is used to repeat the above steps. Specifically, starting with a quadratic polynomial, the points within the first boundary point set are fitted; when the goodness of fit of the quadratic polynomial equation obtained by fitting the quadratic polynomial does not meet the third goodness threshold, a cubic polynomial is used to fit the points within the first boundary point set, and the goodness of fit of the cubic polynomial equation obtained by fitting the cubic polynomial is compared with the third goodness threshold until the goodness of fit of the obtained polynomial equation meets the third goodness threshold. In this embodiment, the polynomial equations of the points within the first boundary point set are first obtained, but of course, the polynomial equations corresponding to the points within the second boundary point set whose goodness of fit meets the third goodness threshold can also be obtained.

[0088] In a specific implementation, the first goodness threshold, the second goodness threshold and the third goodness threshold are all within any range of 0.7-1; preferably, 0.7-1, 0.8-1, 0.85-1, or 0.9-1 can be selected.

[0089] Goodness of fit (G 2 ) is calculated as:

[0090]

[0091] Among them, SSR is the regression sum of squares, which refers to the impact of changes in the independent variable x on the changes in the value of the dependent variable y, or the changes in the values ​​of y caused by the linear relationship between x and y, also known as the explained sum of squares; SST is the total sum of squares, which refers to the total deviation of the n observed values ​​of the dependent variable from its mean; SSE is the residual sum of squares, which reflects the impact of other factors besides x on the value of y, and can also be called the unexplained sum of squares or residual sum of squares, SSR = SST + SSE; represents the fitted value calculated using the formula, is the mean of the fitted points.

[0092] In specific implementation, since it is easy to judge the points on the left and right boundaries as points on the upper and lower boundaries, or to judge the points on the upper and lower boundaries as points on the left and right boundaries in curved text, in order to eliminate the influence of misjudgment on the object to be processed, when using regression direct equations or polynomials to fit the points in the first boundary point set and the points in the second boundary point set, the two most boundary points in the first boundary point set and the two most boundary points in the second boundary point set are not involved in the calculation.

[0093] In a specific implementation, the extracting of a corresponding number of source control points on the first boundary curve and the second boundary curve respectively includes: extracting a preset number of first source control points and second source control points from the first boundary curve and the second boundary curve respectively, with each first source control point corresponding to each second source control point one-to-one; wherein the second source control point is the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve; or, the first source control point is the intersection of the normal of the first boundary curve passing through the second source control point and the first boundary curve.

[0094] In a specific implementation, a preset number of first source control points and second source control points are respectively extracted from the first boundary curve and the second boundary curve, and each first source control point corresponds to each second source control point one by one, including: judging the bending direction of the object to be processed; when the bending direction is bending toward the second boundary point set direction, extracting a preset number of second source control points from the second boundary curve; extracting first source control points corresponding to each second source control point on the first boundary curve, the first source control point being the intersection of the normal of the first boundary curve passing through the second source control point and the first boundary curve; when the bending direction is bending toward the first boundary point set direction, extracting a preset number of first source control points from the first boundary curve; extracting second source control points corresponding to each first source control point on the second boundary curve, the second source control point being the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve.

[0095] In specific implementation, the following four methods can be used to determine the bending direction of the object to be processed. The first method is: calculate the distance between the first boundary point counted in the first boundary point set and the boundary of the text line area, and the distance between the first second boundary point counted in the second boundary point set and the boundary of the text line area, and compare them; when the distance between the first boundary point and the boundary of the text line area is closer, determine that the object to be processed is bent in the direction of the second boundary point set; when the distance between the second boundary point and the boundary of the text line area is closer, determine that the object to be processed is bent in the direction of the first boundary point set.

[0096] The second method for determining the bending direction of the object to be processed is: comparing the time of the first first boundary point included in the first boundary point set with the time of the first second boundary point included in the second boundary point set. When the time of the first first boundary point is earlier than the time of the first second boundary point, it is determined that the object to be processed is bent toward the direction of the second boundary point set; when the time of the first second boundary point is earlier than the time of the first first boundary point, it is determined that the object to be processed is bent toward the direction of the first boundary point set.

[0097] The third method for determining the bending direction of the object to be processed is: calculating the distance between each first boundary point in the first boundary point set, extracting the maximum distance therefrom and recording it as the first distance; calculating the distance between each second boundary point in the second boundary point set, extracting the maximum distance therefrom and recording it as the second distance; comparing the first distance with the second distance; when the first distance is greater than the second distance, determining that the object to be processed is bent in the direction of the second boundary point set; when the second distance is greater than the first distance, determining that the object to be processed is bent in the direction of the first boundary point set.

[0098] The fourth method for determining the bending direction of the object to be processed is: determining the direction of the object to be processed, when the object to be processed is horizontal, extracting the leftmost point in the first boundary point set and the second boundary point set, and recording it as the text line edge point; when the object to be processed is vertical, extracting the topmost point in the first boundary point set and the second boundary point set, and recording it as the text line edge point; calculating the distance between the text line edge point and the first boundary curve and the second boundary curve and comparing them, when the distance between the text line edge point and the first boundary curve is greater than the distance between the text line edge point and the second boundary curve, determining that the object to be processed is curved in the direction of the first boundary point set; when the distance between the text line edge point and the second boundary curve is greater than the distance between the text line edge point and the first boundary curve, determining that the object to be processed is curved in the direction of the second boundary point set.

[0099] When applying the first or second method mentioned above to determine the bending direction of the object to be processed, if the text line area is traversed in the first preset order, the first preset order should be further limited to: starting from the first column on the left, traversing each pixel point of the first column from top to bottom and then traversing the second column, after traversing the second column, traversing the third column, ..., that is, starting from the first column on the left, traversing column by column; if the text line area is traversed in the second preset order, the second preset order should be further limited to: starting from the first row on the upper side, traversing each pixel point of the first row from left to right and then traversing the second row, after traversing the second row, traversing the third row, ..., that is, starting from the first row on the upper side, traversing row by row. For the third and fourth methods mentioned above, there is no need to further limit whether the first preset order is column-by-column traversal or multi-column parallel traversal, and when applying the second preset order, there is no need to limit whether the traversal order is row-by-row or multi-row parallel traversal.

[0100] In a specific implementation, aligning the source control points to obtain the corresponding target control points includes: aligning the first coordinates of the first source control points to obtain the first coordinates of the target control points corresponding to the first source control points; aligning the first coordinates of the second source control points to obtain the first coordinates of the target control points corresponding to the second source control points; and aligning the second coordinates of the corresponding groups of first source control points and second source control points one by one to obtain the second coordinates of the target control points corresponding to the groups of first source control points and second source control points.

[0101] In specific implementation, the first preset order is from left to right for each column and from top to bottom for each column; the second preset order is from top to bottom for each row and from left to right for each row; when the text row area of ​​the object to be processed is traversed in the first preset order to obtain the first boundary point set and the second boundary point set, the first coordinate is the horizontal coordinate and the second coordinate is the vertical coordinate; when the text row area of ​​the object to be processed is traversed in the second preset order to obtain the first boundary point set and the second boundary point set, the first coordinate is the vertical coordinate and the second coordinate is the horizontal coordinate.

[0102] In a specific implementation, the interpolation function is obtained according to each source control point, target control point and minimum energy function, including:

[0103] Assume that in the space region R 2 Internal distribution of q known points set P i , i=1,2,…,q, its coordinates are expressed as: (x i ,y i ,z(x i ,y i )), if z(x i ,y i ) has a second-order continuous derivative, then the energy function is expressed as:

[0104]

[0105] For TPS, it can be obtained by minimizing the energy function:

[0106] Z tps =argminE

[0107] Since the data points are expressed in discrete table form during interpolation, the function to be minimized in formula (1) is expressed as:

[0108]

[0109] In formula (2), ||·|| represents the Euclidean norm, c i is the coefficient, is the kernel function of the thin plate interpolation along the strip, and its value is:

[0110] Among them, r i =(xx i ) 2 +(yy i ) 2 (4)

[0111] In the above formulas (2) to (4), (x, y) is the target control point, (x i ,y i ) is the source control point. According to the thin plate spline interpolation formula calculation method, substituting each target control point and each source control point into the formula yields formula (2), thereby obtaining the interpolation function. The method of obtaining the interpolation function with the minimum energy based on the target control points and the source control points is a prior art method and will not be further elaborated here.

[0112] In specific implementation, the interpolation function is applied to process the text line area corresponding to the object to be processed, including: determining the correspondence between each pixel point in the target image and each pixel point in the text line area of ​​the object to be processed based on the interpolation function; and filling the pixel values ​​of the pixel points of the text line area of ​​the object to be processed into each pixel point of the target image according to the correspondence.

[0113] In this embodiment, the interpolation function is applied to process the text line area corresponding to the object to be processed, and a correspondence is established between each pixel point in the target image and the pixel point in the text line area of ​​the image to be processed. Based on the correspondence, the pixel point in the text line area of ​​the image to be processed corresponding to each pixel point in the target image is found, and the pixel value of the found pixel point is filled in the corresponding pixel point in the target image.

[0114] An embodiment of the present invention also provides a bill processing method, including applying the above-mentioned table or text line processing method to process the bill to determine whether the text line or table in the bill is a curved text line or curved table. When it is determined to be a curved text line or table, the curved text line or table is corrected.

[0115] The bill processing method provided in this embodiment further includes performing character recognition on the corrected text line to obtain the face content of the bill.

[0116] Example 2

[0117] like Figures 2 to 4 As shown, in another embodiment of the present invention, in the table or text line processing method provided, the object of processing is as follows Figure 3The text line area shown is arranged in a horizontal format; the preset order for processing the text line is the first preset order. The flowchart of the processing method of the table or text line is as follows Figure 2 As shown, the specific steps include:

[0118] S200: performing progressive expansion network processing on the text line area corresponding to the text line to obtain an object to be processed;

[0119] S201: Obtain the points of the upper boundary and the lower boundary of the object to be processed, and count them into the first boundary point set (upper boundary point set) and the second boundary point set (lower boundary point set) respectively.

[0120] will be as Figure 3 The text line area image of the object to be processed is converted into a binary image; each column of the binary image is traversed in parallel from left to right, and each pixel value in each column is traversed from top to bottom. Specifically: when the pixel value in each column changes from 0 to 1 for the first time, the pixel point corresponding to 1 is counted into the first boundary point set (upper boundary point set); when the pixel value changes from 1 to 0 for the last time, the pixel point corresponding to 1 is counted into the second boundary point set (lower boundary point set). Figure 4 The first boundary point set shown is composed of the first boundary point 101, and the second boundary point set is composed of the second boundary point 102.

[0121] S202: Determine whether the line on which the points in the first boundary point set lie is a straight line and / or whether the line on which the points in the second boundary point set lie is a straight line.

[0122] The first and second regression lines are fitted using the regression line equation y = φ(x) = a0 + a1x to obtain the first and second regression lines. The parameters of the first and second regression lines are then calculated using Gaussian elimination. Since the left and right boundary points are often misidentified as the upper and lower boundary points in curved text, points in the middle region are selected as fitting objects. In other words, the two outermost points in the first boundary point set (the upper boundary point set) and the two outermost points in the second boundary point set (the lower boundary point set) are not included in the calculation.

[0123] Calculate the goodness of fit G of the regression line 2 , the calculation formula is:

[0124]

[0125] represents the fitted value calculated using the formula, Refers to the average value of the fitting points, the goodness of fit R 2 The value of R is between [0, 1]; 2 ->1(R2 The closer it is to 1), the better the regression equation fits; R 2 ->0, indicating that the regression equation fits worse;

[0126] When R 2 When the value of meets the first goodness threshold (in this embodiment, the first goodness threshold is 0.7-1), the boundary is a straight line, the original image is returned, and the correction is completed.

[0127] When R 2 The value of is less than the first goodness threshold, that is, the goodness of fit does not meet the first goodness threshold, and the boundary is judged to be a curve, and the next step is performed.

[0128] S203: Obtain a first boundary curve based on the first boundary point set, and obtain a second boundary curve based on the second boundary point set; wherein the first boundary curve is a parametric equation of a curve formed by the first boundary points in the first boundary point set; and the second boundary curve is a parametric equation of a curve formed by the second boundary points in the second boundary point set.

[0129] Use the polynomial to fit the points in the first boundary point set to obtain the polynomial boundary equation corresponding to the first boundary point set, and determine whether the goodness of fit of the obtained polynomial boundary equation meets the third goodness threshold (0.75-1 in this embodiment). If the goodness of fit R 2 If the value of meets the third goodness threshold, the second boundary curve (lower boundary curve) is obtained using the polynomial.

[0130] If the goodness of fit R 2 If the value of does not meet the third goodness threshold, a higher-order polynomial is used for fitting until the goodness of fit of the obtained polynomial equation is R 2 When the third goodness threshold is met, the polynomial is used to fit the points in another boundary point set to obtain the corresponding boundary curve.

[0131] S204: extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively;

[0132] S2041: Determine the bending direction

[0133] In this embodiment, the following method is selected to determine the bending direction:

[0134] Extract the leftmost point in the first boundary point set and the second boundary point set, and record them as text line edge points;

[0135] The distances between the text line edge points and the first boundary curve and the second boundary curve are calculated and compared. When the distance between the text line edge points and the first boundary curve is greater than the distance between the text line edge points and the second boundary curve, it is determined that the object to be processed is bent toward the direction of the first boundary point set (bending upward); when the distance between the text line edge points and the second boundary curve is greater than the distance between the text line edge points and the first boundary curve, it is determined that the object to be processed is bent toward the direction of the second boundary point set (bending downward).

[0136] Wherein, when calculating the distance between the text line edge point and the first boundary curve, and the distance between the text line edge point and the second boundary curve, both can be approximated as the distance between a point and a straight line.

[0137] S2042: Select the source control point.

[0138] like Figure 3 As shown in the figure, the text curves upwards. The specific process of selecting the source control point is as follows:

[0139] Extract m first source control points from the first boundary curve (upper boundary curve); extract second source control points corresponding to each first source control point on the second boundary curve (lower boundary curve), and the second source control point is the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve.

[0140] S205: Align each source control point to obtain the corresponding target control point;

[0141] The specific process is as follows: align the horizontal coordinates of the m first source control points on the upper boundary curve, align the horizontal coordinates of the m second source control points on the lower boundary curve, and align the vertical coordinates of each corresponding group of first source control points and second source control points, so as to obtain the coordinates of the target control points corresponding to each source control point.

[0142] S206: Obtain an interpolation function according to each source control point, target control point and minimum energy function.

[0143] According to the following principle of thin plate spline interpolation function: Assume that in the space region R 2 There are q known points in the internal distribution P i , i=1,2,…,q, its coordinates are expressed as: (x i ,y i ,z(x i ,y i )). If z(x i ,y i ) has a second-order continuous derivative, then the energy function is expressed as:

[0144]

[0145] For TPS, it can be obtained by minimizing the energy function:

[0146] Z tps =argminE

[0147] Input the coordinates of the source control points and the target control points (a total of 4m points) into formula (1) to obtain the interpolation function, which is:

[0148] Since the data points are expressed in discrete table form during interpolation, the function to be minimized in formula (1) is expressed as:

[0149]

[0150] In formula (2), ||·|| represents the Euclidean norm, which is c i coefficient, is the kernel function of TPS, and its value is:

[0151]

[0152] Among them, r i =(xx i ) 2 +(yy i ) 2 (4).

[0153] In the above formulas (2) to (4), (x, y) is the target control point, (x i ,y i ) is the source control point. According to the thin plate spline interpolation formula calculation method, substituting each target control point and each source control point into the formula yields formula (2), thereby obtaining the interpolation function. The method of obtaining the interpolation function with the minimum energy based on the target control points and the source control points is a prior art method and will not be further elaborated here.

[0154] S207: Applying an interpolation function to process the text line area corresponding to the text line.

[0155] Extract a pixel point of the target image as the first pixel point, find the pixel point of the text line area where the object to be processed is located corresponding to the first pixel point, and use the pixel value of the second pixel point as the pixel value of the first pixel point; repeat the above steps, traverse the pixel points of each row and column of the target image, and the corrected text line area can be obtained. The corrected text line area is as follows Figure 5 shown.

[0156] An embodiment of the present invention also provides a bill processing method, which includes applying the above-mentioned table or text line processing method to process the bill to determine whether the text line in the bill is a curved text line. When it is determined to be a curved text line, the curved text line is corrected.

[0157] The bill processing method provided in this embodiment further includes performing character recognition on the corrected text line to obtain the face content of the bill.

[0158] like Figure 6 As shown, this embodiment also provides a table or text line processing device, including an electrically connected processor 1020 and a memory 1010. The memory can be a physical storage hard disk or a cloud storage. Computer instructions are stored in the memory. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the text line processing method.

[0159] A computer storage medium stores a computer program, which implements the steps of a text line processing method when the computer program is executed by a processor.

[0160] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for processing the content to be displayed on the display screen. In some embodiments.

[0161] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash storage devices, or cloud storage. In some embodiments, the non-transitory computer-readable storage medium in the memory is used to store at least one instruction, which is executed by the processor to implement the text image processing method provided in the method embodiment of the present application.

Claims

1. A method for processing a table or a text line, characterized in that: The following steps are involved: Obtain points on two opposite boundaries of the object to be processed, and count them into the first boundary point set and the second boundary point set respectively; Obtain a first boundary curve according to the first boundary point set, and obtain a second boundary curve according to the second boundary point set; extracting corresponding numbers of source control points on the first boundary curve and the second boundary curve respectively; Align each source control point to obtain the corresponding target control point; An interpolation function is obtained according to each source control point, target control point and minimum energy function; Apply the interpolation function to process the text line area corresponding to the object to be processed; Before executing the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set, the method further includes: Fitting the points within the first boundary point set with the regression line equation to obtain the corresponding first regression line equation; or fitting the points within the second boundary point set with the regression line equation to obtain the corresponding second regression line equation; Determining whether the goodness of fit of the obtained first regression line equation or the second regression line equation meets a first goodness threshold; When the first goodness threshold is met, the text line area of ​​the object to be processed is directly output; When the first goodness threshold is not met, continuing to perform the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set; or, Use the regression line equation to fit the points in the first boundary point set and the points in the second boundary point set respectively to obtain the corresponding third regression line equation and fourth regression line equation; Determine whether the goodness of fit of the obtained third regression line equation and the fourth regression line equation both meet the second goodness threshold; When the goodness of fit of the third regression line equation and the goodness of fit of the fourth regression line equation both meet the second goodness threshold, directly outputting the text line area of ​​the object to be processed; When the goodness of fit of the third regression line equation and / or the goodness of fit of the fourth regression line equation does not meet the second goodness threshold, continuing to perform the steps of obtaining the first boundary curve according to the first boundary point set and obtaining the second boundary curve according to the second boundary point set; The regression line equation is: y = φ(x) = a0 + a1x.

2. The method according to claim 1, characterized in that The obtaining of points on two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively includes: Traversing the text line area of ​​the object to be processed in a first preset order, counting the first pixel point after the change in the brightness indicator value of the pixel points in each column reaches the preset difference for the first time into the first boundary point set, and counting the last pixel point before the change reaches the preset difference for the last time into the second boundary point set; or, Traversing the text line area of ​​the object to be processed in a second preset order; counting the first pixel point after the change in the brightness indicator value of the pixel points in each line reaches the preset difference for the first time into the first boundary point set, and counting the last pixel point before the change reaches the preset difference for the last time into the second boundary point set; or, The step of obtaining points on two relative boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively further includes: Determining the direction of the object to be processed; When the object to be processed is horizontal, the text line area is traversed in a first preset order; the first pixel point after the change in the brightness indicator value of the pixel points in each column reaches the preset difference for the first time is counted into the first boundary point set, and the last pixel point before the change reaches the preset difference for the last time is counted into the second boundary point set; When the object to be processed is vertical, the text line area is traversed in a second preset order; the first pixel point after the change in the brightness indicator value of the pixel points in each line reaches the preset difference for the first time is counted into the first boundary point set, and the last pixel point before the change reaches the preset difference for the last time is counted into the second boundary point set; The first preset order is from left to right for each column and from top to bottom for each column; the second preset order is from top to bottom for each row and from left to right for each row.

3. The method according to claim 1, characterized in that Before executing the step of obtaining points on two opposite boundaries of the object to be processed and counting them into the first boundary point set and the second boundary point set respectively, the method further includes: Performing progressive expansion network processing on a text line region corresponding to an object to be processed to obtain the object to be processed.

4. The method according to claim 1, wherein The step of obtaining a first boundary curve according to the first boundary point set and obtaining a second boundary curve according to the second boundary point set includes: By successively increasing the degree of the highest order of the polynomial, the polynomials are used to fit the points in the first boundary point set and the points in the second boundary point set, respectively, to obtain a first boundary curve and a second boundary curve whose goodness of fit meets a third goodness threshold, specifically comprising: Fitting the points in the first boundary point set using a polynomial to obtain a corresponding polynomial boundary equation; Determine whether the goodness of fit of the obtained polynomial boundary equation meets a third goodness threshold; When the goodness of fit of the obtained polynomial boundary equation meets the third goodness threshold, the polynomial boundary equation is determined to be the first boundary curve, and the polynomial is used to fit the points in the second boundary point set to obtain the second boundary curve; When the goodness of fit of the obtained polynomial boundary equation does not meet the third goodness threshold, a higher-order polynomial is used to repeat the above steps.

5. The method according to claim 3 or 4, characterized in that The first goodness threshold, the second goodness threshold, and the third goodness threshold are all within a range of 0.7-1; Goodness of fit (G 2 ) is calculated as: , where SSR is the regression sum of squares, which refers to the impact of changes in the independent variable x on the value of the dependent variable y; SST is the total sum of squares, which refers to the total deviation of the n observed values ​​of the dependent variable from its mean; SSE is the residual sum of squares, which reflects the impact of other factors other than x on the value of y, SSR=SST+SSE; represents the fitted value calculated using the formula, is the average value of the fitting points; When using the regression direct equation or polynomial to fit the points in the first boundary point set and the points in the second boundary point set, the two most boundary points in the first boundary point set and the two most boundary points in the second boundary point set are not involved in the calculation.

6. The method according to claim 1, characterized in that The extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively includes: Extracting a preset number of first source control points and second source control points from the first boundary curve and the second boundary curve respectively, wherein each first source control point corresponds to each second source control point one by one; The second source control point is an intersection point of a normal line of the second boundary curve passing through the first source control point and the second boundary curve; or the first source control point is an intersection point of a normal line of the first boundary curve passing through the second source control point and the first boundary curve; Alternatively, extracting a corresponding number of source control points on the first boundary curve and the second boundary curve respectively includes: Extracting a preset number of first source control points and second source control points from the first boundary curve and the second boundary curve respectively, wherein each first source control point corresponds to each second source control point one by one; The second source control point is an intersection point of a normal line of the second boundary curve passing through the first source control point and the second boundary curve; or the first source control point is an intersection point of a normal line of the first boundary curve passing through the second source control point and the first boundary curve; The step of extracting a preset number of first source control points and second source control points from the first boundary curve and the second boundary curve, wherein each first source control point corresponds to each second source control point, includes: Determine the bending direction of the object to be processed; When the bending direction is toward the second boundary point set, a preset number of second source control points are extracted from the second boundary curve; and first source control points corresponding to each second source control point are extracted on the first boundary curve, where the first source control point is an intersection point of the first boundary curve with a normal line passing through the second source control point. When the bending direction is toward the first boundary point set, a preset number of first source control points are extracted from the first boundary curve; second source control points corresponding to each first source control point are extracted on the second boundary curve, and the second source control point is the intersection of the normal of the second boundary curve passing through the first source control point and the second boundary curve.

7. The method according to claim 6, characterized in that The determining of the bending direction of the object to be processed includes: Calculating a distance between a first boundary point counted first in the first boundary point set and the boundary of the text line region, and a distance between a first second boundary point counted first in the second boundary point set and the boundary of the text line region, and comparing them; when the distance between the first boundary point and the boundary of the text line region is closer, determining that the object to be processed is curved toward the direction of the second boundary point set; when the distance between the second boundary point and the boundary of the text line region is closer, determining that the object to be processed is curved toward the direction of the first boundary point set; or, Comparing the time of the first first boundary point included in the first boundary point set with the time of the first second boundary point included in the second boundary point set, and when the time of the first first boundary point is earlier than the time of the first second boundary point, determining that the object to be processed is curved toward the second boundary point set; and when the time of the first second boundary point is earlier than the time of the first first boundary point, determining that the object to be processed is curved toward the first boundary point set; or, Calculating the distances between each first boundary point in the first boundary point set, extracting the maximum distance therefrom and recording it as the first distance; calculating the distances between each second boundary point in the second boundary point set, extracting the maximum distance therefrom and recording it as the second distance; comparing the first distances with the second distances; when the first distance is greater than the second distance, determining that the object to be processed is curved toward the second boundary point set; when the second distance is greater than the first distance, determining that the object to be processed is curved toward the first boundary point set; or, Determine the direction of the object to be processed. When the object to be processed is horizontal, extract the leftmost point in the first boundary point set and the second boundary point set and record it as the text line edge point; when the object to be processed is vertical, extract the topmost point in the first boundary point set and the second boundary point set and record it as the text line edge point; The distances between the edge points of the text line and the first boundary curve and the second boundary curve are calculated and compared. When the distance between the edge points of the text line and the first boundary curve is greater than the distance between the edge points of the text line and the second boundary curve, it is determined that the object to be processed is curved toward the direction of the first boundary point set; when the distance between the edge points of the text line and the second boundary curve is greater than the distance between the edge points of the text line and the first boundary curve, it is determined that the object to be processed is curved toward the direction of the second boundary point set.

8. The method according to claim 6, characterized in that The step of aligning the source control points to obtain corresponding target control points includes: Aligning the first coordinates of each first source control point to obtain the first coordinates of the target control point corresponding to each first source control point; Aligning the first coordinates of each second source control point to obtain the first coordinates of the target control point corresponding to each second source control point; Aligning the second coordinates of the corresponding groups of first source control points and second source control points one by one to obtain the second coordinates of the target control points corresponding to the groups of first source control points and second source control points; in, When the first boundary point set and the second boundary point set are obtained by traversing the text line area of ​​the object to be processed in a first preset order, the first coordinate is the horizontal coordinate and the second coordinate is the vertical coordinate; When the first boundary point set and the second boundary point set are obtained by traversing the text line area of ​​the object to be processed in a second preset order, the first coordinate is the vertical coordinate and the second coordinate is the horizontal coordinate; The first preset order is from left to right for each column and from top to bottom for each column; the second preset order is from top to bottom for each row and from left to right for each row.

9. The method according to claim 1, characterized in that The interpolation function is obtained according to each source control point, target control point and minimum energy function, including: Assume that in the space region R 2 There are q known points in the internal distribution P i , i=1,2,…,q, its coordinates are expressed as: (x i ,y i ,z(x i ,y i ))), if z(x i ,y i ) has a second-order continuous derivative, then the energy function is expressed as: （1） For TPS, it can be obtained by minimizing the energy function: ， Since the data points are expressed in discrete table form during interpolation, the function to be minimized in formula (1) is expressed as: （2） In formula (2), ||·|| represents the Euclidean norm, c i is the coefficient, φ(·) is the kernel function of the thin plate interpolation along the strip, and its value is: φ(r)=r 2 log(r), where r i =(xx i ) 2 +(yy i ) 2 , (x i ,y i ) is the spatial region R 2 The coordinates of the i-th source control point in the space region R, (x, y) 2 The coordinates of the i-th target control point within .

10. The method according to claim 9, characterized in that Applying the interpolation function to process the text line area corresponding to the object to be processed includes: Based on the interpolation function, the correspondence between each pixel point in the target image and the pixel points in the text line area of ​​the object to be processed is determined; according to the correspondence, the pixel values ​​of the pixel points in the text line area of ​​the object to be processed are filled into each pixel point of the target image.

11. A bill processing method, characterized in that: The method comprises applying the table or text line processing method according to any one of claims 1 to 10 to process the bill.

12. A table or text line processing device, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method according to any one of claims 1 to 11.

13. A computer storage medium, characterized in that A computer program is stored thereon, which implements the method steps according to any one of claims 1 to 11 when the computer program is executed by a processor.

Images