A method for correcting perspective transformation of a Demura camera shooting result

Abstract

The present application provides a kind of to Demura camera shooting result perspective transformation correction method, comprising the following steps: image is binarized, binarization result is first dilated then same intensity corrosion, the maximum profile in result is extracted, the convex hull of maximum profile is calculated, the polygon approximation is done to convex hull data, the distortion bounding box is obtained by processing polygon result, and perspective transformation is carried out with reference to bounding box vertex. Distortion of shooting result is reduced by computer graphics processing, and perspective transformation correction is carried out.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer graphics processing, and in particular relates to a method for reducing distortion of shooting results and performing perspective transformation correction. Background Art

[0002] When demuracing a mobile phone screen, one often encounters the problem of how to collect data from the mobile phone display. The method of data collection is generally to use a high-precision industrial camera to shoot the mobile phone display. During the shooting process, it is inevitable that the mobile phone screen is not placed completely horizontally or vertically, and the camera lens imaging itself has a certain degree of distortion, resulting in a certain degree of deformation in the final shooting result. This deformation is likely difficult to observe with the naked eye, but it has an impact that cannot be ignored when calculating data. For example, when the deviation between the beginning and end of a row of pixels exceeds the spacing between a row of pixels, the final mapping result will map some pixels to the values ​​of other adjacent pixels, thereby affecting the demura effect. We have made many attempts to eliminate or reduce this effect as much as possible. Here we disclose a method that can more effectively reduce the impact of deformation on the shooting results. Summary of the Invention

[0003] The present invention provides a method for performing perspective transformation correction on Demura camera shooting results, comprising the following steps:

[0004] Step 1: binarize the image;

[0005] Step 2: dilate the binarized result first and then erode it with the same intensity;

[0006] Step 3, extracting the largest contour from the result of step 2;

[0007] Step 4, calculating the convex hull of the maximum contour;

[0008] Step 5, performing polygonal approximation on the convex hull data;

[0009] Step 6: Process the polygonal result to obtain the distorted bounding box;

[0010] Step 7: Perform perspective transformation based on the bounding box vertices.

[0011] Furthermore, step one includes:

[0012] First, the captured brightness data is dichotomized using the maximum inter-class variance method. The inter-class variance expression is σ 2 =p1p2(m1-m2) 2 , where p1 and p2 are the probabilities of the brightness values ​​being divided into two categories, that is, the proportions, and m1 and m2 are the respective means of the brightness values ​​of the two categories;

[0013] Secondly, the binary result is obtained to make the inter-class variance σ 2 The optimal solution is found by traversing all the luminance as a segmentation point to find the binary result that best fits the expression, or the inter-class variance is compared with the segmentation point value from small to large to show the trend of first increasing and then decreasing to speed up the finding of the optimal solution.

[0014] Finally, after obtaining the segmentation point value, the data on both sides of the segmentation point value is represented by 0 and 1 respectively to complete the binary processing.

[0015] Further, the erosion operation:

[0016] The dilation operation:

[0017] The dilation in step two refers to the expansion of the boundary points of the binary object in morphology, and the erosion is the inverse operation of the dilation. The operation in step two is full-direction,

[0018] The strength of the dilation and the erosion is the same, which is greater than the distance between the adjacent pixel points.

[0019] Further, the result in step two is analyzed to extract the outer contour of all connected regions, exclude noise, and only keep the largest part of the outer contour, that is, the outer contour of the screen area.

[0020] Further, the convex hull in step four is: in a real vector space V, for a given set X, the intersection S of all convex sets containing X is called the convex hull of X. The convex hull of X can be constructed by convex combination of all points (X1,...Xn) in X. The Graham scanning method is used, or the method for calculating the convex hull in various graphics libraries is directly called.

[0021] Further, the discrete point expression of the outer contour is converted into a line segment expression, so that the linear expression of the four sides of the screen area is obtained. Because of the imaging distortion of the camera, the screen area actually enclosed by the linear expression is a very close-to-rectangular irregular quadrilateral. The Douglas-Peucker method is used to simplify the continuous discrete points into line segments. The four rounded corners of the screen will become multiple line segments because the discrete points of the rounded corners are not on a straight line. Any long straight side except the rounded corner part will finally approach a long line segment that fits the side.

[0022] Further, the approximation result in step five is analyzed, first, the short line segment of the rounded corner part is removed, the four line segments closest to the four straight edges of the screen are located, the equation of the straight line where the four line segments are located is calculated from the coordinates of the head and tail points of the line segment, the intersection point of the extended lines of adjacent equations is calculated, and the area surrounded by the four extended intersection points is the bounding box of the screen area after deformation. The four corner coordinates of the bounding box are determined and are floating point types.

[0023] Further, the four vertex coordinates of the bounding box are used in step seven to perform perspective transformation on the irregular quadrilateral, and the content of the irregular area is mapped to a rectangle.

[0024] The perspective transformation matrix is calculated using the four vertexes of the bounding box and the four vertexes of the picture to be mapped, and the data in the bounding box is mapped to the output picture using the perspective transformation matrix. At this time, the pixel arrangement in the output picture is closer to the actual panel position. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flowchart of the present application is shown;

[0026] Figure 2 The effect diagram of step one of the present application is shown;

[0027] Figure 3 The effect diagram of step two of the present application is shown;

[0028] Figure 4 The effect diagram of step three of the present application is shown;

[0029] Figure 5 The effect diagram of step four of the present application is shown;

[0030] Figure 6 The effect diagram of step five of the present application is shown;

[0031] Figure 7 The bounding box diagram of step six of the present application is shown;

[0032] Figure 8 The mapping result of step seven of the present application is shown. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0034] Please refer to Figure 1-8, the present invention provides a method for perspective transformation correction of the shooting results of the Demura camera, first find the screen area, and use binarization to distinguish the brighter screen area from the remaining dark areas (step one). Then, the binarized image is expanded to fill the gaps between pixels, so that the entire screen area becomes a connected whole, and then the whole is eroded with the same intensity to restore it to the actual size of the screen area (step two). After that, all image contours in the result of the previous step are extracted, and all area contours are traversed to find the contour with the largest area, which is the approximate contour of the screen (step three). The convex hull of this approximate contour is then calculated (step four). The purpose of this step is to avoid the notch area that is often found on the upper edge of the screen today, or the depression on other edges. After obtaining the convex hull data, Douglas-Peucker or similar algorithms are used for polygonal approximation (step five), the purpose of which is to obtain a fitting bounding box of the quadrilateral (step six). Use the four vertices of the quadrilateral's bounding box as the four vertices of the perspective transformation, perform perspective transformation on its content (step 7), and map it to a rectangle that is horizontally and vertically aligned with the coordinate system. This completes the correction of the camera's shooting results. See the flowchart of this method for details. Figure 1 , the process is as follows:

[0035] Step 1: binarize the image;

[0036] Step 2: dilate the binarized result first and then erode it with the same intensity;

[0037] Step 3, extracting the largest contour from the result of step 2;

[0038] Step 4, calculating the convex hull of the maximum contour;

[0039] Step 5, performing polygonal approximation on the convex hull data;

[0040] Step 6: Process the polygonal result to obtain the distorted bounding box;

[0041] Step 7: Perform perspective transformation based on the bounding box vertices.

[0042] Step 1 includes:

[0043] First, the captured brightness data is dichotomized using the maximum inter-class variance method. The inter-class variance expression is σ 2 =p1p2(m1-m2) 2 , where p1 and p2 are the probabilities of the brightness values ​​being divided into two categories, that is, the proportions, and m1 and m2 are the respective means of the brightness values ​​of the two categories;

[0044] Secondly, the binarization result is such that the inter-class variance σ 2To find the largest set of results, we can use all brightness as the split point to find the optimal solution of the binary result that best fits the expression, or use the inter-class variance to increase first and then decrease as the split point value increases from small to large to perform binary comparison to accelerate the search for the optimal solution.

[0045] Finally, after obtaining the segmentation point value, the data on both sides of the segmentation point value are represented by 0 and 1 respectively to complete the binarization process.

[0046] Dilation in step 2 refers to the expansion of the boundary points of the binary object in morphology, while erosion is the inverse of dilation. The operations in step 2 are all omnidirectional, and the intensity of dilation and erosion is the same, slightly larger than the boundary spacing between adjacent pixels. The purpose is to connect the discrete pixels in the binary result into a complete area, so that the outer contour of the screen area can be easily calculated. The result of this step is difficult to observe the overall deformation with the naked eye. You need to zoom in and observe the edges to see that the vertical edges will shift by one pixel in the horizontal direction every few hundred or thousand pixels, and the horizontal edges will also shift by one pixel in the vertical direction every few hundred or thousand pixels:

[0047] Corrosion operation:

[0048] Expansion operation:

[0049] Analyze the results of step 2, extract the outer contours of all connected areas, exclude noise, and retain only the largest outer contour, i.e., the outer contour of the screen area. The smaller areas enclosed by the outer contour are considered noise. Figure 4 A shows the outer contour diagram, Figure 4 B shows that the enlarged contour edge is directly composed of discrete points.

[0050] The convex hull in step 4 is a concept from computational geometry (graphics). In a real vector space V, for a given set X, the intersection S of all convex sets containing X is called the convex hull of X. The convex hull of X can be constructed using convex combinations of all points (X1, ...Xn) in X. In two-dimensional Euclidean space, the convex hull can be imagined as a rubber band that just encloses all points. Graham's scan method can be used here, or you can directly call the convex hull calculation methods implemented in various graphics libraries. The purpose of calculating the convex hull is to fill the notch area on the boundary, facilitating subsequent polygon approximation calculations. Figure 5 Displays the calculation of the convex hull to repair the gap and notch area.

[0051] The main function of step five is to convert the previous discrete point expression of the contour into a line segment expression, thereby obtaining the straight line expression of the four sides of the screen area. Due to the imaging distortion of the camera, the screen area actually enclosed by the straight line expression is an irregular quadrilateral very close to a rectangle. The specific method is to use Douglas-Peucker or other similar methods to simplify continuous discrete points into line segments. The four rounded corners of the screen will become multiple line segments because the discrete points themselves are not on a straight line. Any long straight edge other than the rounded corners will eventually be approximated to obtain a long line segment that fits the edge. Figure 6 A shows the polygonal approximation result, which looks the same as the convex hull, but the actual data expression is different. Figure 6 B shows the approximation result after enlargement. The edge is composed of line segments connecting two adjacent points.

[0052] Analyze the approximation results in step five. First, remove the short line segments with rounded corners, locate the four line segments closest to the four straight edges of the screen, and calculate the equations of the straight lines on which the four line segments are located from the coordinates of the first and last points of the line segments. The adjacent equations can be combined to find the intersection of the extended lines of the line segments. The area enclosed by the four extended line intersections in sequence is the bounding box of the deformed screen area. The four corner coordinates of the bounding box are fixed and are floating-point type.

[0053] In step seven, the coordinates of the four vertices of the bounding box are used to perform a perspective transformation on the irregular quadrilateral, mapping the contents of the irregular area into a rectangle. Specifically, the perspective transformation matrix is ​​calculated using the four vertices of the bounding box and the four vertices of the image to be mapped. This matrix is ​​then used to map the data within the bounding box to the output image. The pixel arrangement in the output image is now closer to the actual panel position. In the test sample, the original data had a maximum deviation of approximately three pixels at the beginning and end of each row and column. After the entire correction process, the deviation at the beginning and end of each row and column is less than one pixel.

[0054] The advantages of the present invention mainly lie in its fast calculation, the use of graphics methods to obtain perspective transformation with very good performance optimization, low requirements for data preprocessing, etc., and the effect can meet the needs of general industrial production.

[0055] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is limited by the appended claims and their equivalents.

Claims

1. A method for performing perspective transformation correction on Demura camera shooting results, characterized in that: The following steps are involved: Step 1: binarize the image; Step 2: dilate the binarized result first and then erode it with the same intensity; Step 3, extracting the largest contour from the result of step 2; Step 4, calculating the convex hull of the maximum contour; Step 5, performing polygonal approximation on the convex hull data; Step 6: Process the polygonal result to obtain the distorted bounding box; Step 7: Perform perspective transformation based on the bounding box vertices. Step 1 includes: First, the captured brightness data is dichotomized using the maximum inter-class variance method. The inter-class variance expression is σ 2 =p1p2(m1-m2) 2 , where p1 and p2 are the probabilities of the brightness values ​​being divided into two categories, that is, the proportions, and m1 and m2 are the respective means of the brightness values ​​of the two categories; Secondly, the binarization result is such that the inter-class variance σ 2 To find the largest set of results, we can use all brightness as the split point to find the optimal solution of the binary result that best fits the expression, or use the inter-class variance to increase first and then decrease as the split point value increases from small to large to perform binary comparison to accelerate the search for the optimal solution. Finally, after obtaining the segmentation point value, the data on both sides of the segmentation point value are represented by 0 and 1 respectively to complete the binarization process. Corrosion operation: Expansion operation: Where A is the original image, B is the dilated structure element, represents the corrosion operation, represents the dilation operation, x,y represents the pixel position in the image; The expansion in step 2 refers to the expansion of the boundary points of the binary object in morphology, while erosion is the inverse operation of expansion. The operations in step 2 are all omnidirectional. The strength of dilation and erosion is the same, both of which are greater than the spacing between the boundaries of adjacent pixels; Analyze the results of step 2, extract the outer contours of all connected areas, eliminate noise, and retain only the largest part of the outer contour, that is, the outer contour of the screen area; The convex hull in step 4 is: in a real vector space V, for a given set X, the intersection S of all convex sets containing X is called the convex hull of X. The convex hull of X can be constructed using the convex combination of all points (X1, ...Xn) in X using the Graham scan method or by directly calling the convex hull calculation method in various graphics libraries; The discrete point representation of the outer contour is converted into a line segment representation, thereby obtaining straight line representations for the four sides of the screen area. Due to camera imaging distortion, the screen area actually enclosed by the straight line representation is an irregular quadrilateral very close to a rectangle. The Douglas-Peucker method is used to simplify continuous discrete points into line segments. The four rounded corners of the screen will become multiple line segments because the discrete points themselves are not on a straight line. Any long straight edge other than the rounded corners will ultimately be approximated as a long line segment that fits the edge. Analyze the approximation results in step 5. First, remove the short line segments with rounded corners. Locate the four line segments closest to the four straight edges of the screen. Calculate the equations of the lines containing the four line segments from the coordinates of the first and last points of the line segments. Combine the adjacent equations to find the intersection points of the extended lines of the line segments. The area enclosed by the four consecutive extended line intersections is the bounding box of the deformed screen area. The coordinates of the four corners of the bounding box are fixed and are floating-point data. In step 7, the coordinates of the four vertices of the bounding box are used to perform a perspective transformation on the irregular quadrilateral, mapping the irregular area content into a rectangle: The perspective transformation matrix is ​​calculated using the four vertices of the bounding box and the four vertices of the image to be mapped. The data in the bounding box is mapped to the output image using the perspective transformation matrix. At this time, the pixel arrangement in the output image is closer to the actual panel position.

Images