A real scene morphing digital picture generation method and system

Abstract

This invention discloses a method and system for generating deformed digital images of real-world scenes. The method involves acquiring digital images from real-world scene images and binarizing them; extracting numbers from the images based on the area of ​​white blocks in the binarized images; performing Hough line detection on the numbers to obtain multiple key points after affine transformation; processing these key points to obtain multiple key points before the affine transformation; calculating affine transformation parameters based on the coordinates of these key points; determining the value range of each affine transformation parameter based on the calculated parameters from multiple real-world scene images; and generating multiple deformed digital images by randomly selecting values ​​and performing affine transformation on the images based on the value ranges of each affine transformation parameter. This method uses only a small number of real-world scene digital images for analysis, automatically obtaining the approximate degree and range of deformation of the actual numbers, thus reducing the time spent adjusting deformation parameters.

Description

Technical Field

[0001] This invention relates to the field of OCR technology, specifically to a method and system for generating digital images with real-world scene deformation. Background Technology

[0002] In the field of artificial intelligence, OCR (Optical Character Recognition) is a branch of computer graphics. Traditional OCR has two directions: text detection and text recognition. Text detection is used to detect the location of text in an image, while text recognition is used to identify the content of the text from the detected bounding box. Different methods are used to process the text in an image depending on whether it is on paper or in a real-world scene, and whether it is printed or handwritten. Especially for text in real-world scenes, the deformation of the text (mostly the effect of affine transformation under normal circumstances) will vary depending on the text and the shooting position, making text recognition difficult. Because text recognition in real-world scenes requires a large amount of text data with different degrees of deformation that conform to the current scene for training, and obtaining this data through manual shooting and annotation is too costly and slow, text synthesis must be used to generate data. However, when performing text synthesis, the synthesized data needs to cover the approximate degree and range of text deformation in real-world scenes, which requires a lot of time to debug the affine transformation parameters. Summary of the Invention

[0003] To address this issue, the present invention provides a method and system for generating digital images with real-world deformation, thereby solving the problems that existing OCR text recognition training data acquisition methods, which rely solely on manual shooting and annotation, are too costly and slow, and that adjusting affine transformation parameters during text synthesis requires a significant amount of time.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] According to a first aspect of the present invention, a method for generating a digital image of a real-world scene deformation is provided, the method comprising:

[0006] Digital images are obtained from real-world scene images, and the digital images are then binarized.

[0007] The numbers in the image are obtained by measuring the area of ​​the white block in the binarized image, and Hough line detection is performed on the numbers to obtain multiple key points after affine transformation. The numbers are then processed to obtain multiple key points before affine transformation. The affine transformation parameters are calculated based on the coordinates of the multiple key points before and after affine transformation.

[0008] The range of values ​​for each affine transformation parameter is determined based on the affine transformation parameters calculated from multiple real-world scene images.

[0009] Based on the range of values ​​for each affine transformation parameter, multiple deformed digital images are generated through random value selection and affine transformation of the image.

[0010] Furthermore, digital images are obtained from real-world scene images, specifically including:

[0011] Take photos of existing real numbers that need to be generated into digital images, including as much of the distortion and range as possible; obtain digital images by manually cropping the digital portion of the image or by using an OCR detection model to detect the position of the numbers and cropping the image.

[0012] Furthermore, Hough line detection is performed on the numbers to obtain multiple key points after the affine transformation, specifically including:

[0013] For cases where the top or bottom of the number is a straight line, find the straight line segment 'a' at the top of the number through the highest point of the white block, and find the straight line segment 'b' at the bottom of the number through the lowest point of the white block. Obtain the shorter of the two straight line segments and take the midpoint as the key point 'b'.

[0014] If there is a non-straight line at the top or bottom, and the bottom of the number is curved instead of straight, then draw a straight line perpendicular to the top straight line segment a, and make this straight line the longest in the white block. The intersection of the perpendicular line and the bottom of the number is the key point b. The same applies to the case where the top of the number is curved.

[0015] Ignore cases where neither the top nor the bottom is a straight line;

[0016] The two endpoints of the obtained line segment a and a point b are used as the three key points of the image after affine transformation.

[0017] Furthermore, the process involves processing multiple key points obtained before the affine transformation, specifically including:

[0018] Take the left end of line segment a as a fixed point; rotate the right end of line segment a to a position horizontal with the left end to obtain the position of the right end point after rotation; rotate point b to the perpendicular line of line segment a to make it vertical, thus obtaining the position of point b after rotation; finally, obtain the 3 key points before affine transformation.

[0019] Furthermore, the affine transformation parameters are calculated based on the coordinates of multiple key points before and after the affine transformation, specifically including:

[0020] By substituting the six key points corresponding to each other before and after the affine transformation into the following formula, the values ​​of the parameters a1, b1, and c1 of the affine transformation can be obtained.

[0021]

[0022] Where, x afterand y after Let x represent the x-coordinate and y-coordinate of the point after the affine transformation, respectively; before and y before Let a1, b1, and c1 represent the x and y coordinates of the point before the affine transformation, respectively; a1, b1, and c1 are the parameters of the affine transformation.

[0023] Furthermore, based on the value range of each affine transformation parameter, multiple deformed digital images are generated through random value selection and image affine transformation, specifically including:

[0024] The parameter a will be randomly selected from the ranges A, B, and C of the affine transformation parameters a1, b1, and c1. r b r and c r The coordinates of the three corner points of the digital image to be deformed are substituted into the formula to calculate the coordinates of the three points after deformation; the deformed digital image is obtained by affine transformation based on the coordinates of the six points before and after deformation.

[0025] Furthermore, the digital image is binarized, specifically including binarizing the image according to a set threshold.

[0026] Furthermore, the numbers in the image are obtained based on the area of ​​the white blocks in the binarized image. Specifically, this includes: calculating the area of ​​the white blocks in the binarized image and setting a threshold to retain the larger areas.

[0027] According to a second aspect of the present invention, a system for generating digital images of real-world scene deformation is provided, the system comprising:

[0028] The image processing module is used to acquire digital images based on real scene images and to perform binarization processing on the digital images;

[0029] The parameter calculation module is used to obtain the numbers in the image based on the area of ​​the white block in the binarized image, perform Hough line detection on the numbers, obtain multiple key points after affine transformation, process the multiple key points before affine transformation, and calculate the affine transformation parameters based on the coordinates of the multiple key points before and after affine transformation.

[0030] The parameter range acquisition module is used to determine the value range of each affine transformation parameter based on the affine transformation parameters calculated from multiple real scene images.

[0031] The image generation module is used to generate multiple deformed digital images by randomly selecting values ​​and performing affine transformations on the images, based on the range of values ​​for each affine transformation parameter.

[0032] The present invention has the following advantages:

[0033] This invention proposes a method and system for generating deformed digital images of real-world scenes. The method involves acquiring digital images from real-world scene images and binarizing them. The numbers in the image are then extracted based on the area of ​​white blocks in the binarized image. Hough line detection is performed on the numbers to obtain multiple key points after affine transformation, and these key points are processed to obtain the original key points before the affine transformation. Affine transformation parameters are calculated based on the coordinates of these key points. The value range of each affine transformation parameter is determined based on the parameters calculated from multiple real-world scene images. Multiple deformed digital images are generated by randomly selecting values ​​and performing affine transformation on the images, based on the value range of each affine transformation parameter. This method uses only a small number of real-world scene digital images for analysis, automatically obtaining the approximate degree and range of deformation of the actual numbers, thus reducing the time required to debug deformation parameters. Attached Figure Description

[0034] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a method for generating digital images of real-world scene deformations according to Embodiment 1 of the present invention.

[0036] Figure 2 This is a detailed implementation flowchart of a method for generating digital images of real-world scene deformation provided in Embodiment 1 of the present invention;

[0037] Figure 3 This refers to the case where the top or bottom of the numbers are straight lines in a method for generating deformed digital images of a real scene provided in Embodiment 1 of the present invention.

[0038] Figure 4 This refers to a method for generating a digital image of a real-world scene deformation provided in Embodiment 1 of the present invention, where there is a non-straight line at the top or bottom. Detailed Implementation

[0039] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1

[0041] like Figure 1 As shown in the figure, this embodiment proposes a method for generating digital images of real-world scene deformations, which includes:

[0042] S100. Obtain digital images from real-world scene images and perform binarization on the digital images.

[0043] Take photos of existing real digits that need to be generated into digital images, capturing images that include as many possibilities as possible (degree and range of distortion). Manually crop the digits from the images or use an OCR detection model to detect the digit positions and crop the images. Binarize these images by setting a threshold.

[0044] S200. Obtain the numbers in the image based on the area of ​​the white blocks in the binarized image, perform Hough line detection on the numbers, obtain multiple key points after affine transformation, process them to obtain multiple key points before affine transformation, and calculate the affine transformation parameters based on the coordinates of the multiple key points before and after affine transformation.

[0045] like Figure 2 As shown, the area of ​​the white blocks in the binarized image is calculated, and a threshold is set to retain the larger areas. Then, Hough line detection is performed on the numbers in the image.

[0046] like Figure 3 As shown, for cases where the top or bottom of a number is a straight line, find the straight line segment 'a' at the top of the number through the highest point of the white block, and find the straight line segment 'b' at the bottom of the number through the lowest point of the white block. Obtain the shorter of the two straight line segments and take the midpoint as the key point 'b'.

[0047] like Figure 4 As shown, for cases where there is a non-straight line at the top or bottom, if the bottom of the number is curved and not straight, draw a straight line perpendicular to the top straight line segment a, and make this line the longest in the white block. The intersection of the perpendicular line and the bottom of the number is the key point b. The same applies to cases where the top of the number is curved.

[0048] Ignore cases where neither the top nor the bottom is a straight line;

[0049] The two endpoints of the obtained line segment a and a point b are used as the three key points of the image after affine transformation.

[0050] The three points—the two ends of a straight line 'a' and point 'b'—obtained above can be used as the three key points for the affine transformation of the image. Fix the left end of the line and calculate the rotation of the right end until it is horizontal to the left end. Rotate point 'b' to the perpendicular line from line 'a', making it perpendicular to the image, and calculate the position of point 'b' after rotation. Finally, the three key points before the affine transformation are obtained.

[0051] By substituting the six key points corresponding to each other before and after the affine transformation into the following formula 1, the values ​​of the affine transformation parameters a1, b1, and c1 can be obtained.

[0052]

[0053] Where, x after and y after Let x represent the x-coordinate and y-coordinate of the point after the affine transformation, respectively; before and y before Let a1, b1, and c1 represent the x and y coordinates of the point before the affine transformation, respectively; a1, b1, and c1 are the parameters of the affine transformation.

[0054] S300. Determine the value range of each affine transformation parameter based on the affine transformation parameters calculated from multiple real scene images.

[0055] The value ranges A, B, and C of parameters a1, b1, and c1 can be obtained by using multiple real images.

[0056] S400. Generate multiple deformed digital images by randomly selecting values ​​and performing affine transformations on the images, based on the range of values ​​for each affine transformation parameter.

[0057] Randomly select parameter a from A, B, and C. r b r and c r Substituting into Formula 1, substitute the three corners (0,0), (0,width), and (height,0) of the digital image to be deformed into Formula 1 to obtain the three points after deformation. Based on these six points, the getAffineTransform function of opencv-python can be used to perform an affine transformation on the image, thereby generating the desired deformed digital image.

[0058] Example 2

[0059] Corresponding to Embodiment 1 above, this embodiment proposes a system for generating digital images of real-world scene deformations, the system comprising:

[0060] The image processing module is used to acquire digital images based on real scene images and to perform binarization processing on the digital images;

[0061] The parameter calculation module is used to obtain the numbers in the image based on the area of ​​the white block in the binarized image, perform Hough line detection on the numbers, obtain multiple key points after affine transformation, process the multiple key points before affine transformation, and calculate the affine transformation parameters based on the coordinates of the multiple key points before and after affine transformation.

[0062] The parameter range acquisition module is used to determine the value range of each affine transformation parameter based on the affine transformation parameters calculated from multiple real scene images.

[0063] The image generation module is used to generate multiple deformed digital images by randomly selecting values ​​and performing affine transformations on the images, based on the range of values ​​for each affine transformation parameter.

[0064] The functions performed by each component in the real-scene deformation digital image generation system provided in this embodiment of the invention have been described in detail in the above embodiment 1, so they will not be repeated here.

[0065] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for generating digital images of real-world scene deformation, characterized in that, The method includes: Digital images are obtained from real-world scene images, and the digital images are then binarized. The numbers in the image are obtained by measuring the area of ​​the white block in the binarized image, and Hough line detection is performed on the numbers to obtain multiple key points after affine transformation. The numbers are then processed to obtain multiple key points before affine transformation. The affine transformation parameters are calculated based on the coordinates of the multiple key points before and after affine transformation. The range of values ​​for each affine transformation parameter is determined based on the affine transformation parameters calculated from multiple real-world scene images. Based on the range of values ​​for each affine transformation parameter, multiple deformed digital images are generated through random value selection and affine transformation of the image.

2. The method for generating a digital image of a real-world scene deformation according to claim 1, characterized in that, Obtaining digital images from real-world scene images specifically includes: The process involves photographing real digital numbers that require digital image generation, including all degrees and ranges of deformation; obtaining the digital image by manually cropping the digital portion of the image or by using an OCR detection model to detect the position of the numbers and cropping the image.

3. The method for generating a digital image of a real-world scene deformation according to claim 1, characterized in that, Perform Hough line detection on the numbers to obtain multiple key points after affine transformation, specifically including: For cases where the top or bottom of a number is a straight line, find the straight line segment 'a' at the top of the number through the highest point of the white block, and find the straight line segment 'b' at the bottom of the number through the lowest point of the white block. Obtain the shorter of the two straight line segments and take its midpoint as keypoint 'b'. The two endpoints of the longer of the two straight line segments and one keypoint 'b' are used as the three keypoints after the affine transformation of the image. If the bottom of the number is not a straight line, then draw a straight line perpendicular to the top straight line segment a, and make this line the longest in the white block. The intersection of the perpendicular line and the bottom of the number is the key point b. The two endpoints of the straight line segment a and the key point b are the three key points after the affine transformation of the image. The same process is applied to the case where the top of the number is curved. Ignore cases where neither the top nor the bottom is a straight line.

4. The method for generating a digital image of a real-world scene deformation according to claim 3, characterized in that, And process to obtain multiple key points before the affine transformation, specifically including: Take the left end of line segment a as a fixed point; rotate the right end of line segment a to a position horizontal with the left end to obtain the position of the right end point after rotation; rotate point b to the perpendicular line of line segment a to make it vertical, thus obtaining the position of point b after rotation; finally, obtain the 3 key points before affine transformation.

5. The method for generating a digital image of a real-world scene deformation according to claim 4, characterized in that, The affine transformation parameters are calculated based on the coordinates of multiple key points before and after the affine transformation, specifically including: By substituting the six key points corresponding to each other before and after the affine transformation into the following formula, the values ​​of the parameters a1, b1 and c1, a2, b2 and c2 of the affine transformation can be obtained. Where, x after and y after Let x represent the x-coordinate and y-coordinate of the point after the affine transformation, respectively; before and y before a1, b1, c1, a2, b2, and c2 represent the x and y coordinates of the point before the affine transformation, respectively; a1, b1, c1, a2, b2, and c2 are the parameters of the affine transformation.

6. The method for generating a digital image of a real-world scene deformation according to claim 5, characterized in that, Based on the value range of each affine transformation parameter, multiple deformed digital images are generated through random value selection and image affine transformation, specifically including: The parameter a will be randomly selected from the ranges A, B, and C of the affine transformation parameters a1, b1, and c1. r b r and c r , where a r b r and c r The formula covers the random selection of a2, b2, and c2; and the coordinates of the three corner points of the digital image to be deformed are substituted into the formula to calculate the coordinates of the three points after deformation; and the deformed digital image is obtained by affine transformation based on the coordinates of the six points before and after deformation.

7. The method for generating a digital image of a real-world scene deformation according to claim 1, characterized in that, The digital image is binarized, specifically by binarizing the image according to a set threshold.

8. The method for generating a digital image of a real-world scene deformation according to claim 1, characterized in that, The method involves obtaining numbers from a binary image based on the area of ​​white blocks, specifically by calculating the area of ​​white blocks in the binarized image and setting a threshold to retain the larger areas.

9. A system for generating digital images of real-world scene deformations, characterized in that, The system includes: The image processing module is used to acquire digital images based on real scene images and to perform binarization processing on the digital images; The parameter calculation module is used to obtain the numbers in the image based on the area of ​​the white block in the binarized image, perform Hough line detection on the numbers, obtain multiple key points after affine transformation, process the multiple key points before affine transformation, and calculate the affine transformation parameters based on the coordinates of the multiple key points before and after affine transformation. The parameter range acquisition module is used to determine the value range of each affine transformation parameter based on the affine transformation parameters calculated from multiple real scene images. The image generation module is used to generate multiple deformed digital images by randomly selecting values ​​and performing affine transformations on the images, based on the range of values ​​for each affine transformation parameter.

Images