Method for quickly acquiring image splicing data set with robustness

Abstract

The invention provides a method for quickly acquiring an image splicing data set with high robustness. The method comprises the following steps: S1, adding characters on a blank image: adding characters with various gray levels on one blank image; s2, acquiring key points in the blank image: extracting the key points in the image by using an SIFT algorithm, and storing the positions of the key points for later use; s3, fusing various images with complex backgrounds with the blank background image: carrying out weighted fusion on the images with complex backgrounds and the blank background image in the step S2 by adopting a cv2. AddWeighted function in the OpenCV library to obtain a fused new image; and S4, generating a large number of image splicing training data sets with complex backgrounds and rich key point information through the image fused by the cv2. AddWeighted and the key point coordinate information formed by the SIFT algorithm in the step S2. Through the method, character image splicing training data with high complexity and high robustness can be generated in batches.

Description

Technical Field

[0001] The invention belongs to the technical field of digital image processing, and in particular relates to a method for quickly acquiring a highly robust image splicing data set. Background Art

[0002] Nowadays, deep learning technology has been applied to various fields and has achieved quite good results. The development of deep learning requires not only algorithms and algorithms, but also data support. Good training data can often achieve better results. Especially in the field of image matching, there is no available public data set to use, and manual labeling is time-consuming and laborious.

[0003] Data is obtained by manual annotation, which is also the mainstream method for obtaining training data. Data enhancement is to generate more data by rotating, translating, scaling, and cropping the original data. Synthetic data is generated by placing a virtual camera on a 3D model to generate an image containing camera posture and scene depth information. For example, objects with corner points such as squares, rectangles, and polygons are generated in the scene, and the image corner points are used as key point information for training.

[0004] However, manual labeling requires a lot of manpower and a long labeling cycle when large amounts of data are required. The data generated by data enhancement methods have certain limitations. They are limited to raw data and cannot be expanded to more scenarios. Only key point information of a certain number of shapes (such as squares, rectangles, etc.) can be obtained through 3D model synthesis, and it is impossible to generate a large amount of training data suitable for the field of text and image splicing.

[0005] In addition, technical terms commonly used in this field include:

[0006] Image stitching: The process of stitching two images into a complete image based on the same feature points. SIFT (Scale-Invariant Feature Transform): The SIFT algorithm is a feature extraction algorithm used in image processing, which can match the features of images at different scales and rotations. Summary of the invention

[0007] In order to solve the above problems, the purpose of this application is to use image processing and image synthesis technology to generate highly complex and extremely robust text image splicing training data in large batches.

[0008] Specifically, the present invention provides a method for quickly acquiring a highly robust image stitching dataset, the method comprising the following steps:

[0009] S1. Add text on a blank image: Add text of various gray levels on a blank image;

[0010] S2. Get the key points in the blank image: Use the black text on white background image generated in step S1 and use SIFT algorithm to extract the key points in the image. The key points are found by comparing the difference between each pixel and the surrounding pixels. Therefore, more key points can be found in this way, and the key point positions are saved in a txt document for later use.

[0011] S3. Fusion of the complex background image with the blank background image: Use the cv2.addWeighted function in the OpenCV library to perform weighted fusion of the complex background image and the blank background image in step S2 to obtain a fused new image. The addWeighted function is an image fusion operation, which multiplies the pixels at the same position in the two images by a given coefficient, and the value of the coefficient is between 0 and 1. Then, the two pixel values ​​multiplied by the coefficient are added together, and the result of the addition is the current pixel value of the fused image, which is expressed as follows: output = alphaA + betaB, where A represents the image generated by step S1, B represents the image with a complex background, alpha and beta represent the weight coefficients corresponding to image A and image B, and the two images to be fused must be the same size;

[0012] S4, generate an image stitching training data set. The key point position information of the fused image is the same as the key point position information in step S2 and has been saved by step S2. Use cv2.addWeighted to fuse the image and the key point coordinate information formed by the SIFT algorithm in the corresponding step S2 to generate image stitching training data with complex background and rich key point information. This is just a process of generating one set of training data, and the training data is the synthesized image and the saved key point information. The training set requires a large amount of data, that is, data sufficient for the business scenario. This step can be repeated to batch generate data sets. It is recommended to replace the complex background image required for fusion in each cycle to increase the complexity and diversity of the generated data set. The background image required for fusion needs to be manually collected according to the actual business scenario.

[0013] The SIFT algorithm described in step S2 is a computer vision algorithm for detecting key points and descriptors in an image. The steps of using the SIFT algorithm are as follows:

[0014] 1) Scale space extreme point detection: Use Gaussian filter to build a scale space pyramid and find extreme points in the image at different scale levels. These points are used as candidates for key points;

[0015] 2) Key point positioning: accurately locate the candidate key points to improve accuracy. In the local neighborhood of each candidate point, the Hessian matrix is ​​used to detect the exact position and scale of the key points, and key points with low contrast and weak edge response are eliminated;

[0016] 3),Key point direction matching: In the image area around the key point, build a gradient histogram and find the main direction;

[0017] 4) Generate key point descriptors: In the image area around the key point, construct the key point descriptors according to the main directions. The descriptors usually include gradient information in multiple directions and scales to represent the characteristics of the key point.

[0018] 5), Matching keypoints: Use descriptors to compare keypoints in different images to find similar keypoint pairs.

[0019] Python implementation of the SIFT algorithm:

[0020] 1) Define the scale number and Octave number of the Gaussian difference pyramid, expressed as:

[0021] num_octaves=4

[0022] num_scales_per_octave=5;

[0023] Define the initial scale and Gaussian kernel size in the Gaussian difference pyramid, expressed as:

[0024] sigma = 1.6

[0025] k=2**(1.0 / num_scales_per_octave);

[0026] Create an empty keypoint list, represented as:

[0027] keypoints = [];

[0028] First, generate a Gaussian difference pyramid, expressed as:

[0029] for octave in range(num_octaves):

[0030] for scale in range(num_scales_per_octave):

[0031] Construct the Gaussian kernel, expressed as:

[0032] kernel_size = int(6*sigma)

[0033] if kernel_size%2==0:

[0034] kernel_size+=1

[0035] gaussian_kernel=create_gaussian_kernel(kernel_size,sigma);

[0036] Gaussian filtering is performed on the image, expressed as:

[0037] blurred_image=convolve_with_gaussian(image,gaussian_kernel);

[0038] Calculate the Gaussian difference, expressed as:

[0039] dog=compute_difference_of_gaussians(prev_blurred_image,blurred_image);

[0040] Find local extreme points in the DOG image, expressed as:

[0041] local_extrema=find_local_extrema(dog);

[0042] 2) Filter and locate key points, expressed as:

[0043] keypoints+=localize_keypoints(local_extrema,blurred_image,sigma);

[0044] Update the scale parameter, expressed as:

[0045] sigma*=k;

[0046] Reduce the image size by half and process it to the next octave, expressed as:

[0047] image=downsample_image(image);

[0048] 3) Calculate the main direction and descriptor for each key point, expressed as:

[0049] for keypoint in keypoints:

[0050] keypoint.orientation=compute_keypoint_orientation(keypoint,blurred_image)

[0051] keypoint.descriptor=compute_descriptor(keypoint,blurred_image);

[0052] 4) Return all key points and their descriptors, expressed as:

[0053] return keypoints.

[0054] In step S3, in order to enhance the background complexity of the fused image, alpha, i.e., the value corresponding to the blank background image, is set to 0.1, and beta, i.e., the value corresponding to the complex background image, is set to 0.9.

[0055] Therefore, the advantage of the present application is that the method can generate large batches of highly complex and robust text-image splicing training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0057] Figure 1 Schematic diagram of step S1 in the embodiment of the method.

[0058] Figure 2 is a schematic diagram of step S2 in the embodiment of the method.

[0059] Figure 3(1) to Figure 3(3) show three different backgrounds and Figure 1 The effect of weighted fusion.

[0060] Figure 4 This is the main code diagram of the Python implementation of the SIFT algorithm used in this method.

[0061] Figure 5 is a schematic diagram of the process of this method. DETAILED DESCRIPTION

[0062] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0063] This method belongs to the field of computer vision and deep learning, and proposes a method to quickly obtain a very robust image stitching dataset, such as Figure 5 As shown, the main implementation steps of the method are as follows:

[0064] Step S1. Add text to a blank image: Add text of various gray levels (taking "Hello" as an example) to a blank image, such as Figure 1 As shown;

[0065] Step S2. Get the key points in the blank image: Use the white background and black text image generated in step S1 and use the SIFT algorithm to extract the key points in the image. Since there is a huge contrast between the pure white background and the black text in the image generated in step S1, and the key points are found by comparing the difference between each pixel and the surrounding pixels, more key points can be found in a complex background in this way, and the key point positions can be saved for later use. The key points found are Figure 2 The SIFT algorithm is a computer vision algorithm used to detect key points and descriptors in images. It is well-known for its advantages such as scale invariance, rotation invariance, and the ability to detect key point features at different sizes. The steps for using the SIFT algorithm are as follows:

[0066] 1) Scale space extreme point detection: Use Gaussian filter to build a scale space pyramid and find extreme points in the image at different scale levels. These points are used as candidates for key points;

[0067] 2) Key point positioning: accurately locate the candidate key points to improve accuracy. In the local neighborhood of each candidate point, the Hessian matrix is ​​used to detect the exact position and scale of the key points, and key points with low contrast and weak edge response are eliminated;

[0068] 3),Key point direction matching: In the image area around the key point, build a gradient histogram and find the main direction;

[0069] 4) Generate key point descriptors: In the image area around the key point, construct the key point descriptors according to the main directions. The descriptors usually include gradient information in multiple directions and scales to represent the characteristics of the key point.

[0070] 5), Matching keypoints: Use descriptors to compare keypoints in different images to find similar keypoint pairs.

[0071] The Python implementation of SIFT is as follows Figure 4 As shown:

[0072] 1) Define the scale number and Octave number of the Gaussian difference pyramid, expressed as:

[0073] num_octaves=4

[0074] num_scales_per_octave=5;

[0075] Define the initial scale and Gaussian kernel size in the Gaussian difference pyramid, expressed as:

[0076] sigma = 1.6

[0077] k=2**(1.0 / num_scales_per_octave);

[0078] Create an empty keypoint list, represented as:

[0079] keypoints = [];

[0080] First, generate a Gaussian difference pyramid, expressed as:

[0081] for octave in range(num_octaves):

[0082] for scale in range(num_scales_per_octave):

[0083] Construct the Gaussian kernel, expressed as:

[0084] kernel_size = int(6*sigma)

[0085] if kernel_size%2==0:

[0086] kernel_size+=1

[0087] gaussian_kernel=create_gaussian_kernel(kernel_size,sigma);

[0088] Gaussian filtering is performed on the image, expressed as:

[0089] blurred_image=convolve_with_gaussian(image,gaussian_kernel);

[0090] Calculate the Gaussian difference, expressed as:

[0091] dog=compute_difference_of_gaussians(prev_blurred_image,blurred_image);

[0092] Find local extreme points in the DOG image, expressed as:

[0093] local_extrema=find_local_extrema(dog);

[0094] 2) Filter and locate key points, expressed as:

[0095] keypoints+=localize_keypoints(local_extrema,blurred_image,sigma);

[0096] Update the scale parameter, expressed as:

[0097] sigma*=k;

[0098] Reduce the image size by half and process it to the next octave, expressed as:

[0099] image=downsample_image(image);

[0100] 3) Calculate the main direction and descriptor for each key point, expressed as:

[0101] for keypoint in keypoints:

[0102] keypoint.orientation=compute_keypoint_orientation(keypoint,blurred_image)

[0103] keypoint.descriptor=compute_descriptor(keypoint,blurred_image);

[0104] 4) Return all key points and their descriptors, expressed as:

[0105] return keypoints.

[0106] Because here we only need to find the key points in a single image, no matching is required.

[0107] Step S3. Merge the background image with the blank background image:

[0108] There is no requirement for the number of background images here, they only need to be images in actual application scenarios; the cv2.addWeighted function in the OpenCV library is used to perform weighted fusion of the image with complex background and the blank background image in step S2 to obtain a fused new image. The addWeighted function is an image fusion operation, which multiplies the pixels at the same position in the two images by a given coefficient, the value of the coefficient is between 0 and 1, and then the two pixel values ​​multiplied by the coefficient are added. The result of the addition is the current pixel value of the fused image, which is expressed as follows: output = alphaA + betaB, wherein A represents the image generated by step S1, B represents the image with a complex background, alpha and beta represent the weight coefficients corresponding to image A and image B, and the two images to be fused must be of the same size; in order to enhance the background complexity of the fused image, alpha, i.e., the value corresponding to the blank background image, is set to 0.1, and beta, i.e., the value corresponding to the complex background image, is set to 0.9.

[0109] Step S4, generate image stitching training data set:

[0110] The key point position information of the fused image is the same as the key point position information in step S2 and has been saved by step S2. By using the fused image by cv2.addWeighted and the key point coordinate information formed by the SIFT algorithm in the corresponding step S2, a large number of image stitching training data sets with complex backgrounds and rich key point information can be generated. Image stitching training data with complex backgrounds and rich key point information can be generated. This is just a process for generating one set of training data, and the training data is the synthesized image and the saved key point information. The training set requires a large amount of data, that is, sufficient data for the business scenario. This step can be repeated to generate data sets in batches. It is recommended to replace the complex background image required for fusion in each cycle to increase the complexity and diversity of the generated data set. The background image required for fusion needs to be manually collected according to the actual business scenario.

[0111] The final fused image, as shown in Figure 3(1) to Figure 3(3), is the final fusion effect after using three background images respectively.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for quickly acquiring a highly robust image stitching dataset, characterized in that: The method comprises the following steps: S1. Add text on a blank image: Add text of various gray levels on a blank image; S2. Get the key points in the blank image: Use the black text on white background image generated in step S1 and use SIFT algorithm to extract the key points in the image. The key points are found by comparing the difference between each pixel and the surrounding pixels. Therefore, more key points can be found in this way, and the key point positions are saved in a txt document for later use. S3. Fusion of the complex background image with the blank background image: Use the cv2.addWeighted function in the OpenCV library to perform weighted fusion of the complex background image and the blank background image in step S2 to obtain a fused new image. The addWeighted function is an image fusion operation, which multiplies the pixels at the same position in the two images by a given coefficient, and the value of the coefficient is between 0 and 1. Then, the two pixel values ​​multiplied by the coefficient are added together, and the result of the addition is the current pixel value of the fused image, which is expressed as follows: output = alphaA + betaB, where A represents the image generated by step S1, B represents the image with a complex background, alpha and beta represent the weight coefficients corresponding to image A and image B, and the two images to be fused must be the same size; S4, generate an image stitching training data set. The key point position information of the fused image is the same as the key point position information in step S2 and has been saved by step S2. Use cv2.addWeighted to fuse the image and the key point coordinate information formed by the SIFT algorithm in the corresponding step S2 to generate image stitching training data with complex background and rich key point information: this is just a process of generating a set of training data, and the training data is the synthesized image and the saved key point information; the training set requires a large amount of data, that is, sufficient data for the business scenario; this step can be repeated to batch generate data sets, and it is recommended to replace the complex background image required for fusion in each cycle to increase the complexity and diversity of the generated data set; the background image required for fusion needs to be manually collected according to the actual business scenario.

2. A method for quickly acquiring a highly robust image stitching dataset according to claim 1, characterized in that: The SIFT algorithm described in step S2 is a computer vision algorithm for detecting key points and descriptors in an image. The steps of using the SIFT algorithm are as follows: 1) Scale space extreme point detection: Use Gaussian filter to build a scale space pyramid and find extreme points in the image at different scale levels. These points are used as candidates for key points; 2) Key point positioning: accurately locate the candidate key points to improve accuracy. In the local neighborhood of each candidate point, the Hessian matrix is ​​used to detect the exact position and scale of the key points, and key points with low contrast and weak edge response are eliminated; 3),Key point direction matching: In the image area around the key point, build a gradient histogram and find the main direction; 4) Generate key point descriptors: In the image area around the key point, construct the key point descriptors according to the main directions. The descriptors usually include gradient information in multiple directions and scales to represent the characteristics of the key point. 5), Matching keypoints: Use descriptors to compare keypoints in different images to find similar keypoint pairs.

3. The method for quickly acquiring a highly robust image stitching dataset according to claim 2, characterized in that: Python implementation of the SIFT algorithm: 1) Define the scale number and Octave number of the Gaussian difference pyramid, expressed as: num_octaves=4 num_scales_per_octave=5; Define the initial scale and Gaussian kernel size in the Gaussian difference pyramid, expressed as: sigma = 1.6 k=2**(1.0 / num_scales_per_octave); Create an empty keypoint list, represented as: keypoints = []; First, generate a Gaussian difference pyramid, expressed as: for octave in range(num_octaves): for scale in range(num_scales_per_octave): Construct the Gaussian kernel, expressed as: kernel_size = int(6*sigma) if kernel_size%2==0: kernel_size+=1 gaussian_kernel=create_gaussian_kernel(kernel_size,sigma); Gaussian filtering is performed on the image, expressed as: blurred_image=convolve_with_gaussian(image,gaussian_kernel); Calculate the Gaussian difference, expressed as: dog=compute_difference_of_gaussians(prev_blurred_image,blurred_image); Find local extreme points in the DOG image, expressed as: local_extrema=find_local_extrema(dog); 2) Filter and locate key points, expressed as: keypoints+=localize_keypoints(local_extrema,blurred_image,sigma); Update the scale parameter, expressed as: sigma*=k; Reduce the image size by half and process it to the next octave, expressed as: image=downsample_image(image); 3) Calculate the main direction and descriptor for each key point, expressed as: for keypoint in keypoints: keypoint.orientation=compute_keypoint_orientation(keypoint,blurred_image) keypoint.descriptor=compute_descriptor(keypoint,blurred_image); 4) Return all key points and their descriptors, expressed as: return keypoints.

4. The method for quickly acquiring a highly robust image stitching dataset according to claim 1, characterized in that: In step S3, in order to enhance the background complexity of the fused image, alpha, i.e., the value corresponding to the blank background image, is set to 0.1, and beta, i.e., the value corresponding to the complex background image, is set to 0.9.

Images