A method for quickly obtaining a highly robust image stitching dataset

Abstract

The application provides a method for quickly obtaining a highly robust image splicing data set, comprising the following steps: S1, adding text on a blank image: adding text of various gray levels on a blank image; S2, obtaining key points in the blank image: using a SIFT algorithm to extract key points in the image and saving the key point positions for later use; S3, fusing a plurality of background complex images with the blank background image: using a cv2.addWeighted function in an OpenCV library to perform weighted fusion on the background complex images and the blank background image in step S1 to obtain a new fused image; and S4, generating a large number of image splicing training data sets with complex backgrounds and rich key point information by using the cv2.addWeighted fused image and the key point coordinate information formed by the SIFT algorithm in step S2. A large number of complex and highly robust text image splicing training data can be generated in batches by the method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital image processing, and particularly relates to a method for quickly obtaining an image splicing dataset with high robustness. BACKGROUND

[0002] Nowadays, deep learning technology has been applied to various fields and has achieved quite good results. The development of deep learning not only needs algorithms and algorithms, but also needs data support. Good training data can often achieve better results. Especially in the field of image matching, there is no available public dataset for use, and manual annotation is time-consuming and laborious.

[0003] The data is obtained by manual annotation, which is also the mainstream and commonly used method for obtaining training data. Data augmentation generates more data by rotating, translating, scaling, cropping, etc. on the original data. Synthetic data is generated by placing virtual cameras on 3D models to generate images containing camera poses and scene depth information. For example, generating objects with corner points such as squares, rectangles, and polygons in the scene, and using image corner points as key point information for training.

[0004] However, the manual annotation method requires a large amount of manpower, and in the case of large data volume demand, the annotation period is long. The data generated by the data augmentation method has certain limitations and is limited to the original data, which cannot expand more scenes. The 3D model synthesis method can only obtain a certain number of shape key point information (such as squares, rectangles, etc.), and cannot generate a large amount of training data suitable for the field of text image splicing.

[0005] In addition, the commonly used technical terms in the field include:

[0006] Image splicing: the process of splicing two images into a complete image according to the same feature points.

[0007] SIFT (Scale-Invariant Feature Transform): SIFT algorithm is a feature extraction algorithm used in image processing, which can match features in images at different scales and rotations. SUMMARY

[0008] In order to solve the above problems, the purpose of the present application is to use image processing and image synthesis technology to generate a large number of complex and highly robust text image splicing training data.

[0009] Specifically, the present application provides a method for quickly obtaining an image splicing dataset with high robustness, which comprises the following steps:

[0010] S1. Add text on a blank image: add text of various gray levels on a blank image;

[0011] S2. Obtain key points in the blank image: using the white background black text image generated in step S1, the SIFT algorithm is used to extract the key points in the image. The key points are determined by comparing the difference between each pixel and the surrounding pixels, so more key points can be found in this way, and the key point positions are saved in a txt document for later use;

[0012] S3. Fuse the image with complex background with the blank background image: use the cv2.addWeighted function in the OpenCV library to fuse the image with complex background with the blank background image in step S1 to obtain a new fused image. The addWeighted function is an image fusion operation that multiplies the pixels at the same position in the two images by a given coefficient, the value of the coefficient is between 0 and 1, and then adds the pixel values multiplied by the coefficient, the result of the addition is the current pixel value of the fused image, which is expressed as: output = alphaA+betaB, where A represents the image generated in step S1, B represents the image with complex background, alpha and beta represent the weight coefficients of A image and B image, and the size of the two images to be fused must be the same;

[0013] S4, generate image stitching training data set, the key point position information of the fused image is the same as that in step S2 and has been saved in step S2, using the cv2.addWeighted fused image and the key point coordinate information formed by the SIFT algorithm in step S2, an image stitching training data with complex background and rich key point information can be generated; here is only the generation process of one training data, the training data is the synthesized image and the saved key point information; the training set needs a large amount of data, that is, enough data according to the business scenario; the data set can be generated in batches by repeating this step, and the complex background image needed for fusion is changed every time to increase the complexity and diversity of the generated data set; the background image needed for fusion needs to be manually collected according to the actual business scenario.

[0014] The SIFT algorithm in step S2 is a computer vision algorithm for detecting key points and descriptors in images. The steps of using the SIFT algorithm are as follows:

[0015] 1) Scale space extreme point detection: use a Gaussian filter to build a scale space pyramid, and find extreme points in the image at different scale levels, which are used as candidates for key points;

[0016] 2), keypoint localization: fine-tune the candidate keypoint locations to improve accuracy. In a small neighborhood around each candidate, use the Hessian matrix to detect the precise location and scale of the keypoint. Discard keypoint candidates with low contrast and weak edge responses.

[0017] 3), keypoint orientation matching: construct a histogram of gradient orientations in the image region around the keypoint and find the dominant orientation.

[0018] 4), generate keypoint descriptors: construct a descriptor for the keypoint based on the dominant orientation in the image region around the keypoint. The descriptor typically includes gradient information at multiple orientations and scales, representing the keypoint's features.

[0019] 5), match keypoints: use the descriptors to compare keypoint candidates in different images to find similar keypoint pairs.

[0020] Python implementation of the SIFT algorithm:

[0021] 1) Define the number of scales and octaves for the Gaussian difference pyramid, represented as:

[0022] num_octaves = 4

[0023] num_scales_per_octave = 5;

[0024] Define the initial scale and Gaussian kernel size in the Gaussian difference pyramid, represented as:

[0025] sigma = 1.6

[0026] k = 2 ** (1.0 / num_scales_per_octave);

[0027] Create an empty list of keypoints, represented as:

[0028] keypoints = [];

[0029] First, generate the Gaussian difference pyramid, represented as:

[0030] for octave in range(num_octaves):

[0031] for scale in range(num_scales_per_octave):

[0032] Construct the Gaussian kernel, represented as:

[0033] kernel_size = int(6 * sigma)

[0034] if kernel_size % 2 == 0:

[0035] kernel_size += 1

[0036] gaussian_kernel = create_gaussian_kernel(kernel_size, sigma);

[0037] Apply a Gaussian filter to the image, denoted as:

[0038] blurred_image = convolve_with_gaussian(image, gaussian_kernel);

[0039] Compute the difference of Gaussians, denoted as:

[0040] dog = compute_difference_of_gaussians(prev_blurred_image, blurred_image);

[0041] Find local extrema in the DOG image, denoted as:

[0042] local_extrema = find_local_extrema(dog);

[0043] 2) Filter and localize keypoints, denoted as:

[0044] keypoints += localize_keypoints(local_extrema, blurred_image, sigma);

[0045] Update the scale parameter, denoted as:

[0046] sigma *= k;

[0047] Downsample the image by a factor of two for the next octave, denoted as:

[0048] image = downsample_image(image);

[0049] 3) Compute the dominant orientation and descriptor for each keypoint, denoted as:

[0050] for keypoint in keypoints:

[0051] keypoint.orientation = compute_keypoint_orientation(keypoint,blurred_image)

[0052] keypoint.descriptor = compute_descriptor(keypoint, blurred_image);

[0053] 4) Return all keypoints and their descriptors, denoted as:

[0054] return keypoints.

[0055] In step S3, in order to enhance the background complexity of the fused image, the value of alpha corresponding to the blank background image is set to 0.1, and the value of beta corresponding to the complex background image is set to 0.9.

[0056] Therefore, the method can generate a large number of complex and robust text image splicing training data. BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application.

[0058] Figure 1 is a schematic diagram of step S1 in the method embodiment.

[0059] Figure 2 is a schematic diagram of step S2 in the method embodiment.

[0060] Fig. 3 (1) - Fig. 3 (3) are three different backgrounds and Figure 1 the effect diagram after weighted fusion.

[0061] Figure 4 is a schematic diagram of the main code of the Python implementation of the SIFT algorithm used in the method.

[0062] Figure 5 is a flowchart of the method. DETAILED DESCRIPTION

[0063] In order to more clearly understand the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.

[0064] The method belongs to the field of computer vision and deep learning, and proposes a method for quickly obtaining a highly robust image stitching dataset, as shown in Figure 5 The main implementation steps of the method are as follows:

[0065] Step S1. Adding text on a blank image: adding text of various gray levels (taking “Hello” as an example) on a blank image, as shown in Figure 1 .

[0066] Step S2. Obtaining key points in the blank image: using the white background and black font image generated in step S1, the SIFT algorithm is used to extract the key points in the image. Since there is a great contrast between the pure white background and the black font in the image generated in step S1, and the key points are determined by comparing the difference between each pixel and the surrounding pixels, more key points can be found in complex backgrounds in this way, and the key point positions are saved for later use. The key points found, i.e., the points as shown in Figure 2 , are the SIFT algorithm, a computer vision algorithm for detecting key points and descriptors in images. It is famous for its scale invariance, rotation invariance, and ability to detect key point features at different sizes. The steps for using the SIFT algorithm are as follows:

[0067] 1) Scale space extreme point detection: use a Gaussian filter to construct a scale space pyramid, and find the extreme points in the image at different scale levels, which are used as candidates for key points;

[0068] 2) Key point positioning: accurately position the candidate key points to improve accuracy. In the local neighborhood of each candidate point, the Hessian matrix is used to detect the precise location and scale of the key point, and the key points with low contrast and weak edge response are removed;

[0069] 3) Key point direction matching: construct a gradient histogram in the image region around the key point and find the main direction;

[0070] 4) Generate key point descriptors: construct the key point descriptors around the key point based on the main direction. The descriptors usually include gradient information in multiple directions and scales, which are used to represent the features of the key point;

[0071] 5) Match key points: use the descriptors to compare the key points in different images to find similar key point pairs.

[0072] The Python implementation of the SIFT is shown in Figure 4 .

[0073] 1) Define the number of scales and Octaves for the Gaussian Difference Pyramid, denoted as:

[0074] num_octaves = 4

[0075] num_scales_per_octave = 5;

[0076] Define the initial scale and Gaussian kernel size in the Gaussian Difference Pyramid, denoted as:

[0077] sigma = 1.6

[0078] k = 2 ** (1.0 / num_scales_per_octave);

[0079] Create an empty list of keypoints, denoted as:

[0080] keypoints = [];

[0081] First, generate the Gaussian Difference Pyramid, denoted as:

[0082] for octave in range(num_octaves):

[0083] for scale in range(num_scales_per_octave):

[0084] Build the Gaussian kernel, denoted as:

[0085] kernel_size = int(6 * sigma)

[0086] if kernel_size % 2 == 0:

[0087] kernel_size += 1

[0088] gaussian_kernel = create_gaussian_kernel(kernel_size, sigma);

[0089] Gaussian filter the image, denoted as:

[0090] blurred_image = convolve_with_gaussian(image, gaussian_kernel);

[0091] Compute the Gaussian Difference, denoted as:

[0092] dog = compute_difference_of_gaussians(prev_blurred_image, blurred_image);

[0093] Find local extrema in the DOG image, denoted as:

[0094] local_extrema = find_local_extrema(dog);

[0095] 2) Filter and localize keypoints, denoted as:

[0096] keypoints += localize_keypoints(local_extrema, blurred_image, sigma);

[0097] Update the scale parameter, denoted as:

[0098] sigma *= k;

[0099] Downsample the image by a factor of two for the next octave, denoted as:

[0100] image = downsample_image(image);

[0101] 3) Compute the dominant orientation and descriptor for each keypoint, denoted as:

[0102] for keypoint in keypoints:

[0103] keypoint.orientation = compute_keypoint_orientation(keypoint,blurred_image)

[0104] keypoint.descriptor = compute_descriptor(keypoint, blurred_image);

[0105] 4) Return all keypoints and their descriptors, denoted as:

[0106] return keypoints.

[0107] Since we only need to find keypoints in a single image here, no matching is required.

[0108] Step S3. Merge the background image with the blank background image:

[0109] The number of background images is not required here, only the images in the actual application scenario; the blank background image in step S1 is weighted and fused with the image with complex background to obtain a new fused image by using the cv2.addWeighted function in the OpenCV library, the addWeighted function is an image fusion operation, respectively multiplying the pixels at the same position in the two images by a given coefficient, the value of the coefficient is between 0 and 1, and then adding the two pixel values multiplied by the coefficient, the result of the addition is the current pixel value of the fused image, which is expressed by the formula: output = alphaA+betaB, wherein A represents the image generated by step S1, B represents the image with complex background, alpha and beta represent the weight coefficients corresponding to the A image and the B image, and the sizes of the two images to be fused must be the same; in order to enhance the background complexity of the fused image, here the value of alpha corresponding to the blank background image is set to 0.1, and the value of beta corresponding to the complex background image is set to 0.9.

[0110] Step S4, generating image stitching training dataset:

[0111] The key point position information of the fused image is the same as that in step S2 and has been saved in step S2, and the cv2.addWeighted fused image and the key point coordinate information formed by the SIFT algorithm in step S2 are used, so that a large number of image stitching training datasets with complex background and rich key point information can be generated, that is, image stitching training data with complex background and rich key point information can be generated; here is only the generation process of one training data, the one training data is the synthesized image and the saved key point information; the training set needs a large amount of data, that is, enough data according to the business scenario; the data set can be generated in batches by circulating this step, and it is recommended to replace the complex background image required for fusion each time to increase the complexity and diversity of the generated data set; the background image required for fusion needs to be manually collected according to the actual business scenario.

[0112] The finally fused image is shown in FIGS. 3(1)-3(3).

[0113] The above only describes the preferred embodiments of the present application and is not used to limit the present application. The embodiments of the present application can be variously changed and modified for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for rapidly acquiring a highly robust image stitching dataset, characterized in that, The method includes the following steps: S1. Add text to a blank image: Add text of various grayscale levels to a blank image; S2. Obtain key points in the blank image: Using the white background and black text image generated in step S1, the SIFT algorithm is used to extract key points in the image. The key points are found by comparing the differences between each pixel and the surrounding pixels. Therefore, more key points can be found in this way. The key point positions are saved to a txt document for later use. S3. Merging the image with a complex background with the image with a blank background: The cv2.addWeighted function in the OpenCV library is used to perform a weighted fusion of the image with a complex background and the image with a blank background from step S1 to obtain a new image after fusion. The addWeighted function is an image fusion operation that multiplies the pixels at the same position in the two images by a given coefficient, the value of which is between 0 and 1. Then, the two pixel values ​​multiplied by the coefficient are added together. The result of the sum is the current pixel value of the fused image, which is expressed by the formula: output = alphaA + betaB, where A represents the image generated in step S1, B represents the image with a complex background, and alpha and beta represent the weight coefficients corresponding to the images A and B. The two images to be fused must be the same size. S4: Generate an image stitching training dataset. The keypoint location information of the fused image is the same as that in step S2 and has been saved in step S2. Using cv2.addWeighted to fuse the image and the keypoint coordinate information formed by the SIFT algorithm in step S2, image stitching training data with complex background and rich keypoint information can be generated. This is only the process of generating one training dataset, which is the synthesized image and the saved keypoint information. The training set requires a large amount of data, which is enough for the business scenario. Repeating this step can generate datasets in batches. Each time the complex background image required for fusion is changed, the complexity and diversity of the generated dataset are increased. The background image required for fusion needs to be manually collected according to the actual business scenario.

2. The method for rapidly obtaining a highly robust image stitching dataset according to claim 1, characterized in that, The SIFT algorithm described in step S2 is a computer vision algorithm used to detect key points and descriptors in an image. The steps for using the SIFT algorithm are as follows: 1) Scale-space extremum detection: A scale-space pyramid is constructed using a Gaussian filter to find extrema in the image at different scale levels. These points are used as candidates for key points. 2) Key point localization: Accurately locate candidate key points to improve accuracy. Within the local neighborhood of each candidate point, the Hessian matrix is ​​used to detect the precise location and scale of the key points, and key points with low contrast and weak edge response are eliminated. 3) Key point orientation matching: Construct a gradient histogram within the image region surrounding the key point and find the main orientation; 4) Generate keypoint descriptors: Within the image region surrounding the keypoint, construct a descriptor for the keypoint based on the main orientation. The descriptor typically includes gradient information in multiple orientations and scales to represent the features of the keypoint. 5) Matching key points: Use descriptors to compare key points in different images to find similar key point pairs.

3. The method for rapidly obtaining a highly robust image stitching dataset according to claim 2, characterized in that, The Python implementation of the SIFT algorithm: 1) Define the scale number and octave number of the Gaussian difference pyramid; Define the initial scale and Gaussian kernel size in the difference-of-gaussians pyramid; Create an empty list of key points; First, generate the Gaussian difference pyramid; Construct a Gaussian kernel; Apply Gaussian filtering to the image; Calculate the difference of Gaussians; Find local extrema in a DOG image; 2) Filtering and locating key points; Update scale parameters; Reduce the image size by half and process it for the next octave; 3) Calculate the principal direction and descriptor for each keypoint; 4) Return all keypoints and their descriptors.

4. The method for rapidly obtaining a highly robust image stitching dataset according to claim 1, characterized in that, In step S3, in order to enhance the background complexity of the fused image, the value of alpha, which corresponds to the blank background image, is set to 0.1, and the value of beta, which corresponds to the complex background image, is set to 0.9.

Images