Image stitching method, storage medium and computer

By extracting image feature points and calculating transformation matrix for image registration, the distortion and misstic splicing problems of image splicing under different conditions are solved, and high-quality panoramic splicing is achieved to adapt to image splicing in complex environments.

CN114612298BActive Publication Date: 2024-11-22DSTEK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210109834.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-29
Publication Date
2024-11-22
Estimated Expiration
2042-01-29

AI Technical Summary

Technical Problem

The prior art is difficult to effectively splice images collected by different times, angles, and sensors without reducing the image resolution. Especially in complex environments, distortion and missplicing are prone to problems.

Method used

By extracting feature points in the image, generating feature descriptors, calculating transformation matrix, performing image registration and weighting fusion, a high-quality panoramic stitching image is generated.

Benefits of technology

Seamless stitching of images under different conditions is achieved, the quality of stitching images and the accuracy of subsequent algorithms is improved, and it can adapt to interference factors in complex environments to generate high-quality panoramic stitching images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114612298B_ABST
    Figure CN114612298B_ABST
Patent Text Reader

Abstract

The present invention provides an image stitching method, a storage medium and a computer. The image stitching method comprises: providing a first image and a second image with overlapping parts; extracting feature points in each image and generating a feature descriptor for each feature point; pairing the feature points in the first image with the feature points in the second image according to the feature descriptor; calculating the translation amount and the rotation angle through the paired feature points to obtain a transformation matrix mathematical model of the first image and the second image; performing transformation registration between the first image and the second image according to the transformation matrix mathematical model; stitching and fusing the first image and the second image after transformation registration to obtain a stitched image. In this way, the present invention can perform panoramic stitching and fusion of two overlapping images to generate a high-quality panoramic stitching image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image stitching method, a storage medium and a computer. Background Art

[0002] Image stitching technology is a technology that stitches together several overlapping images (which may be obtained at different times, different perspectives or different sensors) into a large seamless high-resolution image. When using an ordinary camera to obtain a scene image with a wide field of view, because the camera resolution is fixed, the larger the scene is, the lower the image resolution is; and panoramic cameras, wide-angle lenses, etc. are not only very expensive, but also have serious distortion. In order to obtain an ultra-wide perspective or even a 360° panoramic view without reducing the image resolution, the concept of using computers for image stitching has been proposed and gradually developed. Now, image stitching technology has become the research focus of computer graphics and is widely used in space exploration, remote sensing image processing, medical image analysis, video compression and transmission, virtual reality technology, super-resolution reconstruction and other fields. Therefore, stitching together each sub-image realistically and without distortion is the key link to achieve the above fields.

[0003] From the above aspects, it can be seen that the application prospects of image stitching technology are very broad, and in-depth research on image stitching technology is of great significance. Summary of the invention

[0004] The object of the present invention is to provide an image stitching method, a storage medium and a computer, which can perform panoramic stitching and fusion of two overlapping images to generate a high-quality panoramic stitching image.

[0005] To achieve the purpose of the invention, according to one aspect of the invention, the invention provides an image stitching method, which includes: providing a first image and a second image with overlapping parts; extracting feature points in each image and generating a feature descriptor for each feature point; pairing the feature points in the first image with the feature points in the second image according to the feature descriptor; calculating the translation amount and the rotation angle through the paired feature points to obtain a transformation matrix mathematical model of the first image and the second image; performing transformation alignment between the first image and the second image according to the transformation matrix mathematical model; and stitching and fusing the first image and the second image after transformation alignment to obtain a stitched image.

[0006] According to another aspect of the present invention, the present invention provides a storage medium storing program instructions, wherein the image stitching method described above is executed when the program is executed.

[0007] According to yet another aspect of the present invention, the present invention provides a computer, comprising a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the image stitching method described above.

[0008] Compared with the prior art, the present invention can perform panoramic stitching and fusion of two overlapping images obtained at different times, different angles, and different sensor acquisitions to generate a high-quality panoramic stitching image for further processing by subsequent image algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a schematic diagram of a flow chart of an image stitching method in one embodiment of the present invention;

[0010] Figure 2 This is a stitching example of the image stitching method in the present invention. DETAILED DESCRIPTION

[0011] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0012] The present invention provides an image stitching method, which can perform panoramic stitching and fusion of multiple overlapping images to generate a high-quality panoramic stitching image.

[0013] Figure 1 FIG. 1 is a flow chart of an image stitching method 100 in one embodiment of the present invention. Figure 1 As shown, the image stitching method 100 includes the following steps.

[0014] Step 110, providing a first image and a second image with overlapping portions;

[0015] The stitching of multiple images is still based on the stitching of two images, so this article takes the stitching of two images as an example. Obviously, the same stitching principle can be used repeatedly to stitch three or more images.

[0016] like Figure 2 As shown in (a), it shows an example of two images, which have overlapping parts. The two images may be images of different angles collected by different sensors at different times.

[0017] In one embodiment, before subsequent image stitching, the first image and the second image need to be preprocessed.

[0018] The image preprocessing for the images to be stitched mainly includes the following aspects: first, image transformation, including distortion, geometric correction, scaling, etc.; second, from the perspective of illumination, including brightness, contrast, etc.; third, from the perspective of color and details, including noise, clarity, etc.; fourth, special processing: occlusion, deformation, etc. In actual operation, targeted image processing helps to improve the accuracy of subsequent registration algorithms.

[0019] The shooting conditions for industrial inspection are complex, and various interference situations frequently occur: interference in the shooting field of view, which is often caused by stains on the lens surface, people or equipment entering the field of view; the lighting conditions for image shooting are uncertain, which is often caused by changes in the intensity of the light source and the displacement of the light source. The image brightness and contrast parameters can be changed within a certain range, and the values ​​of each pixel of the original image can be changed according to the rules; the shooting field of view is blurred, which is often caused by changes in camera focus, smoke factors in the shooting field of view, and hardware performance decline. The image can be appropriately enhanced, sharpened, blurred, or Gaussian noise can be added, and the original image pixels can be filtered and convolved; the product is out of position and not fixed in the field of view. The original image can be horizontally, vertically flipped, angled, scaled, and randomly cropped. Try to take into account all possible factors so that the subsequent registration algorithm has higher robustness.

[0020] In the present invention, the preprocessing includes one or more of geometric correction, edge extraction, histogram processing, grayscale difference reduction, and noise suppression.

[0021] Step 120 , extracting feature points in each image and generating a feature descriptor for each feature point.

[0022] In one embodiment, the present invention uses the AKAZE algorithm to extract feature points in each image and generate a feature descriptor for each feature point. The feature points extracted using the AKAZE algorithm have illumination invariance, scale invariance, rotation invariance, and robustness to visual changes.

[0023] In one embodiment, step 120 includes: constructing a nonlinear scale space for each image by diffusing the image grayscale; extracting feature points of each image based on the nonlinear scale space of each image; dividing each image into a plurality of grid units, and obtaining features within the grid units where the feature points of each image are located to generate a feature descriptor for the feature points.

[0024] Specifically, constructing the nonlinear scale space of each image by diffusing the image grayscale includes:

[0025] Set the number of layers O and the number of towers S, where:

[0026]

[0027] The nonlinear scale space is constructed using a nonlinear diffusion function, the equation of which is:

[0028]

[0029] Where L is the image, div represents the nonlinear divergence function, represents the gradient solution function, and c is the conduction function in the three directions of x, y, and xy on the image L:

[0030]

[0031] is the Gaussian-smoothed image L, and g is the optimal conduction kernel function for regional diffusion smoothing:

[0032]

[0033] The parameter λ controls the degree of diffusion, and the nonlinear scale space includes multiple layers.

[0034] Specifically, the feature point extraction of each image based on the nonlinear scale space of each image includes: using the Hessian matrix to detect feature points in the nonlinear scale space of each image. Specifically, the Hessian matrix discriminant of each pixel position in the nonlinear scale space is constructed, and the Hessian matrix discriminant of each pixel position is compared with the Hessian matrix discriminants of 3*n*n-1 pixel positions of the upper and lower layers and the current layer of each pixel position. If the Hessian matrix discriminant of the current pixel position is a local maximum in the surrounding neighborhood, it is determined to be an extreme point, and the pixel position is extracted as a feature point, where n is an odd number greater than or equal to 3.

[0035] The upper and lower layers of each pixel position and the 3*n*n-1 pixel positions of the current layer include the n*n pixel positions of the upper layer of each pixel position, the n*n pixel positions of the lower layer of each pixel position, and the n*n-1 (excluding the current pixel position) pixel positions of the current layer of each pixel position. These 3*n*n-1 pixel positions are the surrounding neighborhood of each pixel position.

[0036] Hessian matrix:

[0037]

[0038] In order to describe the feature points, based on the traditional binary descriptor, the AKAZE algorithm of the present invention proposes an improved local difference binary descriptor, namely the M-LDB descriptor. The M-LDB descriptor divides each image into multiple grid units, such as L*L grid units, where L is a natural number greater than or equal to 2. Specifically, the method of obtaining the features within the grid unit where the feature points of each image are located to generate the feature descriptor of the feature point includes: calculating the first-order gradient and grayscale of the grid unit where the feature points of each image are located; obtaining the features inside the grid unit where the feature points of each image are located, the features including edge size and direction; binarizing the features to construct a binary description vector, and using the binary description vector as the feature descriptor of the feature point.

[0039] Step 130 : Pairing feature points in the first image with feature points in the second image according to the feature descriptor.

[0040] In one embodiment, step 130 includes: calculating the Euclidean distance between the feature descriptors of the first image and the second image to obtain the best matching feature point for each feature point; selecting the first m pairs of feature points with the closest distance, where m is a natural number greater than or equal to 2, and m can be set as needed.

[0041] Only the feature points in the overlapping part of the first image will be paired with the feature points in the overlapping part of the second image, or in other words, the pairing is successful. The feature points in the non-overlapping part of the first image will not have paired feature points in the second image, and the feature points in the non-overlapping part of the second image will not have paired feature points in the first image.

[0042] Figure 2 (b) in FIG. 1 schematically shows a plurality of feature points of the first image, a plurality of feature points of the second image, and paired feature points.

[0043] Step 140, calculating the translation amount and the rotation angle through the paired feature points to obtain a transformation matrix mathematical model of the first image and the second image.

[0044] In one embodiment, step 140 includes: calculating the translation distance of the paired feature points in the x and y directions, obtaining the mode, and obtaining the translation amount; selecting two pairs of paired feature points with the closest distance from the paired feature points corresponding to the mode, connecting two feature points in the first image of the two selected pairs of paired feature points to obtain a first pairing line, connecting two feature points in the second image of the two selected pairs of paired feature points to obtain a second pairing line, calculating the angular difference between the first pairing line and the second pairing line to obtain a rotation angle; generating a transformation matrix mathematical model with translation and rotation based on the translation amount and the rotation angle.

[0045] The present invention uses the obvious feature points of the image to estimate the transformation between images instead of using all the image information; these feature points can effectively reduce the image information data calculation amount of the algorithm while retaining the important features of the image, and can effectively improve the calculation speed and reliability of the algorithm.

[0046] Step 150: Perform transformation registration between the first image and the second image according to the transformation matrix mathematical model to provide two images of the same viewing angle for subsequent stitching.

[0047] Step 160: stitching and fusing the first image and the second image after transformation and registration to obtain a stitched image.

[0048] Since the stitching between two images may appear unnatural due to different shooting environments, weighted fusion processing is required to eliminate the accumulated error of global matching and the distortion of the image overlap area. In one embodiment, for the overlapping area, the weighted sum of the value of a pixel position in the first image and the value of the corresponding pixel position in the second image is used as the value of the corresponding pixel position of the stitched image to complete the seamless stitching of the images, wherein for the overlapping area, from the first image to the second image, the weight of the value of the pixel position of the first image gradually decreases, and the weight of the value of the pixel position of the second image gradually increases.

[0049] Specifically, for the overlapping area, from the first image to the second image, the weight of the value of the pixel position of the first image is linearly reduced from 1 to 0, and the weight of the value of the pixel position of the second image is linearly increased from 0 to 1. The weighted sum is the sum of the product of the value of a pixel position in the first image and its weight and the product of the value of the corresponding pixel position in the second image and its weight.

[0050] like Figure 2 (c) shows a stitched image of the first image and the second image.

[0051] The present invention can perform panoramic stitching and fusion of two overlapping images obtained at different times, different angles, and different sensor acquisition situations, and can fully consider various possible external noises, lighting effects and other factors to generate high-quality panoramic stitching images for further processing by subsequent image algorithms.

[0052] The following is an introduction to an application example of the image stitching method 100 of the present invention.

[0053] Defect detection of a product at a company's workshop production site. The top view of the product is a rectangle, and the product is divided into 4 sub-parts by the center point of the product. The 4 cameras are located directly above the center points of the corresponding 4 sub-parts. When the product runs to the current station, the 4 cameras take pictures at the same time and perform panoramic stitching. The shooting environment conditions in the workshop are poor. There may be stains on the lens surface, the light conditions around the camera are sometimes bright and sometimes dark, and there may be a high concentration of dust or water vapor in the shooting field of view, affecting the image quality. The baffle position is not fixed, resulting in incomplete baffle imaging in the shooting scene. The image stitching algorithm is required to adapt to and be compatible with these interference factors.

[0054] Difficulty of the algorithm: The shooting environment is poor. The surface of the product is smooth, and the lighting source is prone to glowing when it hits the product. The defects of the product itself are prone to appear around the photographed surface. Image stitching and fusion are prone to cause mis-stitching, which will make it difficult for the subsequent algorithm to meet the actual detection accuracy requirements.

[0055] When trying to detect the original images separately, change to using panoramic images after image stitching and fusion. On the basis of the original detection process remaining unchanged, the original separately detected images are fused into one image through panoramic stitching, and then targeted with traditional algorithm processing: for images with complex imaging conditions, image registration is performed after pre-processing, which can well fuse multiple sub-images into a panoramic image. Finally, the obtained stitched image is used in subsequent image algorithms, and the overall accuracy of the algorithm is significantly improved.

[0056] According to another aspect of the present invention, the present invention provides a storage medium storing program instructions, which are executed to execute the above-mentioned image stitching method 100. For simplicity, the specific content of the image stitching method 100 is not repeated here.

[0057] According to another aspect of the present invention, the present invention provides a computer, comprising a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the above-mentioned image stitching method 100. For simplicity, the specific content of the image target matching and positioning method is not repeated here.

[0058] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0059] In this document, the directional words such as front, back, top, and bottom are defined by the positions of the components in the drawings and the positions of the components relative to each other, and are only for the sake of clarity and convenience in expressing the technical solution. It should be understood that the use of the directional words should not limit the scope of protection claimed in this application.

[0060] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0061] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

Claims

1. An image stitching method, characterized in that: It includes: providing a first image and a second image having overlapping portions; Extract feature points in each image and generate feature descriptors for each feature point; Pairing feature points in the first image with feature points in the second image; Calculate the translation amount and the rotation angle through the paired feature points to obtain a transformation matrix mathematical model of the first image and the second image; Perform transformation registration between the first image and the second image according to the transformation matrix mathematical model; The first image and the second image after transformation and registration are stitched and fused to obtain a stitched image. The extracting feature points in each image and generating a feature descriptor for each feature point comprises: The nonlinear scale space of each image is constructed by diffusing the image grayscale; Extract feature points of each image based on the nonlinear scale space of each image; Each image is divided into multiple grid units, and the features in the grid unit where the feature points of each image are located are obtained to generate the feature descriptor of the feature point. The nonlinear scale space of each image is constructed by diffusion of image grayscale, including: Set the number of floors O and the number of towers S, where: The nonlinear scale space is constructed using a nonlinear diffusion function, the equation of which is: Where L is the image, div represents the nonlinear divergence function, represents the gradient solution function, time t is used as the scale parameter, and c is the conduction function in the x, y, and xy directions on the image L: is the Gaussian-smoothed image L, and g is the optimal conduction kernel function for regional diffusion smoothing: The parameter λ controls the degree of diffusion; The extracting feature points of each image based on the nonlinear scale space of each image comprises: The Hessian matrix is ​​used to detect feature points in the nonlinear scale space of each image, where the Hessian matrix is: The step of obtaining the features in the grid unit where the feature point of each image is located to generate the feature descriptor of the feature point comprises: Calculate the first-order gradient and grayscale of the grid cell where the feature points of each image are located; Obtaining features inside the grid unit where the feature points of each image are located, wherein the features include edge size and direction; Binarize the feature to construct a binary description vector, and use the binary description vector as the feature descriptor of the feature point. The method of using the Hessian matrix to detect feature points in the nonlinear scale space of each image includes: Construct a Hessian matrix discriminant of each pixel position in the nonlinear scale space, compare the Hessian matrix discriminant of each pixel position with the Hessian matrix discriminants of 3*n*n-1 pixel positions of the upper and lower layers and the current layer of each pixel position, if the Hessian matrix discriminant of the current pixel position is a local maximum in the surrounding neighborhood, it is determined to be an extreme point, and the pixel position is extracted as a feature point, wherein n is an odd number greater than or equal to 3, and the nonlinear scale space includes multiple layers.

2. The image stitching method according to claim 1, characterized in that: It also includes: Preprocessing the first image and the second image, The preprocessing includes one or more of geometric correction, edge extraction, histogram processing, grayscale difference reduction, and noise suppression.

3. The image stitching method according to claim 1, characterized in that: The pairing of the feature points in the first image with the feature points in the second image according to the feature descriptor comprises: Calculate the Euclidean distance between the feature descriptors of the first image and the second image to obtain the best matching feature point for each feature point; Select the first m pairs of feature points with the closest distance, where m is a natural number greater than or equal to 2.

4. The image stitching method according to claim 1, characterized in that: The mathematical model of the transformation matrix of the first image and the second image obtained by calculating the translation amount and the rotation angle through the paired feature points includes: Calculate the translation distance of the paired feature points in the x and y directions, obtain the mode, and obtain the translation amount; Select two pairs of paired feature points with the closest distance from the majority corresponding paired feature points, connect two feature points in the first image of the selected two pairs of paired feature points to obtain a first paired line, connect two feature points in the second image of the selected two pairs of paired feature points to obtain a second paired line, and calculate the angle difference between the first paired line and the second paired line to obtain a rotation angle; Generate a mathematical model of a transformation matrix with translation and rotation based on the translation amount and rotation angle.

5. The image stitching method according to claim 1, characterized in that: The step of splicing and fusing the first image and the second image after transformation and registration comprises: For the overlapping area, the weighted sum of the value of a pixel position in the first image and the value of the corresponding pixel position in the second image is used as the value of the corresponding pixel position in the stitched image, wherein for the overlapping area, the weight of the value of the pixel position of the first image gradually decreases and the weight of the value of the pixel position of the second image gradually increases from the first image to the second image.

6. The image stitching method according to claim 5, characterized in that: For the overlapping area, from the first image to the second image, the weight of the value of the pixel position of the first image decreases linearly from 1 to 0, and the weight of the value of the pixel position of the second image increases linearly from 0 to 1. The weighted sum is the sum of the product of the value of a pixel position in the first image and its weight and the product of the value of the corresponding pixel position in the second image and its weight.

7. A storage medium, characterized in that: It stores program instructions, and when the program is executed, it is run to execute the image stitching method as described in any one of claims 1-6.

8. A computer, characterized in that: It comprises a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the image stitching method according to any one of claims 1 to 6.