Image splicing method applicable to multiple environments based on deep learning
By introducing deep learning algorithms and color gain compensation in image stitching technology, combined with Superpoint and Lightglue algorithms, the feature capture and matching problems of image stitching in the existing technology are solved, and high-resolution perception of large field of view in complex environments is realized , the generated stitching images have higher resolution and natural transitions.
Patent Information
- Application Number
- CN202411118254.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2025-01-10
AI Technical Summary
The existing image stitching technology has problems such as inaccurate feature capture, wrong feature matching, and inadequate fusion transition, which leads to problems such as shadows, misalignment, and blurring of generated stitching images, making it difficult to take into account large field of view and high resolution.
Using a multi-environmental-applicable image stitching method based on deep learning, combined with Superpoint and Lightglue algorithms for feature capture and matching, stitching and fusion through unsupervised deep learning network, and solving the color difference between images through color gain compensation, using UNet The network is trained to improve robustness.
Accurate image registration and stitching in complex environments is achieved, which improves the stitching effect. The generated stitching image has higher resolution and more natural transitions, and improves the robustness and applicability of the algorithm.
Smart Images
Figure CN119295307A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of optoelectronic imaging technology, and in particular relates to an image stitching method applicable to multiple environments based on deep learning. Background Art
[0002] In today's society, digital images have played an increasingly important role, whether as a convenient way to transmit information or as one of the important media for machines to perceive the outside world. They are in the current development of science and technology. Due to the limitations of the hardware performance of the acquisition equipment, the images captured can often only cover a limited field of view. Although the use of special equipment such as fisheye lenses provides a solution for a large field of view, the overall cost of such equipment is relatively high, and due to the serious distortion characteristics of the lens, even after post-image processing, the distortion of the image cannot be completely offset, resulting in local blur and information loss. If a single camera is used for scanning, real-time performance cannot be taken into account; if a camera array is used, the content of each camera is separated from each other, and it is difficult to notice the content in different cameras at the same time. Image stitching refers to the fusion of two or more images that may come from different times, different spaces or different sensors according to similar parts under the premise of ensuring image resolution, so as to stitch different images with limited fields of view into scenes with a large field of view, so as to obtain more comprehensive, more real and more detailed information, so as to process and understand information more simply and quickly. This algorithm has applications in many fields, such as autonomous driving, remote sensing measurement, emergency rescue, medical imaging, surveillance video, virtual reality, etc.
[0003] Existing image stitching technologies often have problems such as inaccurate feature capture, feature matching errors, and unnatural fusion transitions, which result in the final stitched images having problems such as ghosting, misalignment, and blur. Summary of the invention
[0004] In order to solve the above technical problems, the present invention proposes a multi-environment applicable image stitching method based on deep learning, which can expand the field of view while ensuring the original high resolution; combined with the classic image stitching process, the present invention can be divided into four parts: feature capture, feature matching, exposure compensation and stitching fusion. The present invention integrates deep learning algorithms in the three parts of feature capture, feature matching and stitching fusion, which greatly improves the robustness of the algorithm in complex environments; in addition, exposure compensation makes up for the problem that the deep learning algorithm cannot perform color correction on the image, further improves the robustness of the algorithm in complex environments, and effectively solves the problems existing in the existing image stitching technology.
[0005] To achieve the above object, the present invention provides an image stitching method applicable to multiple environments based on deep learning, comprising:
[0006] Acquire several high-resolution images, use a bilinear interpolation algorithm to downsample the high-resolution images to obtain corresponding low-resolution images, and use a Superpoint algorithm based on the low-resolution images to obtain image feature points;
[0007] The image feature points are matched using the Lightglue algorithm, the high-resolution image is projected into the same coordinate system, and a corresponding mask is generated;
[0008] Performing color gain compensation on the overlapping parts of the high-resolution images according to the mask to obtain a high-resolution image after color gain compensation;
[0009] The high-resolution images after color gain compensation are spliced using an unsupervised deep learning network to obtain a spliced image.
[0010] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, a method for obtaining image feature points based on the low-resolution image using the Superpoint algorithm includes:
[0011] Use basic geometric figure training to obtain a pre-trained model;
[0012] Performing secondary training on the pre-trained model by adopting homogeneous adaptation to obtain a trained pre-trained model;
[0013] The low-resolution image is input into the trained pre-trained model, and the image feature points are obtained through the Superpoint algorithm.
[0014] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, the method of matching the image feature points using the Lightglue algorithm includes:
[0015] A Lightglue model is constructed by stacking a plurality of identical layers, and the positions and feature descriptors of two-dimensional points in an image are input into the Lightglue model, wherein each layer of the Lightglue model includes a self-attention unit and a cross-attention unit;
[0016] During the operation, each layer of the network updates the representation of each image feature point, filters and removes points that cannot be matched, and uses a classifier to decide whether to stop reasoning at each layer;
[0017] The matching results are calculated through a lightweight header.
[0018] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, color gain compensation is performed on the overlapping parts between the high-resolution images according to the mask, and the method for obtaining the high-resolution image after color gain compensation includes:
[0019] The image overlapping area is obtained according to the mask calculation, and the average intensity of each image in the overlapping area is calculated based on the image overlapping area. The average difference is defined as the square of the difference after the product of the gain and the average intensity of each image. The partial derivative of the gain of each image is calculated and set to 0 to obtain a high-resolution image after color gain compensation.
[0020] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, the method for obtaining the stitched image includes:
[0021] Inputting the mask and the color gain compensated high-resolution image into the unsupervised deep learning network for splicing;
[0022] The encoder of the unsupervised deep learning network is used to extract semantic features with shared weights, subtract the features of the target image from the reference image, and use the residuals at each resolution to replace the features of the target image and input them into the decoder;
[0023] Set the number of filters in the last layer to 1, set the activation function to the Sigmoid function, and obtain the output floating-point masks of the two images;
[0024] The output floating point masks of the two images are spliced to obtain a spliced image.
[0025] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, the loss function of the unsupervised deep learning network is a boundary term and a smoothing term, wherein the smoothing term is divided into a differential smoothing term and a stitching smoothing term;
[0026] The boundary item is used to ensure that the two endpoints of the image seam are located at the boundary portion where the two images intersect;
[0027] The difference smoothing term is used to describe the chromaticity difference on the difference image;
[0028] The stitching smoothness item is used to describe the continuity of the seam edge of the stitched images.
[0029] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, the boundary term is calculated as follows:
[0030]
[0031] in, is the boundary item; S is the spliced image; Iwr is the distorted reference image; M br and M bt is the boundary of the overlapping part of the mask; || ||1 is the 1-norm.
[0032] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, the differential smoothing term is calculated as follows:
[0033]
[0034] Among them, l D is the difference smoothing term; is the pixel value of the distortion mask at (i, j); is the pixel value of the distortion mask at (i+1,j); D i,j is the chromaticity difference between the two images at (i, j); D i+1,j is the chromaticity difference between the two images at (i+1,j); is the pixel value of the distortion mask at (i, j+1); D i,j+1 is the chromaticity difference between the two images at (i, j+1).
[0035] According to the multi-environment applicable image stitching method based on deep learning provided by the present invention, the stitching smoothing term is calculated as follows:
[0036]
[0037] Among them, l S is the splicing smoothing term; is the pixel value of the distortion mask at (i, j); is the pixel value of the distortion mask at (i+1,j); S i,j is the pixel value of the stitched image at (i, j); S i+1,j is the pixel value of the stitched image at (i+1,j); is the pixel value of the distortion mask at (i, j+1); S i,j+1 is the pixel value of the stitched image at (i, j+1).
[0038] Technical effect of the invention: The invention discloses an image stitching method applicable to multiple environments based on deep learning. In view of the problem that current perception cannot take into account both large field of view and high resolution in complex environments, the effect of image stitching is improved to achieve large field of view and high resolution perception in various environments. For image stitching in complex environments, feature points are detected by the Superpoint algorithm, and then the features are matched by the Lightglue algorithm, so as to achieve the purpose of accurate registration in complex situations. For image stitching in complex environments, the color difference between pictures caused by factors such as different sensors or different times is balanced by adding exposure compensation. The color gain of the overlapping part is calculated according to the mask, and the color gain is multiplied with the mask and the high-resolution image to finally obtain a pre-stitched image. For image stitching in complex environments, an unsupervised neural network is used for stitching, and a UNet-like network is used as the basic network. The training is performed with three angles of boundary endpoints, color continuity and content continuity as loss functions to generate a stitching network to face the stitching in complex environments and improve the robustness of the stitching. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0040] Figure 1 A schematic diagram of a flow chart of an image stitching method applicable to multiple environments based on deep learning according to an embodiment of the present invention;
[0041] Figure 2 It is a detailed flow chart of the algorithm of the embodiment of the present invention;
[0042] Figure 3 This is a flow chart of an exposure compensation algorithm according to an embodiment of the present invention;
[0043] Figure 4 This is a flow chart of the deep learning splicing method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0046] like Figure 1-4As shown, this embodiment provides an image stitching method applicable to multiple environments based on deep learning, including:
[0047] A high-resolution image is collected by using multiple cameras, a bilinear interpolation algorithm is used to downsample the high-resolution image, and a Superpoint algorithm is used to capture image feature points;
[0048] Use the Lightglue algorithm to match the feature points obtained in the previous step, project the high-resolution image into the same coordinate system, and generate the corresponding mask;
[0049] Performing color gain compensation on the overlapping parts of multiple images according to the mask;
[0050] The high-resolution images after color gain compensation are fused and stitched using an unsupervised neural network.
[0051] Furthermore, the downsampling algorithm is specifically as follows:
[0052] The original high-resolution image is downsampled using a bilinear interpolation algorithm to obtain a low-resolution version of each image.
[0053] Furthermore, the specific training method of the neural network of the Superpoint feature point capture algorithm is:
[0054] First, we will use basic geometric figures to train the pre-trained model MagicPoint;
[0055] The pre-trained model is trained again using homogeneous adaptation, that is, the same image is translated, rotated, scaled or perspective transformed to capture features, the MagicPoint model is used to detect feature points respectively, and finally the obtained feature points are added.
[0056] Furthermore, the Lightglue feature matching algorithm is specifically as follows:
[0057] The entire Lightglue model is composed of multiple identical layers stacked together, taking the position p of a two-dimensional point in the image and the feature descriptor d as input. Each layer includes a self-attention unit and a cross-attention unit.
[0058] During the operation, each layer of the network will update the representation of each point, filter and remove points that cannot be matched, and the classifier will decide whether to stop reasoning at each layer, thereby minimizing the amount of calculation;
[0059] Finally, a lightweight head calculates the partial matching results.
[0060] At the same time, Superpoint and Lightglue algorithms are used to perform feature alignment on the image to avoid the distortion of straight line structures caused by using grid homography. At the same time, it also ensures or even improves the alignment accuracy, laying a good foundation for subsequent stitching.
[0061] Furthermore, the specific operation of exposure compensation is:
[0062] First, the pose relationship between the two images is preliminarily determined based on the matching feature points obtained by the above method, and the mask is calculated and generated;
[0063] According to the target definition loss function e is shown in formula (1), where N ij Represents the number of pixels in the overlapping area, g i and g j Represent the gain coefficients of the two graphs respectively, and Respectively represent the average intensity of the two images in the overlapping area. The specific calculation is shown in formula (2), where R(u i ), G(u i ), B(u i ) represent a point u in the overlapping area i The intensity values of the red, green, and blue components of .
[0064]
[0065] In formula (1), N and σ g Respectively represent the standard deviation of error and gain, and in the present invention, σ N =10,σ g =0.1. Finally, apply equation (1) to g i Take the derivative and then make its derivative equal to 0 to obtain the closed form solution, as shown in formula (3),
[0066]
[0067] The image overlap area is calculated by multiplying the obtained image masks, and then the average intensity of each image in the overlap area is calculated. The average difference e is defined as the square of the difference after the product of the gain and the average intensity of each image. In order to minimize the total difference, the partial derivative of the gain of each image is calculated and set to 0. Finally, the gain corresponding to each image can be obtained. Image color gain compensation is used before the neural network is stitched, which makes up for the obvious seams in the spliced image caused by the neural network's inability to adjust when facing large chromaticity differences in the overlapping parts of the images.
[0068] Furthermore, the specific operation of the spliced neural network is:
[0069] First, the mask and input image are used as the input of this part and put into the UNet-like network for splicing;
[0070] Use the network’s encoder to extract semantic features with shared weights, then subtract the target image’s features from the reference image and use the residual at each resolution instead to feed into the decoder;
[0071] Finally, the number of filters in the last layer is set to 1, and the activation function is set to the Sigmoid function, and finally the output floating-point masks of the two images are obtained.
[0072] The loss function of the neural network is expressed as formula (4), where the two terms are the boundary terms and smoothing term The smoothing term is divided into the difference smoothing term l D and the splicing smoothing term l S ,
[0073]
[0074] The boundary term is mainly used to ensure that the two endpoints of the image seam are located at the boundary where the two images intersect. The specific calculation is shown in formula (5):
[0075]
[0076] in, is the boundary term; I wr is the distorted reference image; || ||1 is the 1-norm; S represents the overall spliced image, and M br and M bt represents the boundary of the overlapping part of the mask, and the first and last two terms represent the boundary parts where the two images intersect. This loss restricts the boundary pixels of the overlapping area in S to come from one of the boundaries of the two images, and fixes the endpoints of the "seam" at the intersection of the boundary of the overlapping area as much as possible.
[0077] The smoothing term is divided into the difference smoothing term l D and the splicing smoothing term l S , difference smoothing term l D Describes the chromaticity difference on the differential image, splicing smoothing term l S Describes the continuity of the seam edge of the spliced image. In the difference smoothing term, D = (I wr -I wt ) 2 , so the difference smoothing term is defined as in equation (6),
[0078]
[0079] Among them, l D is the difference smoothing term; is the pixel value of the distortion mask at (i, j); is the pixel value of the distortion mask at (i+1,j); D i,j is the chromaticity difference between the two images at (i, j); D i+1,j is the chromaticity difference between the two images at (i+1,j); is the pixel value of the distortion mask at (i, j+1); D i,j+1 is the chromaticity difference between the two images at (i, j+1).
[0080] The splicing smoothing term is calculated using the first-order difference in the direction and is defined as follows:
[0081]
[0082] Among them, l S is the splicing smoothing term; is the pixel value of the distortion mask at (i, j); is the pixel value of the distortion mask at (i+1,j); S i,j is the pixel value of the stitched image at (i, j); S i+1,j is the pixel value of the stitched image at (i+1,j); is the pixel value of the distortion mask at (i, j+1); S i,j+1 is the pixel value of the stitched image at (i, j+1).
[0083] Using an unsupervised neural network for stitching can generate a floating-point mask to make the final stitching smoother and the stitching traces more concealed; at the same time, it can greatly improve the robustness of the algorithm, allowing the algorithm to handle more scenarios and cope with various complex environments.
[0084] The present invention discloses a multi-environment applicable image stitching method based on deep learning. In view of the problem that current perception cannot take into account both large field of view and high resolution in complex environments, the effect of image stitching is improved to achieve large field of view and high resolution perception in various environments. For image stitching in complex environments, feature points are detected by the Superpoint algorithm, and then the features are matched by the Lightglue algorithm, so as to achieve the purpose of accurate registration in complex situations. For image stitching in complex environments, the color difference between pictures caused by factors such as different sensors or different times is balanced by adding an exposure compensation method, the color gain of the overlapping part is calculated according to the mask, and the color gain is multiplied with the mask and the high-resolution image to finally obtain a pre-stitched image. For image stitching in complex environments, an unsupervised neural network is used for stitching, a UNet-like network is used as the basic network, and training is performed with three angles of boundary endpoints, color continuity and content continuity as loss functions to generate a stitching network to face the stitching in complex environments and improve the robustness of the stitching.
[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A multi-environment applicable image stitching method based on deep learning, characterized in that: include: Acquire several high-resolution images, use a bilinear interpolation algorithm to downsample the high-resolution images to obtain corresponding low-resolution images, and use a Superpoint algorithm based on the low-resolution images to obtain image feature points; The image feature points are matched using the Lightglue algorithm, the high-resolution image is projected into the same coordinate system, and a corresponding mask is generated; Performing color gain compensation on the overlapping parts of the high-resolution images according to the mask to obtain a high-resolution image after color gain compensation; The high-resolution images after color gain compensation are spliced using an unsupervised deep learning network to obtain a spliced image.
2. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 1, characterized in that: The method for obtaining image feature points based on the low-resolution image using the Superpoint algorithm includes: Use basic geometric figure training to obtain a pre-trained model; Performing secondary training on the pre-trained model by adopting homogeneous adaptation to obtain a trained pre-trained model; The low-resolution image is input into the trained pre-trained model, and the image feature points are obtained through the Superpoint algorithm.
3. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 1, characterized in that: The method of matching the image feature points using the Lightglue algorithm includes: A Lightglue model is constructed by stacking a plurality of identical layers, and the positions and feature descriptors of two-dimensional points in an image are input into the Lightglue model, wherein each layer of the Lightglue model includes a self-attention unit and a cross-attention unit; During the operation, each layer of the network updates the representation of each image feature point, filters and removes points that cannot be matched, and uses a classifier to decide whether to stop reasoning at each layer; The matching results are calculated through a lightweight header.
4. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 1, characterized in that: The method of performing color gain compensation on the overlapping parts of the high-resolution images according to the mask to obtain the high-resolution image after color gain compensation comprises: The image overlapping area is obtained according to the mask calculation, and the average intensity of each image in the overlapping area is calculated based on the image overlapping area. The average difference is defined as the square of the difference after the product of the gain and the average intensity of each image. The partial derivative of the gain of each image is calculated and set to 0 to obtain a high-resolution image after color gain compensation.
5. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 1, characterized in that: Methods for obtaining the stitched image include: Inputting the mask and the color gain compensated high-resolution image into the unsupervised deep learning network for splicing; The encoder of the unsupervised deep learning network is used to extract semantic features with shared weights, subtract the features of the target image from the reference image, and use the residuals at each resolution to replace the features of the target image and input them into the decoder; Set the number of filters in the last layer to 1, set the activation function to the Sigmoid function, and obtain the output floating-point masks of the two images; The output floating point masks of the two images are spliced to obtain a spliced image.
6. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 5, characterized in that: The loss function of the unsupervised deep learning network is a boundary term and a smooth term, where the smooth term is divided into a differential smooth term and a splicing smooth term; The boundary item is used to ensure that the two endpoints of the image seam are located at the boundary portion where the two images intersect; The difference smoothing term is used to describe the chromaticity difference on the difference image; The stitching smoothness item is used to describe the continuity of the seam edge of the stitched images.
7. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 6, characterized in that: The boundary term is calculated as follows: in, is the boundary item; S is the spliced image; I wr is the distorted reference image; M br and M bt is the boundary of the overlapping part of the mask; || ||1 is the 1-norm.
8. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 6, characterized in that: The difference smoothing term is calculated as follows: Among them, l D is the difference smoothing term; is the pixel value of the distortion mask at (i, j); is the pixel value of the distortion mask at (i+1,j); D i,j is the chromaticity difference between the two images at (i, j); D i+1,j is the chromaticity difference between the two images at (i+1,j); is the pixel value of the distortion mask at (i, j+1); D i,j+1 is the chromaticity difference between the two images at (i, j+1).
9. The image stitching method applicable to multiple environments based on deep learning as claimed in claim 6, characterized in that: The splicing smoothing term is calculated as follows: Among them, l S is the splicing smoothing term; is the pixel value of the distortion mask at (i, j); is the pixel value of the distortion mask at (i+1,j); S i,j is the pixel value of the stitched image at (i, j); S i+1,j is the pixel value of the stitched image at (i+1,j); is the pixel value of the distortion mask at (i, j+1); S i,j+1 is the pixel value of the stitched image at (i, j+1).