Image stitching method, device, equipment and storage medium

Abstract

This application discloses an image stitching method, apparatus, device, and storage medium. The method includes: acquiring an initial stitching image, which is a stitched image obtained by feature extraction, feature matching, and image transformation of a first image and a second image to be stitched; converting the initial stitching image into a target stitching image based on a trained self-supervised optimization model; wherein the trained self-supervised optimization model is used to determine the optimization residual of the initial stitching image, and the optimization residual is added to the initial stitching image to obtain the target stitching image. By introducing a self-supervised optimization model, the optimization residual of the initial stitching image can be determined, and the optimization residual is added to the initial stitching image to obtain the target stitching image. This effectively mitigates the impact of inaccurate homography matrix calculation, scene depth, and changes in illumination from different viewpoints on image stitching quality, thereby improving image stitching quality and facilitating the subsequent application requirements of the stitched image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image stitching method, apparatus, device and storage medium. Background Technology

[0002] Image stitching is a widely studied problem in computer vision and computer graphics. By stitching and merging multiple images of the same scene, a wider perspective and more scene information can be presented on the same image. This effectively solves the problem that a single ordinary image cannot display enough scene information and is widely used in fields such as virtual display, medical image processing, and autonomous driving.

[0003] In related technologies, image stitching techniques generally include four steps: feature extraction, feature matching, image transformation, and image fusion. Feature extraction is used to extract features from the images to be stitched; feature matching is used to calculate the homography transformation matrix between the two images based on the extracted features; image transformation is used to perform homography transformation on the images, so that the two images to be stitched are in the same reference coordinate system; image fusion is used to find the optimal stitching seam or smooth transition to reduce visual anomalies such as blurring, ghosting, and breaks. However, due to factors such as changes in illumination on the images to be stitched, scene depth, and inaccurate homography matrix calculations, traditional image fusion methods still struggle to address visual anomalies, severely affecting the quality of image stitching and limiting the subsequent application of image stitching algorithms. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image stitching method, apparatus, device, and storage medium, aiming to improve image stitching quality.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an image stitching method, including:

[0007] Obtain an initial stitched image, which is a stitched image obtained after feature extraction, feature matching and image transformation of the first image and the second image to be stitched;

[0008] The initial stitched image is converted into a target stitched image based on a trained self-supervised optimization model; wherein the trained self-supervised optimization model is used to determine the optimization residual of the initial stitched image, and the optimization residual is added to the initial stitched image to obtain the target stitched image.

[0009] The method in the above scheme further includes:

[0010] The self-supervised optimization model to be optimized is trained based on the training samples;

[0011] Determine that the number of training iterations of the self-supervised optimization model has reached a set number or that the loss value of the loss function has converged, and obtain the trained self-supervised optimization model.

[0012] The training samples include pairs of images to be stitched and initial stitched images corresponding to the pairs of images to be stitched. The loss function is determined based on content loss and gradient loss. The content loss represents the pixel difference between the optimized stitched image and the corresponding image to be stitched, and the gradient loss represents the pixel gradient difference between the optimized stitched image and the corresponding image to be stitched.

[0013] In the above scheme, the self-supervised optimization model is a CNN (Convolutional Neural Network) with an autoencoder structure containing skip connections.

[0014] In the above scheme, obtaining the initial stitched image includes:

[0015] Feature extraction is performed on the first and second images to be stitched together, respectively;

[0016] Based on the extracted features, the homography matrix between the first image and the second image is determined;

[0017] Based on the homography matrix, the first image and the second image are mapped to the same image coordinate system to obtain an initial stitched image.

[0018] In the above scheme, the feature extraction of the first image and the second image to be stitched together includes:

[0019] For the first and second images to be stitched together, Harris corner points are extracted respectively to obtain the first corner point set of the first image and the second corner point set of the second image;

[0020] For the first and second images to be stitched together, feature extraction is performed based on a parameter-shared CNN to obtain a first feature map of the first image and a second feature map of the second image;

[0021] Based on each corner point in the first corner point set, position drilling is performed on the first feature map to obtain the feature vector of each corner point in the first corner point set.

[0022] Based on each corner point in the second corner point set, position drilling is performed on the second feature map to obtain the feature vector of each corner point in the second corner point set.

[0023] In the above scheme, determining the homography matrix between the first image and the second image based on the extracted features includes:

[0024] Based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set, feature matching is performed to obtain a matching feature set between the first image and the second image;

[0025] Based on the matching feature set, the homography matrix between the first image and the second image is determined using the Random Sampling Consensus (RANSAC) algorithm.

[0026] In the above scheme, the step of performing feature matching based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set to obtain a matching feature set between the first image and the second image includes:

[0027] Iterate through each corner point in the first corner point set and find the matching corner points in the second corner point set for each corner point.

[0028] Based on the corner point pairs matched between the first corner point set and the corner point set, a matching feature set between the first image and the second image is obtained;

[0029] Among them, the matched corner point pair is the one with the shortest distance between the feature vectors of the two corner points.

[0030] Secondly, embodiments of this application provide an image stitching device, including:

[0031] The acquisition module is used to acquire an initial stitched image, which is a stitched image obtained after feature extraction, feature matching and image transformation of the first image and the second image to be stitched;

[0032] An optimization module is used to convert the initial stitched image into a target stitched image based on a trained self-supervised optimization model; wherein the trained self-supervised optimization model is used to determine the optimization residual of the initial stitched image, and add the optimization residual and the initial stitched image to obtain the target stitched image.

[0033] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor is used to run the computer program, it executes the steps of the method described in the first aspect of embodiments of this application.

[0034] Fourthly, embodiments of this application provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect of embodiments of this application.

[0035] The technical solution provided in this application provides an initial stitched image, which is a stitched image obtained after feature extraction, feature matching, and image transformation of the first and second images to be stitched. The initial stitched image is then converted into a target stitched image based on a trained self-supervised optimization model. The trained self-supervised optimization model is used to determine the optimization residual of the initial stitched image, and the optimization residual is added to the initial stitched image to obtain the target stitched image. By introducing a self-supervised optimization model, the optimization residual of the initial stitched image can be determined, and the optimization residual is added to the initial stitched image to obtain the target stitched image. This effectively mitigates the impact of inaccurate homography matrix calculation, scene depth, and changes in illumination from different viewpoints on image stitching quality, thereby improving image stitching quality and facilitating the subsequent application requirements of the stitched image. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the image stitching method according to an embodiment of this application;

[0037] Figure 2 This is a schematic diagram illustrating the principle of the image stitching method in the application embodiments of this application;

[0038] Figure 3 This is a schematic diagram illustrating the principle of random depth descriptor generation in the application embodiments of this application;

[0039] Figure 4 This is a schematic diagram illustrating the principle of the self-supervised optimization module in the application embodiments of this application;

[0040] Figure 5 This is a schematic diagram of the image stitching device according to an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0044] This application provides an image stitching method that can be applied to electronic devices with data processing capabilities, such as terminal devices and servers, or can be implemented by the cooperation of terminal devices and servers. Specifically, the terminal device can be a computer, smartphone, personal digital assistant (PDA), etc.; the server can be an application server or a web server. In actual deployment, the server can be a standalone server or a cluster server.

[0045] For example, such as Figure 1 As shown, the image stitching method includes:

[0046] Step 101: Obtain the initial stitched image, which is the stitched image obtained after feature extraction, feature matching and image transformation of the first image and the second image to be stitched.

[0047] Step 102: Based on the trained self-supervised optimization model, the initial stitched image is converted into a target stitched image; wherein, the trained self-supervised optimization model is used to determine the optimization residual of the initial stitched image, and the optimization residual is added to the initial stitched image to obtain the target stitched image.

[0048] The image stitching method of this application, by introducing a self-supervised optimization model, can determine the optimization residual of the initial stitched image and add the optimization residual to the initial stitched image to obtain the target stitched image. This effectively alleviates the impact of inaccurate homography matrix calculation, scene depth, and changes in illumination from different viewpoints on image stitching quality, thereby improving image stitching quality and facilitating the subsequent application requirements of the stitched image.

[0049] It's important to note that traditional image fusion methods are complex. For example, considering the influence of different shooting angles, lighting, and shooting environments, directly stitching images often results in noticeable seams, and overlapping areas may exhibit blurring and distortion. Alpha fusion is one technology that uses alpha fusion to achieve image fusion. Alpha fusion involves an important concept—the alpha channel. Images obtained from normal shooting only have three channels: RGB. Besides the three primary color channels used to describe the digital image, the channel used to represent the transparency of each pixel is called the alpha channel. However, in practical applications, the fused stitched image still exhibits visual anomalies such as breaks, blurring, and ghosting. The fused image is not natural enough, severely impacting the requirements of subsequent applications.

[0050] The image stitching method of this application uses a trained self-supervised optimization model to determine the optimization residual of the initial stitched image. This optimization residual can represent the difference between the initial stitched image and the stitched real image. The residual refers to the difference between the predicted value and the actual value. If there exists a mapping f(x) = b, where x = x0, then bf(x0) is the residual. Thus, the target stitched image obtained by adding the optimization residual to the initial stitched image can approximate the stitched real image to the greatest extent, thereby achieving a natural and high-quality image stitching effect. This meets the subsequent application requirements of the stitched image, such as those for VR (Virtual Reality), 360° panoramic video, and 3D reconstruction video tasks. Furthermore, the image stitching method of this application has universality and flexibility, and avoids the complex quantization process in traditional complex image fusion, improving the efficiency and quality of image stitching, and has broad application prospects.

[0051] It is understood that the method in this application embodiment needs to be based on a trained self-supervised optimization model. Based on this, the method further includes:

[0052] The self-supervised optimization model to be optimized is trained based on the training samples;

[0053] Determine that the number of training iterations of the self-supervised optimization model has reached a set number or that the loss value of the loss function has converged, and obtain the trained self-supervised optimization model.

[0054] The training samples include pairs of images to be stitched and initial stitched images corresponding to the pairs of images to be stitched. The loss function is determined based on content loss and gradient loss. The content loss represents the pixel difference between the optimized stitched image and the corresponding image to be stitched, and the gradient loss represents the pixel gradient difference between the optimized stitched image and the corresponding image to be stitched.

[0055] Here, the paired images to be stitched together can be obtained as the corresponding initial stitched images through the aforementioned feature extraction, feature matching and image transformation. The specific process can be referred to in the relevant description below.

[0056] Here, the optimized stitched image is the stitched image output by the self-supervised optimization model after inputting the initial stitched image of the paired images to be stitched. Content loss and gradient loss values ​​are determined based on this optimized stitched image and the paired images to be stitched. The content loss value includes a first content loss value corresponding to the first image to be stitched and a second content loss value corresponding to the second image to be stitched. The gradient loss value can be determined based on the pixel gradient difference between the optimized stitched image and the stitched image used as a reference coordinate system in the paired images to be stitched. For example, the paired images to be stitched include image I1 and image I2. Using image I1 as the reference coordinate system, image I2 is transformed to the coordinate system of image I1 to form the initial stitched image. In this case, the gradient loss value can be determined by the pixel gradient difference between the optimized stitched image and image I1.

[0057] Here, loss convergence means that the loss value is less than or equal to a set threshold. It can be understood that through the above training, the self-supervised optimization model can learn the optimization residuals of the training samples and adjust the model parameters based on the loss function, thus obtaining a well-trained self-supervised optimization model. Since the loss function is determined based on the content loss value and the gradient loss value, the training process includes self-supervised loss-constrained network training for content reconstruction (corresponding to the aforementioned pixel differences) and gradient reconstruction (corresponding to the aforementioned pixel gradient differences). Therefore, the self-supervised optimization model can effectively reduce image artifacts and stitching errors in the stitching region, improve the quality of the stitched image, and solve the problem of difficulty in obtaining ground truth values ​​for image stitching.

[0058] For example, the self-supervised optimization model is a CNN (convolutional neural network) with an autoencoder structure containing skip connections. In this way, feature fusion can be performed based on the autoencoder structure with skip connections, thereby enabling more comprehensive learning of the optimization residuals of the training samples.

[0059] In one application example, the self-supervised optimization model outputs the sum of the optimization residual and the initial stitched image, as shown in the following formula:

[0060] P = R + P i =f cnn (P i )+P i

[0061] Where P is the optimized stitched image (corresponding to the aforementioned target stitched image), R is the optimization residual, and P i For the initial stitched image, R can be derived from f cnn (P i ) indicates that f cnn () is the transformation function that identifies the optimized residual of the initial stitched image.

[0062] Considering the difficulty in obtaining ground truth values ​​for image stitching in real-world scenarios, the loss function of the self-supervised optimization model is exemplarily shown below:

[0063] L=L1+L2+αL g

[0064] Where L is the loss value of the self-supervised optimization model, L1 is the content loss value corresponding to the first image to be stitched, L2 is the content loss value corresponding to the second image to be stitched, and L... g α is the gradient loss value between the optimized stitched image and the image to be stitched, which serves as the reference coordinate system, and α is the weighting coefficient.

[0065] For example, obtaining the initial stitched image includes:

[0066] Feature extraction is performed on the first and second images to be stitched together, respectively;

[0067] Based on the extracted features, the homography matrix between the first image and the second image is determined;

[0068] Based on the homography matrix, the first image and the second image are mapped to the same image coordinate system to obtain an initial stitched image.

[0069] Here, the homography matrix is ​​used to describe the positional mapping relationship between an object in the world coordinate system and the pixel coordinate system.

[0070] In related technologies, feature point locations and corresponding feature descriptors are often extracted manually based on the image domain. This is inefficient and easily affected by noise and lighting changes, which in turn affects the accuracy of subsequent feature matching.

[0071] Based on this, in some embodiments, the feature extraction of the first image and the second image to be stitched together includes:

[0072] For the first and second images to be stitched together, Harris corner points are extracted respectively to obtain the first corner point set of the first image and the second corner point set of the second image;

[0073] For the first and second images to be stitched together, feature extraction is performed based on a parameter-shared CNN to obtain a first feature map of the first image and a second feature map of the second image;

[0074] Based on each corner point in the first corner point set, position drilling is performed on the first feature map to obtain the feature vector of each corner point in the first corner point set.

[0075] Based on each corner point in the second corner point set, position drilling is performed on the second feature map to obtain the feature vector of each corner point in the second corner point set.

[0076] Here, Harris corner extraction is a corner extraction method developed by Chris Harris and Mike Stephens based on the H. Moravec algorithm, which uses the autocorrelation matrix. By extracting Harris corners, we can obtain the first corner set C1 of the first image and the second corner set C2 of the second image.

[0077] In this embodiment, a CNN based on parameter sharing performs feature extraction on the first and second images to be stitched together, obtaining a first feature map of the first image and a second feature map of the second image. This allows the CNN's mapping rules to map low-dimensional image features to high-dimensional depth features. Furthermore, position drilling is performed based on the corner points in the first and second corner point sets to obtain local high-dimensional depth features on the first and second images, thereby improving the robustness and discriminability of subsequent feature matching.

[0078] Here, position drill-up refers to determining the corresponding feature vector (also known as feature descriptor) on the feature map based on the location of the corner point. For example, based on each corner point in the first corner point set, the corresponding feature vector is determined on the first feature map, and based on each corner point in the second corner point set, the corresponding feature vector is determined on the second feature map.

[0079] For example, determining the homography matrix between the first image and the second image based on the extracted features includes:

[0080] Based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set, feature matching is performed to obtain a matching feature set between the first image and the second image;

[0081] Based on the matching feature set, the homography matrix between the first image and the second image is determined using the Random Sampling Consensus (RANSAC) algorithm.

[0082] For example, the step of performing feature matching based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set to obtain a matching feature set between the first image and the second image includes:

[0083] Iterate through each corner point in the first corner point set and find the matching corner points in the second corner point set for each corner point.

[0084] Based on the corner point pairs matched between the first corner point set and the corner point set, a matching feature set between the first image and the second image is obtained;

[0085] Among them, the matched corner point pair is the one with the shortest distance between the feature vectors of the two corner points.

[0086] In one application example, the Brute Force (BF) algorithm can be used for feature matching to obtain a set of matching features between the first and second images. Then, the RANSAC algorithm can be used to calculate the homography matrix between the first and second images. The idea behind the RANSAC algorithm can be understood as: randomly selecting n samples from the matching feature set to calculate the homography matrix H. i And based on the currently calculated homography matrix H i Calculate the error value for the remaining samples. If the error value is less than the threshold t, the sample is considered an interior point; otherwise, it is considered an exterior point. Calculate the number of interior points N. i Repeat the above process until the specified number of iterations is reached, and use all interior point samples from the iteration result with the largest number of interior points to recalculate a new homography matrix as the final homography matrix H.

[0087] The present application will now be described in further detail with reference to an application embodiment.

[0088] This application example provides an image stitching method based on random depth features and self-supervised optimization, such as... Figure 2 As shown, this method includes feature extraction, homography calculation, image mapping, and self-supervised optimization steps. The details of each step are described below:

[0089] 1. Feature Extraction

[0090] Here, feature extraction can be performed by a feature extraction module. This module can extract features from the two input images to be stitched (image 1 and image 2) based on Harris corner detection and random depth descriptor features. Since the feature extraction process for the two images is the same, the specific process of feature extraction will be described in detail below using image 1 as an example:

[0091] 1) Harris Corner Extraction

[0092] According to the Harris corner definition, the input image can be filtered using horizontal and vertical difference operators to obtain the gradient map I in the horizontal direction. x Gradient diagram I in the vertical direction y And calculate the M matrix based on the following formula (1):

[0093]

[0094] The M matrix is ​​further smoothed using Gaussian filtering to obtain the M' matrix, and the corner response matrix R is solved based on the M' matrix, as shown in the following formula (2):

[0095] R = det(M') - k(traceM') 2 (2)

[0096] Where det(M') represents the rank of matrix M', (traceM') 2 The trace of matrix M' is represented by k, which is a set constant. For example, the value of k is 0.05.

[0097] Pixels whose values ​​in the corner response matrix R are greater than a set threshold can be identified as corner locations, and then Harris corners can be extracted. The set threshold can be reasonably set based on the requirements. For example, the threshold can be set to 0.1 times the maximum value of the corner response matrix R (i.e., max(R) × 0.1).

[0098] For example, to avoid concentrated corner point distribution, a non-maximum suppression algorithm can be used to select the pixel position with the largest local response value as the final corner point output, discarding the remaining corner points in the neighborhood. Thus, for the input image 1 and image 2 to be stitched, the first set of detected corner points C1 and the second set of detected corner points C2 can be obtained respectively.

[0099] 2) Generation of random depth descriptors (corresponding to the feature vectors of the aforementioned corner points)

[0100] To further perform feature matching and calculate the homography matrix, feature descriptors need to be generated for each detected corner location. Traditional feature descriptor generation methods are based on manual construction in the local image space and are usually sensitive to noise and illumination changes. Existing deep learning-based semantic feature extraction methods are usually task-specific and require supervised training data, while ground truth values ​​for image stitching in real-world scenes are often difficult to obtain.

[0101] To address the above issues, this application example proposes a method for generating random depth descriptors based on CNN, such as... Figure 3 As shown. This method uses a parameter-sharing Siamese CNN network to extract features from the two images to be stitched (labeled as image I1 and image I2), obtaining corresponding feature maps F1 and F2. In the corner point extraction step, the first corner point set C1 (i.e., ...) is detected on images I1 and I2 respectively. Figure 3 The detection corner set C1 and the second corner set C2 (i.e., ...) are shown. Figure 3 The set of detected corner points shown is 2). Taking the first corner point set C1 as an example, it can be represented as C1 = [(x1,y1),(x2,y2),...(x... n,y n )], where (x n ,y n ) represents the coordinates of the nth Harris corner point detected on the image I1 to be stitched, and C2 is similar.

[0102] To further calculate the feature descriptor corresponding to each detected corner point, position drilling is performed on the corresponding feature maps F1 and F2 using the first corner point set C1 and the second corner point set C2, respectively. Taking the first corner point set C1 as an example, the position drilling process is shown in the following formula (3):

[0103] d i =F1(C1[i]) (3)

[0104] Where, d i This represents the feature descriptor corresponding to the i-th corner point position in the first corner point set C1. The extraction of feature descriptors for corner points in the second corner point set C2 can be performed using the same process as described above, and will not be repeated here.

[0105] It should be noted that the Siamese CNN network only requires parameter sharing. It can utilize untrained CNN networks with random parameters, or pre-trained feature extraction CNN networks from any task; there are no strict restrictions on the network structure. In this application example, a 6-layer 3×3 convolution with a stride of 1 is used as an example, but any of the aforementioned CNN networks can be used in practice. In this application example, based on the aforementioned feature extraction, the same CNN mapping rules and local information can be used to map low-dimensional image features to high-dimensional deep features, thereby improving the robustness and discriminability of subsequent feature matching.

[0106] 2. Uniformity calculation

[0107] Here, homography calculation can be performed by the homography calculation module. Since the aforementioned feature extraction module extracts the first corner point set C1 and the second corner point set C2, as well as the corresponding feature descriptor sets D1 (corresponding to the first corner point set C1) and D2 (corresponding to the second corner point set C2) for the images to be stitched (Image 1 and Image 2), the homography calculation module, based on these features, performs feature matching using the brute-force (BF) algorithm and calculates the homography matrix using the RANSAC algorithm to represent the transformation relationship between the two images to be stitched.

[0108] For the i-th feature position c in the image to be stitched 1 i The feature descriptor is d i The BF algorithm calculates the descriptor distance between the image and all feature points in the image to be stitched (image 2), and returns the feature with the smallest distance as the best matching feature. By traversing all features in image 1, the matching feature set for the two images can be obtained.

[0109] After obtaining the matching feature sets from the two images, the RANSAC algorithm is further used to calculate the homography matrix. The idea behind the RANSAC algorithm is to randomly select n samples from the matching feature set to calculate the homography matrix H. i And based on the currently calculated homography matrix H i Calculate the error value for the remaining samples. If the error value is less than the threshold t, the sample is considered an interior point; otherwise, it is considered an exterior point. Calculate the number of interior points N. i Repeat the above process until the specified number of iterations is reached, and use all interior point samples from the iteration result with the largest number of interior points to recalculate a new homography matrix as the final homography matrix H.

[0110] 3. Image mapping

[0111] Here, image mapping can be performed by the image mapping module. Since the homography calculation module has already calculated the homography matrix between image 1 and image 2 to be stitched (e.g., using image 1 as the reference coordinate system), the image mapping module can use the calculated homography matrix H to map image 2 onto the reference image coordinate system and fuse it with image 1 to obtain the initial stitched image. This image mapping module can be implemented using the `warpPerspective` function in OpenCV.

[0112] 4. Self-supervised optimization

[0113] Here, self-supervised optimization can be performed by a self-supervised optimization module. This module can transform the initial stitched image into a target stitched image based on a trained self-supervised optimization model, thereby optimizing the image quality of the stitched region.

[0114] For example, the principle of the self-supervised optimization module is as follows: Figure 4 As shown, for the initial stitched image P i First, a CNN network can be used to learn and optimize the residual R. The CNN network employs an autoencoder structure that includes skip connections. Second, the predicted optimized residual R is further compared with the initial stitched image P. i The optimized stitched image P is obtained by adding the two images together and is used as the output, as shown in the following formula (4):

[0115] P = R + P i =f cnn (P i )+P i (4)

[0116] Since the ground truth of image stitching in real-world scenarios is difficult to obtain, this application example designs a self-supervised loss for network training, as shown in the following formula (5):

[0117] L=L1+L2+αLg (5)

[0118] Where L is the loss value of the self-supervised optimization model, L1 is the content loss value corresponding to the first image to be stitched, L2 is the content loss value corresponding to the second image to be stitched, and L... g Let α be the gradient loss between the optimized stitched image and the image to be stitched, which serves as the reference coordinate system. L1 and L2 can be understood as the content reconstruction losses corresponding to the images to be stitched, I1 and I2, respectively. L1 and L2 are constrained to ensure that the pixel colors in the optimized stitched image P are consistent with the corresponding pixel colors in the images to be stitched, I1 and I2. g This can be understood as gradient constraint loss, where the gradients of the corresponding pixels in the stitched image P and the image I1 to be stitched are consistent after constraint optimization.

[0119] For example, taking the smooth_L1 loss S as the basic loss function, it is defined as shown in the following formula (6). Then the L1 and L2 loss terms are defined as shown in formula (7) and formula (8) respectively:

[0120]

[0121] L1=S( I1-M1P ) (7)

[0122] L2=S( I w -M2P ) (8)

[0123] Where x is the variable of the basic loss function, I w The image to be stitched is the homography-transformed mapped image of image I2 (corresponding to the processing of the aforementioned image mapping module), calculated as shown in formula (9). P is the optimized stitched image output by the self-supervised optimization module, and M1 and M2 are the masks of the corresponding positions of images I1 and I2 on P, respectively. M2 is calculated as shown in formula (10). Since I1 is used as the reference coordinate system during homography, M1 directly takes the region [0:W, 0:H] on P as 1, and the rest as 0, where W is the width of I1 and H is the height of I1.

[0124] I w =W(H,I2) (9)

[0125] M2=W(H,E) (10)

[0126] Where W is the homography transformation function, which can be implemented using the `warpPerspective` function in OpenCV. E is a matrix of all ones, with a size equal to I. w Consistent.

[0127] L g The loss is defined as shown in formula (11):

[0128]

[0129] Among them, G P These represent the gradient maps corresponding to the optimized stitched image P. This represents the gradient map corresponding to image 1 to be stitched together. For example, the gradient map is obtained by filtering the image along the x and y directions using the Sobel operator, and then weighting the filtering results in the two directions.

[0130] It is understood that the image stitching method in this application embodiment achieves high-quality image stitching through four steps: random depth feature extraction, homography calculation, image mapping, and self-supervised optimization. This method is easy to implement, robust to illumination variations and noise in different stitched images, and avoids the problem of ground truth labeling in depth algorithms.

[0131] In this application embodiment, random depth feature extraction is performed by combining corner detection and depth descriptors. Specifically, feature points are extracted based on Harris corner detection, and a Siamese CNN network is used to construct a random depth vector as a feature descriptor for each detected feature point for feature matching. This feature extraction is simple to construct and easy to implement, avoiding the inefficiency of manually constructing feature descriptors. Furthermore, by introducing local image features and high-dimensional depth feature representations, the problem of low-dimensional image pixel features being affected by illumination changes and noise is alleviated, improving the robustness of subsequent feature matching.

[0132] In this application embodiment, a CNN with an autoencoder structure containing skip connections is used as a self-supervised optimization model to learn and optimize residuals. A self-supervised loss constraint network with content reconstruction and gradient reconstruction is designed for training. This can effectively reduce image artifacts and stitching errors in the stitching region, improve the quality of the stitched image, and solve the problem of difficulty in obtaining the ground truth of image stitching. This effectively solves the problems of lighting artifacts and stitching breaks in the stitching region, and effectively alleviates the impact of inaccurate homography matrix calculation, scene depth, and lighting changes from different viewpoints on image stitching quality. As a result, the image stitching quality is improved, which is conducive to meeting the subsequent application requirements of the stitched image.

[0133] In order to implement the method of the embodiments of this application, the embodiments of this application also provide an image stitching device, which corresponds to the above-described image stitching method. The steps in the above-described image stitching method embodiments are also fully applicable to the embodiments of this image stitching device.

[0134] like Figure 5As shown, the image stitching device includes an acquisition module 501 and an optimization module 502. The acquisition module 501 acquires an initial stitching image, which is a stitched image obtained after feature extraction, feature matching, and image transformation of a first image and a second image to be stitched. The optimization module 502 converts the initial stitching image into a target stitching image based on a trained self-supervised optimization model. The trained self-supervised optimization model determines the optimization residual of the initial stitching image and adds the optimization residual to the initial stitching image to obtain the target stitching image.

[0135] In some embodiments, the optimization module 502 is further configured to:

[0136] The self-supervised optimization model to be optimized is trained based on the training samples;

[0137] Determine that the number of training iterations of the self-supervised optimization model has reached a set number or that the loss value of the loss function has converged, and obtain the trained self-supervised optimization model.

[0138] The training samples include pairs of images to be stitched and initial stitched images corresponding to the pairs of images to be stitched. The loss function is determined based on content loss and gradient loss. The content loss represents the pixel difference between the optimized stitched image and the corresponding image to be stitched, and the gradient loss represents the pixel gradient difference between the optimized stitched image and the corresponding image to be stitched.

[0139] In some embodiments, the self-supervised optimization model is a CNN with an autoencoder structure containing skip connections.

[0140] In some embodiments, the acquisition module 501 is specifically used for:

[0141] Feature extraction is performed on the first and second images to be stitched together, respectively;

[0142] Based on the extracted features, the homography matrix between the first image and the second image is determined;

[0143] Based on the homography matrix, the first image and the second image are mapped to the same image coordinate system to obtain an initial stitched image.

[0144] In some embodiments, the acquisition module 501 performs feature extraction on the first image and the second image to be stitched, respectively, including:

[0145] For the first and second images to be stitched together, Harris corner points are extracted respectively to obtain the first corner point set of the first image and the second corner point set of the second image;

[0146] For the first and second images to be stitched together, feature extraction is performed based on a parameter-shared CNN to obtain a first feature map of the first image and a second feature map of the second image;

[0147] Based on each corner point in the first corner point set, position drilling is performed on the first feature map to obtain the feature vector of each corner point in the first corner point set.

[0148] Based on each corner point in the second corner point set, position drilling is performed on the second feature map to obtain the feature vector of each corner point in the second corner point set.

[0149] In some embodiments, the acquisition module 501 determines the homography matrix between the first image and the second image based on the extracted features, including:

[0150] Based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set, feature matching is performed to obtain a matching feature set between the first image and the second image;

[0151] Based on the matching feature set, the homography matrix between the first image and the second image is determined using the Random Sampling Consensus (RANSAC) algorithm.

[0152] In some embodiments, the acquisition module 501 performs feature matching based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set to obtain a matching feature set between the first image and the second image, including:

[0153] Iterate through each corner point in the first corner point set and find the matching corner points in the second corner point set for each corner point.

[0154] Based on the corner point pairs matched between the first corner point set and the corner point set, a matching feature set between the first image and the second image is obtained;

[0155] Among them, the matched corner point pair is the one with the shortest distance between the feature vectors of the two corner points.

[0156] In practical applications, module 501 (corresponding to) is used to acquire... Figure 2 The feature extraction module, homography calculation module, and image mapping module shown) and optimization module 502 (corresponding to) Figure 2 The self-supervised optimization module shown can be implemented by a processor in the image stitching device. Of course, the processor needs to run a computer program in memory to perform its function.

[0157] It should be noted that the image stitching device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the image stitching device and the image stitching method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0158] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device for executing the aforementioned image stitching method. Figure 6 The diagram shows only an exemplary structure of the electronic device, not the entire structure; implementation is possible as needed. Figure 6 The structure shown may be part or all of the structure.

[0159] like Figure 6 As shown, the electronic device 600 provided in this application embodiment includes: at least one processor 601, a memory 602, a user interface 603, and at least one network interface 604. The various components in the electronic device 600 are coupled together via a bus system 605. It can be understood that the bus system 605 is used to implement communication between these components. In addition to a data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 605.

[0160] The user interface 603 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0161] The memory 602 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0162] The image stitching method disclosed in this application can be applied to or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the image stitching method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically in memory 602. The processor 601 reads information from memory 602 and, in conjunction with its hardware, completes the steps of the image stitching method provided in the embodiments of this application.

[0163] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0164] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0165] In an exemplary embodiment, this application also provides a computer storage medium, specifically a computer-readable storage medium, such as a memory 602 storing a computer program, which can be executed by a processor 601 of an electronic device to complete the steps described in the method of this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0166] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0167] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0168] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image stitching method, characterized by, include: Obtain an initial stitched image, which is a stitched image obtained after feature extraction, feature matching and image transformation of the first image and the second image to be stitched; The self-supervised optimization model to be optimized is trained based on training samples; the training samples include pairs of images to be stitched and initial stitching images corresponding to the pairs of images to be stitched; the self-supervised optimization model is a CNN with an autoencoder structure containing skip connections. The loss function of the self-supervised optimization model is converged to obtain the trained self-supervised optimization model. The loss function is determined based on the content loss value and the gradient loss value. The content loss value represents the pixel difference between the optimized stitched image and the corresponding image to be stitched, and the gradient loss value represents the pixel gradient difference between the optimized stitched image and the corresponding image to be stitched. The initial stitched image is converted into a target stitched image based on a trained self-supervised optimization model; wherein, the trained self-supervised optimization model is used to determine the optimization residual of the initial stitched image, and the optimization residual is added to the initial stitched image to obtain the target stitched image; The process of obtaining the initial stitched image includes: Feature extraction is performed on the first and second images to be stitched together, respectively; Based on the extracted features, the homography matrix between the first image and the second image is determined; Based on the homography matrix, the first image and the second image are mapped to the same image coordinate system to obtain an initial stitched image.

2. The method according to claim 1, characterized in that, The first and second images to be stitched together are subjected to feature extraction, including: For the first and second images to be stitched together, Harris corner points are extracted respectively to obtain the first corner point set of the first image and the second corner point set of the second image; For the first and second images to be stitched together, feature extraction is performed based on a parameter-shared CNN to obtain a first feature map of the first image and a second feature map of the second image; Based on each corner point in the first corner point set, position drilling is performed on the first feature map to obtain the feature vector of each corner point in the first corner point set. Based on each corner point in the second corner point set, position drilling is performed on the second feature map to obtain the feature vector of each corner point in the second corner point set.

3. The method according to claim 2, characterized in that, The step of determining the homography matrix between the first image and the second image based on the extracted features includes: Based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set, feature matching is performed to obtain a matching feature set between the first image and the second image; Based on the matching feature set, the homography matrix between the first image and the second image is determined using a random sampling consensus algorithm.

4. The method according to claim 3, characterized in that, The feature matching based on the feature vectors of each corner point in the first corner point set and the feature vectors of each corner point in the second corner point set is used to obtain a matching feature set between the first image and the second image, including: Iterate through each corner point in the first corner point set and find the matching corner points in the second corner point set for each corner point. Based on the corner point pairs matched between the first corner point set and the second corner point set, a matching feature set between the first image and the second image is obtained; Among them, the matched corner point pair is the one with the shortest distance between the feature vectors of the two corner points.

5. An image stitching device, characterized in that, include: The acquisition module is used to acquire an initial stitched image, which is a stitched image obtained after feature extraction, feature matching and image transformation of the first image and the second image to be stitched; An optimization module is used to train a self-supervised optimization model to be optimized based on training samples. The training samples include pairs of images to be stitched and initial stitching images corresponding to the pairs of images to be stitched. The self-supervised optimization model is a CNN with an autoencoder structure containing skip connections. The module determines the convergence of the loss function of the self-supervised optimization model to obtain a trained self-supervised optimization model. The loss function is determined based on content loss and gradient loss, where the content loss represents the pixel difference between the optimized stitched image and the corresponding image to be stitched, and the gradient loss represents the pixel gradient difference between the optimized stitched image and the corresponding image to be stitched. The module converts the initial stitching image into a target stitching image based on the trained self-supervised optimization model. The trained self-supervised optimization model is used to determine the optimization residual of the initial stitching image and add the optimization residual to the initial stitching image to obtain the target stitching image. The acquisition module is specifically used to: extract features from the first image and the second image to be stitched together, respectively; Based on the extracted features, the homography matrix between the first image and the second image is determined; Based on the homography matrix, the first image and the second image are mapped to the same image coordinate system to obtain an initial stitched image.

6. An electronic device, characterized in that, include: A processor and memory for storing computer programs that can run on the processor, wherein, The processor, when running a computer program, performs the steps of the method according to any one of claims 1 to 4.

7. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images