A large parallax image stitching method

Abstract

The present invention provides a large-parallax image stitching method, comprising: extracting the overlapping area of ​​two large-parallax images through a codec network; selecting one of the overlapping areas as a reference area based on edge detection, and learning the homography matrix of the two overlapping areas; transforming the non-reference large-parallax image based on the homography matrix, and stitching and fusing the transformed large-parallax images; extracting illumination compensation information representing the implicit features of the initially stitched and fused large-parallax images, and performing illumination compensation on the initially stitched image based on the illumination compensation information to obtain a final stitched and fused image. The present invention achieves effective image stitching by learning the homography matrix of the two large-parallax images, and further extracts the implicit feature representation of the large-parallax image pair on this basis to correct for photometric differences in the images, thereby obtaining a higher-quality large-parallax image stitching and fusion result.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more particularly to a large parallax image stitching method. Background Art

[0002] Large-parallax image stitching is an important task in computer vision. It aims to seamlessly merge multiple images with significant perspective differences into a single panoramic image. However, in practical applications, large-parallax images often exhibit significant geometric and photometric differences due to factors such as shooting perspective, lighting conditions, and camera parameters. This can lead to problems such as misalignment, ghosting, and blurring in the stitched image.

[0003] Traditional image stitching methods are mostly based on feature matching and transformation matrix estimation. These methods achieve good results when the parallax is small. However, when stitching images with large parallax, these methods often struggle to accurately match feature points, resulting in inaccurate transformation matrix estimation, which in turn affects the stitching effect. Furthermore, these methods often require manual setting of thresholds and parameters, requiring different adjustments for different images and scenes, and lack adaptability and robustness.

[0004] In recent years, deep learning methods have made significant progress in image processing, providing new insights into large-parallax image stitching. Some deep learning-based image stitching methods achieve relatively accurate stitching by training neural networks to learn the transformation relationships between images. However, most of these methods rely on large amounts of labeled data for supervised learning. In practice, obtaining labeled large-parallax image data is extremely difficult. Furthermore, many methods typically focus solely on the geometric transformation between images while neglecting to account for photometric differences, resulting in noticeable stitching artifacts in areas with inconsistent lighting. Summary of the Invention

[0005] In response to the technical problems existing in the prior art, the present invention provides a large parallax image stitching method, which realizes effective image stitching by learning the homography matrix of two large parallax images. On this basis, the implicit feature representation of the large parallax image pair is further extracted to correct the photometric difference of the images and obtain higher quality large parallax image stitching and fusion results.

[0006] The present invention provides a large parallax image stitching method, comprising:

[0007] Inputting the two large parallax images into a first encoding and decoding network to obtain two overlapping regions, where the two overlapping regions refer to regions where the two large parallax images overlap each other;

[0008] Selecting one of the two overlapping areas as a reference area and the other overlapping area as a non-reference area based on edge detection;

[0009] Inputting the two overlapping areas into a second encoding and decoding network to obtain a homography matrix, wherein the homography matrix represents a transformation relationship from the non-reference area to the reference area;

[0010] Based on the homography matrix, the first large-parallax image is transformed, and the aligned area and the non-overlapping area in the transformed first large-parallax image are fused with image pixels to obtain an aligned first large-parallax image, wherein the first large-parallax image refers to the large-parallax image where the non-reference area is located, the aligned area refers to the overlapping area in the transformed first large-parallax image, and the non-overlapping area refers to the area outside the overlapping area of ​​the first large-parallax image;

[0011] Stitching the aligned first large parallax image and the second large parallax image to obtain a preliminary stitched image, where the second large parallax image refers to the large parallax image where the reference area is located;

[0012] inputting the two overlapping regions into a hidden neural network, outputting illumination compensation information, and performing illumination compensation on the preliminary stitched image based on the illumination compensation information;

[0013] The illumination-compensated stitched image is fused with the preliminary stitched image to obtain a final stitched image.

[0014] The present invention provides a large-parallax image stitching method. The method extracts the overlapping area of ​​two large-parallax images through a codec network. Based on edge detection, one of the overlapping areas is selected as a reference area, and the homography matrix of the two overlapping areas is learned. Based on the homography matrix, a non-reference large-parallax image is transformed and the transformed large-parallax images are stitched and fused. Lighting compensation information representing the implicit features of the initially stitched and fused large-parallax images is extracted, and based on the illumination compensation information, the initial stitched image is subjected to illumination compensation to obtain a final stitched and fused image. The method effectively stitches the images by learning the homography matrix of the two large-parallax images. On this basis, the method further extracts the implicit feature representation of the large-parallax image pair to correct for photometric differences in the images, resulting in higher-quality large-parallax image stitching and fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of a large parallax image stitching method provided by an embodiment of the present invention;

[0016] Figure 2 A diagram of the first codec network architecture provided by an embodiment of the present invention;

[0017] Figure 3 A diagram of a second encoding and decoding network architecture provided by an embodiment of the present invention;

[0018] Figure 4 A diagram of the hidden neural network architecture provided by an embodiment of the present invention;

[0019] Figure 5 A diagram of the feature fusion network architecture provided by an embodiment of the present invention;

[0020] Figure 6 A diagram of the overall network architecture for large parallax image stitching provided by an embodiment of the present invention;

[0021] Figure 7 A schematic structural diagram of a large parallax image stitching system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0023] In response to the urgent needs of the prior art, the present invention provides a large parallax image stitching method that can adaptively learn the transformation relationship between images and simultaneously consider the processing of geometric and photometric differences to achieve high-quality large parallax image stitching. Specifically, through the method of implicit feature representation and homography matrix learning, in which the homography matrix learning mainly processes the geometric transformation of the stitched image, while the implicit feature representation mainly learns the photometric difference between the two images, unsupervised large parallax image stitching is achieved, the adaptability and robustness of the stitching effect are improved, and strong technical support is provided for related applications in the field of computer vision.

[0024] Figure 1 A flow chart of a large parallax image stitching method provided by the present invention is as follows: Figure 1 As shown, the method includes:

[0025] Step 1: Input two large parallax images into a first encoding and decoding network to obtain two overlapping areas, where the two overlapping areas are areas where the two large parallax images overlap.

[0026] It can be understood that for two large parallax RGB images with different illumination, the overlapping areas of the two large parallax images are extracted through the first encoding and decoding network to obtain two overlapping areas, including a first overlapping area in the first large parallax image that overlaps with the second area in the second large parallax image, and these two overlapping areas have the most identical content.

[0027] The first codec network includes three encoders and three decoders, each of the encoders includes a densely connected bottleneck structure residual block and a spatial attention block, and each of the decoders includes a densely connected bottleneck structure residual block and a weighted spatial attention block; the two large disparity images are input into the first codec network to obtain two overlapping areas, including:

[0028] Inputting two large-parallax images into a first encoding / decoding network respectively, extracting local features at three scales of each large-parallax image through three bottleneck structure residual blocks in the encoder, and aggregating the local features at three scales by downsampling through three spatial attention blocks in the encoder;

[0029] The three residual blocks in the decoder are used to restore the aggregated local features of the three scales, and the three weighted spatial attention blocks are used to perform weighted fusion on the local features of the three scales by upsampling, and an attention map of the overlapping area of ​​each large disparity image is output;

[0030] According to each large parallax image and the corresponding overlapping area attention map, the overlapping area of ​​each large parallax image is calculated.

[0031] See also Figure 2 , is the network structure diagram of the first codec network. The first codec network processes large parallax image pairs through an unsupervised densely connected codec network. The main part of the first codec network is a U-Net-based codec network. By extracting and fusing two large parallax images with different illumination, the attention map of the overlapping area is obtained. The first codec network specifically includes three encoders and three decoders. The basic blocks of each encoder and each decoder are composed of densely connected bottleneck structure residual blocks and spatial attention blocks. The specific implementation of the unsupervised densely connected codec network is as follows:

[0032] Encoder: Each encoder's basic block consists of a densely connected bottleneck residual block and spatial attention. Each bottleneck residual block extracts local features from large-parallax images through a series of convolution operations and skip connections, which are then aggregated by the spatial attention block. The three encoders are multi-scale progressive, downsampling the feature maps to reduce spatial resolution and computational complexity.

[0033]

[0034] in, Represents the encoder output feature map of the i-th layer, and DenseBlock represents the densely connected bottleneck structure residual block operation.

[0035] Decoder part: The basic block of each decoder consists of a densely connected bottleneck structure residual block and weighted spatial attention. Each bottleneck structure residual block recovers the local features of the image through a series of operations and uses the weighted spatial attention mechanism to fuse the features transmitted by the encoder.

[0036] The three decoders are in a step-by-step recovery relationship, and the spatial resolution of the image is gradually restored by upsampling the feature map.

[0037]

[0038] in, Denotes the decoder output feature map of layer i, DenseBlock denotes the densely connected bottleneck residual block operation, and Fusion denotes the weighted spatial attention fusion operation, which fuses the output features of the current decoder with the features of the corresponding size of the encoder. The decoder outputs an attention map of the overlapping area of ​​the two large-parallax images.

[0039] Overlap area estimation: Multiply the overlapping area attention map by the input large disparity image and add them together to obtain the overlapping area estimation of the large disparity image.

[0040]

[0041] in, represents the estimated overlapping area, Denotes the input large parallax image, and A denotes the overlapping area attention map. The overlapping area of ​​the two large parallax images is estimated according to this formula.

[0042] To train the first codec network, training and test sets of large-disparity image pairs are first constructed. Specifically, the ELA algorithm is used to stitch images from the UDISD dataset to collect a large number of real stitched images. Images with an extrapolated area less than 10% of the entire image are ignored to ensure that the stitched images have sufficient disparity and irregular boundaries. A content-aware warping algorithm is used to generate a rich variety of grid deformation matrices from these real stitched images. These grid deformations are used in the subsequent rectangularization process. The inverse matrix of the grid deformation is applied to warp the real rectangular images into synthetic stitched images, retaining 5,705 usable samples from over 60,000 samples. The synthetic images are manually screened to remove those with significant distortion. 653 distortion-free samples were selected from the initial large number of real stitched images and added to the training set to improve the dataset's diversity and generalization ability. Overall, the dataset consists of 5,839 training samples and 519 testing samples. Each image in the dataset has a resolution of 512×384.

[0043] Input the training set images into Figure 2 The first encoder-decoder network of the estimated overlap area is trained until the loss function reaches the preset convergence condition. The structure of the encoder-decoder network model for estimating the overlap area includes three stages. The basic blocks of the encoder and decoder are composed of densely connected bottleneck structure residual blocks and spatial attention blocks.

[0044] The input large-disparity image pair first undergoes 3×3 convolution for channel upscaling and preliminary feature extraction, resulting in an H×W×C tensor. This is then passed through a three-stage encoder. In the first stage, this tensor passes through the encoder's first densely connected bottleneck residual block, resulting in a tensor of the same size. In the second stage, this tensor passes through the second residual block for feature extraction, followed by downsampling to a 1 / 2H×1 / 2W×2C tensor. In the third stage, the downsampled tensor passes through the third residual block for feature extraction, followed by further downsampling to a 1 / 4H×1 / 4W×4C tensor. Similarly, the decoder undergoes a similar three-stage process: a 1 / 4H×1 / 4W×4C tensor is first decoded through a residual block, then upsampled, and finally decoded through the next residual block. A skip connection is performed between the encoder and decoder to fuse the features of the encoder and decoder. Specifically, the output of the third-stage encoder is upsampled by a 3×3 convolution and then concatenated with the output of the second-stage encoder. The concatenated tensor is then downsampled by a 1×1 convolution, and the output tensor is then residually connected to the output tensor of the third-stage decoder. In the final stage, the output image is treated as the initial overlap region attention map, which is then multiplied by the input image and residually connected to the input image to obtain the final overlap region estimate for the large disparity image.

[0045] During the training of the first codec network, the loss of the first codec network is calculated based on the L2 loss function and the frequency domain loss function. This is used to measure the difference between the overlap area estimate and the actual overlap area, and to guide the training of the first codec network. The calculation formula for the loss of the first codec network is:

[0046]

[0047] in, represents the total loss of the first codec network, represents the L2 loss function, represents the frequency domain loss function, represents the overlapping area, Represent two large parallax images respectively.

[0048] Regarding loss functions, in the unsupervised large parallax image stitching method of the present invention, in order to ensure the quality and consistency of the images generated at each stage, multiple loss functions are applied during the network training process, including mean square error loss, frequency domain loss, and content alignment loss. The different loss functions are introduced as follows:

[0049] 1. L2 loss (Mean square error loss)

[0050] L2 loss, also known as mean squared error loss, is used to measure the difference between the model's predictions and the true values. It works by calculating the sum of the squares of the differences between the predicted values ​​and the true values, giving a greater penalty for large errors, thereby encouraging the model to predict outputs closer to the true values.

[0051] For the overlapping area estimation of large disparity image pairs, the L2 loss can be expressed as:

[0052]

[0053] in, represents the L2 loss, Indicates that the model is at position The predicted pixel value of Indicates that the true overlapping area is at position The pixel value of .

[0054] 2. Frequency domain loss

[0055] Frequency domain loss is calculated in the frequency domain of an image, taking into account its overall structure and texture. By penalizing the difference between the predicted and true images in the frequency domain, frequency domain loss helps the model better understand the structural relationships between images, leading to more accurate estimation of overlapping regions.

[0056] The frequency domain loss based on Fourier transform can be expressed as:

[0057]

[0058] in, represents the frequency domain loss, and Represents the Fourier transform of the predicted image and the real image in the frequency domain Here the value of Represents the modulus of a complex number.

[0059] Through the processing of the above-mentioned unsupervised densely connected encoder-decoder network, the overlapping area of ​​two large parallax images with different illumination can be effectively estimated, providing a basis for subsequent image fusion or reconstruction tasks.

[0060] Step 2: selecting one overlapping area from the two overlapping areas as a reference area and the other overlapping area as a non-reference area based on edge detection.

[0061] It can be understood that after obtaining the overlap area estimate of the two large parallax images, the vertical edge and horizontal edge information is used to select one overlap area from the overlap areas of the two large parallax images as the reference area for subsequent processing, and the overlap area of ​​the other large parallax image is used as the non-reference area.

[0062] In a possible embodiment of the present invention, the edge detection-based method of selecting one overlapping area from the two overlapping areas as a reference area and the other overlapping area as a non-reference area includes: performing gamma transform on the two overlapping areas respectively to obtain two overlapping areas after gamma transform; detecting edge pixels on the two overlapping areas after gamma transform based on Sobel operator edge detection; counting the number of edge pixels in each overlapping area; and taking the overlapping area with a larger number of edge pixels as a reference area and the overlapping area with a smaller number of edge pixels as a non-reference area.

[0063] Specifically, to effectively improve the legibility of overlapping areas, the output overlap region estimate of the two large-parallax images is first subjected to a gamma transformation with a value of 2. This means that pixels with lower grayscale values ​​(typically corresponding to dark areas or areas rich in detail in the image) are relatively amplified, while pixels with higher grayscale values ​​(bright areas) grow more slowly. This transformation not only enhances the overall contrast of the image, making the image information richer and more distinct, but also helps sharpen and highlight the edges of the overlapping area.

[0064] Gamma transform: Estimate the image in the overlapping area of ​​the large parallax image pair obtained in step 1 1 and 2. Gamma transform is performed separately to enhance the contrast of the image and make the edges of the overlapping areas more prominent. The formula for gamma transform is as follows:

[0065]

[0066] in, Represents the overlapping area image, is the gamma value, is the transformed overlapping area image.

[0067] After completing the gamma transform of the overlapping area of ​​the two large-parallax images to enhance contrast and highlight edge features, the Sobel operator is then used for edge detection. Specifically, the horizontal and vertical Sobel operators are applied to the gamma-transformed image, respectively. The horizontal and vertical gradients of the image at each pixel are calculated through convolution operations, resulting in two results, representing the edge strength of the original image in the horizontal and vertical directions, respectively. The calculated gradient image is then subjected to the Otsu threshold segmentation method. This method analyzes the image's grayscale histogram to find an optimal threshold. Pixels with gradient values ​​above this threshold are marked as edges, while pixels below the threshold are considered non-edges, resulting in a binary edge image. This step ensures that only significant edges are retained, while noise and subtle changes are suppressed.

[0068] Specifically, the Sobel operator is used to perform edge detection on the overlapping areas of the two gamma-transformed images. Edge pixels are extracted from each overlapping area. The edge pixel counts are then counted in each overlapping area. The results are compared and the overlapping area with the highest number of edge pixels is designated as the reference area. The other overlapping area is designated as the non-overlapping area. The reference areas are then visually inspected and subjected to necessary post-processing, such as morphological operations (dilation, erosion, etc.), to remove isolated points or refine edges, ensuring the accuracy and consistency of the reference areas.

[0069] In a possible embodiment of the present invention, based on the Sobel operator edge detection, horizontal edge pixel points and vertical edge pixel points are detected in the two overlapping areas after gamma transformation, including:

[0070] Calculate the horizontal gradient value of each pixel in each overlapping area and longitudinal gradient values ;

[0071] According to the horizontal gradient value of each pixel and longitudinal gradient values , calculate the gradient amplitude of each pixel;

[0072] Based on the gradient magnitude of each pixel, determine whether each pixel is an edge pixel.

[0073] It is understandable that for each pixel in the overlapping area, the horizontal gradient value of each pixel in each overlapping area is calculated. and longitudinal gradient values ,include:

[0074] Get the horizontal gradient operator Gx and the vertical gradient operator Gy;

[0075] Convolve the horizontal gradient operator with each pixel to obtain the horizontal gradient value of each pixel ;

[0076] Convolve the vertical gradient operator with each pixel to obtain the vertical gradient value of each pixel .

[0077] Wherein, the horizontal gradient operator is:

[0078] ;

[0079] The vertical gradient operator is:

[0080] ;

[0081] The horizontal gradient value Gx(x, y) of each pixel is: , the longitudinal gradient value of each pixel for: .

[0082] According to the horizontal gradient value of each pixel and longitudinal gradient values , calculate the gradient amplitude of each pixel, including:

[0083] ;

[0084] in, is the gradient magnitude of each pixel.

[0085] The step of determining whether each pixel is an edge pixel based on the gradient magnitude of each pixel includes:

[0086] When the gradient amplitude of a pixel point is greater than a preset amplitude threshold, the pixel point is an edge pixel point; otherwise, the pixel point is a non-edge pixel point.

[0087] The edge pixels in the two overlapping areas are detected by Sobel operator edge detection. The overlapping area with more edge pixels is selected as the reference area, and the overlapping area with fewer edge pixels is selected as the non-reference area.

[0088] Step 3: Input the two overlapping areas into a second encoding and decoding network to obtain a homography matrix, where the homography matrix represents a transformation relationship from the non-reference area to the reference area.

[0089] It is understandable that in order to effectively stitch two images with large parallax, the second codec network learns the homography matrix of the reference area to transform the non-reference area so that the non-reference area is geometrically aligned with the reference area. The specific implementation steps are as follows:

[0090] Two overlapping regions (one reference region and one non-reference region) are fed into a multi-scale encoder-decoder network (the second encoder-decoder network). The output of the second encoder-decoder network is a multi-grid homography matrix, which describes the transformation from the non-reference region to the reference region. The multi-grid homography matrix can be viewed as a collection of multiple local homography matrices, each responsible for the transformation of a grid region.

[0091] Among them, you can refer to Figure 3 The architecture of the second codec network is shown in Figure 2. Its working principle is as follows: The two estimated overlapping regions are first subjected to feature extraction using a convolutional residual block, resulting in an H×W×C tensor. This tensor is then flattened to an N×C shape (N=H×W). The two N×C tensors are then matrix-multiplied to calculate the correlation between the resulting feature vectors. The distances between the feature point descriptors in each pair of images are compared, and the feature point with the smallest distance to each feature point is selected as the matching point. The multiplied N×N tensor is reshaped to H×W×N and passed through an MLP module consisting of three consecutive convolutional layers and two fully connected layers to obtain the homography matrix. To prevent overfitting of the network model, a dropout layer with a dropout probability of 0.5 is added before each fully connected layer. The MLP module outputs 8 real numbers. The direct linear transform (DLT) algorithm is applied to these 8 real numbers to obtain the 3×3 homography matrix between the two images.

[0092] Step 4: Based on the homography matrix, the first large parallax image is transformed, and the image pixels of the aligned area and the non-overlapping area in the transformed first large parallax image are fused to obtain the aligned first large parallax image, wherein the first large parallax image refers to the large parallax image where the non-reference area is located, the aligned area refers to the overlapping area in the transformed first large parallax image, and the non-overlapping area refers to the area outside the overlapping area in the first large parallax image.

[0093] It is understandable that the non-reference area is transformed to the same perspective as the reference area using the homography matrix to obtain the aligned area. In the present invention, the non-reference large parallax image is transformed using the homography matrix to obtain the transformed non-reference large parallax image. The overlapping area of ​​the transformed non-reference large parallax image is called the alignment area. Due to the difference in perspective, a smooth transition method is used to fuse the image pixels at the junction of the alignment area and the non-overlapping area to eliminate possible seams or inconsistencies. Finally, the aligned large parallax image is updated.

[0094] Among them, the overall architecture of the second codec network is the same as the first codec network architecture in step 1, which is achieved through downsampling, upsampling and feature fusion. This multi-scale feature can better capture the details and context information of the image, thereby improving the accuracy and quality of image reconstruction.

[0095] When performing unsupervised learning on the second codec network, its loss function consists of content alignment loss and smooth transition loss, which are used to measure the difference between the spliced ​​view image and the real view image and guide the training of the second codec network. The loss function is calculated as follows:

[0096]

[0097] in, represents the total loss of the second codec network, represents the content alignment loss function, represents the smooth transition loss function, Represents a stitched image.

[0098] The content alignment loss function and smooth transition loss function are introduced as follows:

[0099] 1. Content alignment loss

[0100] In the encoder-decoder network that learns the homography matrix, the content alignment loss ensures that the transformed non-reference regions maintain high content consistency with the reference regions. By minimizing the content alignment loss, the model can learn a more accurate homography matrix, thereby achieving precise alignment between images.

[0101] The content alignment loss can be calculated by comparing the pixel differences between the reference region and the non-reference region after multi-grid homography transformation, as follows:

[0102]

[0103] in, represents the content alignment loss, N and M represent the height and width of the image respectively, Indicates that the reference area is at position The pixel value of represents the pixel value at the corresponding position of the non-reference area after the multi-grid homography transformation. This formula calculates the squared difference of all pixels between the reference area and the transformed area, and obtains the final content alignment loss by summing and averaging.

[0104] 2. Smooth transition loss

[0105] The smooth transition loss ensures a smooth transition between image pixels at the interface between the aligned and non-overlapping regions, reducing visual artifacts and abruptness during fusion. By minimizing the smooth transition loss, the model learns a more natural way to fuse images, making the aligned images more visually coherent and consistent.

[0106] Based on the smooth transition loss using the gradient difference of pixel values, the formula is as follows:

[0107]

[0108] in, Represents the smooth transition loss, N represents the number of pixels at the junction, and B represents the set of pixels at the junction of the aligned area and the non-overlapping area. represents the aligned image, represents the image of non-overlapping regions, and Represents the gradient calculation of the image in the horizontal and vertical directions respectively. This formula calculates the gradient difference of the pixel at the junction in the horizontal and vertical directions, and obtains the final smooth transition loss by summing and averaging.

[0109] Step 5: Splicing the aligned first large parallax image and the second large parallax image to obtain a preliminary spliced ​​image, where the second large parallax image refers to the large parallax image where the reference area is located.

[0110] It can be understood that, through the selection of the reference area in step 2 and the extraction of the homography matrix, the non-reference large parallax image is aligned and transformed using the homography matrix, and the aligned large parallax image and the reference large parallax image are spliced ​​to form a preliminary spliced ​​image.

[0111] Step 6: Input the two overlapping areas into a hidden neural network, output illumination compensation information, and perform illumination compensation on the preliminary spliced ​​image based on the illumination compensation information.

[0112] As can be understood, the two overlapping regions are fed into an implicit neural representation network. The hidden neural network is trained to learn a photometric adjustment function that adaptively adjusts the image brightness based on its photometric features. The photometrically adjusted implicit feature representation and the aligned image are then fed into a convolutional neural network for feature fusion and image reconstruction.

[0113] In a possible embodiment of the present invention, two overlapping regions are input into a hidden neural network, illumination compensation information is output, and illumination compensation is performed on the preliminary spliced ​​image based on the illumination compensation information, including:

[0114] Get the coordinate position and content window of each pixel point in each overlapping area respectively;

[0115] Each pixel in the two overlapping areas is input into the hidden neural network as the position and content window, and the illumination compensation information is output.

[0116] See Figure 4 , is the architecture diagram of the hidden neural network. The hidden neural representation is to reconstruct or generate the image by learning a mapping function from continuous coordinates (such as spatial coordinates) to image features or attributes. First, the aligned region of the input preliminary spliced ​​large parallax image is converted from the RGB color domain to the HSV color domain. The enhancement process is redefined by mapping the two-dimensional coordinates of the image with uneven brightness to its illumination component (i.e., V).

[0117] Position coordinates are obtained by generating uniformly distributed coordinate points in both height and width and combining these points into a three-dimensional coordinate grid, where each point represents a specific location in the image or feature map. The content window is obtained by convolving the padded input image with a specially crafted convolution kernel (i.e., taking the pixel values ​​of the position coordinate and its surrounding coordinates), thereby extracting a fixed-size image region corresponding to each location. The prepared position coordinates and their content window values ​​are input into a hidden neural network. The hidden neural network first processes the content window features and position coordinates separately, and the output network combines these two information for the final prediction or regression. All three components pass through a nn.Linear layer, which performs a linear transformation on the input. The activation function of this layer is a sine function, but is changed to a sigmoid function in the final layer to accommodate different output requirements. It also accepts a w0 parameter to control the frequency of the sine function, which helps the network capture features at different frequencies. During the training process, different weight decay strategies are applied. The network parameters are divided into different groups. Each group of parameters has its corresponding weight decay value. The default values ​​are 0.1, 0.0001, and 0.001. This strategy helps prevent network overfitting and improve the generalization ability of the model.

[0118] The illumination compensation information of the image is output by the hidden neural network, and based on the illumination compensation information, the illumination compensation is performed on the preliminary spliced ​​image in step 5. The illumination compensation steps include:

[0119] Convert the initial stitched image from the RGB color gamut to the HSV color gamut, and extract the V component in the HSV color gamut;

[0120] Calculating a corrected light intensity based on the light compensation information and the V component, and using the corrected light intensity as a new V component;

[0121] The calculation formula of the corrected light intensity is:

[0122] ;

[0123] The new V component is , is the corrected light intensity, Information for light compensation;

[0124] The new V component is fused with the H component and the S component in the HSV color domain to obtain a light-compensated stitched image, and finally the stitched image is transformed from the HSV color domain to the RGB color domain to obtain a light-compensated stitched image.

[0125] Step 7: Fusing the illumination-compensated stitched image with the preliminary stitched image to obtain a final stitched image.

[0126] The illumination-compensated stitched image and the preliminary stitched image are input into a feature fusion network, and the illumination-compensated stitched image and the preliminary stitched image are residually connected through three residual blocks in the feature fusion network to obtain a final stitched image.

[0127] See Figure 5 , is the architecture diagram of the feature fusion network. The spliced ​​image after illumination compensation in step 6 and the preliminary spliced ​​image output in step 5 are sent to Net4. After three ResBlock residual blocks, the extracted result is residually connected with the spliced ​​image compensated and corrected by the estimated illumination information in step 5 to obtain the final fused spliced ​​image.

[0128] See also Figure 6, is the overall network architecture of the large parallax image stitching method of the present invention, and its working principle is as follows: two large parallax images are input into the first encoding and decoding network, two overlapping areas are output, one overlapping area is selected from the two overlapping areas as the reference area, and the other overlapping area is selected as the non-reference area. The reference area and the non-reference area are input into the second encoding and decoding network, and the homography matrix is ​​output. Based on the homography matrix, the non-reference large parallax image is transformed to a unified perspective with the reference large parallax, and the pixels of the overlapping area and the non-overlapping area are smoothly transitioned to obtain a preliminary stitched image. The two overlapping areas are input into the hidden neural network, and illumination compensation information is output. The preliminary stitched image is subjected to illumination compensation based on the illumination compensation information. Finally, the stitched image after illumination compensation is stitched and fused with the preliminary stitched image to form a final stitched and fused large parallax image.

[0129] See also Figure 7 , a large parallax image stitching system provided by the present invention, the system comprising:

[0130] A first acquisition module 701 is configured to input two large parallax images into a first encoding and decoding network to acquire two overlapping regions, where the two overlapping regions are regions where the two large parallax images overlap.

[0131] A determination module 702 is configured to determine, based on edge detection, one of the two overlapping areas as a reference area and the other overlapping area as a non-reference area;

[0132] A second acquisition module 703 is configured to input the two overlapping regions into a second encoding and decoding network to obtain a homography matrix, where the homography matrix represents a transformation relationship from the non-reference region to the reference region;

[0133] a transformation module 704 configured to transform the first large-parallax image based on the homography matrix, and fuse image pixels of the aligned area and the non-overlapping area in the transformed first large-parallax image to obtain an aligned first large-parallax image, wherein the first large-parallax image refers to the large-parallax image where the non-reference area is located, the aligned area refers to the overlapping area in the transformed first large-parallax image, and the non-overlapping area refers to the area outside the overlapping area of ​​the first large-parallax image;

[0134] An image stitching module 705 is configured to stitch the aligned first large parallax image and the second large parallax image to obtain a preliminary stitched image, where the second large parallax image is the large parallax image of the reference area; and to fuse the illumination-compensated stitched image with the preliminary stitched image to obtain a final stitched image.

[0135] The illumination compensation module 706 is configured to input the two overlapping regions into a hidden neural network, output illumination compensation information, and perform illumination compensation on the preliminary spliced ​​image based on the illumination compensation information.

[0136] It can be understood that the large parallax image stitching system provided by the present invention corresponds to the large parallax image stitching method provided by the aforementioned embodiments. The relevant technical features of the large parallax image stitching system can refer to the relevant technical features of the large parallax image stitching method, which will not be repeated here.

[0137] An embodiment of the present invention provides a large parallax image stitching method. The method extracts the overlapping area of ​​two large parallax images through a codec network; selects one of the overlapping areas as a reference area based on edge detection, and learns the homography matrix of the two overlapping areas; transforms the non-reference large parallax image based on the homography matrix, and stitches and fuses the transformed large parallax images; extracts illumination compensation information representing the implicit features of the initially stitched and fused large parallax images, and performs illumination compensation on the initially stitched image based on the illumination compensation information to obtain a final stitched and fused image. The present invention achieves effective image stitching by learning the homography matrix of the two large parallax images. On this basis, it further extracts the implicit feature representation of the large parallax image pair to correct the photometric differences of the images, thereby obtaining a higher-quality large parallax image stitching and fusion result.

[0138] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0139] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0144] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A large parallax image stitching method, characterized in that: include: Inputting the two large parallax images into a first encoding and decoding network to obtain two overlapping regions, where the two overlapping regions refer to regions where the two large parallax images overlap each other; Selecting one of the two overlapping areas as a reference area and the other overlapping area as a non-reference area based on edge detection; Inputting the two overlapping areas into a second encoding and decoding network to obtain a homography matrix, wherein the homography matrix represents a transformation relationship from the non-reference area to the reference area; Based on the homography matrix, the first large-parallax image is transformed, and the aligned area and the non-overlapping area in the transformed first large-parallax image are fused with image pixels to obtain an aligned first large-parallax image, wherein the first large-parallax image refers to the large-parallax image where the non-reference area is located, the aligned area refers to the overlapping area in the transformed first large-parallax image, and the non-overlapping area refers to the area outside the overlapping area of ​​the first large-parallax image; Stitching the aligned first large parallax image and the second large parallax image to obtain a preliminary stitched image, where the second large parallax image refers to the large parallax image where the reference area is located; Inputting the two overlapping areas into a hidden neural network, outputting illumination compensation information, and performing illumination compensation on the preliminary stitched image based on the illumination compensation information; fusing the illumination-compensated stitched image with the preliminary stitched image to obtain a final stitched image; Inputting the two overlapping areas into a hidden neural network and outputting illumination compensation information includes: Respectively obtain the coordinate position and content window of each pixel point in each overlapping area; Inputting the coordinate position and content window of each pixel point in the two overlapping areas into a hidden neural network, and outputting illumination compensation information, wherein the content window is the neighborhood of the pixel point; Performing illumination compensation on the preliminary stitched image based on the illumination compensation information includes: Converting the preliminary stitched image from the RGB color gamut to the HSV color gamut, and extracting the V component in the HSV color gamut; Calculating a corrected light intensity based on the light compensation information and the V component, and using the corrected light intensity as a new V component; The calculation formula of the corrected light intensity is: ； The new V component is , is the corrected light intensity, Information for light compensation; The new V component is fused with the H component and the S component in the HSV color domain, and restored to the RGB color domain to obtain a spliced ​​image after illumination compensation.

2. The large parallax image stitching method according to claim 1, characterized in that: The first encoding and decoding network includes three encoders and three decoders, each of the encoders includes a densely connected bottleneck structure residual block and a spatial attention block, and each of the decoders includes a densely connected bottleneck structure residual block and a weighted spatial attention block; The step of inputting two large parallax images into a first encoding / decoding network to obtain two overlapping areas includes: Inputting the two large-parallax images into the first encoding and decoding network respectively, extracting local features at three scales of each large-parallax image through the three bottleneck structure residual blocks in the encoder, and aggregating the local features at three scales by downsampling through the three spatial attention blocks in the encoder; The three bottleneck structure residual blocks in the decoder are used to restore the aggregated local features of the three scales, and the three weighted spatial attention blocks are used to perform weighted fusion on the local features of the three scales by upsampling, and an attention map of the overlapping area of ​​each large disparity image is output; According to each large parallax image and the corresponding overlapping area attention map, the overlapping area of ​​each large parallax image is calculated.

3. The large parallax image stitching method according to claim 2, characterized in that: The calculating the overlapping area of ​​each large parallax image according to each large parallax image and the corresponding overlapping area attention map includes: in, represents the overlapping area of ​​large parallax images, represents the input large disparity image, and A represents the overlapping area attention map.

4. The large parallax image stitching method according to claim 1 or 2, characterized in that: When training the first codec network, the loss of the first codec network is calculated based on the L2 loss function and the frequency domain loss function. The calculation formula is: in, represents the loss of the first codec network, represents the L2 loss function, represents the frequency domain loss function, represents the overlapping area, Represent two large parallax images respectively.

5. The large parallax image stitching method according to claim 1, characterized in that: The step of selecting one overlapping area as a reference area and the other overlapping area as a non-reference area from the two overlapping areas based on edge detection includes: performing gamma transformation on the two overlapping regions respectively to obtain the two overlapping regions after gamma transformation; Based on the Sobel operator edge detection, edge pixel points of the two overlapping areas after gamma transformation are detected respectively; Counting the number of edge pixels in each overlapping area; The overlapping area with a larger number of edge pixels is used as a reference area, and the overlapping area with a smaller number of edge pixels is used as a non-reference area.

6. The large parallax image stitching method according to claim 5, characterized in that: The Sobel operator-based edge detection detects horizontal edge pixels and vertical edge pixels in the two overlapping areas after gamma transformation, including: Calculate the horizontal gradient value Gx(x, y) and the vertical gradient value Gy(x, y) of each pixel in each overlapping area; Calculate the gradient amplitude of each pixel according to the horizontal gradient value Gx(x, y) and the vertical gradient value Gy(x, y) of each pixel; Based on the gradient magnitude of each pixel, determine whether each pixel is an edge pixel.

7. The large parallax image stitching method according to claim 6, characterized in that: The calculation of the horizontal gradient value Gx(x, y) and the vertical gradient value Gy(x, y) of each pixel in each overlapping area includes: Get the horizontal gradient operator Gx and the vertical gradient operator Gy; Convolve the horizontal gradient operator with each pixel to obtain the horizontal gradient value Gx(x,y) of each pixel; Convolve the vertical gradient operator with each pixel to obtain the horizontal gradient value Gy(x,y) of each pixel; Wherein, the horizontal gradient operator is: ； The vertical gradient operator is: ； Among them, the horizontal gradient value Gx(x, y) of each pixel is: , the vertical gradient operator of each pixel is: ; The horizontal gradient value of each pixel and the longitudinal gradient value , calculate the gradient amplitude of each pixel, including: ； in, is the gradient amplitude of each pixel; The step of determining whether each pixel is an edge pixel based on the gradient magnitude of each pixel includes: When the gradient amplitude of a pixel point is greater than a preset amplitude threshold, the pixel point is an edge pixel point; otherwise, the pixel point is a non-edge pixel point.

8. The large parallax image stitching method according to claim 1, wherein: The loss of the second codec network is calculated based on the content alignment loss function and the smooth transition loss function. The calculation formula is: in, represents the loss of the first codec network, represents the content alignment loss function, represents the smooth transition loss function, Represents the aligned large-parallax image.

9. The large parallax image stitching method according to claim 1, wherein: The step of fusing the illumination-compensated stitched image with the preliminary stitched image to obtain a final stitched image includes: Inputting the illumination-compensated stitched image and the preliminary stitched image into a feature fusion network, performing a residual connection between the illumination-compensated stitched image and the preliminary stitched image through three residual blocks in the feature fusion network to obtain a final stitched image; Inputting the two overlapping areas into a hidden neural network, outputting illumination compensation information, and performing illumination compensation on the preliminary stitched image based on the illumination compensation information; The illumination-compensated stitched image is fused with the preliminary stitched image to obtain a final stitched image.

Images