Image processing methods based on large AI models

Abstract

This invention proposes an image processing method based on a large AI model, belonging to the field of image processing technology, including the following steps: S1: Acquire multiple images and perform preprocessing; S2: Construct a deep learning model, use the deep learning model to extract feature points from the multiple images respectively, and use a feature matching algorithm to match the feature points of the multiple images to obtain a first matching result. Use a filtering algorithm to screen for mismatches in the first matching pair to obtain a second matching result. Apply an affine transformation to the second matching result to complete the registration of the multiple images; S3: Use an image fusion model to fuse the registered multiple images to generate a panoramic image; S4: Identify blank areas in the panoramic image, and use a completion method to complete the blank areas to obtain the final panoramic image. This invention can generate high-quality, natural panoramic images, improving the visual effect and viewing experience of traveler photos.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method based on a large AI model. Background Technology

[0002] With the widespread use of smartphones and digital cameras, taking photos has become an important way for tourists to record beautiful moments during outdoor travels. However, due to tight schedules or being rushed by others, tourists often cannot allocate enough time to select and adjust their photos, nor can they easily retake them, making it difficult to obtain satisfactory pictures. Furthermore, for many scenic spots, conventional photography methods are insufficient to capture a wide field of view; currently, panoramic shooting or wide-angle cameras are often used.

[0003] Panoramic photography: Panoramic photography typically uses the built-in function of a camera or mobile phone to automatically stitch together multiple photos to generate a panoramic image. While this method can capture a wider field of view, it often encounters problems such as stitching gaps, unnatural transitions, and image distortion when dealing with complex scenes, affecting the final visual effect.

[0004] Wide-angle cameras: Wide-angle cameras can capture a wider field of view, but due to the optical characteristics of wide-angle lenses, they are prone to image distortion, especially at the edges of the image, resulting in poor visual quality. Furthermore, wide-angle cameras often lack detail when shooting distant scenes. Summary of the Invention

[0005] In view of this, the present invention proposes an image processing method based on a large AI model. By using feature point extraction and matching technology of a deep learning model, high-precision registration of multiple images is achieved. Furthermore, an image fusion model is used to generate seamless panoramic images. After intelligently identifying and filling in blank areas in the panoramic image, high-quality and natural panoramic images can be generated, thereby improving the visual effect and viewing experience of traveler photos.

[0006] The technical solution of this invention is implemented as follows:

[0007] This invention provides an image processing method based on a large AI model, comprising the following steps:

[0008] S1: Collect multiple images and perform preprocessing;

[0009] S2: Construct a deep learning model, extract feature points from multiple images using the deep learning model, and use a feature matching algorithm to match the feature points of multiple images to obtain the first matching result. Use a filtering algorithm to screen the first matching pair for mismatches to obtain the second matching result. Apply affine transformation to the second matching result to complete the registration of multiple images.

[0010] S3: Use the image fusion model to fuse the registered multiple images to generate a panoramic image;

[0011] S4: Identify the blank areas in the panoramic image, and complete the blank areas based on the completion method to obtain the final panoramic image.

[0012] Based on the above solution, preferably, step S2 includes:

[0013] S21: Use the VGG network to construct a deep learning model, and input the multiple images into the deep learning model in sequence to extract SIFT feature points;

[0014] S22: Pair the multiple images in pairs to form image groups;

[0015] S23: Select an image group;

[0016] S24: Denote the SIFT feature points of image I1 and image I2 in the image group as the first feature point set {S1} and the second feature point set {S2} respectively, calculate the correlation coefficient between each SIFT feature point in {S1} and all SIFT feature points in {S2}, and obtain the candidate second feature point set for each SIFT feature point in {S1} based on the value of the correlation coefficient, that is, the first matching result;

[0017] S25: Select a first feature point S1 in the first matching result i and its candidate second feature point set

[0018] S26: Sort the candidate second feature point set in descending order according to the value of the correlation coefficient and select the first two candidate second feature points Calculate the distances d1 and d2 between S1 i and respectively. If d1 / d2 < T1, then use the candidate second feature point as the matching point of S1 i and add it to the second matching result;

[0019] S27: Repeat steps S25 - S26 until the first matching result is screened out, and obtain the second matching result of this image group;

[0020] S28: Construct an affine transformation model based on the second matching result, use the affine transformation model to register image I2 with image I1 as the reference image;

[0021] S29: Repeat steps S23 - S28 until all image groups are processed to complete image registration.

[0022] Based on the above scheme, preferably, step S24 includes:

[0023] S241: The SIFT feature points of images I1 and I2 in the image group are respectively denoted as the first feature point set {S1} and the second feature point set {S2}. Extract the descriptor of each feature point to form the first descriptor set {D1} and the second descriptor set {D2}.

[0024] S242: Perform local hashing on the first descriptor set {D1} and the second descriptor set {D2} respectively, and map the first descriptor set {D1} and the second descriptor set {D2} to multiple hash buckets through local hashing;

[0025] S243: Perform preliminary matching within the hash buckets. For each hash bucket, find the feature point pairs to obtain the preliminary matching result {D1}. k D2 m}, where D1 k Represents the descriptors in {D1}, D2 m This refers to the descriptor in {D2};

[0026] S244: Calculate the similarity coefficient of each pair of feature points in the preliminary matching results using the similarity formula. The formula for calculating the correlation coefficient is as follows:

[0027]

[0028] In the formula, R(D1) k D2 m ) is D1 k With D2 m Correlation coefficient, f1 j D1 k The j-th component, f2 j D2 m The j-th component, n is the dimension of the descriptor, ω j Let be the weight of the j-th dimension;

[0029] S245: Set a correlation threshold τ, compare the correlation coefficient with the correlation threshold τ, and filter out the feature point pairs corresponding to the correlation coefficients that are greater than or equal to the correlation threshold τ as the first matching result.

[0030] Based on the above scheme, preferably, step S3 includes:

[0031] Construct an image fusion model, including a first network branch and a second network branch;

[0032] First, select two images from the registered images and input them into the image fusion model to fuse the images and obtain a fused image. Then, input a single image and the fused image in sequence and fuse them again until the registered images are fused to obtain a panoramic image.

[0033] in:

[0034] The first network branch processes the two images to obtain a preliminary reconstructed stitched image. The first network branch adopts an encoder-decoder network. The encoder includes a first network layer, a second network layer, and a third network layer. Each network layer includes one convolutional layer and one pooling layer. The encoder is used to extract low-level features. The decoder includes a first deconvolutional layer, a second deconvolutional layer, and a third deconvolutional layer. The decoder is used to generate high-level features. Skip connections are set between the network layers of the encoder and the deconvolutional layers of the decoder to connect the low-level features and the high-level features.

[0035] The second network branch is used to process the initially reconstructed stitched image to obtain the fused image. The second network branch includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a residual unit. The residual unit includes four residual blocks. Each residual block has the same structure, including convolutional layer 1, ReLU activation function 1, convolutional layer 2, residual connection, and ReLU activation function 2.

[0036] Based on the above scheme, the preferred process for the first network branch to process the two images to obtain the preliminarily reconstructed stitched image is as follows:

[0037] Two images, I1 and I2, are input into the first network branch. The original size of the two images is H*W. The two images are downsampled and their size is converted to 256*256.

[0038] The low-level features of I1 and I2 are extracted sequentially using the first, second, and third network layers of the encoder, resulting in three low-level features of I1 at different scales. and and low-level features at three different scales of I2 and

[0039] Will and splicing and splicing and The features are then concatenated to obtain the first fusion feature F1, the second fusion feature F2, and the third fusion feature F3.

[0040] The first fusion feature F1, the second fusion feature F2 and the third fusion feature F3 are input into the decoder. The third fusion feature F3 is upsampled using the first deconvolution layer so that its size is the same as that of the second fusion feature F2. The upsampled third fusion feature F3 and the second fusion feature F2 are summed through a skip connection to obtain the first updated fusion feature F′1.

[0041] The first updated fusion feature F′1 is upsampled using the second deconvolution layer, making its size the same as the first fusion feature F1. The upsampled first updated fusion feature F′1 and the first fusion feature F1 are then summed via a skip connection to obtain the second updated fusion feature F′2.

[0042] The second updated fused feature F′2 is upsampled using the third deconvolution layer to restore its size to 256*256, thus obtaining the final fused feature.

[0043] A 1*1 convolutional layer is used to map the final fused features to the target number of channels, resulting in a preliminary reconstructed stitched image.

[0044] Based on the above scheme, preferably, the second network branch processes the initially reconstructed stitched image to obtain the fused image as follows:

[0045] The initially reconstructed stitched image is upsampled to its original size H*W using bilinear interpolation;

[0046] The first, second, and third convolutional layers are used to extract features from the upsampled stitched image in sequence to obtain the first layer feature A1, the second layer feature A2, and the third layer feature A3;

[0047] The third-layer feature A3 is input into the residual unit for processing. The first residual block is used to process the third-layer feature A3 to obtain the first residual feature. The first residual feature is input into the second residual block to obtain the second residual feature. The second residual feature is input into the third residual block to obtain the third residual feature. The third residual feature is input into the fourth residual block to obtain the final residual feature.

[0048] The first layer feature A1 is fused with the final residual feature to obtain the fused feature;

[0049] The fused features are convolutionally processed to output the fused image.

[0050] Based on the above scheme, preferably, step S4 includes:

[0051] S41: Use the U-Net segmentation model to segment the panoramic image to obtain blank areas;

[0052] S42: Obtain the location information of the blank area and search the network using a search engine based on the location information;

[0053] S43: If a candidate image is found, the first completion method is used to complete the blank area based on the candidate image; if no candidate image is found, the second completion method is used to complete the blank area.

[0054] S44: Output the final panoramic image.

[0055] Based on the above scheme, preferably, step S42 includes:

[0056] The contour of the blank area is extracted by an edge detection algorithm, and the contour information is optimized by morphological operations to obtain the shape features of the blank area.

[0057] The relative positions of blank areas in the panoramic image are marked using regular grids to obtain positional features;

[0058] Starting from the outline of the blank area, the outer contour is obtained by expanding it outward according to a preset step size. The feature information within the outer contour is then extracted as the environmental features of the blank area.

[0059] The basic features of the blank area are composed of shape features, location features, and environmental features;

[0060] A coarse search is performed in the search engine using basic features to obtain initial candidate images;

[0061] Features are extracted from the initial candidate images to obtain candidate features;

[0062] The basic features of the blank area are compared with the candidate features in the candidate feature set one by one to calculate the similarity, and the most similar image is selected as the candidate image.

[0063] Based on the above scheme, the preferred first completion method includes:

[0064] The candidate images are cropped and deformed to match the shape of the blank area;

[0065] The region of the outer contour is used as a buffer. The color histogram of the candidate image and the buffer is calculated. The color distribution of the candidate image is adjusted according to the histogram matching algorithm.

[0066] The local color difference correction algorithm is used to fine-tune the color histogram matching results to obtain candidate images after color matching.

[0067] Create a mask for the blank area and apply Gaussian blur to the edges of the mask to generate a feathered mask with a gradient effect.

[0068] Define the center of the blank area as the embedding center, use the cv2.seamlessClone function, combined with a feather mask, to perform Poisson blending and generate a preliminary fused image;

[0069] The initial fused image is downsampled using Gaussian pyramids to construct a multi-resolution hierarchy. At each resolution hierarchy, the panoramic image and the initial fused image are fused layer by layer to generate the fusion result of each layer. The multi-resolution hierarchy is then upsampled layer by layer and superimposed to reconstruct the final fused image from the bottom layer to obtain the final panoramic image.

[0070] Based on the above scheme, the preferred second completion method includes:

[0071] In panoramic images, extract the effective non-blank areas surrounding blank areas as reference areas;

[0072] Multiple spatial transformations are performed on the reference region to generate image patches from different perspectives. The structural similarity is used to select the most suitable perspective from the image patches from different perspectives to obtain the optimal perspective image patch.

[0073] Load the pre-trained Context Encoder model, input a panoramic image and an optimal viewpoint image patch, the model fills in the blank areas in the panoramic image based on the context information of the optimal viewpoint image patch, and outputs the completed panoramic image.

[0074] Gaussian blur is used to remove edge effects from the completed panoramic image, resulting in the final panoramic image.

[0075] The system of the present invention has the following advantages over the prior art:

[0076] 1) This invention achieves high-precision image registration through a deep learning model, achieves natural image fusion through multi-layer convolution and residual networks, and solves the problem of handling blank areas in panoramic images through intelligent completion technology, ultimately generating highly consistent panoramic images with excellent visual effects;

[0077] 2) This invention uses a VGG network to construct a deep learning model. By extracting SIFT feature points and performing feature matching, combined with a mismatch screening algorithm, the accuracy of feature point matching is improved. Furthermore, affine transformation is used to accurately register multiple images, effectively eliminating the problems of image misalignment and blurring in traditional methods.

[0078] 3) By constructing an image fusion model that includes an encoder-decoder network and a residual network, image features are extracted and fused layer by layer, achieving a natural transition between images. The skip connection mechanism and multi-scale feature fusion strategy ensure visual coherence and detailed restoration of the images, resulting in natural transitions and no stitching artifacts in the generated panoramic images. Specifically, the first network branch performs encoding and decoding operations on the images to generate a preliminary reconstructed stitched image, while the second network branch optimizes the detail levels of the stitched image through residual units, improving the precision and realism of the final panoramic image.

[0079] 4) In the scheme adopted in this invention, the first network branch fuses low-level features at different scales layer by layer through multi-layer convolution and deconvolution operations to generate a preliminary reconstructed stitched image. This effectively extracts multi-scale information from the image, ensuring efficient capture of detailed features in image stitching, improving video visual quality, and giving the generated stitched image better consistency and clarity. The application of the skip connection mechanism combines the low-level features of the encoder with the high-level features of the decoder, greatly enhancing the connection at the feature level, making the reconstructed stitched image highly consistent across multiple scales, and further improving the image fusion effect.

[0080] 5) This invention utilizes the U-Net segmentation model to accurately identify blank areas in panoramic images and, based on the location information of these blank areas, searches the internet for similar images using a search engine. This step combines edge detection and morphological operations, improving the accuracy and reliability of blank area identification. First, a first completion method based on candidate images achieves high-quality completion of blank areas. Specific techniques include color histogram matching, local color difference correction, and Poisson blending, ensuring seamless integration of color and texture. Second, when no suitable candidate images are found, a second completion method based on the Context Encoder model is employed. By analyzing and utilizing contextual information, intelligent completion in complex scenes is achieved, ensuring the visual consistency and naturalness of the final panoramic image. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a flowchart of the method of the present invention;

[0083] Figure 2This is a schematic diagram of the structure of the first network branch of the present invention;

[0084] Figure 3 This is a schematic diagram of the structure of the second network branch of the present invention. Detailed Implementation

[0085] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0086] Please see Figure 1 This invention provides an image processing method based on a large AI model, comprising the following steps:

[0087] S1: Collect multiple images and perform preprocessing;

[0088] S2: Construct a deep learning model, extract feature points from multiple images using the deep learning model, and use a feature matching algorithm to match the feature points of multiple images to obtain the first matching result. Use a filtering algorithm to screen the first matching pair for mismatches to obtain the second matching result. Apply affine transformation to the second matching result to complete the registration of multiple images.

[0089] S3: The image fusion model is used to fuse the registered images to generate a panoramic image;

[0090] S4: Identify blank areas in the panoramic image, and fill in the blank areas using a completion method to obtain the final panoramic image.

[0091] Specifically, first, multiple photos of the same scenic spot taken by the user using a mobile device are collected, and these images are preprocessed, including noise reduction, brightness and contrast adjustment.

[0092] Specifically, in one embodiment of the present invention, step S2 includes:

[0093] S21: A deep learning model is built using a VGG network. Multiple images are sequentially input into the deep learning model to extract SIFT feature points.

[0094] S22: Pair multiple images together to form an image group;

[0095] S23: Select a group of pictures;

[0096] S24: Denote the SIFT feature points of picture I1 and picture I2 in the picture group as the first feature point set {S1} and the second feature point set {S2} respectively. Calculate the correlation coefficient between each SIFT feature point in {S1} and all SIFT feature points in {S2}, and obtain the candidate second feature point set for each SIFT feature point in {S1} based on the value of the correlation coefficient, that is, the first matching result.

[0097] S25: Select a first feature point S1 in the first matching result i and its candidate second feature point set

[0098] S26: Sort the candidate second feature point set in descending order according to the value of the correlation coefficient, and select the first two candidate second feature points Calculate the distances d1 and d2 between S1 i and respectively. If d1 / d2 < T1, then take the candidate second feature point as the matching point of S1 i and add it to the second matching result.

[0099] S27: Repeat steps S25 - S26 until the first matching result is screened out completely to obtain the second matching result of the picture group.

[0100] S28: Construct an affine transformation model based on the second matching result. Using picture I1 as the reference picture, register picture I2 with the affine transformation model.

[0101] S29: Repeat steps S23 - S28 until all picture groups are processed to complete picture registration.

[0102] Take a specific example to illustrate:

[0103] VGG-16 was chosen as the base model. A pre-trained VGG model was loaded into the deep learning framework to obtain the deep learning model. According to the model requirements, multiple input images were adjusted, including resizing and normalization. The adjusted images were then arranged according to their RGB channels and input into the model to extract feature maps. In this embodiment of VGG-16, the network layers include block3_conv3, block4_conv3, and block5_conv3. The output of block5_conv3 was truncated as the output feature map. This feature map is multi-channel and contains deep convolutional features. Each channel of the feature map can be seen as the representation of the image in different feature dimensions. SIFT identifies salient keypoints on the feature map, ensuring these keypoints are scale- and rotation-invariant. It employs a multi-scale Difference of Gaussian (DoG) pyramid to detect salient points, i.e., keypoints. The feature map is filtered using a multi-scale Gaussian function, and the filtered result serves as the scale-space representation of each point. Local extrema are identified at each scale to determine their precise location and scale. These local extrema are potential SIFT keypoints. A principal orientation is assigned to each keypoint. By calculating the gradient histogram of the keypoint's neighborhood, the standard orientation of the gradient direction and magnitude is determined, ensuring the orientation invariance of the feature. Within the keypoint's neighborhood, the distribution of directional gradients is calculated, generating a 128-dimensional feature vector (descriptor) containing orientation and magnitude. These descriptors describe the local image information of the keypoint. Each extracted keypoint and its corresponding descriptor form a feature point set, i.e., SIFT feature points.

[0104] Multiple images are randomly combined to form multiple image pairs, each pair being an image group. For example, images I1, I2, and I3 can form the image pair (I1, I2)(I1, I3)(I2, I3).

[0105] From the image group, select an image pair to process, for example, select image pair (I1, I2).

[0106] The SIFT feature points of images I1 and I2 in the image group are denoted as the first feature point set {S1} and the second feature point set {S2}, respectively.

[0107] Feature point set: Let {S1} = {S11,S12,...,S1} n} and {S2} = {S21,S22,...,S2 v}

[0108] Calculate the correlation coefficient between each feature point in {S1} and all feature points in {S2}. Based on the correlation coefficient values, obtain the candidate second feature point set for each feature point in {S1}, i.e., the first matching result. This process specifically includes:

[0109] S241: Denote the SIFT feature points of images I1 and I2 in the image group as the first feature point set {S1} and the second feature point set {S2}, respectively. Extract the descriptor for each feature point to form the first descriptor set {D1} and the second descriptor set {D2}. Specifically, each feature pair has a corresponding descriptor, where {S1} = {S11, S12, ..., S1...} n}, each S1 i Corresponding to a descriptor D1 i {S2} = {S21,S22,...,S2} v}, each S2 i Corresponding to a descriptor D2 i .

[0110] S242: Perform local hashing on the first descriptor set {D1} and the second descriptor set {D2} respectively, mapping the first descriptor set {D1} and the second descriptor set {D2} to multiple hash buckets through local hashing; specifically, select Local Sensitive Hash (LSH) to map high-dimensional descriptors to low-dimensional hash buckets. Each descriptor is assigned to one or more hash buckets according to the hash function.

[0111] S243: Perform preliminary matching within the hash buckets. For each hash bucket, find the feature point pairs to obtain the preliminary matching result {D1}. k D2 m}, where D1 k Represents the descriptors in {D1}, D2 m Let {D1} represent the descriptors in {D2}; specifically, within each hash bucket, search for possible descriptor pairs, and consider descriptor pairs within the same hash bucket as preliminary matching candidates. The preliminary matching result is represented as {D1}. k D2 m}, where D1 k and D2 m They come from {D1} and {D2} respectively.

[0112] S244: Calculate the similarity coefficient of each pair of feature points in the preliminary matching results using the similarity formula. The formula for calculating the correlation coefficient is as follows:

[0113]

[0114] In the formula, R(D1) k D2 m ) is D1 k With D2 m Correlation coefficient, f1 j D1 k The j-th component, f2 j D2m The j-th component, where n is the dimension of the descriptor, and ω j is the weight of the j-th dimension. In this embodiment, the range of R(D1 k , D2 m ) is [0, 1]. If D1 k and D2 m are more similar, then the value of R(D1 k , D2 m ) is larger, that is, closer to 1.

[0115] S245: Set the relevant threshold τ, compare the value of the correlation coefficient with the relevant threshold τ, and screen out the feature point pairs corresponding to the correlation coefficients greater than or equal to the relevant threshold τ as the first matching result.

[0116] After that, from the first matching result, select a first feature point and its candidate second feature point set in sequence for screening:

[0117] Sort in descending order according to the correlation coefficient, and select the top two candidate points Calculate the distances d1 and d2 between S1 i and respectively. At this time, the distance can be the Euclidean distance. The smaller the distance, the more similar the points are. Calculate the ratio of d1 to d2. If d1 / d2 < T1, then the candidate second feature point is used as the matching point of S1 i and added to the second matching result.

[0118] In this embodiment, first, through the calculation and sorting of the correlation coefficient, two matching points with the highest similarity are selected. This step is to find the most likely correct matching points After that, by comparing the distances between S1 i and , if is the correct matching point, there should be a significant gap between d1 and d2, and d1 is smaller and d2 is larger. By setting a threshold T1, specifically, T1 can be set to 0.6, to determine whether the distance ratio is less than the threshold T1. If the distance ratio is lower than the threshold T1, it indicates that is significantly better than , which means is more reliable. If the distance ratio is greater than or equal to T1, it means that has little difference, that is, the matching result is not significant enough. At this time, it is determined as an incorrect matching.

[0119] Next, the matching points are repeatedly filtered to obtain the second matching result of the image group. Based on the second matching result, an affine transformation model is constructed. Taking image I1 as the reference, the affine transformation model is applied to transform the coordinates of all pixels in image I2, thereby achieving image registration.

[0120] Finally, repeat the above steps to complete the registration of all images.

[0121] Specifically, in one embodiment of the present invention, step S3 includes:

[0122] Construct an image fusion model, including a first network branch and a second network branch;

[0123] First, select two images from the registered images and input them into the image fusion model to fuse the images and obtain a fused image. Then, input a single image and the fused image in sequence and fuse them again until the registered images are fused to obtain a panoramic image.

[0124] in:

[0125] like Figure 2 As shown, the first network branch processes two images to obtain a preliminarily reconstructed stitched image. The first network branch adopts an encoder-decoder network. The encoder includes a first network layer, a second network layer, and a third network layer. Each network layer includes one convolutional layer and one pooling layer. The encoder is used to extract low-level features. Specifically, in the first network layer, the convolutional layer conv1 has 64 filters, a kernel size of 3*3, a stride of 1, and padding of 1. The pooling layer pool1 is a 2*2 max pooling with a stride of 2. In the second network layer, the convolutional layer conv1 has 128 filters, a kernel size of 3*3, a stride of 1, and padding of 1. The pooling layer pool2 is a 2*2 max pooling with a stride of 2. In the third network layer, the convolutional layer conv3 has 256 filters, a kernel size of 3*3, a stride of 1, and padding of 1. The pooling layer pool3 is a 2*2 max pooling with a stride of 2.

[0126] The decoder includes a first deconvolutional layer, a second deconvolutional layer, and a third deconvolutional layer, and is used to generate high-level features;

[0127] Specifically, the first deconvolutional layer has 128 filters, a kernel size of 3*3, and a stride of 2, resulting in a halved output image size; the second deconvolutional layer has 64 filters, a kernel size of 3*3, and a stride of 2, resulting in a halved output image size; the third deconvolutional layer has 3 filters, a kernel size of 3*3, and a stride of 2, restoring the output image to 256*256.

[0128] To preserve multi-scale information, skip connections are set between the network layers of the encoder and the deconvolutional layers of the decoder to connect low-level features and high-level features. Skip connections connect directly from the convolutional layers of the encoder to the deconvolutional layers of the decoder at the corresponding scale. In this way, high-resolution feature information will be preserved and combined with the high-level features of the decoder.

[0129] like Figure 3 As shown, the second network branch is used to process the initially reconstructed stitched image to obtain the fused image. The second network branch includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a residual unit. The residual unit includes four residual blocks. Each residual block has the same structure, including convolutional layer 1, ReLU activation function 1, convolutional layer 2, residual connection, and ReLU activation function 2.

[0130] Specifically, the first convolutional layer has 64 filters, a 3x3 kernel size, a stride of 1, and padding of 1. The second and third convolutional layers have the same structure as the first convolutional layer. In each residual block, convolutional layer 1 has 64 filters, a 3x3 kernel size, a stride of 1, and padding of 1; convolutional layer 2 has the same structure as convolutional layer 1, and the residual connection adds the input features to the output of the previous convolution.

[0131] The first network branch processes the two images to obtain a preliminary reconstructed stitched image. The processing steps are as follows:

[0132] Two images, I1 and I2, are input into the first network branch. The original size of the two images is H*W. The two images are downsampled and converted to 256*256 to reduce computational complexity and capture large-scale stitching features.

[0133] The low-level features of I1 and I2 are extracted sequentially using the first, second, and third network layers of the encoder, resulting in three low-level features of I1 at different scales. and and low-level features at three different scales of I2 and

[0134] Will and splicing and splicing and The features are then concatenated to obtain the first fusion feature F1, the second fusion feature F2, and the third fusion feature F3. Specifically, the concatenation is performed along the feature channel dimension.

[0135] The first fusion feature F1, the second fusion feature F2 and the third fusion feature F3 are input into the decoder. The third fusion feature F3 is upsampled using the first deconvolution layer so that its size is the same as that of the second fusion feature F2. The upsampled third fusion feature F3 and the second fusion feature F2 are summed through a skip connection to obtain the first updated fusion feature F′1.

[0136] The first updated fusion feature F′1 is upsampled using the second deconvolution layer, making its size the same as the first fusion feature F1. The upsampled first updated fusion feature F′1 and the first fusion feature F1 are then summed via a skip connection to obtain the second updated fusion feature F′2.

[0137] The second updated fused feature F′2 is upsampled using the third deconvolution layer to restore its size to 256*256, thus obtaining the final fused feature.

[0138] A 1*1 convolutional layer is used to map the final fused features to the target number of channels, resulting in a preliminary reconstructed stitched image.

[0139] Specifically, deconvolution starts from the third fusion feature, gradually upsamples and restores the resolution, and combines skip connections to enhance the details, finally obtaining a preliminary stitched image with a size of 256*256.

[0140] The second network branch processes the initially reconstructed stitched image to obtain the fused image. The processing steps are as follows:

[0141] The initially reconstructed stitched image is upsampled to its original size H*W using bilinear interpolation;

[0142] The first, second, and third convolutional layers are used to extract features from the upsampled stitched image in sequence to obtain the first layer feature A1, the second layer feature A2, and the third layer feature A3;

[0143] The third-layer feature A3 is input into the residual unit for processing. The first residual block is used to process the third-layer feature A3 to obtain the first residual feature. The first residual feature is input into the second residual block to obtain the second residual feature. The second residual feature is input into the third residual block to obtain the third residual feature. The third residual feature is input into the fourth residual block to obtain the final residual feature.

[0144] The first layer feature A1 is fused with the final residual feature to obtain the fused feature;

[0145] The fused features are convolutionally processed to output the fused image.

[0146] Specifically, the stitched image is first upsampled to its original size, for example, 1024*1024. Then, features are extracted using the first, second, and third convolutional layers. At this point, the feature sizes output by the three convolutional layers are the same. Residual blocks are then introduced to enhance the network's feature learning ability and avoid gradient vanishing. This embodiment uses four residual blocks, but in other embodiments, six or eight residual blocks can be used, depending on the situation. The third-layer features output from the third convolutional layer are input into the residual units. The four residual blocks are used to effectively refine and supplement the feature information sequentially, resulting in the final residual features. Then, the first-layer feature A1 is fused with the final residual features. At this point, the final residual features are deep features, while the first-layer feature A1 is a shallow feature. By fusing deep and shallow features, the network can effectively utilize rich multi-scale information in the final output, improving the quality of the fused image.

[0147] In this embodiment, after constructing the image fusion model, pre-training is required before processing multiple images. During pre-training, a publicly available dataset is used, the optimizer is Adam, and the initial learning rate is 10. -4 The training loss function is as follows:

[0148] L=ω1L1+ω2L2+ω3L3

[0149]

[0150] In the formula, L is the total loss, L1 is the loss of the first network branch, L2 is the loss of the second network branch, and L3 is the balancing loss of the two network branches. This indicates the loss of image content in the first network branch. G1 represents the image stitching seam loss in the first network branch, and M represents the initially reconstructed stitched image. 1 and M 2 These are the content masks for the two input images, I1 and I2 representing the original two input images. Indicates perceived loss. M represents L1 regularized loss. 1S and M 2S They represent M respectively 1 and M 2 Perform convolution and crop to a mask where all pixel values ​​are between 0 and 1. This indicates the loss of image content in the second network branch. This represents the image stitching seam loss in the second network branch. and The calculation process and and similar, This indicates that the merged image G2 will be adjusted to 256×256 to match the size of G1. ||·||1 indicates l1 normalization, ω1, ω2, ω3, Each represents the corresponding weight.

[0151] Specifically, in one embodiment of the present invention, step S4 includes:

[0152] S41: Use the U-Net segmentation model to segment the panoramic image to obtain blank areas;

[0153] S42: Obtain the location information of the blank area and search the network using a search engine based on the location information;

[0154] S43: If a candidate image is found, the first completion method is used to complete the blank area based on the candidate image; if no candidate image is found, the second completion method is used to complete the blank area.

[0155] S44: Output the final panoramic image.

[0156] In this embodiment, step S41 includes:

[0157] The panoramic image is input into the trained U-Net model to obtain a binarized prediction mask, which marks the blank areas. The U-Net model consists of an encoder and a decoder, forming a symmetrical U-shaped structure.

[0158] The predicted mask is converted into a de facto binary image. Based on the binarized mask, blank areas in the panoramic image are extracted, and a binary image containing the location information of these blank areas is generated.

[0159] In this embodiment, step S42 includes:

[0160] S421: Extract the contour of the blank area using an edge detection algorithm, and optimize the contour information using morphological operations to obtain the shape features of the blank area.

[0161] Specifically, the Canny edge detection algorithm is applied to extract edges in blank areas. The Canny algorithm detects edges by calculating image gradients, mainly including the following steps: Gaussian filtering: First, a Gaussian filter is used to smooth the image and remove noise. Gradient calculation: The gradient intensity and direction of each pixel in the image are calculated. Non-maximum suppression: Edges are thinned, and non-edge points are suppressed. Hysteresis thresholding: The final edges are detected using a double thresholding method.

[0162] Use dilation and erosion operations to refine edge contours. Morphological operations can enhance the continuity and clarity of contours. Dilation: Increases the object's boundary, filling small holes in the edges. Erosion: Shrinks the object's boundary, removing noise points.

[0163] S422: Use a regular grid to mark the relative position of blank areas in the panoramic image to obtain positional features.

[0164] The panoramic image is divided into fixed-size grid cells, and the positions of blank areas on the grid are recorded. The grid size can be adjusted as needed. The positional feature network of the blank areas is output as feature information.

[0165] S423: Using the outline of the blank area as the initial point, expand outward according to the preset step size to obtain the outer contour, and extract the feature information within the outer contour as the environmental features of the blank area.

[0166] Expand the outline of the blank area outwards by a certain step size, and extract image features including the surrounding blank area. For example, set the step size to 10, use a dilation operation to expand the outline, and extract feature information within the expanded outline as environmental features.

[0167] S424: The basic features that make up the blank area by combining shape features, location features and environmental features.

[0168] S425: Use basic features to perform a coarse search in the search engine to obtain initial candidate images.

[0169] Using the aforementioned basic features, a coarse search is performed in the search engine via the API to return initial candidate images. The feature vectors need to be converted to JSON format before being fed into the search engine API.

[0170] S426: Extract features from the initial candidate images to obtain candidate features.

[0171] For each candidate image, extract similar types of features, including edge detection results, location features, and environmental features.

[0172] S427: Calculate the similarity between the basic features of the blank area and the candidate features in the candidate feature set, and select the most similar image as the candidate image.

[0173] Cosine similarity is used to calculate the similarity between the features of each candidate image and the basic features. The candidate image with the highest similarity is selected.

[0174] In this embodiment, the first completion method includes:

[0175] The candidate images are cropped and deformed to match the shape of the blank area.

[0176] Use an image cropping tool, such as the `cv2.getRectSubPix` function in OpenCV, to select a portion that is similar in shape and size to the blank area for cropping. It's important to ensure that the cropped image covers as much of the content as possible to minimize information loss.

[0177] Next, an affine transformation is performed to match the shape of the cropped candidate image to the blank area. The functions `cv2.getAffineTransform` and `cv2.warpAffine` can be used. For larger blank areas, a perspective transformation is applied, using the function `cv2.getPerspectiveTransform`.

[0178] The region of the outer contour is used as a buffer. The color histogram of the candidate image and the buffer is calculated, and the color distribution of the candidate image is adjusted according to the histogram matching algorithm.

[0179] Based on the outline of the blank area, extend outward by a certain number of pixels to generate a buffer. Extract the color information within the buffer and calculate its color histogram, which can be done using the cv2.calcHist function.

[0180] By using a color histogram matching algorithm, such as cv2.calcBackProject in OpenCV, the color distribution of the candidate images is adjusted to closely approximate the colors in the buffer. This reshapes the candidate images to suit the overall style of the panoramic image.

[0181] The local color difference correction algorithm is used to fine-tune the color histogram matching results to obtain candidate images after color matching.

[0182] By combining Gaussian blur and the Laplacian operator, local color adjustments are performed. The color of the candidate image is fine-tuned pixel by pixel by comparing the local mean and variance, so that it has no color difference with the panoramic image in local areas.

[0183] Create a mask for the blank area and apply Gaussian blur to the edges of the mask to generate a feathered mask with a gradient effect.

[0184] Generate a binary mask based on the outline of the blank area. You can use `cv2.fillPoly` to fill the polygon, filling the blank area with white and the rest with black. Apply a Gaussian blur to the mask edges to generate a feathered mask with a gradient effect.

[0185] Define the center of the blank area as the embedding center, use the cv2.seamlessClone function, combined with a feather mask, to perform Poisson blending and generate a preliminary fused image.

[0186] The `cv2.seamlessClone` function is used for seamless cloning, seamlessly embedding the color-matched candidate image into the blank areas of the panoramic image. Poisson Blending achieves seamless fusion by solving the Poisson equation, ensuring that the edges of the candidate image smoothly transition into the panoramic image.

[0187] The initial fused image is downsampled using Gaussian pyramids to construct a multi-resolution hierarchy. At each resolution hierarchy, the panoramic image and the initial fused image are fused layer by layer to generate the fusion result of each layer. The multi-resolution hierarchy is then upsampled layer by layer and superimposed to reconstruct the final fused image from the bottom layer to obtain the final panoramic image.

[0188] The initial fused image is subjected to multiple Gaussian downsampling operations to construct a pyramid hierarchy with multiple resolutions. Gaussian downsampling achieves multi-resolution image representation by recursively applying low-pass filtering and downsampling to the image. Each resolution is fused layer by layer from the bottom layer. The fused results of the lower layers are then upsampled and stacked layer by layer to higher resolution levels, ultimately reconstructing a high-resolution panoramic image layer by layer.

[0189] Specifically, the first completion method covers the entire process from cropping, deformation, color matching, fusion processing to final multi-resolution image fusion. This method ensures that candidate images can be embedded into the blank areas of panoramic images with high quality, achieving natural transitions and visual consistency.

[0190] In this embodiment, the second completion method includes:

[0191] In panoramic images, extract the valid non-blank areas surrounding blank areas as reference areas.

[0192] Specifically, the edges of blank areas are identified, and the area extending outwards along the edges by a certain distance is extracted to extract the surrounding area containing valid information.

[0193] Multiple spatial transformations are performed on the reference region to generate image patches from different perspectives. The most suitable perspective is selected from the image patches from different perspectives using structural similarity to obtain the optimal perspective image patch.

[0194] Multiple spatial transformations (such as rotation, scaling, and affine transformations) are performed to generate image patches from different viewpoints, which are then compared and suitable candidate patches are selected. Specifically, rotation transformations are performed using `cv2.getRotationMatrix2D` and `cv2.warpAffine`. Scaling transformations are performed using `cv2.resize`. Affine transformations are performed using `cv2.getAffineTransform` and `cv2.warpAffine`.

[0195] The structural similarity between each viewpoint image patch and the adjacent regions of the panoramic image is calculated using skimage.metrics.structural_similarity, and the image patch with the highest structural similarity is selected as the best viewpoint image patch.

[0196] Load the pre-trained Context Encoder model, input a panoramic image and an optimal viewpoint image patch, the model fills in the blank areas in the panoramic image based on the context information of the optimal viewpoint image patch, and outputs the completed panoramic image.

[0197] Preprocess the panoramic image and optimal viewpoint image patches, then fill in the blank areas of the panoramic image with these patches. Load the pre-trained model using PyTorch's `torch.load` function. Call the model's `eval` method to switch the model to evaluation mode.

[0198] Fill the blank areas of the panoramic image with the best-viewpoint image patch. Use a mask to locate the blank areas and perform local replacement so that the blank areas are filled with image patches.

[0199] The image is input into a pre-trained Context Encoder model for inference computation and output. Gradient computation is disabled during inference to improve efficiency and save memory.

[0200] Gaussian blur is used to remove edge effects from the completed panoramic image, resulting in the final panoramic image.

[0201] Use the cv2.GaussianBlur function in OpenCV to make the completed part transition smoothly with the original image and reduce edge effects.

[0202] Specifically, the second completion method can achieve high-quality, natural image filling effects, enabling the candidate image and the panoramic image to form a visually consistent and seamless fusion.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image processing method based on a large AI model, characterized in that, Includes the following steps: S1: Collect multiple images and perform preprocessing; S2: Construct a deep learning model, extract feature points from multiple images using the deep learning model, and use a feature matching algorithm to match the feature points of multiple images to obtain the first matching result. Use a filtering algorithm to screen the first matching pair for mismatches to obtain the second matching result. Apply affine transformation to the second matching result to complete the registration of multiple images. S3: The image fusion model is used to fuse the registered images to generate a panoramic image; S4: Identify blank areas in the panoramic image, and fill in the blank areas using a completion method to obtain the final panoramic image; Step S4 includes: S41: Use the U-Net segmentation model to segment the panoramic image to obtain blank areas; S42: Obtain the location information of the blank area and search the network using a search engine based on the location information; S43: If a candidate image is found, the first completion method is used to complete the blank area based on the candidate image; if no candidate image is found, the second completion method is used to complete the blank area. S44: Output the final panoramic image; Step S42 includes: The contour of the blank area is extracted by an edge detection algorithm, and the contour information is optimized by morphological operations to obtain the shape features of the blank area. The relative positions of blank areas in the panoramic image are marked using regular grids to obtain positional features; Starting from the outline of the blank area, the outer contour is obtained by expanding it outward according to a preset step size. The feature information within the outer contour is then extracted as the environmental features of the blank area. The basic features of the blank area are composed of shape features, location features, and environmental features; A coarse search is performed in the search engine using basic features to obtain initial candidate images; Features are extracted from the initial candidate images to obtain candidate features; The basic features of the blank area are compared with the candidate features in the candidate feature set one by one to calculate the similarity, and the most similar image is selected as the candidate image. The first method of completion includes: The candidate images are cropped and deformed to match the shape of the blank area; The region of the outer contour is used as a buffer. The color histogram of the candidate image and the buffer is calculated. The color distribution of the candidate image is adjusted according to the histogram matching algorithm. The local color difference correction algorithm is used to fine-tune the color histogram matching results to obtain candidate images after color matching. Create a mask for the blank area and apply Gaussian blur to the edges of the mask to generate a feathered mask with a gradient effect. Define the center of the blank area as the embedding center, use the cv2.seamlessClone function, combined with a feather mask, to perform Poisson blending and generate a preliminary fused image; The initial fused image is downsampled using Gaussian pyramid to construct a multi-resolution layer. At each resolution layer, the panoramic image and the initial fused image are fused layer by layer to generate the fusion result of each layer. The multi-resolution layers are then upsampled and superimposed layer by layer to reconstruct the final fused image from the bottom layer to obtain the final panoramic image. The second completion method includes: In panoramic images, extract the effective non-blank areas surrounding blank areas as reference areas; Multiple spatial transformations are performed on the reference region to generate image patches from different perspectives. The most suitable perspective is selected from the image patches from different perspectives using structural similarity to obtain the optimal perspective image patch. Load the pre-trained Context Encoder model, input a panoramic image and an optimal viewpoint image patch, the model fills in the blank areas in the panoramic image based on the context information of the optimal viewpoint image patch, and outputs the completed panoramic image. Gaussian blur is used to remove edge effects from the completed panoramic image, resulting in the final panoramic image.

2. The image processing method based on a large AI model as described in claim 1, characterized in that, Step S2 includes: S21: A deep learning model is constructed using a VGG network. Multiple images are sequentially input into the deep learning model to extract SIFT feature points. The VGG network uses VGG-16, and the network layers include block3_conv3, block4_conv3, and block5_conv3. The output of block5_conv3 is truncated as the output feature map. This feature map is multi-channel and contains deep convolutional features. S22: Pair multiple images together to form an image group; S23: Select a group of pictures; S24: The SIFT feature points of images I1 and I2 in the image group are respectively denoted as the first feature point set {S1} and the second feature point set {S2}. Calculate the correlation coefficient between each SIFT feature point in {S1} and all SIFT feature points in {S2}. Based on the value of the correlation coefficient, obtain the candidate second feature point set for each SIFT feature point in {S1}, which is the first matching result. S25: Select a first feature point from the first matching results. and its candidate second feature point set ; S26: Arrange the candidate second feature point set in descending order of correlation coefficient values. Sort the data and select the top two candidate second feature points. , ,calculate respectively with , The distances d1 and d2, if Then the candidate second feature point As The matching points are added to the second matching result; S27: Repeat steps S25-S26 until the first matching result is completely filtered to obtain the second matching result of the image group; S28: Construct an affine transformation model based on the second matching result, and use image I1 as the reference image to register image I2 using the affine transformation model; S29: Repeat steps S23-S28 until all image groups have been processed and image registration is complete.

3. The image processing method based on a large AI model as described in claim 2, characterized in that, Step S24 includes: S241: The SIFT feature points of images I1 and I2 in the image group are respectively denoted as the first feature point set {S1} and the second feature point set {S2}. Extract the descriptor of each feature point to form the first descriptor set {D1} and the second descriptor set {D2}. S242: Perform local hashing on the first descriptor set {D1} and the second descriptor set {D2} respectively, and map the first descriptor set {D1} and the second descriptor set {D2} to multiple hash buckets through local hashing; S243: Perform preliminary matching within the hash buckets. For each hash bucket, find the feature point pairs to obtain the preliminary matching result {D1}. k D2 m }, where D1 k Represents the descriptors in {D1}, D2 m This refers to the descriptor in {D2}; S244: Calculate the similarity coefficient of each pair of feature points in the preliminary matching results using the similarity formula. The formula for calculating the correlation coefficient is as follows: In the formula, for and The correlation coefficient, for The j-th component, for The j-th component, where n is the dimension of the descriptor. Let be the weight of the j-th dimension; S245: Set relevant thresholds Based on the correlation coefficient value and the correlation threshold The comparison will be greater than or equal to the relevant threshold. The feature point pairs corresponding to the correlation coefficients are selected as the first matching results.

4. The image processing method based on a large AI model as described in claim 1, characterized in that, Step S3 involves a pre-training process for the image fusion model, with the following loss function used for training: In the formula, For the total loss, For the loss of the first network branch, For the loss of the second network branch, To balance the loss of the two network branches, This indicates the loss of image content in the first network branch. This represents the image stitching seam loss in the first network branch. This represents a stitched image representing the initial reconstruction. and These are the content masks for the two input images, respectively. and This represents the two images that were originally input. Indicates perceived loss. Indicates L1 regularized loss. and They respectively represent the following: and Perform convolution and crop to a mask where all pixel values ​​are between 0 and 1. This indicates the loss of image content in the second network branch. This represents the image stitching seam loss in the second network branch. and The calculation process and and similar, This indicates that the images will be merged. Adjusted to , in order to Same size Indicates l1 normalization, , , , , , , Each represents the corresponding weight.

5. The image processing method based on a large AI model as described in claim 4, characterized in that, Step S3 includes: Construct an image fusion model, including a first network branch and a second network branch; First, select two images from the registered images and input them into the image fusion model to fuse the images and obtain a fused image. Then, input a single image and the fused image in sequence and fuse them again until the registered images are fused to obtain a panoramic image. The first network branch processes two images to obtain a preliminarily reconstructed stitched image. The first network branch adopts an encoder-decoder network. The encoder includes a first network layer, a second network layer, and a third network layer. Each network layer includes one convolutional layer and one pooling layer. The encoder is used to extract low-level features. The decoder includes a first deconvolutional layer, a second deconvolutional layer, and a third deconvolutional layer. The decoder is used to generate high-level features. Skip connections are set between the network layers of the encoder and the deconvolutional layers of the decoder to connect low-level features and high-level features. The second network branch is used to process the initially reconstructed stitched image to obtain the fused image. The second network branch includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a residual unit. The residual unit includes four residual blocks. Each residual block has the same structure, including convolutional layer 1, ReLU activation function 1, convolutional layer 2, residual connection, and ReLU activation function 2.

6. The image processing method based on a large AI model as described in claim 5, characterized in that, The first network branch processes the two images to obtain a preliminary reconstructed stitched image. The processing steps are as follows: Two images, I1 and I2, are input into the first network branch. The original size of the two images is H*W. The two images are downsampled and their size is converted to 256*256. The low-level features of I1 and I2 are extracted sequentially using the first, second, and third network layers of the encoder, resulting in three low-level features of I1 at different scales. , and and low-level features at three different scales of I2. , and ; Will and splicing and splicing and The features are then concatenated to obtain the first fusion feature F1, the second fusion feature F2, and the third fusion feature F3. The first fused feature F1, the second fused feature F2, and the third fused feature F3 are input into the decoder. The third fused feature F3 is upsampled using the first deconvolution layer, making its size the same as that of the second fused feature F2. The upsampled third fused feature F3 and the second fused feature F2 are then summed via a skip connection to obtain the first updated fused feature. ; The first update fused features are applied using the second deconvolution layer. Upsample the feature to the same size as the first fused feature F1, and then update the first fused feature using the upsampled feature. The second updated fusion feature is obtained by summing the first fusion feature F1 with the first fusion feature via a skip connection. ; The second update fused features are applied using the third deconvolution layer. Upsampling is performed to restore its size to 256*256, resulting in the final fused feature; A 1*1 convolutional layer is used to map the final fused features to the target number of channels, resulting in a preliminary reconstructed stitched image.

7. The image processing method based on a large AI model as described in claim 6, characterized in that, The second network branch processes the initially reconstructed stitched image to obtain the fused image. The processing steps are as follows: The initially reconstructed stitched image is upsampled to its original size H*W using bilinear interpolation; The first, second, and third convolutional layers are used to extract features from the upsampled stitched image in sequence to obtain the first layer feature A1, the second layer feature A2, and the third layer feature A3; The third-layer feature A3 is input into the residual unit for processing. The first residual block is used to process the third-layer feature A3 to obtain the first residual feature. The first residual feature is input into the second residual block to obtain the second residual feature. The second residual feature is input into the third residual block to obtain the third residual feature. The third residual feature is input into the fourth residual block to obtain the final residual feature. The first layer feature A1 is fused with the final residual feature to obtain the fused feature; The fused features are convolutionally processed to output the fused image.

Images