Picture processing method based on AI large model

Through the AI-based large-scale image processing method, the stitching gap and image distortion problems during panoramic shooting in outdoor tourism are solved, high-precision image registration and natural panoramic image generation are achieved, and visual effects and detailed expression are improved.

CN119048337AActive Publication Date: 2024-11-29WUHAN HUA XIAOPAN COMPUTING TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411159673.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The prior art is difficult to capture natural and seamless field of view when taking panoramic photos during outdoor travel. It is often encountered with stitching gaps, image distortion and image distortion, especially in complex scenes and details.

Method used

Using the image processing method based on AI big model, high-precision registration of multiple pictures is carried out through the feature point extraction and matching technology of the deep learning model, and seamless panoramic images are generated by combining the picture fusion model, and intelligently identify and complement them in blank areas. Full, feature extraction and fusion is used for VGG network and encoder-decoder network, and high-quality completion of blank areas is combined with U-Net segmentation model and Context Encoder model.

Benefits of technology

It realizes high-precision image registration and natural panoramic image generation, eliminates splicing traces, improves visual effects and detailed expression, and ensures the consistency and clarity of panoramic images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048337A_ABST
    Figure CN119048337A_ABST
Patent Text Reader

Abstract

The invention proposes a picture processing method based on an AI large model, and relates to the technical field of image processing, and the method comprises the following steps: S1, collecting and preprocessing a plurality of pictures; s2, constructing a deep learning model, performing feature point extraction on the plurality of pictures by using the deep learning model, matching the feature points of the plurality of pictures by using a feature matching algorithm to obtain a first matching result, and performing mismatching screening on the first matching pair by using a screening algorithm to obtain a second matching result, affine transformation is applied to the second matching result, and registration of the multiple pictures is completed; s3, fusing the plurality of registered pictures by adopting a picture fusion model to generate a panoramic picture; and S4, carrying out blank area identification on the panoramic picture, and complementing the blank area based on a complementing method to obtain a final panoramic picture. According to the invention, high-quality and natural panoramic pictures can be generated, and the visual effect and viewing experience of passenger pictures are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method based on an AI large model. Background Art

[0002] With the popularity of smartphones and digital cameras, taking photos has become an important way for tourists to record beautiful moments during outdoor travel. However, due to the tight schedule of outdoor tours or the urging of people around them, tourists usually cannot reserve enough time to screen and adjust the photos they took, and it is difficult to retake them, making it difficult to obtain satisfactory photos. In addition, for many scenic spots, conventional photography methods are difficult to capture a wide field of view. Currently, panoramic shooting or wide-angle cameras are mostly used for shooting:

[0003] Panoramic shooting: Panoramic shooting usually uses the built-in function of the camera or mobile phone to automatically stitch multiple photos to generate a panoramic picture. Although this method can capture a wider field of view, when processing complex scenes, problems such as stitching gaps, unnatural transitions and image distortion often occur, affecting the final visual effect.

[0004] Wide-angle camera: A wide-angle camera can capture a wider field of view, but due to the optical characteristics of wide-angle lenses, image distortion is prone to occur, especially at the edges of the picture, resulting in poor visual effects. In addition, wide-angle cameras often lack detail expression when shooting distant scenes. Summary of the invention

[0005] In view of this, the present invention proposes an image processing method based on an AI large model, which realizes high-precision alignment of multiple images through feature point extraction and matching technology of a deep learning model; and uses an image fusion model to generate seamless panoramic images; after intelligently identifying and completing blank areas in the panorama, it can generate high-quality and natural panoramic images, thereby improving the visual effects and viewing experience of travelers' photos.

[0006] The technical solution of the present invention is achieved in this way:

[0007] The present invention provides an image processing method based on an AI large model, comprising the following steps:

[0008] S1: collect multiple images and perform preprocessing;

[0009] S2: construct a deep learning model, use the deep learning model to extract feature points of multiple images respectively, and use a feature matching algorithm to match the feature points of multiple images to obtain a first matching result, use a screening algorithm to screen the first matching pair for mismatching, obtain a second matching result, apply affine transformation to the second matching result, and complete the registration of multiple images;

[0010] S3: Use an image fusion model to fuse the registered multiple images to generate a panoramic image;

[0011] S4: Identify the blank areas in the panoramic image, and complement the blank areas based on the complement method to obtain the final panoramic image.

[0012] Based on the above solution, preferably, step S2 includes:

[0013] S21: Use the VGG network to construct a deep learning model, and input the multiple images into the deep learning model in sequence to extract SIFT feature points;

[0014] S22: Pair the multiple images in pairs to form image groups;

[0015] S23: Select an image group;

[0016] S24: Denote the SIFT feature points of image I1 and image I2 in the image group as the first feature point set {S1} and the second feature point set {S2} respectively, calculate the correlation coefficient between each SIFT feature point in {S1} and all SIFT feature points in {S2}, and obtain the candidate second feature point set for each SIFT feature point in {S1} based on the value of the correlation coefficient, that is, the first matching result;

[0017] S25: Select a first feature point S1 in the first matching result i and its candidate second feature point set

[0018] S26: Sort the candidate second feature point set in descending order according to the value of the correlation coefficient, select the first two candidate second feature points Calculate the distances d1 and d2 between S1 i and respectively. If d1 / d2 < T1, then use the candidate second feature point as the matching point of S1 i and add it to the second matching result;

[0019] S27: Repeat steps S25 - S26 until the first matching result is screened out to obtain the second matching result of the image group;

[0020] S28: Construct an affine transformation model based on the second matching result, use image I1 as the reference image, and register image I2 using the affine transformation model;

[0021] S29: Repeat steps S23 - S28 until all image groups are processed to complete image registration.

[0022] Based on the above solution, preferably, step S24 includes:

[0023] S241: SIFT feature points of the picture I1 and the picture I2 in the picture group are recorded as a first feature point set {S1} and a second feature point set {S2} respectively, and a descriptor of each feature point is extracted to form a first descriptor set {D1} and a second descriptor set {D2};

[0024] S242: performing local hash processing on the first descriptor set {D1} and the second descriptor set {D2} respectively, and mapping the first descriptor set {D1} and the second descriptor set {D2} to multiple hash buckets through local hashing;

[0025] S243: Perform preliminary matching in the hash buckets, find the feature point pairs in each hash bucket, and obtain preliminary matching results {D1 k ,D2 m}, where D1 k Denotes the descriptor in {D1}, D2 m represents the descriptor in {D2};

[0026] S244: Calculate the similarity coefficient of each pair of feature points in the preliminary matching result using a similarity formula, wherein the calculation formula of the correlation coefficient is:

[0027]

[0028] In the formula, R(D1 k ,D2 m ) is D1 k With D2 m The correlation coefficient, f1 j D1 k The jth component of j For D2 m The jth component of , n is the dimension of the descriptor, ω j is the weight of the j-th dimension;

[0029] S245: Setting a correlation threshold τ, comparing the correlation coefficient value with the correlation threshold τ, and selecting feature point pairs corresponding to correlation coefficients greater than or equal to the correlation threshold τ as the first matching result.

[0030] Based on the above solution, preferably, step S3 includes:

[0031] Constructing an image fusion model, including a first network branch and a second network branch;

[0032] First, two images are selected from the multiple registered images and input into the image fusion model for image fusion to obtain a fused image. Then, the single images and the fused image are input in sequence for fusion again until the multiple registered images are fused to obtain a panoramic image.

[0033] in:

[0034] The first network branch is used to process the two images to obtain a preliminarily reconstructed spliced ​​image. The first network branch adopts an encoder-decoder network. The encoder includes a first network layer, a second network layer and a third network layer. Each network layer includes a convolution layer and a pooling layer. The encoder is used to extract low-level features; the decoder includes a first deconvolution layer, a second deconvolution layer and a third deconvolution layer. The decoder is used to generate high-level features; a jump connection is set between the network layer of the encoder and the deconvolution layer of the decoder to connect the low-level features with the high-level features;

[0035] The second network branch is used to process the initially reconstructed spliced ​​image to obtain a fused image. The second network branch includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a residual unit. The residual unit includes 4 residual blocks. Each residual block has the same structure, including convolutional layer 1, ReLU activation function 1, convolutional layer 2, residual connection and ReLU activation function 2.

[0036] Based on the above solution, preferably, the first network branch processes the two images to obtain a preliminarily reconstructed spliced ​​image as follows:

[0037] Input two images I1 and I2 into the first network branch. The original size of the two images is H*W. Downsample the two images and convert the size to 256*256.

[0038] The first, second, and third network layers of the encoder are used to extract the low-level features of I1 and I2 in turn, and three low-level features of I1 at different scales are obtained. and And three low-level features of different scales of I2 and

[0039] Will and To splice, and To splice, and Perform splicing to obtain the first fusion feature F1, the second fusion feature F2 and the third fusion feature F3;

[0040] The first fused feature F1, the second fused feature F2 and the third fused feature F3 are input into the decoder, the third fused feature F3 is upsampled by the first deconvolution layer to make its size the same as the second fused feature F2, and the upsampled third fused feature F3 is added to the second fused feature F2 via a jump connection to obtain the first updated fused feature F′1;

[0041] The first updated fusion feature F′1 is upsampled by the second deconvolution layer to make its size the same as the first fusion feature F1, and the upsampled first updated fusion feature F′1 is added to the first fusion feature F1 via a jump connection to obtain the second updated fusion feature F′2;

[0042] The third deconvolution layer is used to upsample the second updated fusion feature F′2 to restore its size to 256*256 to obtain the final fusion feature;

[0043] A 1*1 convolutional layer is used to map the final fused features to the target number of channels to obtain a preliminary reconstructed spliced ​​image.

[0044] Based on the above solution, preferably, the second network branch processes the initially reconstructed spliced ​​image to obtain a fused image as follows:

[0045] Use bilinear interpolation to upsample the initially reconstructed spliced ​​image to the original size H*W;

[0046] The first convolution layer, the second convolution layer, and the third convolution layer are used to extract features of the upsampled spliced ​​image in turn, and the first layer features A1, the second layer features A2, and the third layer features A3 are obtained;

[0047] The third layer feature A3 is input into the residual unit for processing, the third layer feature A3 is processed by the first residual block to obtain the first residual feature, the first residual feature is input into the second residual block, the second residual feature is input into the third residual block, the third residual feature is output, the third residual feature is input into the fourth residual block, and the final residual feature is output;

[0048] The first layer feature A1 is fused with the final residual feature to obtain the fused feature;

[0049] Perform convolution operation on the fused features and output the fused image.

[0050] Based on the above solution, preferably, step S4 includes:

[0051] S41: Use the U-Net segmentation model to segment the panoramic image and obtain the blank area;

[0052] S42: Obtaining location information of the blank area, and searching on the Internet through a search engine according to the location information;

[0053] S43: if a candidate image is found, the first completion method is used to complete the blank area based on the candidate image; if no candidate image is found, the second completion method is used to complete the blank area;

[0054] S44: Output the final panoramic image.

[0055] Based on the above solution, preferably, step S42 includes:

[0056] The contour of the blank area is extracted by edge detection algorithm, and the contour information is optimized by morphological operation to obtain the shape characteristics of the blank area;

[0057] Use a regular grid to mark the relative position of the blank area in the panoramic image to obtain position features;

[0058] Taking the outline of the blank area as the initial point, the outer contour is obtained after expanding according to the preset step length, and the feature information in the outer contour is extracted as the environmental feature of the blank area;

[0059] The shape features, location features and environmental features are combined into the basic features of the blank area;

[0060] Use basic features to perform a rough search in the search engine to obtain initial candidate images;

[0061] Extract the features of the initial candidate images to obtain candidate features;

[0062] The similarity between the basic features of the blank area and the candidate features in the candidate feature set is calculated one by one, and the most similar picture is taken as the candidate picture.

[0063] Based on the above solution, preferably, the first completion method includes:

[0064] Crop and deform the candidate image to make it the same shape as the blank area;

[0065] The area surrounding the outline is used as a buffer, the color histogram of the candidate image and the buffer is calculated, and the color distribution of the candidate image is adjusted according to the histogram matching algorithm;

[0066] Use the local color difference correction algorithm to fine-tune the color histogram matching results to obtain the candidate images after color matching;

[0067] Create a mask for the blank area and perform Gaussian blur on the mask edge to generate a feathered mask with a gradient effect;

[0068] Define the center of the blank area as the embedding center, use the cv2.seamlessClone function, combined with the feathering mask, perform Poisson Blending, and generate a preliminary fused image;

[0069] The preliminary fused image is downsampled using a Gaussian pyramid to construct a multi-resolution hierarchy. At each resolution level, the panoramic image and the preliminary fused image are fused layer by layer to generate the fusion result of each layer. The image is upsampled layer by layer and multi-resolution levels are superimposed to reconstruct the final fused image from the bottom layer to obtain the final panoramic image.

[0070] Based on the above solution, preferably, the second completion method includes:

[0071] In the panoramic image, extract the valid non-blank area around the blank area as the reference area;

[0072] Perform multiple spatial transformations on the reference area to generate image blocks of different perspectives, and use structural similarity to select the most suitable perspective from the image blocks of different perspectives to obtain the image block with the best perspective;

[0073] Load the pre-trained Context Encoder model, input the panoramic image and the best view image block, the model completes the blank area in the panoramic image according to the context information of the best view image block, and outputs the completed panoramic image;

[0074] Gaussian blur is used to eliminate the edge effect of the completed panoramic image to obtain the final panoramic image.

[0075] The system of the present invention has the following beneficial effects compared with the prior art:

[0076] 1) The present invention realizes high-precision image registration through a deep learning model, realizes natural image fusion through multi-layer convolution and residual networks, and solves the problem of processing blank areas of panoramic images through intelligent completion technology, and finally generates a highly consistent panoramic image with excellent visual effects;

[0077] 2) The present invention uses the VGG network to build a deep learning model, extracts SIFT feature points and performs feature matching, and combines the mismatch screening algorithm to improve the accuracy of feature point matching. On this basis, affine transformation is used to accurately align multiple images, effectively eliminating the problems of image dislocation and blur in traditional methods;

[0078] 3) By constructing an image fusion model including an encoder-decoder network and a residual network, the image features are extracted and fused layer by layer to achieve a natural transition of the image. The skip connection mechanism and multi-scale feature fusion strategy ensure the visual coherence and delicate detail restoration of the image, so that the generated panoramic image has a natural transition and no stitching traces. Specifically, the first network branch encodes and decodes the image to generate a preliminary reconstructed stitched image, while the second network branch optimizes the detail level of the stitched image through the residual unit, thereby improving the fineness and realism of the final panoramic image;

[0079] 4) In the scheme adopted by the present invention, the first network branch fuses low-level features of different scales layer by layer through multi-layer convolution and deconvolution operations to generate a preliminary reconstructed spliced ​​picture. It can effectively extract multi-scale information in the picture, ensure efficient capture of detail features in picture splicing, improve video visual quality, and make the generated spliced ​​picture have better consistency and clarity; the application of the jump connection mechanism combines the low-level features of the encoder with the high-level features of the decoder, greatly enhancing the connection at the feature level, so that the reconstructed spliced ​​picture has a high degree of consistency at multiple scales, further improving the effect of picture fusion;

[0080] 5) The present invention uses the U-Net segmentation model to accurately identify blank areas in panoramic pictures, and searches and obtains similar pictures on the Internet through a search engine based on the location information of the blank areas. This step combines edge detection and morphological operations to improve the accuracy and reliability of blank area recognition; first, through the first completion method based on candidate pictures, high-quality completion of blank areas is achieved. Specific technologies include color histogram matching, local color difference correction, and Poisson Blending, which ensure the seamless fusion of color and texture; secondly, when a suitable candidate picture cannot be found, the second completion method based on the Context Encoder model is adopted. Through the analysis and utilization of contextual information, intelligent completion in complex scenes is achieved, ensuring the visual consistency and naturalness of the final panoramic picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 is a flow chart of the method of the present invention;

[0083] Figure 2is a schematic structural diagram of a first network branch of the present invention;

[0084] Figure 3 It is a schematic diagram of the structure of the second network branch of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] See also Figure 1 The present invention provides an image processing method based on an AI large model, comprising the following steps:

[0087] S1: collect multiple images and perform preprocessing;

[0088] S2: construct a deep learning model, use the deep learning model to extract feature points of multiple images respectively, and use a feature matching algorithm to match the feature points of multiple images to obtain a first matching result, use a screening algorithm to screen the first matching pair for mismatching, obtain a second matching result, apply affine transformation to the second matching result, and complete the registration of multiple images;

[0089] S3: Use the image fusion model to fuse multiple registered images to generate a panoramic image;

[0090] S4: Identify the blank areas of the panoramic image, and complete the blank areas based on a completion method to obtain a final panoramic image.

[0091] Specifically, first, multiple photos of the same scenic spot taken by users using mobile devices are collected, and these pictures are preprocessed, including denoising, brightness and contrast adjustment.

[0092] Specifically, in one embodiment of the present invention, step S2 includes:

[0093] S21: Use VGG network to build a deep learning model, input multiple pictures into the deep learning model in sequence, and extract SIFT feature points;

[0094] S22: Pair the multiple pictures into pairs to form a picture group;

[0095] S23: Select a picture group;

[0096] S24: Denote the SIFT feature points of picture I1 and picture I2 in the picture group as the first feature point set {S1} and the second feature point set {S2} respectively. Calculate the correlation coefficient between each SIFT feature point in {S1} and all SIFT feature points in {S2}, and obtain the candidate second feature point set for each SIFT feature point in {S1} based on the value of the correlation coefficient, that is, the first matching result;

[0097] S25: Select a first feature point S1 in the first matching result i and its candidate second feature point set

[0098] S26: Sort the candidate second feature point set in descending order according to the value of the correlation coefficient, and select the first two candidate second feature points Calculate the distances d1 and d2 between S1 i and respectively. If d1 / d2 < T1, then use the candidate second feature point as the matching point of S1 i and add it to the second matching result;

[0099] S27: Repeat steps S25 - S26 until the first matching result is screened out completely, and obtain the second matching result of the picture group;

[0100] S28: Construct an affine transformation model based on the second matching result. Taking picture I1 as the reference picture, register picture I2 using the affine transformation model;

[0101] S29: Repeat steps S23 - S28 until all picture groups are processed, and complete picture registration.

[0102] Illustrate with a specific example:

[0103] Select VGG-16 as the base model, load the pre-trained VGG model in the deep learning framework, obtain the deep learning model, adjust the input multiple pictures according to the model requirements, including size and normalization, arrange the adjusted pictures according to the RGB channels, and then input them into the model to extract the feature map. In the VGG-16 of this embodiment, the network layer includes block3_conv3, block4_conv3 and block5_conv3, and the output of block5_conv3 is intercepted as the output feature map, which is multi-channel and contains deep convolution features. Each channel of the feature map can be regarded as the representation of the image in different feature dimensions. Find recognizable key points on the feature map (, which should be scale-invariant and rotation-invariant. SIFT uses a multi-scale Gaussian difference (DoG) pyramid to detect salient points, i.e., key points. The feature map is filtered based on a multi-scale Gaussian function, and the filtering results are used as the scale space representation of each point in the feature map. Find local extreme points at each scale and determine their exact position and scale. These local extreme points are potential SIFT key points. Assign a main direction to each key point. By calculating the gradient histogram of the neighborhood of the key point, the standard direction of the gradient direction and amplitude is determined to ensure the directional invariance of the feature. In the neighborhood of the key point, the distribution of the directional gradient is calculated to generate a 128-dimensional feature vector (descriptor) containing the direction and amplitude. These descriptors are used to describe the local image information of the key point. Each extracted key point and its corresponding descriptor form a feature point set, i.e., SIFT feature point.

[0104] Multiple pictures are randomly combined to form multiple picture pairs, each pair is a picture group. For example, pictures I1, I2, and I3 can form a picture pair (I1, I2)(I1, I3)(I2, I3).

[0105] From the image group, select an image pair for processing, for example, select the image pair (I1, I2).

[0106] The SIFT feature points of the images I1 and I2 in the image group are recorded as the first feature point set {S1} and the second feature point set {S2} respectively.

[0107] Feature point set: Let {S1} = {S11, S12, ..., S1 n} and {S2} = {S21, S22, ..., S2 v}.

[0108] Calculate the correlation coefficient between each feature point in {S1} and all feature points in {S2}, and obtain a candidate second feature point set for each feature point in {S1} based on the value of the correlation coefficient, i.e., the first matching result. The process specifically includes:

[0109] S241: The SIFT feature points of the images I1 and I2 in the image group are recorded as the first feature point set {S1} and the second feature point set {S2}, respectively, and the descriptors of each feature point are extracted to form the first descriptor set {D1} and the second descriptor set {D2}; specifically, each feature pair has a corresponding descriptor, where {S1} = {S11, S12, ..., S1 n}, each S1 i Corresponding to a descriptor D1 i ,{S2}={S21,S22,...,S2 v}, each S2 i Corresponding to a descriptor D2 i .

[0110] S242: Perform local hashing on the first descriptor set {D1} and the second descriptor set {D2} respectively, and map the first descriptor set {D1} and the second descriptor set {D2} to multiple hash buckets through local hashing; specifically, select local sensitive hashing LSH to map high-dimensional descriptors to low-dimensional hash buckets. Each descriptor is assigned to one or more hash buckets according to the hash function.

[0111] S243: Perform preliminary matching in the hash buckets, find the feature point pairs in each hash bucket, and obtain preliminary matching results {D1 k ,D2 m}, where D1 k Denotes the descriptor in {D1}, D2 m Denotes the descriptor in {D2}; Specifically, in each hash bucket, possible descriptor pairs are searched, and the descriptor pairs in the same hash bucket are taken as preliminary matching candidates. The preliminary matching result is represented as {D1 k ,D2 m}, where D1 k and D2 m From {D1} and {D2} respectively.

[0112] S244: Calculate the similarity coefficient of each pair of feature points in the preliminary matching result using a similarity formula, wherein the calculation formula of the correlation coefficient is:

[0113]

[0114] In the formula, R(D1 k ,D2 m ) is D1 k With D2 m The correlation coefficient, f1 j D1 k The jth component of j For D2m The j-th component, where n is the dimension of the descriptor, and ω j is the weight of the j-th dimension. In this embodiment, the range of R(D1 k , D2 m ) is [0, 1]. If D1 k is more similar to D2 m , then the value of R(D1 k , D2 m ) is larger, that is, closer to 1.

[0115] S245: Set the relevant threshold τ, compare the value of the correlation coefficient with the relevant threshold τ, and screen out the feature point pairs corresponding to the correlation coefficients greater than or equal to the relevant threshold τ as the first matching result.

[0116] After that, from the first matching result, select a first feature point and its candidate second feature point set in sequence for screening:

[0117] Sort in descending order according to the correlation coefficient, and select the first two candidate points Calculate the distances d1 and d2 between S1 i and respectively. At this time, the distance can be the Euclidean distance. The smaller the distance, the more similar the points are. Calculate the ratio of d1 to d2. If d1 / d2 < T1, then the candidate second feature point is used as the matching point of S1 i and added to the second matching result.

[0118] In this embodiment, first, by calculating and sorting the correlation coefficients, two matching points with the highest similarity are selected. This step is to find the most likely correct matching points After that, by comparing the distances between S1 i and , if is the correct matching point, there should be a significant difference between d1 and d2, and d1 is smaller and d2 is larger. By setting a threshold T1, specifically, T1 can be set to 0.6, to determine whether the distance ratio is less than the threshold T1. If the distance ratio is lower than the threshold T1, it indicates that is significantly better than , which means is more reliable. If the distance ratio is greater than or equal to T1, it means that has little difference, that is, the matching result is not significant enough. At this time, it is determined as a wrong match.

[0119] Next, the matching points are repeatedly screened to obtain the second matching result of the picture group, and an affine transformation model is constructed based on the second matching result. Taking picture I1 as a reference, the affine transformation model is applied to transform the coordinates of all pixel points in picture I2, thereby achieving picture registration.

[0120] Finally, repeat the above steps to complete the registration of all images.

[0121] Specifically, in one embodiment of the present invention, step S3 includes:

[0122] Constructing an image fusion model, including a first network branch and a second network branch;

[0123] First, two images are selected from the multiple registered images and input into the image fusion model for image fusion to obtain a fused image. Then, the single images and the fused image are input in sequence for fusion again until the multiple registered images are fused to obtain a panoramic image.

[0124] in:

[0125] like Figure 2 As shown, the first network branch is used to process the two images to obtain a preliminary reconstructed spliced ​​image. The first network branch adopts an encoder-decoder network. The encoder includes a first network layer, a second network layer and a third network layer. Each network layer includes 1 convolution layer and 1 pooling layer, and the encoder is used to extract low-level features; specifically, the number of filters of the convolution layer conv1 in the first network layer is 64, the convolution kernel size is 3*3, the step size is 1, the padding is 1, and the pooling layer pool1 is a maximum pooling of 2*2 with a step size of 2; the number of filters of the convolution layer conv1 in the second network layer is 128, the convolution kernel size is 3*3, the step size is 1, the padding is 1, and the pooling layer pool2 is a maximum pooling of 2*2 with a step size of 2; the number of filters of the convolution layer conv3 in the third network layer is 256, the convolution kernel size is 3*3, the step size is 1, the padding is 1, and the pooling layer pool3 is a maximum pooling of 2*2 with a step size of 2.

[0126] The decoder includes a first deconvolution layer, a second deconvolution layer and a third deconvolution layer, and the decoder is used to generate high-level features;

[0127] Specifically, the number of filters in the first deconvolution layer is 128, the convolution kernel size is 3*3, the step size is 2, and the output image size is reduced by half; the number of filters in the second deconvolution layer is 64, the convolution kernel size is 3*3, the step size is 2, and the output image size is reduced by half; the number of filters in the third deconvolution layer is 3, the convolution kernel size is 3*3, the step size is 2, and the output image is restored to 256*256;

[0128] In order to maintain multi-scale information, a skip connection is set between the network layer of the encoder and the deconvolution layer of the decoder to connect low-level features and high-level features; the skip connection is directly connected from the convolution layer of the encoder to the deconvolution layer of the corresponding scale of the decoder, so that the high-resolution feature information will be preserved and combined with the high-level features of the decoder.

[0129] like Figure 3 As shown, the second network branch is used to process the initially reconstructed spliced ​​image to obtain a fused image. The second network branch includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a residual unit. The residual unit includes 4 residual blocks. Each residual block has the same structure, including convolutional layer 1, ReLU activation function 1, convolutional layer 2, residual connection and ReLU activation function 2.

[0130] Specifically, the number of filters in the first convolutional layer is 64, the convolution kernel size is 3*3, the stride is 1, and the padding is 1. The second and third convolutional layers have the same structure as the first convolutional layer. In each residual block, the number of filters in convolutional layer 1 is 64, the convolution kernel size is 3*3, the stride is 1, and the padding is 1; the structure of convolutional layer 2 is the same as that of convolutional layer 1, and the residual connection adds the input features to the convolution output of the previous step.

[0131] The first network branch processes the two images to obtain a preliminary reconstructed spliced ​​image as follows:

[0132] Two images I1 and I2 are input into the first network branch. The original size of the two images is H*W. The two images are downsampled and converted to 256*256 to reduce the computational complexity and capture large-scale splicing features.

[0133] The first, second, and third network layers of the encoder are used to extract the low-level features of I1 and I2 in turn, and three low-level features of I1 at different scales are obtained. and And three low-level features of different scales of I2 and

[0134] Will and To splice, and To splice, and The first fusion feature F1, the second fusion feature F2 and the third fusion feature F3 are obtained by splicing; specifically, the splicing is performed on the feature channel dimension.

[0135] The first fused feature F1, the second fused feature F2 and the third fused feature F3 are input into the decoder, the third fused feature F3 is upsampled by the first deconvolution layer to make its size the same as the second fused feature F2, and the upsampled third fused feature F3 is added to the second fused feature F2 via a jump connection to obtain the first updated fused feature F′1;

[0136] The first updated fusion feature F′1 is upsampled by the second deconvolution layer to make its size the same as the first fusion feature F1, and the upsampled first updated fusion feature F′1 is added to the first fusion feature F1 via a jump connection to obtain the second updated fusion feature F′2;

[0137] The third deconvolution layer is used to upsample the second updated fusion feature F′2 to restore its size to 256*256 to obtain the final fusion feature;

[0138] A 1*1 convolutional layer is used to map the final fused features to the target number of channels to obtain a preliminary reconstructed spliced ​​image.

[0139] Specifically, the deconvolution starts from the third fusion feature, gradually upsampling to restore the resolution, and combining the jump connection to enhance the detail features, and finally obtains a preliminary spliced ​​image with a size of 256*256.

[0140] The second network branch processes the initially reconstructed spliced ​​image to obtain the fused image as follows:

[0141] Use bilinear interpolation to upsample the initially reconstructed spliced ​​image to the original size H*W;

[0142] The first convolution layer, the second convolution layer, and the third convolution layer are used to extract features of the upsampled spliced ​​image in turn, and the first layer features A1, the second layer features A2, and the third layer features A3 are obtained;

[0143] The third layer feature A3 is input into the residual unit for processing, the third layer feature A3 is processed by the first residual block to obtain the first residual feature, the first residual feature is input into the second residual block, the second residual feature is input into the third residual block, the third residual feature is output, the third residual feature is input into the fourth residual block, and the final residual feature is output;

[0144] The first layer feature A1 is fused with the final residual feature to obtain the fused feature;

[0145] Perform convolution operation on the fused features and output the fused image.

[0146] Specifically, the spliced ​​image is first upsampled to the original size, such as 1024*1024, and then the first convolutional layer, the second convolutional layer and the third convolutional layer are used to extract features. At this time, the feature sizes output by the three convolutional layers are the same. Then, the residual block is introduced to enhance the feature learning ability of the network and avoid gradient disappearance. In this embodiment, 4 residual blocks are set, but in other embodiments, 6 or 8 residual blocks can also be set, which can be determined according to the specific situation. The third layer features output by the third convolutional layer are input into the residual unit, and the 4 residual blocks are used to effectively refine and supplement the feature information in turn to obtain the final residual features. Then, the first layer features A1 are fused with the final residual features. At this time, the final residual features are deep-level features, and the first layer features A1 are shallow-level features. By fusing the deep-level features with the shallow-level features, the network can effectively utilize the rich multi-scale information in the final output and improve the quality of the fused image.

[0147] In this embodiment, after building the image fusion model, pre-training is required, and then multiple images are processed. During pre-training, a public data set is used for training, the optimizer is Adam, and the initial value of the learning rate is 10 -4 , the training loss function is as follows:

[0148] L=ω1L1+ω2L2+ω3L3

[0149]

[0150] In the formula, L is the total loss, L1 is the loss of the first network branch, L2 is the loss of the second network branch, and L3 is the balance loss of the two network branches. represents the image content loss in the first network branch, represents the image splicing loss in the first network branch, G1 represents the initially reconstructed splicing image, and M 1 and M 2 They are the content masks of the two input images, I1 and I2 represent the two original input images, represents the perceived loss, represents L1 regularization loss, M 1S and M 2S Respectively represent M 1 and M 2 Convolution and cropping to a mask where pixel values ​​are all between 0 and 1, represents the image content loss in the second network branch, represents the image splicing loss in the second network branch, and The calculation process and and similar, represents adjusting the fused image G2 to 256×256 to be the same size as G1, ||·||1 represents l1 normalization, ω1, ω2, ω3, All represent corresponding weights.

[0151] Specifically, in one embodiment of the present invention, step S4 includes:

[0152] S41: Use the U-Net segmentation model to segment the panoramic image and obtain the blank area;

[0153] S42: Obtaining location information of the blank area, and searching on the Internet through a search engine according to the location information;

[0154] S43: if a candidate image is found, the first completion method is used to complete the blank area based on the candidate image; if no candidate image is found, the second completion method is used to complete the blank area;

[0155] S44: Output the final panoramic image.

[0156] In this embodiment, step S41 includes:

[0157] The panoramic image is input into the trained U-Net model to obtain a binary prediction mask and mark the blank area. The U-Net model consists of an encoder and a decoder, forming a symmetrical U-shaped structure.

[0158] The predicted mask is converted into a clear binary image. Based on the binary mask, the blank areas in the panoramic image are extracted to generate a binary image containing the location information of the blank areas.

[0159] In this embodiment, step S42 includes:

[0160] S421: extracting the outline of the blank area through an edge detection algorithm, and optimizing the outline information using morphological operations to obtain shape features of the blank area.

[0161] Specifically, the Canny edge detection algorithm is applied to extract the edge of the blank area. The Canny algorithm detects edges by calculating the image gradient, which mainly includes the following steps: Gaussian filtering: first use a Gaussian filter to smooth the image and remove noise. Calculate the gradient: calculate the gradient strength and direction of each pixel in the image. Non-maximum suppression: refine the edge and suppress non-edge points. Hysteresis threshold: detect the final edge through double thresholds.

[0162] Use dilation and erosion operations to optimize edge contours. Morphological operations can enhance the continuity and clarity of contours. Dilation: Increase the boundaries of objects and fill small holes in the edges. Erosion: Reduce the boundaries of objects and remove noise points.

[0163] S422: Use a regular grid to mark the relative position of the blank area in the panoramic image to obtain a position feature.

[0164] Divide the panoramic image into fixed-size grid cells and record the position of the blank area on the grid. The grid size can be adjusted as needed. Output the position feature network of the blank area as feature information.

[0165] S423: Taking the outline of the blank area as the initial point, expanding it outward according to a preset step length to obtain an outer contour, and extracting feature information within the outer contour as the environmental feature of the blank area.

[0166] Expand outward from the outline of the blank area by a certain step length to extract image features including those around the blank area. For example, set the step length to 10, use the dilation operation to expand the outline, and extract the feature information within the expanded outline as the environmental feature.

[0167] S424: Shape features, position features and environmental features are combined into basic features of the blank area.

[0168] S425: Perform a rough search in a search engine using basic features to obtain initial candidate images.

[0169] Use the above basic features to perform a rough search in the search engine through the API and return the initial candidate images. The feature vector needs to be converted into JSON format and given as input to the search engine API.

[0170] S426: Extract features of the initial candidate image to obtain candidate features.

[0171] For each candidate image, similar types of features are extracted, including edge detection results, location features, and environmental features.

[0172] S427: Calculate similarity between the basic features of the blank area and the candidate features in the candidate feature set one by one, and take the most similar picture as the candidate picture.

[0173] Use cosine similarity to calculate the similarity between the features of each candidate image and the basic features. Select the candidate image with the highest similarity.

[0174] In this embodiment, the first completion method includes:

[0175] The candidate image is cropped and deformed to make it the same shape as the blank area.

[0176] Use image cropping tools, such as the cv2.getRectSubPix function in OpenCV, to select a portion that is similar in shape and size to the blank area for cropping. You need to ensure that the cropped image covers the entire content as much as possible to reduce information loss.

[0177] After that, affine transformation is performed to make the cropped candidate image match the shape of the blank area. The cv2.getAffineTransform and cv2.warpAffine functions can be used. For blank areas with larger areas, perspective transformation is used, that is, cv2.getPerspectiveTransform function.

[0178] The area surrounding the outline is used as a buffer, the color histogram of the candidate image and the buffer is calculated, and the color distribution of the candidate image is adjusted according to the histogram matching algorithm.

[0179] Based on the outline of the blank area, expand a certain pixel distance outward to generate a buffer. Extract the color information in the buffer and calculate its color histogram. You can use the cv2.calcHist function.

[0180] Through the color histogram matching algorithm, such as cv2.calcBackProject in OpenCV, the color distribution of the candidate image is adjusted to be close to the color of the buffer. The candidate image is adjusted to fit the overall style of the panoramic image.

[0181] The local color difference correction algorithm is used to fine-tune the color histogram matching results to obtain the candidate images after color matching.

[0182] The Gaussian blur and Laplacian operators are combined to perform local color adjustment, and the color of the candidate image is fine-tuned pixel by pixel by comparing the local mean and variance, so that there is no color difference between it and the panoramic image in the local area.

[0183] Create a mask for the blank area and perform Gaussian blur on the mask edge to generate a feathered mask with a gradient effect.

[0184] Generate a binary mask based on the outline of the blank area. You can use cv2.fillPoly to fill the polygon, fill the blank area with white and the rest with black. Perform Gaussian blur on the mask edge to generate a feathered mask with a gradient effect.

[0185] Define the center of the blank area as the embedding center, use the cv2.seamlessClone function, combined with the feathering mask, perform Poisson Blending, and generate a preliminary fused image.

[0186] Use the cv2.seamlessClone function to perform seamless cloning and embed the color-matched candidate image seamlessly into the blank area of ​​the panoramic image. Poisson Blending achieves seamless fusion by solving the Poisson equation to ensure that the edges of the candidate image smoothly transition into the panoramic image.

[0187] The preliminary fused image is downsampled using a Gaussian pyramid to construct a multi-resolution hierarchy. At each resolution level, the panoramic image and the preliminary fused image are fused layer by layer to generate the fusion result of each layer. The image is upsampled layer by layer and multi-resolution levels are superimposed to reconstruct the final fused image from the bottom layer to obtain the final panoramic image.

[0188] Perform multiple Gaussian downsampling on the preliminary fused image to construct a pyramid level with multiple resolutions. Gaussian downsampling achieves multi-resolution representation of the image by recursively low-pass filtering and downsampling the image. Each resolution is fused layer by layer from the bottom layer. The fusion results of the lower layers are stacked layer by layer to higher resolution layers through upsampling, and finally a high-resolution panoramic image is reconstructed layer by layer.

[0189] Specifically, the first completion method covers the entire process from cropping, deformation, color matching, fusion processing to the final multi-resolution image fusion. This method ensures that the candidate image can be embedded into the blank area of ​​the panoramic image with high quality, achieving natural transition and visual consistency.

[0190] In this embodiment, the second completion method includes:

[0191] In the panoramic image, the valid non-blank area around the blank area is extracted as the reference area.

[0192] Specifically, the edge of the blank area is identified, and a certain distance is expanded outward along the edge to extract the surrounding area containing valid information.

[0193] The reference area is spatially transformed multiple times to generate image blocks of different perspectives. The structural similarity is used to select the most suitable perspective from the image blocks of different perspectives to obtain the image block with the best perspective.

[0194] Perform multiple spatial transformations (such as rotation, scaling, and affine transformation) to generate image blocks of different perspectives for comparison and selection of suitable candidate blocks. Specifically, the rotation transformation uses cv2.getRotationMatrix2D and cv2.warpAffine. The scaling transformation uses cv2.resize. The affine transformation uses cv2.getAffineTransform and cv2.warpAffine.

[0195] Use skimage.metrics.structural_similarity to calculate the structural similarity between each perspective image block and the adjacent area of ​​the panoramic image, and select the image block with the highest structural similarity as the best perspective image block.

[0196] Load the pre-trained Context Encoder model, input the panoramic image and the best viewing angle image block, and the model completes the blank area in the panoramic image according to the context information of the best viewing angle image block, and outputs the completed panoramic image.

[0197] Preprocess the panoramic image and the best view image blocks, and fill the image blocks into the blank areas of the panoramic image. Use PyTorch's torch.load function to load the pre-trained model. Call the model's eval method to switch the model to evaluation mode.

[0198] The best viewing angle image block is filled into the blank area of ​​the panoramic image. The blank area is located by mask and replaced locally so that the blank area is filled with image blocks.

[0199] Input the image into the pre-trained Context Encoder model, perform inference calculations, and obtain the output results. Disable gradient calculations during inference to improve efficiency and save memory.

[0200] Gaussian blur is used to eliminate the edge effect of the completed panoramic image to obtain the final panoramic image.

[0201] Use the cv2.GaussianBlur function in OpenCV to make the completed part and the original image transition smoothly and reduce the edge effect.

[0202] Specifically, according to the second completion method, a high-quality and natural image filling effect can be achieved, so that the candidate image and the panoramic image form a visually consistent and seamless fusion.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. The image processing method based on AI large model is characterized by: The steps include: S1: collect multiple images and perform preprocessing; S2: construct a deep learning model, use the deep learning model to extract feature points of multiple images respectively, and use a feature matching algorithm to match the feature points of multiple images to obtain a first matching result, use a screening algorithm to screen the first matching pair for mismatching, obtain a second matching result, apply affine transformation to the second matching result, and complete the registration of multiple images; S3: Use the image fusion model to fuse multiple registered images to generate a panoramic image; S4: Identify the blank areas of the panoramic image, and complete the blank areas based on a completion method to obtain a final panoramic image.

2. The image processing method based on the AI ​​large model according to claim 1, characterized in that: Step S2 includes: S21: Use VGG network to build a deep learning model, input multiple pictures into the deep learning model in sequence, and extract SIFT feature points; S22: Pair the multiple pictures into pairs to form a picture group; S23: Select a picture group; S24: The SIFT feature points of the pictures I1 and I2 in the picture group are recorded as the first feature point set {S1} and the second feature point set {S2}, respectively, and the correlation coefficients between each SIFT feature point in {S1} and all SIFT feature points in {S2} are calculated, and a candidate second feature point set for each SIFT feature point in {S1} is obtained based on the values ​​of the correlation coefficients, i.e., the first matching result; S25: Select a first feature point S1 in the first matching result i and its candidate second feature point set S26: Sort the candidate second feature point set in descending order according to the value of the correlation coefficient and select the first two candidate second feature points Calculate S1 i respectively with to obtain the distances d1 and d2. If d1 / d2 < T1, then use the candidate second feature point as the matching point of S1 i and add it to the second matching result; S27: repeating steps S25-S26 until the first matching result is screened and a second matching result of the picture group is obtained; S28: constructing an affine transformation model based on the second matching result, taking the image I1 as the reference image, and registering the image I2 using the affine transformation model; S29: Repeat steps S23-S28 until all image groups are processed and image registration is completed.

3. The image processing method based on the AI ​​large model according to claim 2, characterized in that: Step S24 includes: S241: SIFT feature points of the picture I1 and the picture I2 in the picture group are recorded as a first feature point set {S1} and a second feature point set {S2} respectively, and a descriptor of each feature point is extracted to form a first descriptor set {D1} and a second descriptor set {D2}; S242: performing local hash processing on the first descriptor set {D1} and the second descriptor set {D2} respectively, and mapping the first descriptor set {D1} and the second descriptor set {D2} to multiple hash buckets through local hashing; S243: Perform preliminary matching in the hash buckets, find the feature point pairs in each hash bucket, and obtain preliminary matching results {D1 k ,D2 m }, where D1 k Denotes the descriptor in {D1}, D2 m represents the descriptor in {D2}; S244: Calculate the similarity coefficient of each pair of feature points in the preliminary matching result using a similarity formula, wherein the calculation formula of the correlation coefficient is: In the formula, R(D1 k ,D2 m ) is D1 k With D2 m The correlation coefficient, f1 j D1 k The jth component of j For D2 m The jth component of , n is the dimension of the descriptor, ω j is the weight of the j-th dimension; S245: Set a correlation threshold T, compare the correlation coefficient value with the correlation threshold τ, and select feature point pairs corresponding to correlation coefficients greater than or equal to the correlation threshold τ as the first matching result.

4. The image processing method based on the AI ​​large model according to claim 2, characterized in that: Step S3 includes: Constructing an image fusion model, including a first network branch and a second network branch; First, two images are selected from the multiple registered images and input into the image fusion model for image fusion to obtain a fused image. Then, the single images and the fused image are input in sequence for fusion again until the multiple registered images are fused to obtain a panoramic image. in: The first network branch is used to process the two images to obtain a preliminarily reconstructed spliced ​​image. The first network branch adopts an encoder-decoder network. The encoder includes a first network layer, a second network layer and a third network layer. Each network layer includes a convolution layer and a pooling layer. The encoder is used to extract low-level features; the decoder includes a first deconvolution layer, a second deconvolution layer and a third deconvolution layer. The decoder is used to generate high-level features; a jump connection is set between the network layer of the encoder and the deconvolution layer of the decoder to connect the low-level features with the high-level features; The second network branch is used to process the initially reconstructed spliced ​​image to obtain a fused image. The second network branch includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a residual unit. The residual unit includes 4 residual blocks. Each residual block has the same structure, including convolutional layer 1, ReLU activation function 1, convolutional layer 2, residual connection and ReLU activation function 2.

5. The image processing method based on the AI ​​large model according to claim 4, characterized in that: The first network branch processes the two images to obtain a preliminary reconstructed spliced ​​image as follows: Input two images I1 and I2 into the first network branch. The original size of the two images is H*W. Downsample the two images and convert the size to 256*256. The first, second, and third network layers of the encoder are used to extract the low-level features of I1 and I2 in turn, and three low-level features of I1 at different scales are obtained. and And three low-level features of different scales of I2 and Will and To splice, and To splice, and Perform splicing to obtain the first fusion feature F1, the second fusion feature F2 and the third fusion feature F3; The first fused feature F1, the second fused feature F2 and the third fused feature F3 are input into the decoder, the third fused feature F3 is upsampled by the first deconvolution layer to make its size the same as the second fused feature F2, and the upsampled third fused feature F3 is added to the second fused feature F2 via a jump connection to obtain the first updated fused feature F′1; The first updated fusion feature F′1 is upsampled by the second deconvolution layer to make its size the same as the first fusion feature F1, and the upsampled first updated fusion feature F′1 is added to the first fusion feature F1 via a jump connection to obtain the second updated fusion feature F′2; The third deconvolution layer is used to upsample the second updated fusion feature F′2 to restore its size to 256*256 to obtain the final fusion feature; A 1*1 convolutional layer is used to map the final fused features to the target number of channels to obtain a preliminary reconstructed spliced ​​image.

6. The image processing method based on the AI ​​large model according to claim 5, characterized in that: The second network branch processes the initially reconstructed spliced ​​image to obtain the fused image as follows: Use bilinear interpolation to upsample the initially reconstructed spliced ​​image to the original size H*W; The first convolution layer, the second convolution layer, and the third convolution layer are used to extract features of the upsampled spliced ​​image in turn, and the first layer features A1, the second layer features A2, and the third layer features A3 are obtained; The third layer feature A3 is input into the residual unit for processing, the third layer feature A3 is processed by the first residual block to obtain the first residual feature, the first residual feature is input into the second residual block, the second residual feature is input into the third residual block, the third residual feature is output, the third residual feature is input into the fourth residual block, and the final residual feature is output; The first layer feature A1 is fused with the final residual feature to obtain the fused feature; Perform convolution operation on the fused features and output the fused image.

7. The image processing method based on the AI ​​large model according to claim 1, characterized in that: Step S4 includes: S41: Use the U-Net segmentation model to segment the panoramic image and obtain the blank area; S42: Obtaining location information of the blank area, and searching on the Internet through a search engine according to the location information; S43: if a candidate image is found, the first completion method is used to complete the blank area based on the candidate image; if no candidate image is found, the second completion method is used to complete the blank area; S44: Output the final panoramic image.

8. The image processing method based on the AI ​​large model according to claim 7, characterized in that: Step S42 includes: The contour of the blank area is extracted by edge detection algorithm, and the contour information is optimized by morphological operation to obtain the shape characteristics of the blank area; Use a regular grid to mark the relative position of the blank area in the panoramic image to obtain position features; Taking the outline of the blank area as the initial point, the outer contour is obtained after expanding according to the preset step length, and the feature information in the outer contour is extracted as the environmental feature of the blank area; The shape features, location features and environmental features are combined into the basic features of the blank area; Use basic features to perform a rough search in the search engine to obtain initial candidate images; Extract the features of the initial candidate images to obtain candidate features; The similarity between the basic features of the blank area and the candidate features in the candidate feature set is calculated one by one, and the most similar picture is taken as the candidate picture.

9. The image processing method based on the AI ​​large model according to claim 8, characterized in that: The first completion method includes: Crop and deform the candidate image to make it the same shape as the blank area; The area surrounding the outline is used as a buffer, the color histogram of the candidate image and the buffer is calculated, and the color distribution of the candidate image is adjusted according to the histogram matching algorithm; Use the local color difference correction algorithm to fine-tune the color histogram matching results to obtain the candidate images after color matching; Create a mask for the blank area and perform Gaussian blur on the mask edge to generate a feathered mask with a gradient effect; Define the center of the blank area as the embedding center, use the cv2.seamlessClone function, combined with the feathering mask, perform Poisson Blending, and generate a preliminary fused image; The preliminary fused image is downsampled using a Gaussian pyramid to construct a multi-resolution hierarchy. At each resolution level, the panoramic image and the preliminary fused image are fused layer by layer to generate the fusion result of each layer. The image is upsampled layer by layer and multi-resolution levels are superimposed to reconstruct the final fused image from the bottom layer to obtain the final panoramic image.

10. The image processing method based on AI large model according to claim 8, characterized in that: The second completion method includes: In the panoramic image, extract the valid non-blank area around the blank area as the reference area; Perform multiple spatial transformations on the reference area to generate image blocks of different perspectives, and use structural similarity to select the most suitable perspective from the image blocks of different perspectives to obtain the image block with the best perspective; Load the pre-trained Context Encoder model, input the panoramic image and the best view image block, the model completes the blank area in the panoramic image according to the context information of the best view image block, and outputs the completed panoramic image; Gaussian blur is used to eliminate the edge effect of the completed panoramic image to obtain the final panoramic image.

Citation Information

Patent Citations

  • Semispherical panorama photographing method, apparatus and terminal

    CN105827978A

  • Improved image matching and mismatching elimination algorithm

    CN110443295A

  • Multi-temporal remote sensing image automatic registration method based on improved SIFT algorithm

    CN114494378A

  • Model training method, video processing method, computer equipment and medium

    CN114677623A

  • Image splicing method, system and device based on deep learning and medium

    CN116934592A