End-to-end image registration and black edge completion method based on deep learning

Abstract

The invention relates to an end-to-end image registration and black edge completion method based on deep learning. According to the method, image registration and black edge completion tasks are integrated into a combined deep learning network, and the image registration precision and the completion effect are optimized through end-to-end training, so that the naturalness and consistency of image quality are improved. According to the method, the completion strategy of the generative adversarial network is combined with the lightweight feature matching enhancement module, the completion region consistent with the content of the original image can be efficiently and accurately generated, and seamless fusion of the completion region and the original image is ensured. Meanwhile, a joint optimization strategy is adopted, and through a multi-scale feature extraction module and a joint loss function, when registration and completion tasks are optimized, the calculation complexity is reduced, and the overall effect is improved. The method can effectively improve the overall quality of image registration and completion, and is widely applied to multiple fields of medical image processing, remote sensing image analysis, computer vision and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an end-to-end image registration and black border completion method based on deep learning. BACKGROUND

[0002] Image registration is one of the basic tasks in computer vision, and its goal is to align two or more images for comparison or fusion in the same coordinate system. Traditional image registration methods, such as feature point matching-based registration and frequency domain-based methods, usually rely on geometric transformations (such as rotation, translation, and scaling) to align images. However, in these methods, due to the transformation of pixel coordinates and interpolation calculation, black regions often appear at the edges or backgrounds of the images, which are called "black borders". These black border regions cannot obtain effective information from the original image, and filling these regions often relies on additional post-processing steps, such as simple interpolation methods or texture synthesis-based completion techniques. Such a process not only is inefficient, but also easily leads to the fragmentation of image registration and completion tasks, and cannot form a consistent processing process.

[0003] Existing image completion methods mainly include interpolation methods, texture synthesis-based methods, and deep learning completion methods that have emerged in recent years. Interpolation methods such as bilinear interpolation and cubic interpolation can effectively fill black borders in simple scenarios, but in complex scenarios, especially in areas with a large amount of texture or structure, interpolation methods often cannot maintain the natural consistency of the completed region with the original image. Texture synthesis-based methods fill black borders by extracting texture information from other regions of the image, but this method has high computational complexity and is difficult to handle large-scale images or high-dimensional data, and relies on the local structure of image content, often leading to poor completion results.

[0004] With the development of deep learning technology, methods based on convolutional neural networks (CNN) and generative adversarial networks (GAN) have gradually become the mainstream in the field of image completion. These methods can generate more natural completed regions by learning the underlying patterns in large amounts of data. However, current deep learning models still optimize image registration and completion tasks independently, often ignoring the inherent relationship between the two tasks. The registration task focuses on calculating the geometric transformation between images, while the completion task focuses on filling the missing regions after transformation. Due to the lack of end-to-end joint optimization, these two tasks often need to be handled separately and optimized in different networks, which leads to the accumulation of registration errors and the inconsistency of completion results, thereby affecting the quality of the final image.

[0005] In addition, existing deep learning methods usually train each task separately and use different loss functions to optimize, which cannot fully utilize the registration information to improve the quality of completion. With the increase of image resolution, the computational complexity and resource consumption also present exponential growth, making the application of these methods in large-scale image processing limited.

[0006] In summary, the prior art has the problems of independent optimization of registration and completion tasks, low efficiency and poor consistency. SUMMARY

[0007] To solve the problems in the prior art, the present application proposes an end-to-end image registration and black border completion method based on deep learning, which combines registration and completion tasks into a unified optimization process through a deep learning framework. This end-to-end joint optimization can fully utilize the registration information, optimize the registration accuracy and improve the completion effect through multi-scale feature extraction modules and joint loss functions, solving the problem of fragmentation between registration and completion tasks. The method of the present application can be used to accurately complete the images whose content is missing due to deformation caused by registration, achieving advanced completion performance. The method of the present application provides a new idea for efficient solution of image registration and completion tasks, and has important technical value and application potential. The method not only has wide application prospects in high-precision application fields such as medical images and remote sensing images, but also can play an important role in large-scale image processing tasks.

[0008] The present application adopts the following technical solutions:

[0009] A method of end-to-end image registration and black border completion combination is realized based on an end-to-end image registration and black border completion model, and the construction method of the end-to-end image registration and black border completion model specifically includes the following steps:

[0010] S1: Obtain multi-scale features of an input image and a target image respectively;

[0011] S2: Perform feature fusion and feature enhancement on the two kinds of multi-scale features extracted in step S1 respectively in sequence to obtain enhanced features;

[0012] S3: Calculate a geometric transformation matrix between the input image and the target image based on the enhanced features, register the input image to the target image according to the geometric transformation matrix, and generate a registered image;

[0013] S4: Based on the registered image generated in step S3, use a generative adversarial network to complete the black border area in the registered image;

[0014] S5: Optimize the end-to-end image registration and black border completion model through a joint loss function to obtain a trained end-to-end image registration and black border completion model.

[0015] Specifically, in the step S1, the acquisition of the multi-scale features is realized based on a multi-scale feature extraction module, and the specific method is as follows: assuming that an input image is H0xW0 is the height and width of the image, the channel number is C0, and after a plurality of convolution and pooling operations, multi-scale features are obtained wherein l represents the number of the convolution layer. The convolution operation of each layer can be expressed as: I l = σ(Conv l (I l-1 )) wherein wherein, I l is the output of the convolution layer l, Conv l (I l-1 ) represents the convolution operation on the output I l-1 of the previous convolution layer, and σ is an activation function such as ReLU. For the pooling operation, the maximum pooling (MaxPooling) is used, and the calculation formula is as follows:

[0016]

[0017] wherein p is the size of the pooling window, usually 2x2 or 3x3, and the maximum pooling operation takes the maximum value of each region in the image as the output, thereby realizing the dimension reduction and scale invariance of the features.

[0018] Specifically, in the step S2, the registration sub-network comprises a lightweight feature matching enhancement module for improving the matching accuracy of the key points. The two kinds of multi-scale features extracted in the step S1 are subjected to feature fusion and feature enhancement in sequence by using the registration sub-network to obtain enhanced features, and the specific operation steps are as follows:

[0019] 1) the multi-scale features obtained in S1 are uniformly up-sampled to the same resolution by using the bilinear interpolation, and are subjected to channel splicing to form a fusion feature map containing multi-scale information.

[0020] 2) the fusion feature map is input into the lightweight feature matching enhancement module, and is subjected to channel attention and spatial attention processing in sequence to obtain enhanced features.

[0021] The operation steps of the channel attention and spatial attention processing are as follows:

[0022] 1) channel attention calculation: the fusion feature map F is subjected to global average pooling and global maximum pooling in the spatial dimension to obtain two vectors, the two vectors are respectively input into a shared MLP to generate a channel weight matrix W c ; W c is multiplied with the fusion feature map F in the channel to obtain a channel-weighted feature map F c ,

[0023]

[0024] 2) Spatial attention calculation: feature map F after channel weighting c Perform average pooling and maximum pooling on the channel dimension respectively to obtain two feature maps, which are then concatenated along the channel dimension to form a composite feature map, and a spatial weight matrix W is generated through convolution operation. s ;W s And the channel-weighted feature map F c Multiply by channel to get the spatially weighted feature map F s This is the enhanced feature.

[0025] The step S3 specifically includes the following steps:

[0026] 1) Calculate the cosine similarity matrix S(x) between the input image and the enhanced features corresponding to the target image i ,x j ):

[0027]

[0028] Among them, F A (x i ) and F B (x j ) are the enhanced features corresponding to the input image at position x i The enhanced feature corresponding to the target image is at position x j vector.

[0029] 2) Filter matching pairs with confidence ≥ 0.8, and use the RANSAC algorithm (iteration 1000 times, inlier threshold 2 pixels) to remove outliers and retain inlier matching pairs.

[0030] 3) Based on the interior point matching pairs, the transformation matrix is ​​solved by the least squares method:

[0031]

[0032] in, and H k They represent the final estimated value of the geometric transformation matrix and the geometric transformation matrix corresponding to the kth iteration respectively;

[0033] 4) Generate the registered image according to the geometric transformation matrix. Specifically, the input image will be geometrically transformed according to H to obtain the registered image I registered , and its calculation formula is:

[0034]

[0035] wherein, is the inverse matrix of the transformation matrix , representing the mapping of the coordinates of the target image back to the coordinates of the input image; I represents the target image; I registered (x ′ ,y ′ ) is the pixel value of the registered image I registered at (x ′ ,y ′ ).

[0036] Specifically, step S4 inputs the registered image generated in step S3 into the completion subnetwork, and uses a generative adversarial network (GAN) to complete the black border region in the registered image, and the steps are as follows:

[0037] 1) input the registered image I registered generated in step S3 into the completion subnetwork;

[0038] 2) the generator in the completion subnetwork (generative adversarial network) receives the registered image I registered and the corresponding mask M as input, wherein M is a binary mask, and the black border region in the image is 1, and other regions are 0, and is specifically represented as:

[0039]

[0040] wherein, M1 is a mask with all 1s and has the same resolution as the registered image I registered , and M(x ′ ,y ′ ) is the pixel value of the mask M at (x ′ ,y ′ );

[0041] 3) the generator completes the black border region in the registered image through an encoder-decoder architecture (such as U-Net), and outputs the completed image I generated .

[0042] Specifically, in step S5, the joint loss function specifically includes a registration loss and a completion loss, and each loss function is as follows:

[0043] 1) registration loss L registration : use mean square error (MSE) to measure the pixel difference between the registered image and the true value image:

[0044]

[0045] wherein, N is the total number of pixels of the image; I registered (i,j) and I gt are the registered image Iregistered and ground truth image I gt Pixel value at position (i, j).

[0046] 2) Completion loss: including structural similarity loss and adversarial loss.

[0047] Structural similarity loss (L SSIM ): measures the structural similarity of the completed region and the corresponding part of the ground truth image, the formula is:

[0048]

[0049] Where, μ I , μ J are the mean of the completed image and the ground truth image, σ I , σ J are the variance of the completed image and the ground truth, σ IJ is the covariance of the completed image and the ground truth, c1, c2 are constants to prevent the denominator from being zero; adversarial loss L GAN : uses a generative adversarial network (GAN) to optimize the authenticity of the completed region. The adversarial loss is calculated by the output of the discriminator:

[0050]

[0051] Where, I gt is the ground truth image, I generated is the generated image, I generated = G(I registered , M); D(·) is the discriminator, which receives the completed image I registered and judges the similarity of the completed region and the corresponding region of the ground truth image, outputting the true or false score of each small block; the output of the discriminator is used to optimize the generator, so that the completed region is more natural and seamlessly connected with other parts of the image; D(I gt ) is the score of the discriminator judging whether I gt is real or not; D(G(I registered , M)) is the true or false score, which is used to judge the similarity of the completed region and the corresponding region of the ground truth image I gt .

[0052] 3) Joint loss function: combine the registration loss, structural similarity loss and adversarial loss to get the final joint loss function L total :

[0053] L total = λ1L registration + λ2L SSIM + λ3L GAN

[0054] Wherein, λ1, λ2, λ3 are hyperparameters, respectively used for controlling the weights of the registration loss, the structural similarity loss and the adversarial loss.

[0055] Specifically, the above-mentioned trained end-to-end image registration and black border completion model is used for image registration and black border region completion, thereby generating a final complete image. Specifically:

[0056] 1) load the optimized model parameters;

[0057] 2) input each pair of input image and target image into the end-to-end image registration and black border completion model;

[0058] 3) extract multi-scale features of the input image and the target image respectively;

[0059] 4) calculate a geometric transformation matrix based on the multi-scale features, register the input image to the target image, and generate a registered image;

[0060] 5) generate a completed image based on the registered image, which is a complete image.

[0061] Compared with the prior art, the present application has the following beneficial effects:

[0062] In the prior art, image registration and black border completion tasks are usually optimized independently, resulting in accumulated registration errors and inconsistent completion effects. The present application integrates image registration and black border completion tasks into a joint deep learning network, optimizes through end-to-end training and joint loss function, and simultaneously improves image registration accuracy and completion effect; reduces error accumulation, improves consistency between registration and completion tasks, and makes the completed region more natural and seamlessly connected to the original image. In complex image scenarios such as medical images and remote sensing images, the present application can more accurately align images and complete black borders, providing a high-quality image basis for subsequent analysis and application.

[0063] Secondly, the present application uses a multi-scale feature extraction module to avoid repeated feature extraction processes, and the design of the joint loss function simplifies the overall network structure. Compared with the prior art of training different networks and using different loss functions, the present application greatly reduces computational complexity and resource consumption. In processing large-scale image data, the present application significantly reduces the time and computational resources required for model training and inference, improves processing efficiency, and is more suitable for real-time requirements in practical applications.

[0064] Finally, based on the completion strategy of the generative adversarial network (GAN), combined with the lightweight feature matching enhancement module, the black border area can be effectively filled, so that the generated completion area is more natural and consistent with the content of the original image, and the seamless docking of the completion area and other parts of the image can be realized, thereby improving the visual quality of the image. In the panoramic image stitching scene, the black border area after completion is naturally transitioned with the surrounding environment, and there is no obvious stitching mark, which effectively improves the visual quality and integrity of the image and enhances the usability of the image in various applications. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The construction flowchart of the end-to-end image registration and black border completion model provided by the embodiment of the application is shown in the figure.

[0066] Figure 2 The completion effect diagram of the application on the map-satellite image dataset is shown in (a), (b) and (c) from left to right, which are modal A image, modal B image (partially missing), and completed modal B image. DETAILED DESCRIPTION

[0067] The specific embodiment of the application will be further described below in combination with related principles and processes.

[0068] The end-to-end image registration and black border completion method based on deep learning of the application is realized based on an end-to-end image registration and black border completion model. As shown in the figure, the end-to-end image registration and black border completion model of the application is constructed. Figure 1 The construction method of the end-to-end image registration and black border completion model of the application specifically includes the following steps:

[0069] 1. Data preparation: collect image data for training and testing, covering medical images, remote sensing images, panoramic images and other types. Adjust the collected images to a uniform resolution, such as 256x256 pixels, and perform normalization processing to map the pixel values to the [0,1] interval. Divide the training set and the test set in a ratio of 8:2.

[0070] 2. Construction of network model: multi-scale feature extraction module: a multi-scale feature extraction module is constructed using a convolutional neural network (CNN). The convolution kernel size of the convolution layer is set to 3x3, the step is 1, and the padding is 1; the activation function is ReLU. The pooling layer uses the maximum pooling operation, and the pooling window size is 2x2 and the step is 2. Specifically, the multi-scale feature map includes three levels. Through three layers of convolution and pooling operations, shallow, middle and deep feature maps are extracted respectively. The shallow feature map extracted by the first convolution layer, the size is 1 / 4 of the input image (i.e. 64x64 pixels), and the number of channels is 64. The middle layer feature map Through the second convolutional layer extraction, the size is 1 / 8 (i.e. 32x32 pixels), and the number of channels is 128. Deep feature map Through the third convolutional layer extraction, the size is 1 / 16 (i.e. 16x16 pixels), and the number of channels is 256.

[0071] Registration sub-network: in the lightweight feature matching enhancement module, the shallow, medium and deep three-layer feature maps are uniformly up-sampled to 64x64 resolution through bilinear interpolation, and then channel splicing is performed to form a feature map that fuses multi-scale information When calculating the channel attention, the outputs of the global average pooling and the global maximum pooling are two 1x1x4481x1x448 vectors, which are respectively input into the shared MLP (multi-layer perceptron) to generate the channel weight matrix The number of neurons in the hidden layer of the MLP is set to 128. W c is multiplied with the feature map F in the channel to obtain the channel-weighted feature map F c . When calculating the spatial attention, the channel-weighted feature map F c is subjected to average pooling and maximum pooling in the channel dimension to obtain two 64x64x1 feature maps, the two feature maps are spliced along the channel dimension to form a 64x64x2 composite feature map, and a spatial weight matrix is generated through a 7x7 convolution kernel W c is multiplied with the feature map F in the channel to obtain the spatial-weighted feature map f s . The number of iterations of the RANSAC algorithm is set to 1000 times, and the inlier threshold is set to 2 pixels.

[0072] The completion sub-network: the generator adopts the U-Net architecture, and the convolution kernel size of the encoder part is 3x3, 3x3, 3x3, and 3x3 in turn, and the step size is 2; the transposed convolution kernel size of the decoder part is 2x2, 2x2, 2x2, and 2x2 in turn, and the step size is also 2. The discriminator can adopt the PatchGAN structure, and the convolution kernel size is 4x4, and the step size is 2.

[0073] 3. Training network: during the training process, the image pairs (input image and target image) in the training set are input into the network in turn. The joint loss function L total = λ1L registration + λ2L SSIM + λ3L GAN , where the hyperparameters λ1, λ2 and λ3 are set to 0.5, 0.3 and 0.2 respectively. The Adam optimizer is used, the initial learning rate is set to 0.0001, and the exponential decay strategy is adopted, the decay rate is 0.99, and the decay is performed once every 10 epochs. During the training process, 100 batches of data are trained every epoch, and each batch of data contains 16 image pairs.

[0074] 4. Testing and Application: After network training is complete, load the trained model parameters into the model. Input the image pairs in the test set into the network sequentially, and perform image registration and black edge completion operations according to the process in the training phase. In actual application scenarios, such as medical image processing, medical images of different modalities (such as CT and MRI images) are registered and black edge completion is performed; in remote sensing image analysis, remote sensing images acquired at different times and from different perspectives are processed; in panoramic image stitching, the stitched panorama is registered and black edge completion is performed.

[0075] Image registration and black edge completion are performed based on the trained end-to-end joint image registration and black edge completion model. The specific method is as follows: First, the input and target images enter the multi-scale feature extraction module, where a convolutional neural network (CNN) is used to extract shallow, mid-level, and deep feature maps. These feature maps are upsampled and concatenated to generate a fused feature map. This fused feature map then enters the lightweight feature matching enhancement module, where keypoint matching accuracy is improved through mechanisms such as channel attention and spatial attention. Next, the registration subnetwork calculates the geometric transformation matrix based on the processed features to generate the registered image. The registered image is then fed into the completion subnetwork, which utilizes a generative adversarial network (GAN) consisting of a U-Net generator and a PatchGAN discriminator to fill in the black edge regions of the image. Finally, the complete image is output, achieving the joint processing of image registration and black edge completion, effectively solving the black edge filling problem in traditional methods and improving image quality.

[0076] like Figure 2 Figures (a), (b), and (c) show, from left to right, the Modality A image, the registered Modality B image (with black edges generated by the registration operation), and the completed Modality B image. The completed Modality B image effectively completes the black edges and is well aligned with the Modality A image, showing consistent content. This demonstrates that the method of the present invention effectively achieves transmembrane image registration and completion.

[0077] The present invention has the following advantages: 1) Jointly optimizing the registration and completion tasks can significantly improve the accuracy and completion effect of image registration; 2) The design of the multi-scale feature extraction module and the joint loss function simplifies the network structure and reduces computational overhead; 3) The completion strategy based on the generative adversarial network (GAN) makes the generated image more natural, and the completed area is seamlessly connected with the rest of the image; 4) The method is applicable to a variety of complex scenes such as medical images, remote sensing images, and panoramic image stitching, and has broad application prospects. The present invention can effectively improve the overall quality of image registration and completion, and is widely used in multiple fields such as medical image processing, remote sensing image analysis, and computer vision.

[0078] The above describes the present application and its embodiments, which are not limited, and the drawings only show one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired thereby, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution are not creative, and should belong to the protection scope of the present application.

Claims

1. A deep learning-based end-to-end image registration and black edge completion method, characterized by: Based on an end-to-end image registration and black edge completion model, the construction method of the end-to-end image registration and black edge completion model specifically includes the following steps: S1: Obtain multi-scale features of the input image and target image respectively; S2: performing feature fusion and feature enhancement on the two multi-scale features extracted in step S1 to obtain enhanced features; S3: Calculate the geometric transformation matrix between the input image and the target image based on the enhanced features, register the input image to the target image according to the geometric transformation matrix, and generate a registered image; S4: Based on the registered image generated in step S3, the black edge area in the registered image is completed using a generative adversarial network; S5: Optimizing the end-to-end image registration and black edge completion model through a joint loss function to obtain a trained end-to-end image registration and black edge completion model.

2. The method according to claim 1, characterized in that In step S1, the acquisition of multi-scale features is achieved based on a multi-scale feature extraction module, and the specific method is as follows: For images H0×W0 is the height and width of the image, C0 is the number of channels, and after several convolution and pooling operations, multi-scale features are obtained. Where l represents the number of the convolution layer; the convolution operation of each layer is expressed as: I l =σ(Conv l (I l-1 )); among them, I l is the output of convolutional layer l, Conv l (I l-1 ) represents the output I of the previous convolutional layer l-1 Perform convolution operation, σ is the activation function.

3. The method according to claim 1, characterized in that In step S2, the feature fusion operation is specifically as follows: The multi-scale features obtained by S1 are uniformly upsampled to the same resolution through bilinear interpolation, and channel splicing is performed to form a fused feature map containing multi-scale information.

4. The method according to claim 3, characterized in that In step S2, the feature enhancement is implemented based on the lightweight feature matching enhancement module. The specific operation steps are as follows: The fused feature map is input into the lightweight feature matching enhancement module, and is processed by channel attention and spatial attention in sequence to obtain enhanced features.

5. The method according to claim 4, characterized in that The enhanced features are obtained by sequentially processing channel attention and spatial attention. The specific operation steps are as follows: 1) Channel attention calculation: Perform global average pooling and global maximum pooling on the fusion feature map F in the spatial dimension to obtain two vectors, which are input into the shared MLP to generate the channel weight matrix W. c ;W c Multiply the fusion feature map F by channel to obtain the channel-weighted feature map F c , 2) Spatial attention calculation: feature map F after channel weighting c Perform average pooling and maximum pooling on the channel dimension respectively to obtain two feature maps, which are then concatenated along the channel dimension to form a composite feature map, and a spatial weight matrix W is generated through convolution operation. s ;W s And the channel-weighted feature map F c Multiply by channel to get the spatially weighted feature map F s This is the enhanced feature.

6. The method according to claim 1, characterized in that The step S3 specifically includes the following steps: 1) Calculate the cosine similarity matrix S(x) between the input image and the enhanced features corresponding to the target image i ,x j ): Among them, F A (x i ) and F b (x j ) are the enhanced features corresponding to the input image at position x i The enhanced feature corresponding to the target image is at position x j vector of 2) Filter matching pairs x with cosine similarity ≥ 0.8 j and x i And use the RANSAC algorithm to remove outliers and retain the inlier matching pairs; 3) Based on the interior point matching pairs, the geometric transformation matrix is ​​solved by the least squares method: in, and H k They represent the final estimated value of the geometric transformation matrix and the geometric transformation matrix corresponding to the kth iteration respectively; 4) Generate the registered image I according to the geometric transformation matrix registered , and its calculation formula is: in, is the geometric transformation matrix The inverse matrix of I represents the target image; I registered (x′, y′) is the image after registration I registered The pixel value at (x′, y′).

7. The method according to claim 6, characterized in that Step S4 is specifically as follows: 1) The registered image I generated in step S3 registered Enter the Generative Adversarial Network; 2) The generator in the generative adversarial network receives the registered image I registered And the corresponding mask M as input, where the mask M is a binary mask, marking the black edge area in the image as 1 and other areas as 0, specifically expressed as: Among them, M1 is a mask with all 1s and its resolution is the same as the registered image I registered Similarly, M(x′, y′) is the pixel value of mask M at (x′, y′); 3) The generator completes the black edge area in the registered image through the encoder-decoder architecture and outputs the completed image I generated .

8. The method according to claim 7, characterized in that In step S5, the joint loss function specifically includes registration loss and completion loss, which are as follows: 1) Registration loss L registration : Use the mean square error to measure the pixel difference between the registered image and the true value image: Where N is the total number of pixels in the image, I registered (i, j) and I gt (i,j) are the registered images I registered and the true value image I gt The pixel value at position (i, j); 2) Completion loss: including structural similarity loss and adversarial loss; Structural similarity loss L SSIM : Measures the structural similarity between the completed area and the corresponding part of the true image. The formula is: Among them, μ I ,μ J is the mean of the completed image and the true image, σ I ,σ J is the variance between the completed image and the true value, σ IJ is the covariance between the completed image and the true value, c1 and c2 are constants used to prevent the denominator from being zero; Adversarial loss L GAN : Generate adversarial networks to optimize the authenticity of the completed area, and the adversarial loss is calculated through the output of the discriminator: Among them, I TG is the true value image, I generated For the generated image, I generated =G(I registered ,M), D(·) is the discriminator; D(I gt ) is the discriminator's judgment I gt Is the rating real? registered ,M)) is the true and false score, which is used to judge the completed area and the true value image I gt Similarity of corresponding regions; 3) Joint loss function: Combine the registration loss, structural similarity loss and adversarial loss to obtain the final joint loss function L total : L total =λ1L registration +λ2L SSIM +λ3L GAN Among them, λ1, λ2, and λ3 are hyperparameters, which are used to control the weights of registration loss, structural similarity loss, and adversarial loss, respectively.

9. The method according to any one of claims 1 to 8, characterized in that The specific image registration and black edge completion methods are as follows: 1) Load the optimized model parameters; 2) inputting each pair of input image and target image into the end-to-end image registration and black edge completion model; 3) Extract multi-scale features of the input image and target image respectively; 4) Calculate the geometric transformation matrix based on the multi-scale features, register the input image to the target image, and generate the registered image; 5) Generate a completed image based on the registered image, that is, a complete image.

Images