Image completion method based on fast Fourier transform and selective attention mechanism

Abstract

The present invention relates to an image completion method based on deep learning. The image completion method proposes an efficient completion network with a built-in Fourier transform operation, which ensures the high-efficiency operation of the model while realizing long-distance dependence. At the same time, the completion network is embedded with a fast Fourier transform module of an aggregated selective convolution kernel network, wherein the fast Fourier transform module is responsible for the completion network to obtain the global receptive field, and the aggregated selective convolution kernel network adaptively adjusts the proportion according to the contribution of each scale, providing the model with more accurate local information supplement, and finally achieving image completion with higher global fusion and richer local details. Compared with the existing common completion methods, the method of the present invention generates images with better restoration and clarity, especially when the occlusion rate is large, and generates local semantic coherence of the image and global overall fit.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image completion method based on fast Fourier transform and selective attention mechanism. Background Art

[0002] Image completion involves using existing information and model algorithms to infer and generate a complete image when part of the image data is missing. Image completion technology based on fast Fourier transform modules and aggregated selective convolutional networks can be applied to multiple fields, such as cultural relic restoration, object removal, and criminal investigation. In the field of cultural relics, this technology can be used to restore damaged paintings, improving the quality and accuracy of restoration. In the field of image object removal, this technology can be used for intelligent video analysis and image restoration, improving the real-time performance and accuracy of surveillance systems. In the field of criminal investigation, it can be used for evidence photo restoration and face completion to assist in case investigations.

[0003] With the advancement of science and technology and the widespread adoption of artificial intelligence, this technology has broad application prospects across various fields. In the restoration of cultural relics, it can improve efficiency and quality, reducing the difficulty of repairing large areas of damaged artifacts. In the field of object removal, it can enhance the effectiveness of object removal, ensuring that the image maintains a high degree of global consistency after the object is removed. In criminal investigation, it can reduce errors in evidence restoration while improving the efficiency of the investigation process. Therefore, image completion technology based on fast Fourier transform and selective attention mechanism has broad application prospects and market demand.

[0004] Image completion technology involves using existing information and model algorithms to infer the missing image data to generate a complete image. Fast Fourier transform and selective attention mechanism are two key technologies currently widely used in the field of image completion.

[0005] The Fast Fourier Transformer (FFT) is an efficient and fast algorithm. This algorithm primarily refers to the discrete Fourier transform (DFT), which transforms signals from one domain to another. The FFT's operational goal is to map spatial features to a spectral domain, then effectively globally update the resulting spectral data, and finally convert the processed data back to spatial features. Compared to convolution, the FFT can help convert images between the Fourier (i.e., frequency) and spatial domains. Since spectral domain transformations are performed by updating all pixels, the FFT helps the model expand the receptive field of a network layer to all pixels in the image, achieving a global receptive field. This improves the information correlation across all pixels, ultimately enabling efficient information exchange between distant pixels. Therefore, this technology is suitable for solving problems such as image completion, where globally consistent results are crucial.

[0006] Furthermore, numerous experiments have demonstrated that image completion algorithms incorporating an attention mechanism can improve completion efficiency and quality. The attention mechanism, in a neural network, weights input information to focus attention on the most useful parts for the task at hand. The attention mechanism primarily achieves two functional goals: determining which parts of the input image require attention, and allocating limited information processing resources to the most important parts of the object being processed. The attention mechanism can selectively prioritize critical information while ignoring non-critical information, which improves network performance. Extensive experiments have demonstrated that image completion algorithms incorporating an attention mechanism can improve completion efficiency and quality. The present invention also incorporates an attention mechanism. Furthermore, to enhance the anti-interference capabilities of the completion network, the present invention employs a clustered selective convolutional kernel network as its attention mechanism module. This attention mechanism weights features processed by convolution kernels of different scales to adaptively generate corresponding convolution kernel combinations, thereby adjusting the network's receptive field to help the network adapt to changes in the input image scale and missing regions. This attention mechanism effectively addresses the complex input challenges inherent in image completion tasks.

[0007] In the field of image completion, attention mechanisms can be used to enhance the network's focus on missing parts and complete them based on existing information and model algorithms. In addition to selective attention mechanisms, other image completion techniques exist, such as interpolation-based image completion methods and sparse coding-based image completion methods. These techniques have a wide range of applications in the field.

[0008] Deficiencies in existing technical solutions:

[0009] 1. Although current deep learning-based image completion methods achieve better completion results than traditional methods, most of them ignore the establishment of long-distance information interaction between pixels, resulting in poor performance of these models when processing large-area missing images and insufficient overall fit of the generated images.

[0010] 2. On the other hand, some solutions that retain image detail information by fusing features of different scales ignore the differences in contributions of each scale, resulting in insufficient model accuracy and considerable artifact errors in the generated results. Summary of the Invention

[0011] To address the shortcomings of existing technologies, an image completion method based on fast Fourier transform and selective attention mechanism is proposed. The image completion method proposes an efficient completion network with built-in Fourier transform operation. The completion network is embedded with a fast Fourier transform module of an aggregated selective convolution kernel network. The fast Fourier transform module is responsible for completing the network to obtain the global receptive field. The aggregated selective convolution kernel network adaptively adjusts the proportion according to the contribution of each scale to help the model retain detailed information while enhancing the model's robustness to input scale changes. The specific methods include:

[0012] Step 1: Prepare image completion datasets, namely the Cele-A HQ dataset and the Place2 dataset;

[0013] Step 2: Randomly extract a set number of images from the Cele-A HQ dataset and the Place2 dataset as training sets, and the remaining images as validation sets;

[0014] Step 3: For the images in the training set and the validation set, generate irregular masks in a random manner to obtain mask images;

[0015] Step 4: Input the images in the training set into the completion network for training. The main framework of the completion network is a generative adversarial network. The model includes a generator and a block discriminator. The generator is composed of a convolution module, a deconvolution module, a multi-scale convolution fusion module, and a fast Fourier transform module. The discriminator adopts a block discriminator.

[0016] The generator takes the mask image and the binary mask containing the missing area as input, and the output of the generator is the completed image. The process of the generator generating the completed image includes:

[0017] Step 41: Encode the input into a low-dimensional space using three stacked ordinary convolutions. This process reduces the dimensionality of the input and obtains a feature vector that contains important information about the input image.

[0018] Step 42: The feature vector encoded in step 41 is sent to a multi-scale convolution fusion module. The multi-scale convolution fusion module splits a normal convolution into four sub-convolutions with dilation rates of 1, 2, 4, and 8, respectively, and then uses a normal convolution to integrate the feature information of multiple convolution kernels.

[0019] Step 43: Inputting the feature information integrated by the multi-scale fusion module into a module combination composed of at least three stacked fast Fourier transform modules, each fast Fourier transform module having an embedded aggregated selective convolution kernel network;

[0020] Step 431: The feature information entering the Fast Fourier Transform module is first subjected to a 3×3 convolution layer, and then the feature information is mapped to the frequency domain using the real Fourier Transform module and then mapped back to the spatial domain;

[0021] Step 432: A 3×3 convolution operation is then performed, and then the network enters the aggregated selective convolution kernel network. The aggregated selective convolution kernel network has a built-in learning mechanism for dynamic fusion of multiple receptive fields, which provides local information supplement for the fast Fourier transform module. At the same time, by dynamically assigning receptive field weights, it helps the generator adapt to the impact of changes in the input image scale.

[0022] Step 433: The features processed by the aggregated selective convolution kernel network module are input into the real inverse Fourier transform module to restore the features to a spatial structure to obtain a spatial structure feature. The spatial structure feature is then element-wise added to the input feature of the fast Fourier transform module using a skip connection. The feature is then subjected to a 1×1 convolution to obtain a real feature map.

[0023] Step 434: input the real feature map obtained by the first fast Fourier transform module into the subsequent fast Fourier transform modules in sequence, and repeat the fast Fourier transform operations from step 431 to step 433 to obtain the final real feature map;

[0024] Step 44: Input the final real feature map obtained in step 43 into the multi-scale convolution fusion module to obtain a fused feature map;

[0025] Step 44: Finally, multiple deconvolution layers are used to decode the fused feature map into a completed generated image, and the generated image is used as the output of the generator;

[0026] Step 45: The block discriminator performs three-layer convolution processing on the generated image to obtain a scoring matrix. The block discriminator distinguishes the generated image from the real image based on the scoring matrix, and feeds back the parameters to the generator to guide the generator to generate a generated image that is closer to the real image. The training ends until the block discriminator can no longer distinguish between the real image and the generated image.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. Inspired by the fast Fourier transform module with parameter efficiency advantages, this paper proposes an efficient completion network with built-in fast Fourier transform operations, which can achieve long-range dependencies while ensuring the high-efficiency operation of the model.

[0029] 2. The present invention also adds an improved aggregated selective convolution kernel network to the fast Fourier transform module. This module will adaptively adjust the proportion of contributions at each scale, attempting to provide more accurate local information supplementation for the model, and ultimately achieve image completion with higher global fusion and richer local details. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the overall framework of the image completion network model of the present invention;

[0031] Figure 2 Schematic diagram of the structure of the multi-scale convolution fusion module of the present invention;

[0032] Figure 3 Schematic diagram of the structure of the fast Fourier transform module of the present invention;

[0033] Figure 4 Schematic diagram of the structure of the aggregated selective convolution kernel network of the present invention;

[0034] Figure 5 This is the experimental result of applying regular mask to the Celeb-A dataset;

[0035] Figure 6 This is the experimental result of applying irregular masks to the Celeb-A dataset;

[0036] Figure 7 This is a comparison of image restoration effects after large-scale damage. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0038] The following is a detailed description with reference to the accompanying drawings.

[0039] The purpose of the present invention is to improve the quality and efficiency of image completion by combining a fast Fourier transform with a clustered selective convolutional kernel network, and to address some of the defects and deficiencies in the prior art, such as slow processing of large-scale, high-resolution images and poor image completion for complex textures and structures. The method of the present invention can more accurately restore the details and structure of the original image, thereby improving the visual and application effects of image completion. The solution to the technical problems created by the present invention gives image completion based on the Fourier transform and selective attention mechanism broader application prospects and market value.

[0040] In response to the shortcomings of the existing technology, the present invention proposes an image completion method based on fast Fourier transform and selective attention mechanism. The image completion method proposes an efficient completion network with built-in Fourier transform operation. The completion network is embedded with a fast Fourier transform module of an aggregated selective convolution kernel network. The fast Fourier transform module is responsible for completing the network to obtain the global receptive field. The selective convolution kernel network adaptively adjusts the proportion according to the contribution of each scale feature to help the model retain detailed information while enhancing the model's robustness to input scale transformations. The specific method includes:

[0041] Step 1: Prepare image completion datasets, namely the Cele-A HQ dataset and the Place2 dataset.

[0042] Step 2: Randomly extract a set number of images from the Cele-A HQ dataset and the Place2 dataset as training sets, and the remaining images as validation sets.

[0043] Specifically, 20,000 portrait images from the Celeb-A HQ dataset are used as the training set, and 5,000 images are selected from the remaining images as the test set. In the Place2 dataset, 8 million randomly selected images are used as the training set, and the remaining 20,000 images are used as the test set.

[0044] Step 3: For the images in the training set and the validation set, generate irregular masks in a random manner to obtain mask images.

[0045] Specifically, 5,000 irregular masks are generated for training the model, and finally 1,000 of the remaining images are randomly selected as mask inputs for the testing process.

[0046] Step 4: Input the images in the training set into the completion network for training. The main framework of the completion network is a generative adversarial network. The model consists of a generator and a block discriminator. The generator is composed of a convolution module, a deconvolution module, a multi-scale convolution fusion module (MSFM), and a fast Fourier transform module (FSKAM). It is a highly modular deep network. The discriminator uses the same block discriminator as Patch-GAN, which is suitable for processing image data.

[0047] Figure 1 This paper presents the overall framework of the proposed image completion network model. The generator portion of this model employs an encoder-decoder framework. The completion network first processes the input, consisting of the concatenation of the mask and the damaged image along the channel dimension, using three layers of convolution operations to generate a high-dimensional feature. The network then passes this high-dimensional feature to a bottleneck layer, which consists of multiple core modules stacked together. The multi-scale convolutional fusion module is embedded at the front and end of the bottleneck layer as a core module. The bottleneck layer's center consists of a series of multiple Fast Fourier Transform (FFT) modules with embedded Selective Attention (SAT) mechanisms. The FFT modules primarily perform FFT operations, achieving high parameter efficiency per operation. To achieve this, this model modestly increases the number of FFT modules while maintaining predictable computational cost. Furthermore, the FFT SAT module embeds an Aggregative Selective Kernels Network (ASKNET). Finally, the network performs three deconvolution operations on the bottleneck layer's processed features, decoding them into an output consistent with the true image scale.

[0048] The generator takes the mask image and the binary mask containing the missing area as input, and the output of the generator is the completed image. The process of the generator generating the restored image includes:

[0049] Step 41: Encode the input into a low-dimensional space using three stacked ordinary convolutions. This process reduces the dimensionality of the input and obtains a feature vector that contains important information about the input image.

[0050] Step 42: The feature vector encoded in step 41 is fed into a multi-scale convolution fusion module. The multi-scale convolution fusion module splits a normal convolution into four sub-convolutions with dilation rates of 1, 2, 4, and 8, respectively, and then uses a normal convolution to integrate the feature information of multiple convolution kernels. This module helps the generator obtain more long-range context information, thereby enriching the details of the generated image and enhancing the rationality of the generated content.

[0051] Figure 2 This figure shows the architecture of the multi-scale convolution kernel fusion module, which employs a multi-scale feature fusion mechanism. To reduce parameters, this module introduces four dilated convolutions with dilation rates of 1, 2, 4, and 8 in the four branches of the module network. The features obtained after the dilated convolutions are output with a channel count that is one-quarter the number of input feature channels. These dilated convolution features are then concatenated in the channel dimension. Following this, a standard convolution operation is performed on the concatenated feature map using a 3×3 convolution to integrate the features of the multi-scale convolution kernels.

[0052] The multi-scale convolution kernel fusion module uses dilated convolutions with larger dilation rates to capture broader contextual information, while dilated convolutions with smaller dilation rates capture narrower contextual information. Multi-scale convolution kernels can more fully utilize pixel information from both the far and near ends of the missing region. This helps the network integrate features from different levels, ultimately improving the refinement of the completed image.

[0053] Step 43: The feature information processed by the multi-scale fusion module is input into a module combination consisting of at least three stacked Fast Fourier Transform (FFT) modules. It is important to note that each FFT module embeds an Aggregate Selective Kernels Network (ASKNET). The FFT module helps the generator establish long-range dependencies between pixels to ensure global consistency of the generated image, while the ASKNET improves the generator's receptive field adaptability.

[0054] Figure 3This is a schematic diagram of the Fast Fourier Transform (FFT) module. The FFT-based Selective Attention (SAT) module first extracts features using convolution with a kernel size of 3×3, then performs batch normalization and activation on these features. Next, a real FFT is performed on these processed features to obtain real features. Unlike the FFT, the real FFT uses only half of the spectrum and retains half of the real values. The real and imaginary parts of the tensor are then fused to convert them into real numbers. These real features are then input into the ASKNET, an aggregated selective convolutional kernel network. ASKNET selectively assigns different weights to different convolution kernels based on their contributions, dynamically adjusting the receptive field. Finally, a real inverse FFT is performed on the resulting real features to restore them to a spatial structure.

[0055] Step 431: The feature information entering the Fast Fourier Transform module is first subjected to a 3×3 convolution layer, and then the feature information is mapped to the frequency domain using the real Fourier Transform module and then mapped back to the spatial domain;

[0056] Step 432: A 3×3 convolution operation is then performed before entering the aggregated selective convolutional kernel network ASKNET. The aggregated selective convolutional kernel network ASKNET has a built-in learning mechanism for dynamic fusion of multiple receptive fields, providing local information supplementation for the fast Fourier transform module FSKAM. It also helps the generator adapt to changes in input image scale by dynamically assigning receptive field weights.

[0057] Figure 4 This paper presents a schematic diagram of the proposed aggregated selective convolutional kernel network. This network generates convolutional kernels of varying importance for different images. This dynamic learning design helps the network adapt to changes in the input image scale and the scale of objects within the image. This module helps image completion networks adapt to changes in the scale of the entire missing image and the scale of missing objects within the image. Notably, this network replaces the multi-scale feature fusion of the original selective convolutional kernel network with a feature aggregation operation. Figure 4As shown in the figure, the feature map is first fed into four dilated convolutional layers, which are derived from ordinary convolutions. Dilated convolutional layer 1 performs a dilated convolution operation with a dilation ratio of 1, dilated convolutional layer 2 performs a dilated convolution operation with a dilation ratio of 2, dilated convolutional layer 3 performs a dilated convolution operation with a dilation ratio of 4, and dilated convolutional layer 4 performs a dilated convolution operation with a dilation ratio of 8. After the feature map is processed by these four dilated convolutional layers, four features with different receptive fields are generated. These four features are then element-wise summed to obtain a multi-scale fused feature. This feature is then passed to the fusion module, which uses an average pooling layer to obtain a 1×c feature vector. This feature vector is then compressed into a 1×(c / r) feature vector using a fully connected layer. This feature vector is then upscaled into four 1×c feature vectors using a four-way fully connected layer. These four feature vectors are then distributed to four different selective modules: Selective Module 1, Selective Module 2, Selective Module 3, and Selective Module 4. Next, the features originally passed through the four dilated convolutional layers are element-wise multiplied with the four dimensionally upgraded feature vectors to obtain four weighted feature maps. A 1×1 convolution is then used to reduce the dimensionality of these four different-scale features along the channel dimension, reducing the number of channels for each feature to 1 / 4 of the original. Finally, these features are concatenated along the channel dimension and aggregated using a 1×1 convolution. Compared to direct summation, this design more effectively preserves detailed information while ensuring network parameter efficiency.

[0058] Step 433: The features processed by the aggregated selective convolution kernel network module ASKNET enter the real inverse Fourier transform module to restore the features to the spatial structure to obtain spatial structure features, and then use the jump connection to add the spatial structure features and the input features of the fast Fourier transform module FAKAM element by element. Then the features undergo a 1×1 convolution to obtain a real feature map.

[0059] Step 434: Input the feature map obtained by the first fast Fourier transform module into the subsequent fast Fourier transform modules in sequence, repeat the fast Fourier transform operations from step 431 to step 433 to obtain the final real feature map.

[0060] This fast Fourier transform operation is repeated three times to enhance the global information integration of features. The above process is part of the feature processing process of the fast Fourier transform module combination. The feature map obtained in this process continues to enter the multi-scale fusion module, which helps the network retain more detailed information.

[0061] Step 44: Input the final real feature map obtained in step 43 into the multi-scale convolution fusion module to obtain a fused feature map;

[0062] Step 45: Finally, multiple deconvolution layers are used to decode the fused feature maps into a restored generated image, which will be used as the output of the generator.

[0063] Step 46: The block discriminator performs three layers of convolution on the generated image to obtain a scoring matrix. The block discriminator uses this scoring matrix to distinguish between the generated image and the real image. The block discriminator then feeds back the parameters to the generator, guiding it to produce images that are closer to the real image. This process continues until the block discriminator can no longer distinguish between the real and generated images, concatenating the parameters. This completes the training phase, and the parameters are saved after training. Next, we enter the application phase, where the mask to be processed and the image obscured by the mask are concatenated and used as input for the application model. Finally, the generator uses the trained parameters to produce the completion result.

[0064] Figure 5 Comparison of the completion results of applying regular masks to the Celeb-A dataset. Figure 5 (a) is the original image, Figure 5 (b) is the damaged image, Figure 5 (c) Completion results of the Aggregated Contextual-Transformation GAN (AOT GAN) method for generating adversarial networks. Figure 5 (d) is the completion result of the Pyramid Context Encoder Network (PEN) method. Figure 5 (e) is the completion result of the gated convolution network GC (Gated Convolution Network) method. Figure 5 (f) is the completion result of the proposed method. Compared with other methods, the proposed method generates richer texture details and achieves a globally consistent effect.

[0065] Figure 6 This is a comparison of the completion results of applying irregular masks to the Celeb-A dataset. Figure 6 (a) is the original image, Figure 6 (b) is a damaged image. Figure 6 (c) is the AOT GAN completion result. Figure 6 (d) is the completion result of the PEN method, Figure 6 (e) is the completion result of GC method, Figure 6(f) is the completion result of the method of the present invention. The restoration results of the GC model and the PEN model show that these two models tend to generate blurred textures and details with color differences. In addition, PEN has relatively obvious restoration traces. Compared with the above schemes, the restoration content of the AOT GAN model is clearer, but the facial features generated by this model deviate from the realism. For example, Figure 6 (c) The right eye generated by the second row of characters deviates greatly from the real human eye, and the restoration result of this model has a higher degree of restoration than other methods.

[0066] Figure 7 The comparison results of image restoration after large-scale damage are shown. Figure 7 (a) is the damaged image, Figure 7 (b) is the original image, 7 (c) is the completion result of the method of the present invention, Figure 7 (d) is the completion result of the AOT method. Figure 7 (e) is the completion result of the GC method. When the image occlusion rate is as low as 20%-40%, the method of the present invention obtains clearer results than other methods. The model of the present invention generates an image with more consistent texture, and its generated results have certain quality advantages, such as Figure 7 As shown in the first and second rows. When the occlusion ratio reaches 70% to 90%, the present invention generates more reasonable details, and its advantage in authenticity is very obvious. It is not difficult to see that the model of the present invention is less dependent on the information of neighboring pixels, such as Figure 7 As shown in the third and fourth rows. Figure 7 (c) Figure 7 As shown in (d), for images with missing areas exceeding 80%, the proposed model's generation capabilities are not significantly affected, and the generated images even maintain local semantic coherence and global consistency. Other methods, on the other hand, overly rely on the effective information of neighboring pixels, resulting in large-scale artifacts and texture discontinuities in the missing areas, and perform poorly when processing images with minimal visible information.

[0067] It should be noted that the above-described specific embodiments are illustrative only. Those skilled in the art may devise various solutions based on the disclosure of the present invention, and such solutions fall within the scope of the present invention and are intended to be protected by the present invention. Those skilled in the art should understand that the present description and its accompanying drawings are intended to be illustrative only and are not intended to limit the scope of the claims. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. An image completion method based on fast Fourier transform and selective attention mechanism, characterized in that: The image completion method proposes an efficient completion network with a built-in Fourier transform operation. The completion network is embedded with a fast Fourier transform module of an aggregated selective convolutional kernel network. The fast Fourier transform module is responsible for completing the network to obtain a global receptive field. The aggregated selective convolutional kernel network adaptively adjusts the proportion according to the contribution of each scale to help the model retain detailed information while enhancing the model's robustness to input scale changes. The specific method includes: Step 1: Prepare image completion datasets, namely the Cele-A HQ dataset and the Place2 dataset; Step 2: Randomly extract a set number of images from the Cele-A HQ dataset and the Place2 dataset as training sets, and the remaining images as validation sets; Step 3: For the images in the training set and the validation set, generate irregular masks in a random manner to obtain mask images; Step 4: Input the images in the training set into the completion network for training. The main framework of the completion network is a generative adversarial network, which includes a generator and a block discriminator. The generator is composed of a convolution module, a deconvolution module, a multi-scale convolution fusion module, and a fast Fourier transform module. The discriminator adopts a block discriminator. The generator takes the mask image and the binary mask containing the missing area as input, and the output of the generator is the completed image. The process of the generator generating the completed image includes: Step 41: Use three stacked ordinary convolutions to encode the input into a low-dimensional space, thereby reducing the dimensionality of the input and obtaining a feature vector containing important information about the input image. Step 42: The feature vector encoded in step 41 is sent to a multi-scale convolution fusion module. The multi-scale convolution fusion module splits a normal convolution into four sub-convolutions with dilation rates of 1, 2, 4, and 8, respectively, and then uses a normal convolution to integrate the feature information of multiple convolution kernels. Step 43: Inputting the feature information integrated by the multi-scale convolution fusion module into a module combination composed of at least three stacked fast Fourier transform modules, each fast Fourier transform module having an embedded aggregated selective convolution kernel network; Step 431: The feature information entering the Fast Fourier Transform module is first subjected to a 3×3 convolution layer, and then the feature information is mapped to the frequency domain using the real Fourier Transform module and then mapped back to the spatial domain; Step 432: A 3×3 convolution operation is then performed, and then the network enters the aggregated selective convolution kernel network. The aggregated selective convolution kernel network has a built-in learning mechanism for dynamic fusion of multiple receptive fields, which provides local information supplement for the fast Fourier transform module. At the same time, by dynamically assigning receptive field weights, it helps the generator adapt to the impact of changes in the input image scale. Step 433: The features processed by the aggregated selective convolution kernel network module are input into the real inverse Fourier transform module to restore the features to a spatial structure to obtain a spatial structure feature. The spatial structure feature is then element-wise added to the input feature of the fast Fourier transform module using a skip connection. The feature is then subjected to a 1×1 convolution to obtain a real feature map. Step 434: input the real feature map obtained by the first fast Fourier transform module into the subsequent fast Fourier transform modules in sequence, and repeat the fast Fourier transform operations from step 431 to step 433 to obtain the final real feature map; Step 44: Input the final real feature map obtained in step 43 into the multi-scale convolution fusion module to obtain a fused feature map; Step 45: Finally, multiple deconvolution layers are used to decode the fused feature map into a completed generated image, which is used as the output of the generator. Step 46: The block discriminator performs three-layer convolution processing on the generated image to obtain a scoring matrix. The block discriminator distinguishes the generated image from the real image based on the scoring matrix, and feeds back the parameters to the generator to guide the generator to generate a generated image that is closer to the real image. The training ends until the block discriminator can no longer distinguish between the real image and the generated image.

Images