An image completion method based on multi-stage complex neural network

Abstract

The present invention proposes an image completion method based on a multi-stage complex neural network, which aims to improve the completion quality of damaged images. The method comprises the following steps: first, constructing a multi-stage complex neural network generator, each stage comprising a real-to-complex block, a complex convolution block, a complex transposed convolution block and a complex-to-real block to achieve conversion from the real domain to the complex domain, feature extraction, upsampling and final real domain conversion. Secondly, through adversarial training, the generator and discriminator are trained using a training set to optimize the completion effect of the generator. Finally, the trained network is used to complete the new input image and mask to generate a complete image. The method of the present invention can achieve high-quality image completion effects, and the method of the present invention has fewer parameters under the same number of features.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to an image completion method based on a multi-stage complex-valued neural network (MS-CN). Background Art

[0002] Images are stored digitally in computers, but during the actual capture and transmission process, they are inevitably subject to noise interference. Furthermore, due to factors such as the condition of the transmission equipment, the resulting image is often damaged, severely interfering with subsequent image analysis and processing. To meet the needs of image-based applications, various image processing techniques have been proposed, such as image denoising, image super-resolution, and image completion. Image completion aims to provide visually appreciable fills for missing areas in damaged images, becoming an important research direction in the field of computer vision and widely used in image enhancement and medical image processing.

[0003] Current image completion algorithms are often based on real numbers. However, complex numbers are often used in practical applications in fields such as communications, bioinformatics, speech recognition, and image processing. This suggests that using complex numbers to represent the inputs, outputs, and parameters of neural networks has potential appeal in these related fields. Complex neural networks utilize complex parameters and variables to process related information. Their primary difference from real neural networks lies in the representation of multiplication operations—complex multiplication functions involve phase rotation and amplitude modulation, thus reducing degrees of freedom. Real neural networks are characterized by self-organization and high degrees of learning freedom. Complex neural networks can mitigate the potential risks of excessive degrees of freedom by using a priori knowledge of the amplitude and phase of the data. Because phase information details the shape, edges, and orientation of objects in an image, the information stored in phase information can recover much of the information encoded in its amplitude. Complex neural networks also hold great promise for analyzing and processing information. In recent years, complex neural networks have garnered increasing attention in the field of machine learning.

[0004] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0005] The main purpose of the present invention is to provide an image completion method based on a multi-stage complex neural network to solve the technical problem of low quality of image completion results in the prior art.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A multi-stage complex neural network-based image completion method includes the following steps:

[0008] A1. Establish a multi-stage complex neural network generator. The generator includes at least three stages, and each stage is processed by the following modules:

[0009] One-stage network: Input image I and mask M are converted to the complex domain through the real-to-complex block (RCB), and then features are extracted through the complex convolution block (CCB). The intermediate features are stored for skip connection. The second half converts the features back to the real domain through the complex transposed convolution block (CTCB) and the complex-to-real block (CRB) to form a preliminary completed image;

[0010] Two-stage network: input the output image of the first stage, without using skip connections, directly refine the image of the first stage output, and further refine the image through complex convolution block (CCB), complex transposed convolution block (CTCB) and complex to real block (CRB).

[0011] Three-stage network: The output image of the second stage is input, similar to the first stage, and the final image refinement and completion is performed through the real-to-complex block (RCB), complex convolution block (CCB), complex transposed convolution block (CTCB) and complex-to-real block (CRB);

[0012] A2. Adversarial training: Using the training set, the multi-stage complex neural network generator and the simple real neural network discriminator are trained adversarially. The multi-stage complex neural network is trained using the images and masks in the training set. The generator is responsible for filling in the missing parts of the image, and the discriminator evaluates the consistency of the image features output by the generator with the original image to optimize the completion quality of the generator.

[0013] A3. Image completion: Use the trained multi-stage complex neural network generator to complete the new input image and mask, and output the complete completed image.

[0014] Furthermore, in the three-stage network, the real-to-complex block generates the real and imaginary parts of the complex number from the input image through two paths. Both paths use the Conv3 convolution layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and the BatchNorm normalization layer in sequence, repeated twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm batch normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer in sequence to restore it to a real image.

[0015] Furthermore, the complex convolution blocks in both the one-stage and three-stage networks include a complex convolution layer, a complex activation function, and a complex batch normalization layer. The convolution kernel size of the complex convolution layer is 4, the stride is 2, and the padding is 1. In addition to extracting features, it also has a downsampling function.

[0016] Furthermore, the first half of the one-stage and three-stage networks consists of 8 complex convolution blocks, and the second half includes 8 complex transposed convolution blocks. Their parameters are the same as the complex convolution blocks, except that the convolution is changed to transposed convolution. In addition to extracting features, they also have upsampling functions. The second half of the network is symmetrical with the first half.

[0017] Furthermore, in the first half of the two-stage network, the first complex convolution block has a size of 7, a stride of 1, and only extracts features. The subsequent blocks have a size of 3, a stride of 2, and a padding of 1, and have both feature extraction and upsampling functions. There are two of them. The middle part includes three complex residual convolution blocks, which sequentially include a complex convolution layer, a complex activation function, and a complex normalization layer, and only extract features. The second half includes a complex transposed convolution block that is symmetrical to the first half.

[0018] Furthermore, in step A2, the complete image and mask of the training dataset are used as the input of the network, and multi-level and multi-scale feature extraction is performed through complex convolution blocks. The features are then restored to the original image through complex transposed convolution layers, and the final network model is obtained through multiple training steps.

[0019] Furthermore, in step A3, for each complete image, a corresponding mask is given to represent the area to be completed, and the mask is fed into the trained multi-stage complex neural network. The network finally outputs the completed complete image.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the image completion method based on a multi-stage complex neural network.

[0021] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the image completion method based on a multi-stage complex neural network is implemented.

[0022] In some embodiments of the present invention, the following steps are included:

[0023] Step 1: Build a multi-stage complex neural network consisting of a Real-Complex Block (RC), a Complex-Valued Convolution Block (CCB), a Complex-Valued Transposed Convolution Block (CTCB), and a Complex-Valued Convolution Residual Block (CCRB). The one-stage and three-stage networks extract feature information at various scales through the complex convolution blocks in the first half and transmit it to the second half using skip connections. Each complex transposed convolution block in the second half receives the output of the previous module and further processes the corresponding feature information from the first half, fully utilizing information from both low-level and high-level layers. A simple discriminator is also built to determine whether an image is legitimate and is trained adversarially against the multi-stage complex neural network. Step 2: Adversarially train the generator and discriminator using images and masks from the training set. In this process, the generator is responsible for filling in the missing parts of the image, while the discriminator evaluates the quality of the generator's output, guiding the generator to use a complex neural network to extract image features in multiple stages, complete the occluded areas, and improve their detail and quality. Step 3: Use the trained generator to complete the input image and the mask, and output the complete image.

[0024] The first step specifically includes: establishing a network structure based on a multi-stage complex neural network, with all networks starting and ending with a real-to-complex block and a complex-to-real block respectively. The real-to-complex block generates the real part and imaginary part of the complex number from the input image through two paths respectively. Both paths use a Conv3 layer with a convolution kernel size of 3 to extract features, and then pass through a ReLU activation layer and a BatchNorm normalization layer in sequence, repeated twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through Conv3, BatchNorm, ReLU, Conv3 and Tanh activation layers in sequence. The intermediate body of the first and third stage networks is composed of a complex convolution block and a complex transposed convolution block. The complex convolution blocks are all composed of a complex convolution layer, a complex activation function and a complex batch normalization layer, where the convolution kernel size of the complex convolution layer is 4, the step size is 2, and the step size is 1. In addition to extracting features, it also has a downsampling function. The first half of the network consists of eight complex convolutional blocks, and the second half consists of eight complex transposed convolutional blocks. These blocks have the same parameters as the complex convolutional blocks, except that they use transposed convolutions instead of convolutions, performing upsampling in addition to feature extraction. The second half of the network is symmetrical to the first. In the first half of the second-stage network, the first complex convolutional block has a size of 7 and a stride of 1, and only extracts features. Subsequent blocks have a size of 3, a stride of 2, and padding of 1, performing both feature extraction and upsampling. There are two of these blocks, each with a size of 3, a stride of 2, and padding of 1. They perform both feature extraction and upsampling. The middle section consists of three complex residual convolutional blocks, which sequentially consist of a complex convolutional layer, a complex activation function, and a complex normalization layer, and only extract features. The second half consists of complex transposed convolutional blocks that are symmetrical to the first half.

[0025] The second step specifically includes: taking the complete image and mask of the training dataset as the input of the network, performing multi-level and multi-scale feature extraction through complex convolution blocks, and then restoring the features back to the original image through complex transposed convolution layers, and obtaining the final network model through multiple training cycles.

[0026] The third step specifically involves providing a corresponding mask for each complete image to represent the area to be completed. These two masks are then fed into a trained multi-stage complex neural network. The network ultimately outputs the completed image.

[0027] The present invention has the following beneficial effects:

[0028] The present invention proposes an image completion method based on a multi-stage complex neural network, which can achieve high-quality image completion effects. The present invention uses a combination of multi-stage networks and complex neural modules in an image completion method for the first time, adding real-complex conversion blocks at the beginning and end of each stage network, and the first half uses complex convolution modules to extract multi-level and multi-scale features. The second half constructs a network by stacking complex transposed convolution blocks, and uses the extracted features to gradually restore the image. In addition, the multi-stage complex neural network proposed in the present invention can take into account both rough image repair and detailed optimization through stage networks with different receptive fields, and is more suitable for image completion. The multi-level and multi-scale features extracted by modules with different receptive fields have better feature extraction capabilities and can achieve high-quality image completion results. In addition, the method of the present invention has fewer parameters under the same number of features.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a simplified flow chart of the image completion method based on a multi-stage complex neural network according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the network structure of the image completion method based on a multi-stage complex neural network according to an embodiment of the present invention.

[0032] Figure 3 The reconstruction process of the image completion method based on the multi-stage complex neural network of the embodiment of the present invention is DETAILED DESCRIPTION

[0033] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0034] The image completion method of the present invention mainly includes the following processes: the first step: establishing a multi-stage complex-valued neural network (MS-CN) based on the basic structure of a complex neural network as an image completion generator, and a simple real neural network as a discriminator. The generator contains a total of three stages of networks, consisting of a real-complex block (RC), a complex convolution block (CCB), a complex transposed convolution block (CTCB) and a complex residual convolution block (CCRB). In the first stage, the preliminary results of image completion are generated by a complex neural network with a large receptive field; in the second stage, the preliminary results are refined by a complex neural network with a small receptive field; in the third stage, the results of the second stage are refined by a complex neural network with a large receptive field. The second step: using the data set to form adversarial training between the generator and the discriminator, pre-training a multi-stage complex neural network with reasonable weights. Step 3: Accept the image and mask as model input, use the multi-stage complex neural network obtained by adversarial training to complete it, and obtain the complete image as output.

[0035] See Figures 1 to 3 , an embodiment of the present invention provides an image completion method based on a multi-stage complex neural network, comprising the following steps:

[0036] A1. Establish a multi-stage complex neural network generator. The generator includes at least three stages, and each stage is processed by the following modules:

[0037] One-stage network: Input image I and mask M are converted to the complex domain through the real-to-complex block (RCB), and then features are extracted through the complex convolution block (CCB). The intermediate features are stored for skip connection. The second half converts the features back to the real domain through the complex transposed convolution block (CTCB) and the complex-to-real block (CRB) to form a preliminary completed image;

[0038] Two-stage network: input the output image of the first stage, without using skip connections, directly refine the image of the first stage output, and further refine the image through complex convolution block (CCB), complex transposed convolution block (CTCB) and complex to real block (CRB).

[0039] Three-stage network: The output image of the second stage is input, similar to the first stage, and the final image refinement and completion is performed through the real-to-complex block (RCB), complex convolution block (CCB), complex transposed convolution block (CTCB) and complex-to-real block (CRB);

[0040] A2. Adversarial training: Using the training set, the multi-stage complex neural network generator and the simple real neural network discriminator are trained adversarially. The multi-stage complex neural network is trained using the images and masks in the training set. The generator is responsible for filling in the missing parts of the image, and the discriminator evaluates the consistency of the image features output by the generator with the original image to optimize the completion quality of the generator.

[0041] A3. Image completion: Use the trained multi-stage complex neural network generator to complete the new input image and mask, and output the complete completed image.

[0042] In step A1, the first half of the one-stage and three-stage networks converts the image and mask from the real domain to the complex domain using a real-to-complex block (RCB). Feature extraction is then performed using several CCBs, and intermediate features are stored for skip connections. The CTCBs in the second half first concatenate the output features of the previous step with the intermediate features corresponding to the first half along the feature dimension, and then further recover the features. Finally, the complex features are converted to an image using a complex-to-real block (CRB).

[0043] The second-stage network needs to have a low receptive field, so no skip connections are used. Instead, the output image of the first stage is refined by passing it through a real-to-complex block, a complex convolution block, a complex transposed convolution block, and a complex-to-real block.

[0044] The discriminator's task is primarily for training and has no direct relevance to image completion. Therefore, a simple real neural network consisting of real convolutional blocks is used. Its purpose is to determine whether the generator's output has the same characteristics as the original image, thus providing a partial optimization objective.

[0045] like Figure 2As shown, in a preferred embodiment, step A1 specifically includes: establishing a multi-stage complex neural network structure, including a real-complex conversion block, a complex convolution block, a complex transposed convolution block and a complex residual convolution block. The real-complex conversion blocks of the three-stage network are divided into two types, a real-to-complex block and a complex-to-real block. The real-to-complex block generates the real part and the imaginary part of the complex number of the input image through two paths. Both paths use the Conv3 convolution layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and the BatchNorm normalization layer in sequence, and repeat twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm batch normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer in sequence to restore it to a real image.

[0046] like Figure 2 As shown, in a preferred embodiment, the complex convolution blocks in both the one-stage and three-stage networks include a complex convolution layer, a complex activation function, and a complex batch normalization layer. The convolution kernel size of the complex convolution layer is 4, the stride is 2, and the padding is 1. In addition to feature extraction, it also has a downsampling function. Furthermore, the first half of the one-stage and three-stage networks consists of 8 complex convolution blocks, and the second half includes 8 complex transposed convolution blocks. Their parameters are the same as those of the complex convolution blocks, except that the convolution is changed to transposed convolution. In addition to feature extraction, they also have an upsampling function. The second half of the network is symmetrical with the first half.

[0047] like Figure 2 As shown, in a preferred embodiment, in the first half of the two-stage network, the first complex convolution block has a size of 7, a step size of 1, and only extracts features. The subsequent block size is 3, the step size is 2, and the padding is 1. It has feature extraction and upsampling functions, and the number is 2; the middle part includes 3 complex residual convolution blocks, which sequentially include a complex convolution layer, a complex activation function, and a complex normalization layer, and only extracts features; the second half includes a complex transposed convolution block symmetrical to the first half.

[0048] In step A2, the generator and discriminator are adversarially trained using the training set, and a multi-stage complex neural network is trained using the images and masks in the training set. The completion is completed through rough patching and detailed optimization.

[0049] like Figure 1 and Figure 3 As shown in Figure 2, in step A2, the complete image and mask of the training dataset are used as the input of the network, and multi-level and multi-scale feature extraction is performed through complex convolution blocks. The features are then restored to the original image through complex transposed convolution layers, and the final network model is obtained through multiple training steps.

[0050] In step A3, the trained complex neural network is used to complete the input image and mask to obtain a complete image.

[0051] like Figure 1 and Figure 3 As shown in Figure 3, in step A3, for each complete image, a corresponding mask is given to represent the area to be completed, and the mask is fed into the trained multi-stage complex neural network. The network finally outputs the completed complete image.

[0052] The following further describes specific embodiments of the present invention and algorithm examples.

[0053] A method for image completion based on a multi-stage complex neural network. Components of the first and third-stage network structures: The network consists of a real-to-complex block and a complex-to-real block at the beginning and end, respectively. The front and back parts of the main network consist of eight complex convolution blocks and complex transposed convolution blocks, respectively. Skip connections exist before and after the network to enable feature transfer. Components of the second-stage network structure: The network consists of the same components as the first and third stages. The front and back parts of the main network consist of three complex convolution blocks and complex transposed convolution blocks, respectively. The middle part of the network consists of three complex residual convolution blocks, which are used to further process the extracted features.

[0054] Real to complex block: such as Figure 2 As shown in the figure (marked by light green arrows), the module includes two identical paths that generate the real and imaginary parts of complex features respectively. Each path passes the image through a real convolution layer Conv3 with a convolution kernel size of 3, a stride of 1, and a padding of 1, a BatchNorm normalization layer, and a ReLU activation layer, repeated twice. Complex to real block: Figure 2 As shown in the figure (marked by the light yellow arrow), the module concatenates the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer.

[0055] Complex convolution block: such as Figure 2 As shown (marked by light blue arrows), this module consists of a complex convolutional layer, a complex normalization layer, and a complex activation layer. The first and third stage complex convolutional layers have a kernel size of 4, a stride of 2, and a padding of 1, providing both feature extraction and downsampling. The second stage complex convolutional block has the same structure, except that the first kernel size is 7, the stride of 1, and the padding of 0, while the next two kernel sizes are 3, the stride of 2, and the padding of 1, providing both feature extraction and downsampling.

[0056] Complex transposed convolution block: such as Figure 2As shown (marked by pink arrows), this module consists of a complex transposed convolutional layer, a complex normalization layer, and a complex activation layer. The convolution kernel size in the three-stage network is 4, with a stride of 2 and padding of 1. This block processes the output of the previous module and the corresponding block in the first half of the network. In the two-stage network, the first two convolution kernels have a size of 3, a stride of 2, a padding of 1, and an external padding of . The final convolution kernel has a size of 7, a stride of 1, and a padding of 0.

[0057] Complex residual convolution block: such as Figure 2 As shown in the figure (marked by the brown-red arrow), this module consists of a complex convolution layer, a complex normalization layer, and a complex activation layer. The convolution kernel size is 3, the stride is 1, and the padding is 1. This module incorporates a residual addition operation during the calculation process.

[0058] Network training: The first-stage network first uses the first half of the complex neural network to extract the multi-level, multi-scale features of the image, and transmits them to the corresponding second half of the network via skip connections. Each module in the second half combines the features of the two paths to roughly restore the image's general appearance. The second and third-stage networks further refine and supplement the image's details at different receptive fields. This method combines the advantages of complex representation and networks with different receptive fields to achieve good completion results on the image to be completed. The network operation process is as follows:

[0059] O=Net3(Net2(Net1(I,M),M),M)

[0060] Among them, Net i represents the i-th stage network, I and M represent the input image and mask respectively.

[0061] Finally, the network is trained using the loss function. The three-stage network has a loss function, where the loss function of Net1 is λ g The value is 0.1.

[0062] Pixel loss is defined as follows:

[0063]

[0064] Where O1 is the output of the first-stage network, ⊙ represents the dot product, ||||1 represents the 1-norm, and λ h The value is 6.

[0065] Fighting Losses The definition of is as follows, where D represents the discriminator network.

[0066] M1=I⊙(1-M)+O1⊙M

[0067]

[0068] The training loss of Net2 is defined as follows, where λ tv ,λ per and λ sty The values ​​are 0.1, 0.05 and 120.

[0069]

[0070] In the formula Represents the pixel loss of Net2, defined by analogy.

[0071] In the formula The definition is as follows, M2 is defined by analogy with M1, and M2(i,j) represents the value of the corresponding coordinate in the image.

[0072]

[0073] In the formula and There are two losses defined on VGG-16, as follows:

[0074]

[0075] Where F i (·) represents the feature map of the i-th layer (i∈{5,10,17}) when the data is input to the pre-trained network VGG-16, and G i (·)=F i (·)F i (·) T .

[0076] The training loss definition of Net3 is L3, which is derived from L2.

[0077] Gradient optimization uses the Adam optimizer with adaptive learning rate. For the three sub-networks of the generator and the discriminator, they each have their own Adam optimizer, and training can obtain the optimal network parameters.

[0078] Image completion implementation: Input the original image and the corresponding mask into a trained multi-stage complex neural network model to obtain a high-quality complete completed image.

[0079] In summary, the present invention proposes an image completion method based on a multi-stage complex neural network, which innovatively uses a combination of a multi-stage network and a complex neural module to complete the image. A real-complex conversion block is added at the beginning and end of each stage network. The first half uses a complex convolution module to extract multi-level and multi-scale features. The second half constructs a network by stacking complex transposed convolution blocks, and uses the extracted features to gradually restore the image. By using networks with different receptive fields at different stages, the present invention can simultaneously achieve rough repair and detailed optimization of the image. The multi-level and multi-scale features extracted by the different receptive field modules have better feature extraction capabilities and can obtain high-quality image completion results. The method of the present invention not only improves the quality of image completion, but also, under the same number of features, can use fewer parameters due to its efficient feature extraction capability, which makes the model more compact and efficient. Therefore, the present invention has broad application prospects in application fields such as image enhancement and medical image processing, and due to its high-quality image completion effect, it is of great significance to promote the development of the field of computer vision.

[0080] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0081] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0082] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0083] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0084] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0085] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0087] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0088] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0089] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0090] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0091] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0092] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for image completion based on a multi-stage complex neural network, characterized in that: The following steps are involved: A1. Establish a multi-stage complex neural network generator. The generator includes at least three stages, and each stage is processed by the following modules: One-stage network: Input image I and mask M are converted to the complex domain through the real-to-complex block (RCB), and then features are extracted through the complex convolution block (CCB). The intermediate features are stored for skip connection. The second half converts the features back to the real domain through the complex transposed convolution block (CTCB) and the complex-to-real block (CRB) to form a preliminary completed image; Two-stage network: takes the output image of the first stage as input, directly refines the image of the first stage output without using skip connections, and further refines the image through complex convolution block (CCB), complex transposed convolution block (CTCB) and complex to real block (CRB); Three-stage network: The output image of the second stage is input, similar to the first stage, and the final image refinement and completion is performed through the real-to-complex block (RCB), complex convolution block (CCB), complex transposed convolution block (CTCB) and complex-to-real block (CRB); A2. Adversarial training: Using the training set, the multi-stage complex neural network generator and the simple real neural network discriminator are trained adversarially. The multi-stage complex neural network is trained using the images and masks in the training set. The generator is responsible for filling in the missing parts of the image, and the discriminator evaluates the consistency of the image features output by the generator with the original image to optimize the completion quality of the generator. A3. Image completion: Use the trained multi-stage complex neural network generator to complete the new input image and mask, and output the complete completed image.

2. The image completion method based on a multi-stage complex neural network according to claim 1, wherein: In the three-stage network, the real-to-complex block generates the real and imaginary parts of the input image through two paths respectively. Both paths use the Conv3 convolution layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and BatchNorm normalization layer in sequence, repeated twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm batch normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer in sequence to restore it to a real image.

3. The image completion method based on a multi-stage complex neural network according to claim 1 or 2, characterized in that: The complex convolution blocks in both the one-stage and three-stage networks include a complex convolution layer, a complex activation function, and a complex batch normalization layer. The convolution kernel size of the complex convolution layer is 4, the stride is 2, and the padding is 1. In addition to extracting features, it also has a downsampling function.

4. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 3, characterized in that: The first half of the one-stage and three-stage networks consists of 8 complex convolution blocks, and the second half includes 8 complex transposed convolution blocks. Their parameters are the same as the complex convolution blocks, except that the convolution is changed to transposed convolution. In addition to extracting features, they also have upsampling functions. The second half of the network is symmetrical with the first half.

5. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 4, characterized in that: In the first half of the two-stage network, the first complex convolution block has a size of 7, a stride of 1, and only extracts features. The subsequent blocks have a size of 3, a stride of 2, and a padding of 1. They have both feature extraction and upsampling functions, and there are two of them. The middle part includes three complex residual convolution blocks, which sequentially include a complex convolution layer, a complex activation function, and a complex normalization layer, and only extract features. The second half includes a complex transposed convolution block that is symmetrical to the first half.

6. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 5, characterized in that: In step A2, the complete image and mask of the training dataset are used as the input of the network, and multi-level and multi-scale feature extraction is performed through complex convolution blocks. The features are then restored to the original image through complex transposed convolution layers, and the final network model is obtained through multiple training cycles.

7. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 6, characterized in that: In step A3, for each complete image, a corresponding mask is given to represent the area that needs to be completed, and the mask is passed into the trained multi-stage complex neural network. Finally, the network outputs the completed complete image.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image completion method based on a multi-stage complex neural network is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image completion method based on a multi-stage complex neural network is implemented.

Images