Image completion method based on multi-stage complex neural network

Abstract

The invention provides an image completion method based on a multi-stage complex neural network, and aims to improve the completion quality of damaged images. The method comprises the following steps that firstly, a multi-stage complex number neural network generator is constructed, each stage comprises a real number-to-complex number block, a complex number convolution block, a complex number transposed convolution block and a complex number-to-real number block, so that conversion from a real number field to a complex number field, feature extraction, up-sampling and final real number field conversion are achieved; secondly, training the generator and the discriminator by using a training set through adversarial training so as to optimize the completion effect of the generator; and finally, complementing the new input image and the mask by using the trained network to generate a complete image. By using the method provided by the invention, a high-quality image completion effect can be realized, and meanwhile, the method provided by the invention has fewer parameters under the condition of the same feature number.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to an image completion method based on a multi-stage complex-valued neural network (MS-CN). Background Art

[0002] Images are stored in computers in digital form, but in the actual shooting and transmission process, they are always inevitably interfered by noise. In addition, due to factors such as the condition of the transmission equipment, the final image is often damaged, which seriously interferes with subsequent image analysis and processing. In order to meet the needs of image-based applications, people have proposed various image processing technologies, such as image denoising, image super-resolution, and image completion. Among them, the purpose of image completion is to provide visually appreciable filling for the missing areas of the damaged image, which has become an important research direction in the field of computer vision and is also widely used in image enhancement, medical image processing, etc.

[0003] Current image completion algorithms are often based on real numbers. However, complex numbers are often used in practical applications in the fields of communications, bioinformatics, speech recognition, image processing, etc. This shows that using complex numbers to represent the input, output, and parameters of neural networks has potential appeal in these related fields. Complex neural networks are neural networks that use complex parameters and complex variables to process related information. The main difference between them and real neural networks is the expression of multiplication operations - the multiplication function of complex numbers involves phase rotation and amplitude modulation, so a certain degree of freedom can be reduced. Self-organization and high learning freedom are the characteristics of real neural networks. Complex neural networks can reduce the potential dangers caused by high freedom by knowing the amplitude and phase in the data a priori. Since the phase information describes the shape, edge, and direction of the object in the image in detail, the amount of information stored in the phase information can recover most of the information encoded by its amplitude. Complex neural networks also have good development prospects in analyzing and processing information. In recent years, complex neural networks have gained increasing attention in the field of machine learning.

[0004] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0005] The main purpose of the present invention is to provide an image completion method based on a multi-stage complex neural network to solve the technical problem of low quality of image completion results in the prior art.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for image completion based on a multi-stage complex neural network comprises the following steps:

[0008] A1. Establish a multi-stage complex neural network generator, which includes at least three stages, and each stage is processed by the following modules:

[0009] One-stage network: input image I and mask M, convert the input to the complex domain through the real-to-complex block (RCB), then extract features through the complex convolution block (CCB), the intermediate feature storage is used for skip connection, and the second half converts the features back to the real domain through the complex transposed convolution block (CTCB) and the complex-to-real block (CRB) to form a preliminary completed image;

[0010] Two-stage network: Input the output image of the first stage, without using skip connections, directly refine the image of the first stage output, and further refine the image through complex convolution block (CCB), complex transposed convolution block (CTCB) and complex to real block (CRB).

[0011] Three-stage network: The output image of the second stage is input, similar to the first stage, and the final image refinement and completion is performed through the real-to-complex block (RCB), complex convolution block (CCB), complex transposed convolution block (CTCB) and complex-to-real block (CRB);

[0012] A2. Adversarial training: using a training set to perform adversarial training on the multi-stage complex neural network generator and the simple real neural network discriminator, and using the images and masks in the training set to train a multi-stage complex neural network, wherein the generator is responsible for repairing the missing parts in the image, and the discriminator evaluates the consistency of the image features output by the generator with the original image, so as to optimize the completion quality of the generator;

[0013] A3. Image completion: Use the trained multi-stage complex neural network generator to complete the new input image and mask and output the complete completed image.

[0014] Furthermore, in the three-stage network, the real-to-complex block generates the real part and imaginary part of the complex number from the input image through two paths. Both paths use the Conv3 convolution layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and the BatchNorm normalization layer in turn, and repeat twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm batch normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer in turn to restore it to a real image.

[0015] Furthermore, the complex convolution blocks in the one-stage and three-stage networks include complex convolution layers, complex activation functions, and complex batch normalization layers, where the convolution kernel size of the complex convolution layer is 4, the step size is 2, and the padding is 1. In addition to extracting features, it also has a downsampling function.

[0016] Furthermore, the first half of the one-stage and three-stage networks consists of 8 complex convolution blocks, and the second half includes 8 complex transposed convolution blocks. Their parameters are the same as the complex convolution blocks, except that the convolution is changed to transposed convolution, which has an upsampling function in addition to feature extraction. The second half of the network is symmetrical with the first half.

[0017] Furthermore, in the first half of the two-stage network, the first complex convolution block has a size of 7, a stride of 1, and only extracts features. The subsequent block size is 3, a stride of 2, and a padding of 1. It has feature extraction and upsampling functions, and the number is 2. The middle part includes 3 complex residual convolution blocks, which sequentially include a complex convolution layer, a complex activation function, and a complex normalization layer, and only extracts features. The second half includes a complex transposed convolution block that is symmetrical to the first half.

[0018] Furthermore, in step A2, the complete image and mask of the training data set are used as the input of the network, and multi-level and multi-scale feature extraction is performed through complex convolution blocks. The features are then restored back to the original image through complex transposed convolution layers, and the final network model is obtained through multiple trainings.

[0019] Furthermore, in step A3, for each complete image, a corresponding mask is given to indicate the area to be completed, and the mask is passed to the trained multi-stage complex neural network. Finally, the network outputs the completed complete image.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the image completion method based on a multi-stage complex neural network.

[0021] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the image completion method based on a multi-stage complex neural network is implemented.

[0022] In some embodiments of the present invention, the following steps are included:

[0023] Step 1: Establish a multi-stage complex neural network, which consists of a complex conversion block (Real-Complex Block, RC), a complex convolution block (Complex-Valued ConvolutionBlock, CCB), a complex transposed convolution block (Complex-Valued Transposed ConvolutionBlock, CTCB) and a complex residual convolution block (Complex-Valued ConvolutionResidual Block, CCRB). The one-stage and three-stage networks extract feature information of each size through the complex convolution block in the first half and transmit it to the second half using skip connections. Each complex transposed convolution block in the second half receives the output of the previous module and the feature information corresponding to the first half for further processing, making full use of the information at the bottom and high levels. At the same time, establish a simple discriminator to determine whether the image is reasonable, and conduct adversarial training with the multi-stage complex neural network; Step 2: Use the images and masks in the training set to conduct adversarial training on the generator and discriminator. In this process, the generator is responsible for filling in the missing parts of the image, while the discriminator evaluates the output quality of the generator, thereby guiding the generator to use complex neural networks to extract image features in multiple stages, complete the occluded areas, and improve their details and quality. Step 3: Use the trained generator to complete the input image and mask to obtain the output complete image.

[0024] The first step specifically includes: establishing a network structure based on a multi-stage complex neural network, and all networks are headed and tailed by real-to-complex blocks and complex-to-real blocks respectively. The real-to-complex block generates the real part and imaginary part of the complex number of the input image through two paths respectively. The paths all use the Conv3 layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and the BatchNorm normalization layer in turn, and repeat twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3, BatchNorm, ReLU, Conv3 and Tanh activation layers in turn. The intermediate body of the first and third stage networks is composed of complex convolution blocks and complex transposed convolution blocks. The complex convolution blocks are all composed of complex convolution layers, complex activation functions and complex batch normalization layers, wherein the complex convolution layer has a convolution kernel size of 4, a step size of 2, and a step size of 1. In addition to extracting features, it also has a downsampling function. The first half of the network consists of 8 complex convolution blocks, and the second half consists of 8 complex transposed convolution blocks. Their parameters are the same as the complex convolution blocks, except that they are changed from convolution to transposed convolution, which has the function of upsampling in addition to feature extraction. The second half of the network is symmetrical to the first half. In the first half of the second stage of the main body of the network, the first complex convolution block has a size of 7 and a stride of 1, which only extracts features. The subsequent blocks have a size of 3, a stride of 2, and a padding of 1, which have the functions of feature extraction and upsampling, and the number is 2. The middle part consists of 3 complex residual convolution blocks, which are composed of a complex convolution layer, a complex activation function, and a complex normalization layer in sequence, and only extract features. The second half consists of complex transposed convolution blocks that are symmetrical to the first half.

[0025] The second step specifically includes: taking the complete image and mask of the training data set as the input of the network, performing multi-level and multi-scale feature extraction through complex convolution blocks, and then restoring the features back to the original image through complex transposed convolution layers, and obtaining the final network model through multiple trainings.

[0026] The third step specifically includes: for each complete image, a corresponding mask is provided to represent the area to be completed, and the two are passed together into a trained multi-stage complex neural network. The network finally outputs the completed complete image.

[0027] The present invention has the following beneficial effects:

[0028] The present invention proposes an image completion method based on a multi-stage complex neural network, which can achieve high-quality image completion effects. The present invention uses a combination of a multi-stage network and a complex neural module in an image completion method for the first time, adds a real-complex conversion block at the beginning and end of each stage of the network, and uses a complex convolution module to extract multi-level and multi-scale features in the first half. The second half builds a network by stacking complex transposed convolution blocks, and uses the extracted features to gradually restore the image. In addition, the multi-stage complex neural network proposed in the present invention can take into account both rough image repair and detailed optimization through stage networks with different receptive fields, and is more suitable for image completion. The multi-level and multi-scale features extracted by different receptive field modules have better feature extraction capabilities and can obtain high-quality image completion results. In addition, the method of the present invention has fewer parameters under the same number of features.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 : is a flowchart of an image completion method based on a multi-stage complex neural network according to an embodiment of the present invention.

[0031] Figure 2 Schematic diagram of the network structure of the image completion method based on a multi-stage complex neural network according to an embodiment of the present invention

[0032] Figure 3 The reconstruction process of the image completion method based on a multi-stage complex neural network in the embodiment of the present invention is DETAILED DESCRIPTION

[0033] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0034] The image completion method of the present invention mainly includes the following processes: the first step: establishing a multi-stage complex-valued neural network (MS-CN) based on the basic structure of a complex neural network as an image completion generator, and a simple real neural network as a discriminator. The generator contains a total of three stages of networks, consisting of a real-complex block (RC), a complex convolution block (CCB), a complex transposed convolution block (CTCB) and a complex residual convolution block (CCRB). In the first stage, the preliminary results of image completion are generated by a complex neural network with a large receptive field; in the second stage, the preliminary results are refined by a complex neural network with a small receptive field; in the third stage, the second stage results are refined by a complex neural network with a large receptive field. The second step: using a data set to form adversarial training in the generator and the discriminator, pre-training a multi-stage complex neural network with reasonable weights. Step 3: Accept the image and mask as model input, use the multi-stage complex neural network obtained by adversarial training to complete it, and get the complete image as output.

[0035] See also Figures 1 to 3 , an embodiment of the present invention provides an image completion method based on a multi-stage complex neural network, comprising the following steps:

[0036] A1. Establish a multi-stage complex neural network generator, which includes at least three stages, and each stage is processed by the following modules:

[0037] One-stage network: input image I and mask M, convert the input to the complex domain through the real-to-complex block (RCB), then extract features through the complex convolution block (CCB), the intermediate feature storage is used for skip connection, and the second half converts the features back to the real domain through the complex transposed convolution block (CTCB) and the complex-to-real block (CRB) to form a preliminary completed image;

[0038] Two-stage network: Input the output image of the first stage, without using skip connections, directly refine the image of the first stage output, and further refine the image through complex convolution block (CCB), complex transposed convolution block (CTCB) and complex to real block (CRB).

[0039] Three-stage network: The output image of the second stage is input, similar to the first stage, and the final image refinement and completion is performed through the real-to-complex block (RCB), complex convolution block (CCB), complex transposed convolution block (CTCB) and complex-to-real block (CRB);

[0040] A2. Adversarial training: using a training set to perform adversarial training on the multi-stage complex neural network generator and the simple real neural network discriminator, and using the images and masks in the training set to train a multi-stage complex neural network, wherein the generator is responsible for repairing the missing parts in the image, and the discriminator evaluates the consistency of the image features output by the generator with the original image, so as to optimize the completion quality of the generator;

[0041] A3. Image completion: Use the trained multi-stage complex neural network generator to complete the new input image and mask and output the complete completed image.

[0042] In step A1, the first half of the one-stage and three-stage networks convert the image and mask from the real domain to the complex domain through the real-to-complex block (RCB), then extract features through several CCBs, and store intermediate features for skip connections. The CTCB in the second half first connects the output features of the previous step with the intermediate features corresponding to the first half in the feature dimension, and then further restores the features. Finally, the complex features are converted to images through the complex-to-real block (CRB).

[0043] The second-stage network needs to meet the low receptive field, so no skip connection is used. The first-stage output image is only refined by passing the real-to-complex block, the complex convolution block, the complex transposed convolution block, and the complex-to-real block in sequence.

[0044] The discriminator is mainly used for training and has no direct relationship with image completion, so a simple real neural network consisting of real convolution blocks is used. Its function is to determine whether the output of the generator has the same features as the original image, so as to provide part of the optimization target.

[0045] like Figure 2As shown, in a preferred embodiment, step A1 specifically includes: establishing a multi-stage complex neural network structure, including a real-complex conversion block, a complex convolution block, a complex transposed convolution block and a complex residual convolution block. The real-complex conversion blocks of the three-stage network are divided into two types, real-to-complex blocks and complex-to-real blocks. The real-to-complex block generates the real part and imaginary part of the complex number of the input image through two paths. Both paths use the Conv3 convolution layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and the BatchNorm normalization layer in turn, repeating twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm batch normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer in turn to restore it to a real image.

[0046] like Figure 2 As shown, in a preferred embodiment, the complex convolution blocks in the one-stage and three-stage networks include a complex convolution layer, a complex activation function, and a complex batch normalization layer, wherein the convolution kernel size of the complex convolution layer is 4, the step size is 2, and the padding is 1. In addition to extracting features, it also has a downsampling function. Further, the first half of the one-stage and three-stage networks is composed of 8 complex convolution blocks, and the second half includes 8 complex transposed convolution blocks, whose parameters are the same as the complex convolution blocks, only the convolution is changed to a transposed convolution, which has an upsampling function in addition to extracting features, and the second half of the network is symmetrical with the first half.

[0047] like Figure 2 As shown, in a preferred embodiment, in the first half of the two-stage network, the first complex convolution block has a size of 7, a step size of 1, and only extracts features; the subsequent block size is 3, the step size is 2, and the padding is 1, which has feature extraction and upsampling functions, and the number is 2; the middle part includes 3 complex residual convolution blocks, which sequentially include a complex convolution layer, a complex activation function, and a complex normalization layer, and only extracts features; the second half includes a complex transposed convolution block symmetrical to the first half.

[0048] In step A2, the generator and discriminator are adversarially trained using the training set, and a multi-stage complex neural network is trained using the images and masks in the training set. The completion is completed through rough patching and detailed optimization.

[0049] like Figure 1 and Figure 3 As shown, in step A2, the complete image and mask of the training data set are used as the input of the network, and multi-level and multi-scale feature extraction is performed through complex convolution blocks. The features are then restored back to the original image through complex transposed convolution layers, and the final network model is obtained through multiple trainings.

[0050] In step A3, the input image and mask are complemented using the trained complex neural network to obtain a complete image.

[0051] like Figure 1 and Figure 3 As shown, in step A3, for each complete image, a corresponding mask is given to indicate the area to be completed, and the mask is passed into the trained multi-stage complex neural network. Finally, the network outputs the completed complete image.

[0052] The specific embodiments of the present invention and algorithm examples thereof are further described below.

[0053] A method for image completion based on a multi-stage complex neural network. Components of the one- and three-stage network structure: The network consists of real-to-complex blocks and complex-to-real blocks at the beginning and end, respectively. The front and back parts of the main body of the network consist of 8 complex convolution blocks and complex transposed convolution blocks, respectively. There are jump connections before and after the network to achieve feature transfer. Components of the two-stage network structure: The network is the same as the one- and three-stage. The front and back parts of the main body of the network consist of 3 complex convolution blocks and complex transposed convolution blocks, respectively. The middle part of the network consists of 3 complex residual convolution blocks, which are used to further process the extracted features.

[0054] Real to complex block: Figure 2 As shown in the figure (light green arrows), the module includes two identical paths that generate the real and imaginary parts of complex features respectively. Each path passes the image through a real convolution layer Conv3 with a convolution kernel size of 3, a stride of 1, and a padding of 1, a BatchNorm normalization layer, and a ReLU activation layer, repeated twice. Complex to real block: Figure 2 As shown in the figure (marked by the light yellow arrow), the module connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer.

[0055] Complex convolutional block: such as Figure 2 As shown in the figure (light blue arrow), the module consists of a complex convolution layer, a complex normalization layer, and a complex activation layer. The convolution kernel size of the first and third stage complex convolution layers is 4, the step size is 2, and the padding is 1, which has both feature extraction and downsampling functions. The structure of the second stage complex convolution block is the same, except that the first convolution kernel size is 7, the step size is 1, and the padding is 0, while the next two convolution kernel sizes are 3, the step size is 2, and the padding is 1, which has both feature extraction and downsampling functions.

[0056] Complex transposed convolution block: Figure 2As shown in the figure (marked by pink arrows), the module consists of a complex transposed convolution layer, a complex normalization layer, and a complex activation layer. The convolution kernel size in the first three-stage network is 4, the stride is 2, and the padding is 1. This block accepts the output of the previous module and the corresponding block in the first half of the network as input and processes it. In the two-stage network, the first two convolution kernels have a size of 3, a stride of 2, a padding of 1, and an external padding of . The last convolution kernel has a size of 7, a stride of 1, and a padding of 0.

[0057] Complex residual convolution block: such as Figure 2 As shown in the figure (marked by brown-red arrows), the module consists of a complex convolution layer, a complex normalization layer, and a complex activation layer. The convolution kernel size is 3, the stride is 1, and the padding is 1. The module adds a residual addition operation during the calculation process.

[0058] Network training part: The first-stage network first uses the first half of the complex neural network to extract the multi-level and multi-scale features of the image, and transmits them to the corresponding second half of the network through jump connections. Each module in the second half combines the features of the two paths to roughly restore the general appearance of the image. The second and third-stage networks further refine and supplement the details of the image on different receptive fields. This method can combine the advantages of complex representation and networks with different receptive fields to achieve good completion effects on the image to be completed. The operation process of the network is:

[0059] O=Net3(Net2(Net1(I,M),M),M)

[0060] Among them, Net i represents the i-th stage network, I and M represent the input image and mask respectively.

[0061] Finally, the network is trained using the loss function. The three-stage network has a loss function, among which the loss function of Net1 is λ g The value is 0.1.

[0062] Pixel loss is defined as follows:

[0063]

[0064] Where O1 is the output of the first-stage network, ⊙ represents the dot product, ||||1 represents the 1-norm, and λ h The value is 6.

[0065] Fighting Losses The definition of is as follows, where D represents the discriminator network.

[0066] M1=I⊙(1-M)+O1⊙M

[0067]

[0068] The training loss of Net2 is defined as follows, where λ tv , per With λ sty The values ​​are 0.1, 0.05 and 120.

[0069]

[0070] In the formula Represents the pixel loss of Net2, defined by analogy.

[0071] In the formula The definition is as follows, M2 is defined by analogy with M1, and M2(i,j) represents the value of the corresponding coordinate in the image.

[0072]

[0073] In the formula and There are two losses defined on VGG-16, as follows:

[0074]

[0075] Where F i (·) represents the feature map of the i-th layer (i∈{5,10,17}) when the data is input into the pre-trained network VGG-16 for calculation, and G i (·)=F i (·)F i (·) T .

[0076] The training loss definition of Net3 L3 is derived from L2.

[0077] Gradient optimization uses the Adam optimizer with adaptive learning rate. For the three sub-networks of the generator and the discriminator, each has its own Adam optimizer, and training can obtain the optimal network parameters.

[0078] Image completion implementation: Input the original image and the corresponding mask into the trained multi-stage complex neural network model to obtain a high-quality complete completed image.

[0079] In summary, the present invention proposes an image completion method based on a multi-stage complex neural network, which innovatively uses a combination of a multi-stage network and a complex neural module to complete the image, adds a real-complex conversion block at the beginning and end of each stage of the network, uses a complex convolution module to extract multi-level and multi-scale features in the first half, and constructs a network by stacking complex transposed convolution blocks in the second half, and gradually restores the image using the extracted features. By using networks with different receptive fields at different stages, the present invention can simultaneously achieve rough repair and detailed optimization of the image. The multi-level and multi-scale features extracted by different receptive field modules have better feature extraction capabilities and can obtain high-quality image completion results. The method of the present invention not only improves the quality of image completion, but also, under the same number of features, due to its efficient feature extraction capability, can use fewer parameters, which makes the model more compact and efficient. Therefore, the present invention has broad application prospects in application fields such as image enhancement and medical image processing, and due to its high-quality image completion effect, it is of great significance to promote the development of the field of computer vision.

[0080] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0081] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0082] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0083] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0084] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0085] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0086] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0087] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0088] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0089] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0090] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0091] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0092] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for image completion based on a multi-stage complex neural network, characterized in that: The following steps are involved: A1. Establish a multi-stage complex neural network generator, which includes at least three stages, and each stage is processed by the following modules: One-stage network: input image I and mask M, convert the input to the complex domain through the real-to-complex block (RCB), then extract features through the complex convolution block (CCB), the intermediate feature storage is used for skip connection, and the second half converts the features back to the real domain through the complex transposed convolution block (CTCB) and the complex-to-real block (CRB) to form a preliminary completed image; Two-stage network: Input the output image of the first stage, without using skip connections, directly refine the image of the first stage output, and further refine the image through complex convolution block (CCB), complex transposed convolution block (CTCB) and complex to real block (CRB). Three-stage network: The output image of the second stage is input, similar to the first stage, and the final image refinement and completion is performed through the real-to-complex block (RCB), complex convolution block (CCB), complex transposed convolution block (CTCB) and complex-to-real block (CRB); A2. Adversarial training: using a training set to perform adversarial training on the multi-stage complex neural network generator and the simple real neural network discriminator, and using the images and masks in the training set to train a multi-stage complex neural network, wherein the generator is responsible for repairing the missing parts in the image, and the discriminator evaluates the consistency of the image features output by the generator with the original image, so as to optimize the completion quality of the generator; A3. Image completion: Use the trained multi-stage complex neural network generator to complete the new input image and mask and output the complete completed image.

2. The image completion method based on a multi-stage complex neural network as claimed in claim 1, characterized in that: In the three-stage network, the real-to-complex block generates the real part and imaginary part of the complex number through two paths. Both paths use the Conv3 convolution layer with a convolution kernel size of 3 to extract features, and then pass through the ReLU activation layer and BatchNorm normalization layer in turn, repeated twice in total. The complex-to-real block directly connects the real part of the data with the complex number, and then passes through the Conv3 convolution layer, BatchNorm batch normalization layer, ReLU activation layer, Conv3 convolution layer and Tanh activation layer in turn to restore it to a real image.

3. The image completion method based on a multi-stage complex neural network according to claim 1 or 2, characterized in that: The complex convolution blocks in the one-stage and three-stage networks include complex convolution layers, complex activation functions, and complex batch normalization layers. The convolution kernel size of the complex convolution layer is 4, the step size is 2, and the padding is 1. In addition to extracting features, it also has a downsampling function.

4. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 3, characterized in that: The first half of the one-stage and three-stage networks consists of 8 complex convolution blocks, and the second half includes 8 complex transposed convolution blocks. Their parameters are the same as the complex convolution blocks, except that the convolution is changed to transposed convolution. In addition to extracting features, it also has an upsampling function. The second half of the network is symmetrical with the first half.

5. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 4, characterized in that: In the first half of the two-stage network, the first complex convolution block has a size of 7, a step size of 1, and only extracts features. The subsequent block size is 3, the step size is 2, and the padding is 1. It has feature extraction and upsampling functions, and there are 2 of them. The middle part includes 3 complex residual convolution blocks, which sequentially include a complex convolution layer, a complex activation function, and a complex normalization layer, and only extracts features. The second half includes a complex transposed convolution block that is symmetrical to the first half.

6. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 5, characterized in that: In step A2, the complete image and mask of the training data set are used as the input of the network, and multi-level and multi-scale feature extraction is performed through complex convolution blocks. The features are then restored back to the original image through complex transposed convolution layers, and the final network model is obtained through multiple trainings.

7. The image completion method based on a multi-stage complex neural network according to any one of claims 1 to 6, characterized in that: In step A3, for each complete image, a corresponding mask is given to indicate the area to be completed, and the mask is passed to the trained multi-stage complex neural network. Finally, the network outputs the completed complete image.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image completion method based on a multi-stage complex neural network as described in any one of claims 1 to 7 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image completion method based on a multi-stage complex neural network as described in any one of claims 1 to 7 is implemented.

Images