Image wrinkle removal processing method, system, terminal device and storage medium

Abstract

The present invention provides an image wrinkle removal method, system, terminal device, and storage medium. The method comprises: training a mapping network based on a defect atlas and a wrinkle atlas, and training a coarse-repair network based on the trained mapping network and wrinkle atlas; inputting a to-be-processed image and a mask image into the trained coarse-repair network for image restoration, thereby obtaining a first restored image; inputting the first restored image and the mask image into a fine-repair network for image restoration, thereby obtaining a second restored image; and superimposing the second restored image with the first restored image to obtain a wrinkle-removed image. By training the mapping network, the present invention reduces the impact of the distribution difference between real and simulated data on the restoration model. Using the coarse-repair network and the fine-repair network, the first restored image is restored from coarse to fine wrinkles on a human portrait, thereby reducing the smearing effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image wrinkle removal method, system, terminal device and storage medium. Background Art

[0002] As the imaging resolution of digital imaging devices increases, images captured by these devices can now clearly display facial wrinkles. Consequently, users are increasingly demanding the ability to process facial wrinkles. Therefore, how to process facial wrinkles in images has become a technical challenge facing those skilled in the art.

[0003] In the existing image wrinkle removal process, after wrinkles are identified, similarity replacement or Poisson fusion is used to remove wrinkles in the wrinkle area. However, due to the lack of understanding of the overall semantic information of the image, severe smearing and wrinkle faults are prone to occur, which reduces the effect of image wrinkle removal. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide an image wrinkle removal method, system, terminal device and storage medium, aiming to solve the problem of poor image wrinkle removal effect caused by the lack of understanding of the overall semantic information of the image in the existing image wrinkle removal process.

[0005] The embodiment of the present invention is implemented as follows: a method for image wrinkle removal, the method comprising:

[0006] Training a mapping network based on a defect atlas and a wrinkle atlas, and training a coarse repair network based on the trained mapping network and the wrinkle atlas;

[0007] Inputting the image to be processed and the mask image into the trained coarse repair network to perform image repair, thereby obtaining a first repaired image;

[0008] The first repaired image and the mask image are input into a refinement network for image repair to obtain a second repaired image, and the second repaired image is superimposed on the first repaired image to obtain a wrinkle-removed image.

[0009] Furthermore, the training of the mapping network based on the defect atlas and the wrinkle atlas includes:

[0010] Sampling the defect atlas and the wrinkle atlas respectively to obtain a defect sampling image and a wrinkle simulation sampling image;

[0011] Inputting the defect sampling image and the wrinkle simulation sampling image into the encoder in the mapping network for image encoding respectively to obtain a defect latent vector and a wrinkle simulation latent vector;

[0012] Inputting the defect latent vector and the wrinkle simulation latent vector into a decoder in the mapping network for image decoding to obtain a defect decoded image and a wrinkle simulation decoded image;

[0013] Performing loss calculation based on the defect sampling graph, the wrinkle simulation sampling graph, the defect decoded image, and the wrinkle simulation decoded image to obtain an image reconstruction loss value of the mapping network;

[0014] performing similarity loss calculation based on the defect latent vector and the wrinkle simulation latent vector to obtain a feature loss value of the mapping network, and constructing a model loss value of the mapping network based on the feature loss value and the image reconstruction loss value;

[0015] Parameters of the mapping network are updated according to the model loss value until the mapping network converges.

[0016] Furthermore, the formula used for calculating the loss based on the defect sampling image, the wrinkle simulation sampling image, the defect decoded image, and the wrinkle simulation decoded image is:

[0017]

[0018]

[0019] The formula used for calculating the similarity loss based on the defect latent vector and the wrinkle simulation latent vector is:

[0020] L D =E r～R [ReLU(1-D(E(r)))]+E x～X [ReLU(D((x)))];

[0021] L E =-E x～x [D(E(x))]

[0022] Where r is the wrinkle simulation sampling image, x is the defect sampling image, x′ is the defect decoded image, r′ is the wrinkle simulation decoded image, G k is the kth layer of the VGG network, D is the discriminator, and E is the encoder.

[0023] Furthermore, the training of the rough network based on the trained mapping network and the training atlas includes:

[0024] Sampling the training atlas to obtain a wrinkle pairing map, wherein the wrinkle pairing map includes a wrinkle map, a wrinkle removal map, and a wrinkle mask map;

[0025] Inputting the wrinkle pairing graph into the encoder of the rough-repair network for encoding to obtain a latent vector, and inputting the latent vector into the generator of the rough-repair network;

[0026] For the generator in the rough-repair network, instruct each decoding layer to merge and convolve the feature map output by the corresponding encoding layer with the output feature map of the previous layer decoder to obtain the feature input of the current layer and the residual map of the current layer;

[0027] The features of the current layer are convolved and sent to the next decoding layer;

[0028] The current layer residual map is merged with the previous layer residual map to obtain the current layer decoded output image, and the parameters of the generator in the coarse repair network are updated according to the wrinkle removal map and the decoded output image until the coarse repair network converges.

[0029] Furthermore, inputting the first restoration image and the mask image into a refinement network for image restoration includes:

[0030] Encoding the first repair image and the mask image to obtain a feature map, and scaling and translating the feature map;

[0031] The scaled and translated feature map is decoded to obtain the second repaired map.

[0032] Furthermore, the formula used to regularize the feature map is:

[0033]

[0034]

[0035]

[0036] Among them, F in is the feature map, i, j, k are all array subscripts, C represents the number of channels, H, W are the height and width of the wrinkle mask, then i∈[0, C], j∈[0, H], k∈[0, W].

[0037] Furthermore, the feature map is scaled and translated:

[0038]

[0039] Among them, I n represents the input image of the nth layer of the refinement network, and ψ and Ψ represent convolution operations.

[0040] Another object of an embodiment of the present invention is to provide an image wrinkle removal processing system, the system comprising:

[0041] A network training module, configured to train a mapping network based on a defect atlas and a wrinkle atlas, and to train a rough repair network based on the trained mapping network and wrinkle atlas;

[0042] An image restoration module is configured to input the image to be processed and the mask image into the trained coarse restoration network to perform image restoration, thereby obtaining a first restored image.

[0043] The image wrinkle removal module is used to input the first repair image and the mask image into the refinement network to perform image repair to obtain a second repair image, and superimpose the second repair image with the first repair image to obtain a wrinkle-removed image.

[0044] Another object of an embodiment of the present invention is to provide a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0045] Another object of an embodiment of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0046] In an embodiment of the present invention, a coarse repair network is trained using a trained mapping network and a wrinkle atlas, and the image to be processed and the mask image are input into the trained coarse repair network for image restoration, thereby making fuller use of the simulated data set and reducing the sample collection cost. A feature fusion module is used to perform feature fusion and image prediction from the bottom up to improve the accuracy of coarse repair. A feature scaling module further reduces the difficulty of training the fine repair network. The coarse repair network and the fine repair network are used to perform portrait wrinkle restoration from coarse to fine on the first repair image, thereby reducing the smearing phenomenon and improving the image quality of the wrinkle-removed image after image wrinkle removal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of an image wrinkle removal method provided by a first embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the structure of a mapping network provided by the first embodiment of the present invention;

[0049] Figure 3 is an overview diagram of the refined network provided by the first embodiment of the present invention;

[0050] Figure 4 2 is a schematic structural diagram of a self-normalization (self-Nonrm) module provided in the first embodiment of the present invention;

[0051] Figure 5is a flow chart of an image wrinkle removal method provided by a second embodiment of the present invention;

[0052] Figure 6 2 is a schematic diagram of the structure of a refined network provided by the second embodiment of the present invention;

[0053] Figure 7 is a schematic structural diagram of a coarse repair network provided by a second embodiment of the present invention;

[0054] Figure 8 2 is a schematic diagram of a coarse-block structure provided by the second embodiment of the present invention;

[0055] Figure 9 is a schematic structural diagram of a discriminator provided by a second embodiment of the present invention;

[0056] Figure 10 is a structural diagram of an image wrinkle removal processing system provided by a third embodiment of the present invention;

[0057] Figure 11 It is a schematic structural diagram of a terminal device provided in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0060] Example 1

[0061] See also Figure 1 , is a flow chart of an image wrinkle removal method provided by a first embodiment of the present invention. The image wrinkle removal method can be applied to any terminal device. The image wrinkle removal method includes the following steps:

[0062] Step S10, training a mapping network based on the defect atlas and the wrinkle atlas, and training a rough repair network based on the trained mapping network and the wrinkle atlas;

[0063] Among them, the blemish atlas is a set of pictures with wrinkles on the face or neck, represented by R, the wrinkle atlas is a set of pictures that helps portrait-related information simulate wrinkles on flawless pictures, represented by X, and the picture set that does not contain any facial or neck wrinkles is called a flawless picture.

[0064] Optionally, the training of the mapping network based on the defect atlas and the wrinkle atlas includes:

[0065] Sampling the defect atlas and the wrinkle atlas respectively to obtain a defect sampling image and a wrinkle simulation sampling image;

[0066] Among them, by adopting an unsupervised method to train the mapping network, the number of sampling of the defect atlas and the wrinkle atlas can be set according to demand, the defect sampling map obtained by sampling is r, r∈R, and the wrinkle simulation sampling map obtained by sampling is x, x∈X;

[0067] Inputting the defect sampling image and the wrinkle simulation sampling image into the encoder in the mapping network for image encoding respectively to obtain a defect latent vector and a wrinkle simulation latent vector;

[0068] Among them, see Figure 2 The mapping network (mappingNet) is an "encoder (E)-generator" structure, which consists of stacked convolutional layers. The encoder in the mapping network encodes the features of the defect sampling image and the wrinkle simulation sampling image respectively to obtain the defect latent vector and the wrinkle simulation latent vector;

[0069] The mapping network has a U-shaped structure. The decoder receives two signals from the encoder at the same time. One of the two signals comes from real image data (defect latent vector) and the other is simulated data (wrinkle simulation latent vector).

[0070] Inputting the defect latent vector and the wrinkle simulation latent vector into a decoder in the mapping network for image decoding to obtain a defect decoded image and a wrinkle simulation decoded image;

[0071] Performing loss calculation based on the defect sampling graph, the wrinkle simulation sampling graph, the defect decoded image, and the wrinkle simulation decoded image to obtain an image reconstruction loss value of the mapping network;

[0072] A similarity loss calculation is performed based on the defect latent vector and the wrinkle simulation latent vector to obtain a feature loss value of the mapping network, and a model loss value of the mapping network is constructed based on the feature loss value and the image reconstruction loss value.

[0073] In this step, when the mapping network is trained, r∈R and x∈X are input into the encoder to obtain the corresponding latent vectors. The latent vectors of both are input into the decoder at the same time. The decoder attempts to reconstruct the input signal, and its output is x', r'. To ensure that the encoder can effectively encode the input image while minimizing the difference between the simulated image and the real image in the feature space, the following loss design is made:

[0074] The formula used for calculating the loss based on the defect sampling image, the wrinkle simulation sampling image, the defect decoded image, and the wrinkle simulation decoded image is:

[0075] Reconstruction loss:

[0076]

[0077] Alpha_Perceptual_loss:

[0078]

[0079] Adv-loss:

[0080] Discriminator loss: L D =E r～R [ReLU(1-D(r))]+E x～x [ReLU(D(x))];

[0081] Encoder: L E =-E x～x [D(E(x))]

[0082] Where r is the wrinkle simulation sampling image, x is the defect sampling image, x′ is the defect decoded image, r′ is the wrinkle simulation decoded image, G k is the kth layer of the VGG model. D is the discriminator. Using Alpha_Perceptual_loss allows for more targeted intervention in the model's final output. Adv-loss is used to narrow the gap between simulated and real data in the latent space. Adv-loss consists of two parts: the discriminator loss, which optimizes the discriminator, and the encoder loss. Since the encoder is optimized, G is used to represent the encoder in the formula for easy distinction. Of the three loss functions, reconstruction loss and Alpha_Perceptual_loss are used to reconstruct the image, while Adv-loss is used to narrow the gap between the real and simulated images in the latent space.

[0083] Step S20, inputting the image to be processed and the mask image into the trained coarse repair network to perform image repair, thereby obtaining a first repaired image;

[0084] The coarse repair network takes the wrinkle-erased image and the wrinkle mask as input, and attempts to fill the erased part to achieve a preliminary repair effect of the image, thereby obtaining the first repaired image c.

[0085] Step S30: Input the first repaired image and the mask image into a refinement network for image repair to obtain a second repaired image, and superimpose the second repaired image with the first repaired image to obtain a wrinkle-removed image.

[0086] Among them, the output of the coarse repair network is the result of the initial image restoration. It takes the wrinkle-erased image as input and attempts to fill in these erased parts. Since it pays more attention to the missing areas of the image, when the stripes are coarse or wrinkles are concentrated, using only the coarse repair network is likely to give a smearing effect, so it is necessary to connect to the fine repair network for secondary image restoration.

[0087] See also Figure 3 This is an overview of the refinement network. The refinement network requires a larger receptive field, allowing the model to perform calculations based on a larger image range. It also considers the contextual information of the wrinkle area (surrounding skin texture and color) as well as the global characteristics of the image. Therefore, the contextual attention module (ContextureAttention) and dilated convolution (DilatedConv) are introduced. To facilitate model training, the refinement network also uses the wrinkle mask as part of the model input to guide model optimization. The loss functions used in the refinement network are Alpha_Perceptual_loss, adv-loss, and partial-weighted loss.

[0088] Regarding the local weighted loss function (partial-weighted Loss): L1-Loss is widely used in the field of image reconstruction. Since wrinkles account for a relatively small proportion of the entire image, using L1 to average the entire image can easily lead to a smearing effect in the result. In order to improve the wrinkle removal effect of the model, it is hoped that the model can pay more attention to the wrinkle mask area. Considering that wrinkles generally only cover the obvious crease area, while actual wrinkles have a certain transition area, both the transition area and the mask area need to be paid attention to in model optimization, and these need to be reflected in the loss. Partial-weighted Loss can solve this problem well:

[0089]

[0090]

[0091] Where k is the search radius, which is 3 in this embodiment, β=1.0, m i，j is the value of the wrinkle MASK at a specific pixel position, 1 represents normal skin, 0 represents wrinkles, H and W are the height and width of the image respectively. It can be seen that partial-weighted takes wrinkles together with their surrounding pixels into consideration and gives different evaluation weights according to their distance from the wrinkles.

[0092] Optionally, in this step, inputting the first restoration image and the mask image into a refinement network for image restoration includes:

[0093] Encoding the first repair image and the mask image to obtain a feature map, and scaling and translating the feature map;

[0094] Decoding the scaled and translated feature map to obtain the second repaired map;

[0095] Among them, the first repair image and mask image are encoded, mapped into feature maps through convolution operations, and the feature maps are scaled and translated, and the feature maps are decoded. The decoder consists of convolution, void convolution and context attention modules, and its decoding result is the second repaired image.

[0096] In this step, the image and mask are input into the refinement network, and feature extraction is performed through a stacked convolution-self-normalization (Conv-SNorm) module. The contextual information of the area to be repaired and the global image information are respectively extracted through the ContexturAttention branch and the Dilated-Conv branch. The contextual information and the global information are superimposed, and the refinement network output is obtained through convolution. Finally, the refined network output is superimposed with the first output to obtain the wrinkle-removed image. Preferably, the convolution-self-normalization (Conv-SNorm) consists of two parts: "convolution" and "self-normalization". The output of the convolution is a "feature map", and the feature map needs to be "translated and scaled" through SNorm to obtain the second repaired image.

[0097] Among them, for Figure 6 The Conv-SNorm operation in , consists of convolution (conv) and regularization term (self-Norm), and this embodiment provides a self-Nonrm structure (see Figure 4 ), in this step, the formula used to regularize the feature map is:

[0098]

[0099]

[0100]

[0101] Among them, F in is a feature map, i, j, k are all array subscripts, C represents the number of channels, H, W are the height and width of the wrinkle mask, then i∈[0, C], j∈[0, H], k∈[0, W].

[0102] Furthermore, assuming that the input feature map is NCHW, the ratio is the size of each batch, and the width and height of the feature map are set to N=1 for the convenience of representation. After normalization, re-scaling and translation are performed:

[0103]

[0104] In this embodiment, I n represents the input image of the nth layer of the refinement network, and ψ and Ψ represent convolution operations.

[0105] The trained mapping network and wrinkle atlas are used to train the coarse repair network, and the image to be processed and the mask map are input into the trained coarse repair network for image restoration. This makes fuller use of the simulated data set and reduces the sample collection cost. The feature fusion module performs bottom-up feature fusion and image prediction to improve the accuracy of coarse repair. The convolution-self-normalization module further reduces the difficulty of fine repair network training, and uses the PartialWeighted-Loss method to enable the model to hierarchically focus on the defective parts of the image to be processed for restoration, reducing the smearing phenomenon and improving the image quality of the wrinkle-removed image after image wrinkle removal.

[0106] Example 2

[0107] See also Figure 5 , is a flow chart of an image wrinkle removal method provided by a second embodiment of the present invention. This embodiment is used to further refine step S10, including the steps of:

[0108] Step S11, sampling the training atlas to obtain a wrinkle pairing map;

[0109] The wrinkle matching map includes: wrinkle map, wrinkle removal map, and wrinkle mask map.

[0110] Step S12, inputting the wrinkle pairing map into the encoder in the rough-refinement network for encoding to obtain a latent vector, and inputting the latent vector into the generator in the rough-refinement network;

[0111] Among them, see Figure 6 , the coarse-refinement network shares the same encoder with MappingNet, and introduces feature maps from each layer of the encoder into the decoder;

[0112] Step S13: for the generator in the rough-repair network, instruct each decoding layer to merge and convolve the feature map output by the corresponding encoding layer with the output feature map of the previous layer decoder to obtain the feature input of the current layer and the residual map of the current layer;

[0113] Facial wrinkles vary in thickness, length, and width, and different wrinkles are best processed at different scales. Therefore, the decoder is constructed using a "coarse-to-fine" approach. With the exception of L0, the output of each layer is obtained by adding the output of the current layer to the output of the previous layer. Furthermore, a coarse synthesis module (Coarse-Block) is designed at the feature fusion point.

[0114] Coarse-Block has two functions: one is to generate the image output of the current layer, and the other is to fuse the current image information with the feature map of the encoder to generate a new feature map. Its structure is as follows Figure 7 shown.

[0115] Step S14, performing convolution processing on the current layer features and sending them to the next decoding layer;

[0116] Step S15: Merge the current layer residual map with the previous layer residual map to obtain the current layer decoded output image, and update the parameters of the generator in the coarse repair network according to the wrinkle removal map and the decoded output image until the coarse repair network converges.

[0117] Figure 8 The Coarse-Block diagram in the illustration uses L1 as an example. L0 is actually the lowest-resolution layer in the model, so it only has one output. The Coarse-Block first merges the two feature images together for convolution, producing a three-channel image output (coarse image). To leverage the information in the Coarse-Image, the convolved merged image is combined with the corresponding wrinkle pairing image or wrinkle simulation sample image for subsequent calculations. The Coarse-Block output can be expressed as:

[0118] F a =ψ(f(x a ))

[0119] F b =θ(g(x b ))

[0120] F=F s *F b

[0121] Among them, f and g are convolution kernels, ψ is the sigmoid activation function, θ is the ReLU function, and the final output result is the product of the two feature maps.

[0122] The Loss in Coarse-Block is multi-scale Loss and Alpha_Perceptual_los. Among them, Alpha_Perceptual_los is consistent with MappingNet and will not be repeated.

[0123] Multi-scale Loss:

[0124]

[0125] Optionally, in this embodiment, adv-loss is used in many places, and this loss depends on the discriminator. In order to improve the image details, the discriminator in this embodiment splits the image into NxN patches and judges whether each patch is true or false. Its structure is as follows Figure 9 shown.

[0126] In this embodiment, through the design of Coarse-Block, the image output of the current layer can be effectively generated, and the current image information can be fused with the feature map of the encoder to generate a new feature map. Based on the generated new feature map, the image restoration effect of the coarse repair network is effectively improved.

[0127] Example 3

[0128] See also Figure 10 , is a schematic diagram of the structure of an image wrinkle removal processing system 100 provided in a third embodiment of the present invention, comprising: a network training module 10, an image restoration module 11 and an image wrinkle removal module 12, wherein:

[0129] The network training module 10 is used to train the mapping network based on the defect atlas and the wrinkle atlas, and to train the rough repair network based on the trained mapping network and the wrinkle atlas.

[0130] The network training module 10 is further configured to: sample the defect atlas and the wrinkle atlas respectively to obtain a defect sampling image and a wrinkle simulation sampling image;

[0131] Inputting the defect sampling image and the wrinkle simulation sampling image into the encoder in the mapping network for image encoding respectively to obtain a defect latent vector and a wrinkle simulation latent vector;

[0132] Inputting the defect latent vector and the wrinkle simulation latent vector into a decoder in the mapping network for image decoding to obtain a defect decoded image and a wrinkle simulation decoded image;

[0133] Performing loss calculation based on the defect sampling graph, the wrinkle simulation sampling graph, the defect decoded image, and the wrinkle simulation decoded image to obtain an image reconstruction loss value of the mapping network;

[0134] performing similarity loss calculation based on the defect latent vector and the wrinkle simulation latent vector to obtain a feature loss value of the mapping network, and constructing a model loss value of the mapping network based on the feature loss value and the image reconstruction loss value;

[0135] Parameters of the mapping network are updated according to the model loss value until the mapping network converges.

[0136] Optionally, the formula used for calculating the loss based on the defect sampling image, the wrinkle simulation sampling image, the defect decoded image, and the wrinkle simulation decoded image is:

[0137]

[0138]

[0139] L E =-E x～x [D(E(x))]

[0140] The formula used for calculating the similarity loss based on the defect latent vector and the wrinkle simulation latent vector is:

[0141] L D =E r～R [ReLU(1-D(E(r)))]+E x～x [ReLU(D((x)))];

[0142] Where r is the wrinkle simulation sampling image, x is the defect sampling image, x′ is the defect decoded image, r′ is the wrinkle simulation decoded image, G k is the generator of the kth layer, D is the discriminator, and E is the encoder.

[0143] Furthermore, the network training module 10 is further configured to: sample the training atlas to obtain a wrinkle simulation sampling image, and generate a wrinkle pairing image based on the wrinkle simulation sampling image, wherein the wrinkle pairing image includes a wrinkle image, a wrinkle removal image, and a wrinkle mask image;

[0144] Inputting the wrinkle pairing graph and the wrinkle simulation sampling graph into the encoder of the rough-repair network for encoding to obtain a latent vector, and inputting the latent vector into the generator of the rough-repair network;

[0145] For the generator in the rough-repair network, instruct each decoding layer to merge and convolve the feature map output by the corresponding encoding layer with the output feature map of the previous layer decoder to obtain the feature input of the current layer and the residual map of the current layer;

[0146] The features of the current layer are convolved and sent to the next decoding layer;

[0147] The current layer residual map is merged with the previous layer residual map to obtain the current layer decoded output image, and the parameters of the generator in the coarse repair network are updated according to the wrinkle removal map and the decoded output image until the coarse repair network converges.

[0148] The image restoration module 11 is configured to input the image to be processed and the mask image into the trained coarse restoration network to perform image restoration and obtain a first restored image.

[0149] The image wrinkle removal module 12 is configured to input the first repaired image and the mask image into a refinement network to perform image repair to obtain a second repaired image, and to superimpose the second repaired image with the first repaired image to obtain a wrinkle-removed image.

[0150] The image wrinkle removal module 12 is further configured to: encode the first repair image and the mask image to obtain a feature map, and scale and translate the feature map;

[0151] The scaled and translated feature map is decoded to obtain the second repaired map.

[0152] Optionally, the formula used to regularize the feature map is:

[0153]

[0154]

[0155]

[0156] Among them, F in is the feature map, i, j, k are all array subscripts, C represents the number of channels, H, W are the height and width of the wrinkle mask, then i∈[0, C], j∈[0, H], k∈[0, W].

[0157] Furthermore, the feature map is scaled and translated:

[0158]

[0159] Among them, I n represents the input image of the nth layer of the refinement network, and ψ and Ψ represent convolution operations.

[0160] In this embodiment, a coarse repair network is trained using a trained mapping network and a wrinkle atlas, and the image to be processed and the mask image are input into the trained coarse repair network for image restoration. This makes fuller use of the simulated data set and reduces the sample collection cost. The feature fusion module performs bottom-up feature fusion and image prediction to improve the accuracy of coarse repair. The feature scaling module further reduces the difficulty of training the fine repair network. The Partial Weighted-Loss method is used to enable the model to hierarchically focus on the defective parts of the image to be processed for restoration, thereby reducing the smearing phenomenon and improving the image quality of the wrinkle-removed image after image wrinkle removal.

[0161] Example 4

[0162] Figure 11 This is a structural block diagram of a terminal device 2 provided in the fourth embodiment of the present application. Figure 11 As shown, the terminal device 2 of this embodiment includes: a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for an image wrinkle removal method. When the processor 20 executes the computer program 22, the steps of each embodiment of the above-mentioned image wrinkle removal method are implemented, such as Figure 1 S10 to S40 as shown, or Figure 5 Alternatively, the processor 20 executes the computer program 22 to implement the above Figure 10 The functions of each unit in the corresponding embodiment are, for example, Figure 10 For details on the functions of units 10 to 12, please refer to Figure 10 The relevant descriptions in the corresponding embodiments are not repeated here.

[0163] Exemplarily, the computer program 22 may be divided into one or more units, which are stored in the memory 21 and executed by the processor 20 to complete the present application. The one or more units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 22 in the terminal device 2. For example, the computer program 22 may be divided into a network training module 10, an image restoration module 11, a texture prediction module 12, and an image overlay module 13, with the specific functions of each unit being as described above.

[0164] The terminal device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that Figure 10 It is only an example of the terminal device 2 and does not constitute a limitation on the terminal device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0165] The processor 20 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0166] The memory 21 may be an internal storage unit of the terminal device 2, such as a hard disk or memory of the terminal device 2. The memory 21 may also be an external storage device of the terminal device 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 2. Furthermore, the memory 21 may include both an internal storage unit of the terminal device 2 and an external storage device. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 may also be used to temporarily store data that has been output or is about to be output.

[0167] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium can be non-volatile or volatile. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in computer-readable storage media can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunications signals.

[0169] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for image wrinkle removal, characterized in that: The method comprises: Training a mapping network based on a defect atlas and a wrinkle atlas, and training a coarse repair network based on the trained mapping network and the wrinkle atlas; Inputting the image to be processed and the mask image into the trained coarse repair network to perform image repair, thereby obtaining a first repaired image; Inputting the first repaired image and the mask image into a refinement network to perform image repair to obtain a second repaired image, and superimposing the second repaired image with the first repaired image to obtain a wrinkle-removed image; The method of training a mapping network based on a defect atlas and a wrinkle atlas includes: Sampling the defect atlas and the wrinkle atlas respectively to obtain a defect sampling image and a wrinkle simulation sampling image; Inputting the defect sampling image and the wrinkle simulation sampling image into the encoder in the mapping network for image encoding respectively to obtain a defect latent vector and a wrinkle simulation latent vector; Inputting the defect latent vector and the wrinkle simulation latent vector into a decoder in the mapping network for image decoding to obtain a defect decoded image and a wrinkle simulation decoded image; Performing loss calculation based on the defect sampling graph, the wrinkle simulation sampling graph, the defect decoded image, and the wrinkle simulation decoded image to obtain an image reconstruction loss value of the mapping network; performing similarity loss calculation based on the defect latent vector and the wrinkle simulation latent vector to obtain a feature loss value of the mapping network, and constructing a model loss value of the mapping network based on the feature loss value and the image reconstruction loss value; Parameters of the mapping network are updated according to the model loss value until the mapping network converges.

2. The image wrinkle removal method according to claim 1, wherein: The formula used for calculating the loss based on the defect sampling image, the wrinkle simulation sampling image, the defect decoded image, and the wrinkle simulation decoded image is: The formula used for calculating the similarity loss based on the defect latent vector and the wrinkle simulation latent vector is: ； Wherein, r is the wrinkle simulation sampling graph, is the defect sampling diagram, is the defective decoded image, is the wrinkle simulation decoded image, is the kth layer of the VGG model, D is the discriminator, and E is the encoder.

3. The image wrinkle removal method according to claim 1, wherein: The step of training the rough repair network based on the trained mapping network and the training atlas includes: Sampling the training atlas to obtain a wrinkle pairing map, wherein the wrinkle pairing map includes a wrinkle map, a wrinkle removal map, and a wrinkle mask map; Inputting the wrinkle pairing graph into the encoder of the rough-repair network for encoding to obtain a latent vector, and inputting the latent vector into the generator of the rough-repair network; For the generator in the rough-repair network, instruct each decoding layer to merge and convolve the feature map output by the corresponding encoding layer with the output feature map of the previous layer decoder to obtain the feature input of the current layer and the residual map of the current layer; The features of the current layer are convolved and sent to the next decoding layer; The current layer residual map is merged with the previous layer residual map to obtain the current layer decoded output image, and the parameters of the generator in the coarse repair network are updated according to the wrinkle removal map and the decoded output image until the coarse repair network converges.

4. The image wrinkle removal method according to claim 1, wherein: Inputting the first restoration image and the mask image into a refinement network to perform image restoration includes: Encoding the first repair image and the mask image to obtain a feature map, and scaling and translating the feature map; The scaled and translated feature map is decoded to obtain the second repaired map.

5. The image wrinkle removal method according to claim 4, wherein: The formula used to regularize the feature map is: in, is the feature map, i, j, k are all array subscripts, C represents the number of channels, H, W are the height and width of the wrinkle mask, then i∈[0, C], j∈[0, H], k∈[0, W].

6. The image wrinkle removal method according to claim 5, wherein: The feature map is scaled and translated: ； in, represents the input image of the nth layer of the refinement network, and Represents a convolution operation.

7. An image wrinkle removal processing system, characterized in that: The system comprises: A network training module, configured to train a mapping network based on a defect atlas and a wrinkle atlas, and to train a rough repair network based on the trained mapping network and wrinkle atlas; An image restoration module is configured to input the image to be processed and the mask image into the trained coarse restoration network to perform image restoration, thereby obtaining a first restored image. an image wrinkle removal module, configured to input the first repaired image and the mask image into a refinement network to perform image repair to obtain a second repaired image, and superimpose the second repaired image with the first repaired image to obtain a wrinkle-removed image; The network training module is further used to: sample the defect atlas and the wrinkle atlas respectively to obtain a defect sampling image and a wrinkle simulation sampling image; Inputting the defect sampling image and the wrinkle simulation sampling image into the encoder in the mapping network for image encoding respectively to obtain a defect latent vector and a wrinkle simulation latent vector; Inputting the defect latent vector and the wrinkle simulation latent vector into a decoder in the mapping network for image decoding to obtain a defect decoded image and a wrinkle simulation decoded image; Performing loss calculation based on the defect sampling graph, the wrinkle simulation sampling graph, the defect decoded image, and the wrinkle simulation decoded image to obtain an image reconstruction loss value of the mapping network; performing similarity loss calculation based on the defect latent vector and the wrinkle simulation latent vector to obtain a feature loss value of the mapping network, and constructing a model loss value of the mapping network based on the feature loss value and the image reconstruction loss value; Parameters of the mapping network are updated according to the model loss value until the mapping network converges.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images