Training methods and apparatus, image restoration methods and apparatus

Abstract

This application discloses a training method and apparatus, and an image restoration method and apparatus. The image restoration model includes multiple feature extraction layers connected in sequence. The training method first obtains an image training set, which includes sample images, original images corresponding to the sample images, and sample masks corresponding to the sample images. The sample images include invalid pixel regions. Then, the image restoration model is trained based on the sample images, original images, and sample masks. In this method, the same feature extraction layer uses a shared convolution kernel to convolve the image and mask input to the feature extraction layer respectively. The pixel value type of the convolution kernel is floating point, which can improve the semantic consistency and fidelity of the restored image, so that the restoration quality meets the quality requirements.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method and apparatus, and an image restoration method and apparatus. Background Technology

[0002] With the rise of deep learning, especially the development of deep convolutional networks, significant breakthroughs have been achieved in various traditional image processing problems, including image inpainting. Image inpainting is a traditional image processing problem where the input image is partially damaged, and the damaged parts need to be repaired based on information from the normal regions in the image and statistical information from a large number of normal images.

[0003] For specific application scenarios of image restoration, deep learning methods are mainly used for image restoration, such as using convolutional neural networks to process and avoid interference from invalid pixels. However, in the process of implementing this application, the inventors found that the semantic consistency and fidelity of the restored images obtained by the current image restoration methods are poor, and the restoration quality is difficult to meet the quality requirements.

[0004] It should be noted that the information disclosed in this background section is only for understanding the background technology of the present application concept, and therefore may include information that does not constitute prior art. Summary of the Invention

[0005] The first objective of this application is to propose an image restoration method that can improve the semantic consistency and fidelity of the restored image, so that the restoration quality meets the quality requirements.

[0006] The second objective of this application is to propose an image restoration method.

[0007] The third objective of this application is to propose a training device for an image restoration model.

[0008] The fourth objective of this application is to provide an image restoration apparatus.

[0009] To achieve the above objectives, a first aspect of this application proposes a training method for an image restoration model. The image restoration model includes multiple sequentially connected feature extraction layers. The training method includes: acquiring an image training set, wherein the image training set includes sample images, an original image corresponding to the sample images, and a sample mask corresponding to the sample images, wherein the sample images include invalid pixel regions; training the image restoration model based on the sample images, the original images, and the sample masks, wherein the same feature extraction layer uses a shared convolution kernel to convolve the image and the mask input to the feature extraction layer respectively, and the pixel value type of the convolution kernel is floating point.

[0010] According to the training method of the image restoration model proposed in the embodiments of this application, a mask-guided convolution operator is proposed to enable the mask branch and the image branch to share the convolution kernel by sharing the convolution kernel. The mask branch and the image branch are combined and optimized, reducing the number of parameters. The image channel and the mask channel can use each other's information, which promotes the information sharing between the two branches. Furthermore, by using a floating-point mask, in addition to indicating whether a pixel is valid, it can also finely describe the degree of validity and invalidity of pixels. Therefore, it can improve the semantic consistency and fidelity of the restored image, so that the restoration quality meets the quality requirements, and realize more refined processing of invalid pixels in the image restoration task.

[0011] According to one embodiment of this application, training the image restoration model based on the sample image, the original image, and the sample mask includes: for the same feature extraction layer, convolving the input mask with the convolution kernel to obtain the mask convolution result; obtaining the updated mask output by the current feature extraction layer based on the mask convolution result; convolving the input image with the convolution kernel to obtain the image convolution result; and obtaining the feature map output by the current feature extraction layer based on the image convolution result.

[0012] According to one embodiment of this application, training the image restoration model based on the sample image, the original image, and the sample mask further includes: obtaining the output of the image restoration model based on the feature map and the updated mask; and adjusting the loss function based on the original image and the output of the image restoration model.

[0013] According to one embodiment of this application, obtaining the updated mask output of the current feature extraction layer based on the mask convolution result includes: inputting the mask convolution result into a nonlinear activation function to obtain the function output; performing a matrix multiplication on the function output and the mask convolution result to obtain the updated mask output of the current feature extraction layer.

[0014] According to one embodiment of this application, obtaining the feature map output by the current feature extraction layer based on the image convolution result includes: performing matrix multiplication on the image convolution result and the function output to obtain the image multiplication result; and obtaining the feature map output by the current feature extraction layer based on the image multiplication result.

[0015] According to one embodiment of this application, the nonlinear activation function f(x) is:

[0016]

[0017] Where x is a scaling factor positively correlated with the number of effective pixels, 0≤x≤1, μ is the scaling factor threshold, γ is the scaling factor, and h(γ)=4πexp(8γ) 3).

[0018] According to one embodiment of this application, before convolving the input mask and the input image using the convolution kernel, the method further includes: dividing the convolution kernel into negative coefficient convolution kernels and non-negative coefficient convolution kernels; when convolving the input mask, convolution is performed using the negative coefficient convolution kernel and the non-negative coefficient convolution kernel respectively to obtain two corresponding mask convolution results, and a corresponding function output is obtained based on the two mask convolution results; after performing matrix multiplication of the function output with the corresponding mask convolution results, the two mask multiplication results are added using a corresponding weighting factor to obtain the updated mask output of the current feature extraction layer; when convolving the input image, convolution is performed using the negative coefficient convolution kernel and the non-negative coefficient convolution kernel respectively to obtain two corresponding image convolution results; after performing matrix multiplication of the function output with the two image convolution results respectively, the two image multiplication results are added to obtain the sum of the image multiplication results, and the feature map output of the current feature extraction layer is obtained based on the sum of the image multiplication results.

[0019] To achieve the above objectives, a second aspect of this application also proposes an image restoration method, the method comprising: acquiring an image to be restored and a mask corresponding to the image to be restored; inputting the image to be restored and the mask into an image restoration model obtained by training the image restoration model according to any of the above embodiments, so as to obtain a restored image.

[0020] To achieve the above objectives, a third aspect of this application also proposes a training apparatus for an image restoration model. The image restoration model includes multiple feature extraction layers connected in sequence. The training apparatus includes: a training set acquisition module for acquiring an image training set, wherein the image training set includes a sample image, an original image corresponding to the sample image, and a sample mask corresponding to the sample image, and the sample image includes invalid pixel regions; and a model training module for training the image restoration model based on the sample image, the original image, and the sample mask, wherein the model training module uses a shared convolution kernel to convolve the sample image and update the sample mask for the same feature extraction layer.

[0021] To achieve the above objectives, a fourth aspect of this application also proposes an image restoration apparatus, the apparatus comprising: an image acquisition module for acquiring an image to be restored and a mask corresponding to the image to be restored; and an image restoration module for inputting the image to be restored and the mask into an image restoration model obtained by training the image restoration model according to any of the above embodiments, so as to obtain a restored image.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a training method for an image restoration model according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of a model framework for training using the U-net network model in one embodiment of this application.

[0025] Figure 3 This is a schematic diagram of the convolution process of a mask-guided convolution operator in one embodiment of this application.

[0026] Figure 4 This is an output curve of the nonlinear activation function f(x) in one embodiment of this application when the scaling factor γ = 0.5.

[0027] Figure 5 This is an output curve of the nonlinear activation function f(x) in one embodiment of this application when the scaling factor threshold μ = 0.5.

[0028] Figure 6 This is a scatter plot of the statistical distribution of parameters μ and γ of the second layer of the encoder in one embodiment of this application.

[0029] Figure 7 This is a scatter plot of the statistical distribution of parameters μ and γ of the fourth layer of the encoder in one embodiment of this application.

[0030] Figure 8 This is a scatter plot of the statistical distribution of parameters μ and γ of the second layer of the decoder in one embodiment of this application.

[0031] Figure 9 This is a scatter plot of the statistical distribution of parameters μ and γ of the fourth layer of the encoder in one embodiment of this application.

[0032] Figure 10 This is a schematic diagram of the convolution process based on another mask-guided convolution operator in one embodiment of this application.

[0033] Figure 11 This is a schematic flowchart of an image restoration method according to an embodiment of this application.

[0034] Figure 12 This is a structural block diagram of a training device for an image restoration model according to an embodiment of this application.

[0035] Figure 13 This is a structural block diagram of an image restoration apparatus according to an embodiment of this application.

[0036] Figure 14 This is a structural block diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of the embodiments described below are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0038] The training method and apparatus, and the image restoration method and apparatus of this application are described below with reference to the accompanying drawings.

[0039] Please see Figure 1 The training method for the image restoration model in this application embodiment includes the following steps:

[0040] 1000, Obtain the image training set, which includes sample images, original images corresponding to the sample images, and sample masks corresponding to the sample images. The sample images include invalid pixel regions.

[0041] In 2000, the image restoration model was trained based on sample images, original images, and sample masks. In this model, the same feature extraction layer used a shared convolution kernel to convolve the input image and mask of the same feature extraction layer respectively. The pixel value type of the convolution kernel k was floating point.

[0042] The sample image is partially incomplete, meaning it contains invalid pixel regions. Invalid pixel regions include holes or locations with abnormal pixel values. Invalid pixels will output abnormal values ​​when participating in convolution operations. The original image is an intact image, while the sample image can be obtained by altering or destroying the original image.

[0043] Please see Figure 2 Image restoration models can use the U-Net network model, which consists of an encoder and a decoder. The feature extraction layer includes convolutional layers and deconvolutional layers. The encoder includes multiple convolutional layers connected in sequence, and the decoder includes multiple deconvolutional layers connected in sequence. The output of each feature extraction layer is the input of the next feature extraction layer.

[0044] The encoder's first convolutional layer takes sample image 100 as input. Through convolutions across several layers, it extracts a compressed feature representation of the image. This compressed feature representation is then input into the decoder. The decoder decodes the compressed feature representation through deconvolution operations across several deconvolution layers, obtaining the restored image 300. Here, 'c' within the circle represents channel concatenation. This represents the addition of elements at the corresponding positions.

[0045] Understandably, to preserve good texture information in the restored image, the U-Net network model can employ skip connections. This allows shallow texture details to be retained in deeper layers of the network, and this direct connection method avoids the gradient vanishing problem. Skip connections involve cropping the downsampled feature map and concatenating it with the upsampled feature map, essentially concatenating them channel by channel.

[0046] During the training of the image restoration model, each feature extraction layer involves convolution operations on the feature image and mask output by the previous feature extraction layer. Convolution operations on the sample image and feature image are called image branches, and convolution operations on the mask are called mask branches. The mask represents the mask, which is usually a matrix used to identify whether a pixel is valid.

[0047] Please see Figure 3 For each feature extraction layer, when performing convolution / deconvolution operations, the convolution kernel used for convolution / deconvolution of the input image p1 and the convolution kernel used for convolution / deconvolution of the input mask m1 are shared convolution kernel k. Therefore, in this embodiment, the operation of using the same convolution kernel k to convolve the input image p1 and the input mask m1 in the same feature extraction layer is called mask-guided convolution (hereinafter referred to as MagConv). Thus, the image branch and the mask branch share the same network, and therefore can refer to each other to obtain a better compressed feature representation.

[0048] It's understandable that deconvolution operations use deconvolution kernels, and different feature extraction layers use different kernels. These kernels change through network learning; therefore, different kernels are used for convolution operations between two different feature extraction layers. Similarly, for deconvolution layers, the same deconvolution layer can use a shared kernel to deconvolve the input image and mask separately. The pixel values ​​of the deconvolution kernel are also floating-point type.

[0049] Because floating-point kernels are used for convolution, the mask convolution also uses floating-point kernels, resulting in an updated mask m3 that is a floating-point mask (soft mask). The numerical values ​​in the mask can be represented by color bars, which indicate the degree of validity. Therefore, they can express more information about validity. The values ​​range from 0.0 to 1.0, and any value greater than zero is considered a valid pixel. Since the validity of the original valid pixels is definitely higher than that of the valid pixels generated by the previous layer of the multi-layer neural network, using floating-point values ​​as an indicator of validity provides more valuable information than using integer values ​​to identify pixel validity (0 for invalid pixels, 1 for valid pixels). This information can more accurately guide the processing of image branches and directly connect to subsequent operations. The result of mask convolution serves as the mask input for the next feature extraction layer without requiring changes to subsequent operations or specific mask update rules.

[0050] Please see Figure 2 The U-Net network model may include: a first convolutional layer 211, second convolutional layers 221-226, a first deconvolutional layer 231-235, a second deconvolutional layer 241, and a third deconvolutional layer 251. The connection relationship between each layer is as follows: first convolutional layer 211 - second convolutional layer 221 - second convolutional layer 222 - second convolutional layer 223 - second convolutional layer 224 - second convolutional layer 225 - second convolutional layer 226 - first deconvolutional layer 231 - first deconvolutional layer 232 - first deconvolutional layer 233 - first deconvolutional layer 234 - second deconvolutional layer 241 - first deconvolutional layer 235 - third deconvolutional layer 251.

[0051] The convolution operations in each layer are not entirely the same. Specifically, the convolution operations in the first and second convolutional layers include MagConv and ReLU; the deconvolution operations in the first and second deconvolutional layers include MagConv, LeakyReLU, and ReLU; and the deconvolution operations in the third deconvolutional layer include MagConv and Tanh. MagConv is... Figure 3The mask-guided convolution or deconvolution process shown in the diagram uses MagConv in each convolutional layer for convolution and MagConv in each deconvolutional layer for deconvolution. Whether convolution or deconvolution, the same floating-point convolutional or deconvolutional kernel is used within the same MagConv convolution process. ReLU (Rectified Linear Unit) is used for computation via an activation function. The mathematical expression for ReLU can be: a = max(0, z), where a is the output and z is the input. LeakyReLU (Leaky Rectified Linear Unit) is also used for computation via an activation function. The mathematical expression for LeakyReLU can be: y = max(0, x) + leak * min(0, x), where y is the output, x is the input, and leak is a small constant to retain some negative axis values ​​so that the negative axis information is not completely lost. Tanh is the hyperbolic tangent function, also used for computation via an activation function.

[0052] For each feature extraction layer, the mask output from the mask branch in the MagConv process is directly used as the input mask for the next layer, while the feature image output from the image branch in the MagConv process needs to be input into the subsequent function of this layer for computation. The result of the computation is the output image of this layer and is used as the input image for the next layer. For example, in the second convolutional layer 221, the MagConv process outputs a mask and a feature image. The mask is directly used as the input mask for the second convolutional layer 222, while the feature image needs to be input into the ReLU activation function for computation. The result of the computation is used as the input image for the second convolutional layer 222. As another example, for the first deconvolutional layer 232, the MagConv process outputs a mask and a feature image. The mask is directly used as the input mask for the first deconvolutional layer 233, while the feature image needs to be input into the LeakyReLU activation function to obtain the first computation result. The first computation result is then input into the ReLU activation function to obtain the second computation result, which is used as the input image for the first deconvolutional layer 233.

[0053] At the start of training, the sample image 100 and the corresponding initial mask are input to the first convolutional layer 211. For the first convolutional layer 211, the sample image 100 is the input image p1, and the corresponding initial mask is the input mask m1. A MagConv convolution operation is performed on the input image p1 and the input mask m1, that is, the input image p1 and the input mask m1 are convolved by the convolution kernel k respectively, resulting in the corresponding feature image p3 and the updated mask m3. The image output after the feature image p3 is input to the ReLU activation function is used as the input image p1 of the second convolutional layer 221, and the updated mask m3 is used as the input mask m1 of the second convolutional layer 221. The MagConv convolution operation of the second convolutional layer 221 then begins. Subsequent convolutions and deconvolutions follow the same pattern. Here, f is the activation function. ⊙ represents the convolutional operation, and ⊙ represents element-wise multiplication of matrices. It's understandable that the operation of the activation function f occurs within the MagConv process, while the operations of the ReLU and LeakyReLU activation functions occur outside the MagConv process, and they operate on different objects.

[0054] The images repaired using the image inpainting model trained in this embodiment achieve state-of-the-art performance on several classic image quality evaluation metrics, including Frechet Inception Score (FID), LPIPS scores, structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR), compared to current classic algorithms such as EC, LBAM, MEDFE, and CTSDG, on the ParisStreetView, CelebA, and Places2 datasets. From a visual comparison perspective, the repair results obtained by this invention are more realistic, with a more natural transition between the repaired area and other areas, and better semantic consistency.

[0055] According to the training method of the image restoration model proposed in the embodiments of this application, a mask-guided convolution operator is proposed to enable the mask branch and the image branch to share the convolution kernel by sharing the convolution kernel. The mask branch and the image branch are combined and optimized, reducing the number of parameters. The image channel and the mask channel can use each other's information, which promotes the information sharing between the two branches. Furthermore, by using a floating-point mask, in addition to indicating whether a pixel is valid, it can also finely describe the degree of validity and invalidity of pixels. Therefore, it can improve the semantic consistency and fidelity of the restored image, so that the restoration quality meets the quality requirements, and realize more refined processing of invalid pixels in the image restoration task.

[0056] In some embodiments, step 2000 may include the following steps:

[0057] 2100, For the same feature extraction layer, the input mask is convolved by a convolution kernel to obtain the mask convolution result;

[0058] 2200, based on the mask convolution result, the updated mask output by this feature extraction layer is obtained;

[0059] 2300, the input image is convolved using a convolution kernel to obtain the image convolution result;

[0060] 2400, based on the image convolution result, the feature map output by this feature extraction layer is obtained.

[0061] Please see Figure 3 If the current feature extraction layer is the second convolutional layer 221, then the input mask m1 and the input image p1 are convolved by the convolution kernel k to obtain the mask convolution result m2 and the image convolution result p2. After further operation on the mask convolution result m2, the mask m3 is obtained, which is the input mask of the second convolutional layer 222. After further operation on the image convolution result p2, the feature image p3 is obtained. Then, after the operation of the ReLU activation function, the feature map output by the MagConv operation is obtained. After further operation on the feature map output by the MagConv operation, the compressed feature map output by this feature extraction layer is obtained, which is the input image of the second convolutional layer 222.

[0062] Understandably, the same principle applies to deconvolution layers. For the same deconvolution layer, the input mask is deconvolved using a deconvolution kernel to obtain the mask convolution result. Based on the mask convolution result, the updated mask output by this deconvolution layer is obtained. The input image is deconvolved using a deconvolution kernel to obtain the image deconvolution result. Based on the image deconvolution result, the decoded feature map output by this deconvolution layer is obtained.

[0063] In some embodiments, step 2000 may further include the following steps:

[0064] 2500, based on the feature map and the updated mask, the output of the image inpainting model is obtained;

[0065] 2600, adjust the loss function based on the original image and the output of the image inpainting model.

[0066] The feature map and updated mask output by each feature extraction layer serve as inputs to the next layer. Specifically, the compressed feature map and updated mask output by each convolutional layer serve as inputs to the next layer, until the next layer is a deconvolutional layer. The decoded feature map and updated mask output by the deconvolutional layer serve as inputs to the next layer, until the third deconvolutional layer 251 is reached. The third deconvolutional layer 251 outputs the restored image 300, which is the output of the image restoration model. Then, image 300 is compared with the original images in the image training set, and the error / loss between the two is calculated. The steepest descent method is then used to continuously modify the network parameters, making the distance between the neural network output and the ground truth (GT) get closer and closer, until the loss is less than a certain threshold or the set number of iterations is reached.

[0067] loss function It can be: in, This is a pixel-level reconstruction loss function. The pixel-level loss measures the fidelity between the reconstructed image and the original image. Specifically, either L1 or L2 loss can be used; this embodiment uses L1 loss. Let λ be the perceptual loss function. p The weighting factors for the perceptual loss function, Similarly, L1 or L2 loss can be used; Let λ be the style loss function. s The weighting factor of the style loss function describes the style features of a class of images, such as people, landscapes, oil paintings, and mountains and rivers. Style features are usually described by Gram matrix transformation (input image, output matrix of the same size). Let λ be the smoothness loss function. t The weighting factors of the smoothness loss function, By constraining the differences between adjacent pixels to be not too large, the final generated image has better visual continuity.

[0068] Based on the U-net network model, an optimization function was constructed using both pixel-level and feature-level loss functions, combining image losses from two image domains to train the network model and achieve pixel-level and feature-level image restoration effects. First, the L1 loss function of the image domain was adopted, which characterizes pixel-level image distortion. Second, a pre-trained model on ImageNet was used to extract image features, constructing an L1 loss function in the feature domain. Third, the style loss function from image style transfer or image translation tasks was adopted, which is the difference in feature map output on feature distribution (L1 of Gram matrix). In addition, TV loss for image smoothing was adopted to eliminate checkerboard noise in the restored area.

[0069] ImageNet is a classic database in computer vision, used for evaluating image classification, localization, and detection tasks. It contains over one million labeled images across 1000 image categories. Therefore, the ImageNet database is commonly used to pre-train neural network models, such as VGG, ResNet, U-Net, and DenseNet. These networks can serve as a backbone for user-designed networks, reducing the burden of massive training data and computational demands required for user network training. With pre-trained models, users only need a small number of samples and computations to obtain a good model.

[0070] In some embodiments, step 2200 may include the following steps:

[0071] 2210. Input the mask convolution result into a nonlinear activation function to obtain the function output;

[0072] 2220. Perform matrix dot product on the function output and the mask convolution result to obtain the updated mask output of this feature extraction layer.

[0073] Step 2400 may include the following steps:

[0074] 2410. Perform matrix dot product on the image convolution result and the function output to obtain the image dot product result;

[0075] 2420. Based on the image dot product result, the feature map output by this feature extraction layer is obtained.

[0076] Please see Figure 3After obtaining the mask convolution result m2 through the convolution kernel k, the mask convolution result m2 is input into the activation function f. The activation function f is a non-linear activation function. The function output is obtained through the activation function f. Then, the function output and the mask convolution result are multiplied by matrix dot product, that is, the corresponding elements of the matrix are multiplied (⊙ in the figure) to obtain the updated mask m3.

[0077] After obtaining the image convolution result p2 through convolution kernel k, and after obtaining the function output, the function output is multiplied by the image convolution result p2 to obtain the image dot product result of the MagConv operation. The image dot product result is the feature image p3. The image dot product result is then passed through activation functions such as the ReLU function to obtain the compressed feature map output by this convolutional layer.

[0078] Therefore, the mask branch provides a non-linearly activated mask to the image branch, which is used to modulate the convolution output of the image branch to correct the convolution output of the image branch.

[0079] Understandably, the same principle applies to the deconvolution process. The deconvolution result of the mask is input into a nonlinear activation function to obtain the function output. Then, a matrix multiplication is performed on the function output and the mask deconvolution result to obtain the updated mask of the current deconvolution layer output. In addition, a matrix multiplication is performed on the image deconvolution result and the function output to obtain the decoded feature map of the current deconvolution layer output.

[0080] In some embodiments, the nonlinear activation function f(x) is:

[0081]

[0082] Where x is a scaling factor positively correlated with the number of effective pixels, 0≤x≤1, μ is the scaling factor threshold, γ is the scaling factor, and h(γ)=4πexp(8γ) 3 The nonlinear activation function is controlled by parameters μ and γ, which are learnable and obtained through end-to-end optimization of the deep network, achieving adaptive compensation capability and more accurate compensation effect.

[0083] Specifically, if the reciprocal function is used as f(x), the result of partial convolution is scaled using the reciprocal of the number of effective pixels. However, when the number of invalid pixels is too large, the reliability and credibility of the convolution output of this block will be low. Therefore, this application uses a piecewise nonlinear activation function. As can be seen from the formula of f(x), when the number of invalid pixels reaches the scaling factor threshold μ, the scaling factor used for convolution output is suppressed. When the number of invalid pixels exceeds the scaling factor threshold μ, the scaling factor used for convolution output will change from activation to suppression.

[0084] Therefore, this function takes into account the number of valid / invalid pixels and the reliability of the entire convolution block. That is, when there are too many invalid pixels, the reliability of the convolution result will be reduced. In this case, the convolution result should be suppressed rather than activated.

[0085] Please see Figure 4 It can be seen that the parameters μ and γ control the inflection points and contours of the curve f(x). Figure 4 Curves corresponding to different μ values ​​were plotted while keeping γ = 0.5. Specifically, b1 is the curve output by the activation function f(x) when μ = 0.2 and γ = 0.5, and g1 is the inflection point of b1; b2 is the curve output by the activation function f(x) when μ = 0.3 and γ = 0.5, and g2 is the inflection point of b2; b3 is the curve output by the activation function f(x) when μ = 0.5 and γ = 0.5, and g3 is the inflection point of b3; b4 is the curve output by the activation function f(x) when μ = 0.7 and γ = 0.5, and g4 is the inflection point of b4; b5 is the curve output by the activation function f(x) when μ = 1.0 and γ ∈ [0,1].

[0086] When the independent variable x is less than or equal to the threshold μ, the activation function curve is a complex exponential function and a monotonically increasing function, corresponding to the part to the left of the inflection points of curves b1 to b4. From the inflection point of the piecewise function to the left, the number of invalid pixels increases while the scaling factor decreases, which is the opposite trend to the right half of the piecewise function. When the independent variable x is greater than the threshold μ, the activation function is a reciprocal function, i.e., 1 / x (a monotonically decreasing function), corresponding to the part to the right of the inflection points of curves b1 to b5. When μ = 1, the value of f(x) of the corresponding curve b5 is always 1, i.e., no scaling, which corresponds to normal convolution. When μ = 0, the corresponding curve f(x) becomes 1 / x, which corresponds to the partial convolution algorithm.

[0087] Please see Figure 5 It can be seen that the parameters μ and γ control the inflection points and contours of the curve f(x). Figure 5 Curves corresponding to different values ​​of γ while keeping μ = 0.5 are plotted. Clearly, μ determines the boundary point of the piecewise function, while γ determines the contour of the left side of the curve, representing different scaling factors. γ = 0 and γ = 1 correspond to two constant factors, 2.0 and 1.0, respectively. Specifically, b6 is the curve output by the activation function f(x) when μ = 0.5 and γ = 0.0; b7 is the curve output by the activation function f(x) when μ = 0.5 and γ = 0.3; b8 is the curve output by the activation function f(x) when μ = 0.5 and γ = 0.5; b9 is the curve output by the activation function f(x) when μ = 0.5 and γ = 0.7; b10 is the curve output by the activation function f(x) when μ = 0.5 and γ = 1.0; and g0 is the common inflection point of b6 to b10.

[0088] pass Figure 4and Figure 5 It can be seen that the activation function f(x) has learnable parameters, which makes it more adaptive and can degenerate to two trivial values ​​(μ=1 and μ=0), corresponding to normal convolution and partial convolution algorithms, respectively.

[0089] Therefore, this embodiment comprehensively considers the number of effective pixels within the convolution block and the overall reliability of the convolution output, and designs a more refined, learnable nonlinear function to modulate the output of partial convolution. Based on reliability and energy consistency, the convolution output is corrected, which can achieve more refined compensation for partial convolution results and activate or suppress the convolution output. Moreover, it is obtained through end-to-end learning and has adaptive capabilities. Its degradation form includes normal convolution and partial convolution.

[0090] In some embodiments, before convolving the input mask and the input image with the convolution kernel, the convolution kernel can be divided into negative coefficient convolution kernels and non-negative coefficient convolution kernels.

[0091] When convolving the input mask, convolution is performed using negative coefficient convolution kernels and non-negative coefficient convolution kernels respectively to obtain two corresponding mask convolution results. Based on the two mask convolution results, the corresponding function output is obtained. After performing matrix multiplication between the function output and the corresponding mask convolution results, the two mask multiplication results are added together using the corresponding weighting factors to obtain the updated mask output of this feature extraction layer.

[0092] When convolving the input image, convolution is performed using negative coefficient convolution kernels and non-negative coefficient convolution kernels respectively to obtain two corresponding image convolution results. After performing matrix dot product between the function output and the two image convolution results respectively, the sum of the two image dot product results is obtained by adding the two image dot product results. The feature map output by this feature extraction layer is obtained based on the sum of the image dot product results.

[0093] Specifically, for each feature extraction layer, before performing convolution / deconvolution operations, the convolution kernel k is first separated into a negative coefficient convolution kernel k2 and a non-negative coefficient convolution kernel k1. The negative coefficient convolution kernel k2 is a negative coefficient matrix, where the pixel values ​​are floating-point negative numbers with a range greater than or equal to -1.0 and less than 0.0. The mask branch operated on by the negative coefficient convolution kernel k2 is the negative mask branch. The non-negative coefficient convolution kernel k1 is a non-negative coefficient matrix, where the pixel values ​​are floating-point non-negative numbers with a range greater than or equal to 0.0 and less than or equal to 1.0. The mask branch operated on by the non-negative coefficient convolution kernel k1 is the non-negative mask branch.

[0094] Therefore, due to the separation of the convolution kernel k, both the image branch and the mask branch are divided into two parts. Each part is executed independently, and the outputs are merged to obtain the overall output of each branch. Each of the two branches corresponds to a nonlinear activation function f1 and f2, and the parameters μ and γ of functions f1 and f2 are different.

[0095] The convolution / deconvolution operation through the feature extraction layer specifically includes the following six processes.

[0096] The first process is to normalize the non-negative coefficient convolution kernel k1 to obtain the normalized convolution kernel k1'. Then, the input mask m1 of the current feature extraction layer is convolved / deconvolved through the convolution kernel k1' to obtain the first mask convolution result m21. The first mask convolution result m21 is then input into the corresponding first activation function f1 to obtain the output of the first activation function f1. Finally, the output of the first activation function f1 is multiplied by the first mask convolution result m21 to obtain the first mask dot product result m31.

[0097] The second process is to normalize the negative coefficient convolution kernel k2 to obtain the normalized convolution kernel k2'. Then, the input mask m1 of the current feature extraction layer is convolved / deconvolved through the convolution kernel k2' to obtain the second mask convolution result m22. The second mask convolution result m22 is then input into the corresponding second activation function f2 to obtain the output of the second activation function f2. Finally, the output of the second activation function f2 is multiplied by the second mask convolution result m22 to obtain the second mask multiplication result m32.

[0098] The third process is, after completing the first and second processes mentioned above, to multiply the first mask dot product result m31 by the first weighting factor λ. p Perform matrix dot product to obtain the first weighted result, and then combine the second mask dot product result m32 with the second weighting factor λ. n Perform matrix dot product to obtain the second weighted result, then add the first weighted result and the second weighted result together element by element to obtain the updated mask m3 output by the feature extraction layer of this layer.

[0099] The fourth process is to use a non-negative coefficient convolution kernel k1 to convolve / deconvolve the input image p1 of the current feature extraction layer to obtain the first image convolution result p21. Then, after obtaining the output of the first activation function f1 in the first process, the output of the first activation function f1 is used to perform matrix multiplication with the first image convolution result p21 to obtain the first image multiplication result p31.

[0100] The fifth process involves convolving / deconvolving the input image p1 of the current feature extraction layer using a negative coefficient convolution kernel k2 to obtain the second image convolution result p22. Then, after obtaining the output of the second activation function f2 in the second process, the output of the second activation function f2 is multiplied by the second image convolution result p22 to obtain the second image dot product result p32.

[0101] The sixth process is to add the first image dot product result p31 and the second image dot product result p32 element by element to obtain the sum of the image dot product results p3, which is the feature image p3 output by the MagConv operation. By inputting the sum of the image dot product results p3 into the activation function ReLU or LeakyReLU, the compressed feature map output by this feature extraction layer is obtained.

[0102] It is understandable that, apart from the limitation on the order of the processes caused by the conditions for their occurrence, there are no other restrictions on the order of the above six processes. Furthermore, among the above six processes, the first to fifth processes, as well as the process in the sixth process until the feature image p3 is obtained, all belong to the MagConv process.

[0103] Please see Figures 6 to 10 In this diagram, equilateral triangles represent positions under non-negative mask branches, and inverted triangles represent positions under negative mask branches. Different positions represent different values ​​of μ and γ. Statistical analysis of the experimental results shows that in image inpainting tasks, the proportions of μ=1 and μ=0 (normal convolution and partial convolution) are very small. This is reflected in the scatter plot, where most points are concentrated in the center, while points in the surrounding areas (μ / γ=0 or 1) are very sparse. This indicates that the optimal convolution (the learned optimal μ and γ) in image inpainting tasks is neither partial convolution nor normal convolution. Therefore, the adaptive mask-guided convolution disclosed in this application is more suitable for image inpainting tasks.

[0104] The output energy of convolutional blocks containing invalid pixels is measured by summing the kernel coefficients. This facilitates scaling to maintain energy consistency with normal convolutional blocks. The convolutional kernel is separated into negative and non-negative coefficient matrices. Correspondingly, both the image and mask blocks are divided into two sub-blocks according to the kernel splitting method. Positive and negative convolutional kernels are convolved with the corresponding mask branches and, after non-linear activation, used as scaling factors to correct the image branch convolution results. This allows for accurate energy measurement and proper convolution compensation, more precisely characterizing convolutional energy changes. Simultaneously, this scaling factor also applies to the mask branches, resulting in two modulated masks. These masks are then weighted by two factors λ. p and λ n Merge them into a single mask, which will serve as the updated mask.

[0105] This embodiment expresses the image inpainting problem as a joint optimization process of encoding and decoding, and proposes a mask-guided convolutional neural network for image inpainting tasks. It excludes invalid pixels in damaged images from convolution, avoids the participation of invalid pixels, and compensates for the convolution results. At the same time, it scales the convolution blocks with partial pixel participation to keep their energy at the same level as normal convolution blocks, so that their output on valid pixels is consistent with standard convolution. This achieves the purpose of eliminating abnormal (invalid) pixel interference and adjusting the energy balance of convolution output, which greatly improves the performance of image inpainting and obtains image reconstruction results with high fidelity, realistic image feel, and semantic consistency.

[0106] Additionally, please see Figure 11 This application also provides an image restoration method, which includes the following steps:

[0107] 3000, obtain the image to be repaired and the mask corresponding to the image to be repaired;

[0108] 4000, input the image to be repaired and the mask into the image repair model obtained by the training method of the image repair model in any of the above embodiments, so as to obtain the repaired image.

[0109] It should be noted that the image restoration model in this embodiment is described in detail in the embodiment of the training method of the image restoration model in this application, and will not be repeated here.

[0110] Additionally, please see Figure 12 This application embodiment also provides a training device 400 for an image restoration model, which includes a training set acquisition module 410 and a model training module 420.

[0111] The training set acquisition module 410 is used to acquire an image training set, wherein the image training set includes a sample image, an original image corresponding to the sample image, and a sample mask corresponding to the sample image, and the sample image includes invalid pixel regions.

[0112] The model training module 420 is used to train the image restoration model based on the sample image, the original image, and the sample mask. The image restoration model includes multiple feature extraction layers connected in sequence. The model training module uses a shared convolution kernel to convolve the sample image and update the sample mask for the same feature extraction layer.

[0113] It should be noted that for details not disclosed in the training device of the image restoration model in this embodiment, please refer to the details disclosed in the embodiment of the training method of the image restoration model in this application, which will not be repeated here.

[0114] Additionally, please see Figure 13 This application also provides an image restoration apparatus 500, which includes an image acquisition module 510 and an image restoration module 520. The image acquisition module 510 acquires the image to be restored and a mask corresponding to the image to be restored. The image restoration module 520 inputs the image to be restored and the mask into an image restoration model obtained using the training method of the image restoration model in any embodiment of this application, to obtain a restored image.

[0115] It should be noted that for details not disclosed in the image restoration device in this embodiment, please refer to the details disclosed in the embodiment of the image restoration model training method in this application, which will not be repeated here.

[0116] Additionally, please see Figure 14 This application also provides a computer device 600, which includes a processor 610 and a memory 620. The memory 620 is used to store at least one computer program. The at least one computer program is loaded by the processor 610 and executed as a training method for an image restoration model in any of the above embodiments, or the at least one computer program is loaded by the processor 610 and executed as an image restoration method in the above embodiments.

[0117] It should be noted that for details not disclosed in the computer device in this embodiment, please refer to the details disclosed in the embodiment of the training method for the image restoration model in this application, and the details disclosed in the embodiment of the image restoration method in this application, which will not be repeated here.

[0118] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method of the image restoration model in any of the above embodiments, or the image restoration method in the above embodiments.

[0119] It should be noted that for details not disclosed in the computer-readable storage medium in this embodiment, please refer to the details disclosed in the embodiment of the training method for the image restoration model in this application, and the details disclosed in the embodiment of the image restoration method in this application, which will not be repeated here.

[0120] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0121] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise expressly specified. Relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, and computer-readable storage media are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0125] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for training an image inpainting model, characterized in that, The image inpainting model comprises a plurality of feature extraction layers connected in sequence, and the training method comprises: An image training set is obtained, wherein the image training set comprises a sample image, an original image corresponding to the sample image, and a sample mask corresponding to the sample image, and the sample image comprises an invalid pixel region; The image inpainting model is trained according to the sample image, the original image, and the sample mask, wherein the same feature extraction layer adopts a shared convolution kernel to respectively convolve an image and a mask input into the feature extraction layer, and the pixel value type of the convolution kernel is floating point type; The nonlinear activation function f(x) is: wherein x is a scaling factor positively correlated with the number of effective pixels, 0≤x≤1, μ is a scaling factor threshold, γ is a scaling factor, ; Before the input mask and the input image are convolved by the convolution kernel, the method further comprises: dividing the convolution kernel into a negative coefficient convolution kernel and a non-negative coefficient convolution kernel; When the input mask is convolved, the negative coefficient convolution kernel and the non-negative coefficient convolution kernel are respectively used for convolution to obtain two corresponding mask convolution results, the two mask convolution results are respectively input into the segmented nonlinear activation function to obtain two corresponding function outputs, the two corresponding function outputs are respectively multiplied by the corresponding mask convolution results, and then the two mask multiplication results are added by using the corresponding weighting factor to obtain an updated mask output by the feature extraction layer; When the input image is convolved, the negative coefficient convolution kernel and the non-negative coefficient convolution kernel are respectively used for convolution to obtain two corresponding image convolution results, the two image convolution results are respectively input into the segmented nonlinear activation function to obtain two corresponding function outputs, the two corresponding function outputs are respectively multiplied by the corresponding image convolution results, and then the sum of the two image multiplication results is obtained by adding the two image multiplication results, and the feature map output by the feature extraction layer is obtained according to the sum of the image multiplication results.

2. The training method of claim 1, wherein, The training of the image inpainting model according to the sample image, the original image, and the sample mask comprises: For the same feature extraction layer, the input mask is convolved by the convolution kernel to obtain a mask convolution result; The updated mask output by the feature extraction layer is obtained according to the mask convolution result; The input image is convolved by the convolution kernel to obtain an image convolution result; The feature map output by the feature extraction layer is obtained according to the image convolution result.

3. The training method of claim 2, wherein, The training of the image inpainting model according to the sample image, the original image, and the sample mask further comprises: The output of the image inpainting model is obtained according to the feature map and the updated mask; The loss function is adjusted according to the original image and the output of the image inpainting model.

4. An image inpainting method characterized by, The method comprises: An image to be inpainted and a mask corresponding to the image to be inpainted are obtained; The image to be inpainted and the mask are input into the image inpainting model obtained by using the training method of any one of claims 1-3 to obtain an inpainted image.

5. An apparatus for training an image inpainting model, comprising: The image inpainting model comprises a plurality of feature extraction layers connected in sequence, and the training device comprises: The training set acquisition module is configured to acquire an image training set, wherein the image training set comprises a sample image, an original image corresponding to the sample image, and a sample mask corresponding to the sample image, and the sample image comprises an invalid pixel region. The model training module is configured to train the image inpainting model according to the sample image, the original image, and the sample mask, wherein the model training module uses a shared convolution kernel to convolve the sample image and update the sample mask for the same feature extraction layer. The nonlinear activation function f(x) is: wherein x is a scaling factor positively correlated with the number of effective pixels, 0≤x≤1, μ is a scaling factor threshold, γ is a scaling factor, ; Before the convolution kernel is used to convolve the input mask and the input image, the convolution kernel is divided into a negative coefficient convolution kernel and a non-negative coefficient convolution kernel. When the input mask is convolved, the negative coefficient convolution kernel and the non-negative coefficient convolution kernel are used to convolve the input mask to obtain two mask convolution results, the two mask convolution results are input into the segmented nonlinear activation function to obtain two corresponding function outputs, the two corresponding function outputs are respectively multiplied by the corresponding mask convolution results, and then the two mask multiplication results are added by using the corresponding weighting factor to obtain the updated mask output by the feature extraction layer. When the input image is convolved, the negative coefficient convolution kernel and the non-negative coefficient convolution kernel are used to convolve the input image to obtain two image convolution results, the two image convolution results are input into the segmented nonlinear activation function to obtain two corresponding function outputs, the two corresponding function outputs are respectively multiplied by the corresponding image convolution results, and then the two image multiplication results are added to obtain the sum of the image multiplication results, and the feature map output by the feature extraction layer is obtained according to the sum of the image multiplication results.

6. An image inpainting apparatus characterized by comprising: The device comprises: The image acquisition module is configured to acquire an image to be repaired and a mask corresponding to the image to be repaired. The image inpainting module is configured to input the image to be repaired and the mask into the image inpainting model obtained by using the training method of any one of claims 1-3 to obtain a repaired image.

Images