Model Training Method, Image Generation Method and Related Devices

Abstract

The present application provides a model training method, an image generation method and related devices, which are applied to the field of artificial intelligence. The method includes: obtaining a training sample including multiple groups of images, each group of images including a lossless image and its corresponding damaged image; inputting the multiple groups of images into a generation model to train the generation model; wherein, the generation model based on DCGAN is used to generate the corresponding lossless image based on the damaged image, and the model includes an attention module for performing a convolution operation on the received first feature map to obtain a second feature map, and the second feature map after being processed by a mask image is used to perform a weighting operation on the first feature map to obtain a third feature map, and a convolution operation is performed on the third feature map to obtain a fourth feature map. The method of the present application solves the problem that the pixels of the repaired image are discontinuous, alleviates the poor visual experience caused by the discontinuity between pixels, and is beneficial to obtaining good visual effects.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and particularly to a model training method, an image generation method, and related devices. Background Art

[0002] Image restoration is a processing technology for improving image quality. It speculates and repairs the content of damaged or missing areas based on known image content, making the content information of the restored image reasonable and subjectively visually realistic. The core challenge of this technology is to synthesize visually realistic and semantically reasonable pixels for the missing areas to be consistent with the existing pixels. Image restoration has important practical significance, especially in the protection of artworks, the restoration of old photos, and many applications in image-based rendering and computer photography.

[0003] Currently, the principles of a large number of image restoration algorithms are implemented based on deep convolutional generative adversarial networks (DCGAN).

[0004] However, when using DCGAN to restore images, there will be a problem that the pixels in the restored image are discontinuous. Summary of the Invention

[0005] This application provides a model training method, an image generation method, and related devices to solve the problem of discontinuous pixels in the restored image.

[0006] In a first aspect, this application provides a model training method, which includes: obtaining training samples, where the training samples include multiple groups of images, each group of images includes a lossless image and its corresponding damaged image, and the damaged image is obtained by processing the corresponding lossless image with a preset mask image; inputting the multiple groups of images into a generation model to train the generation model; where the generation model is used to generate the corresponding lossless image based on the damaged image, the generation model is trained based on DCGAN, the generation model includes an attention module, the attention module is used to perform a convolution operation on the received first feature map to obtain a second feature map, after the second feature map is processed by the mask image, it is used to perform a weighting operation on the first feature map to obtain a third feature map, and the attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map.

[0007] In this application, multiple groups of images in the obtained training samples are sequentially input into a generation model including an attention module. Convolution operations (to obtain a second feature map) and processing of a mask image are respectively performed on the received first feature map. The feature map after mask processing is used to perform a weighting operation on the first feature map, and then a convolution operation is performed on the feature map after the weighting operation. This can solve the problem of discontinuous pixels at the damaged boundary when repairing a damaged image, thereby alleviating the adverse visual experience caused by the discontinuity between pixels and facilitating obtaining a good visual effect.

[0008] In combination with the first aspect, in some implementation manners of the first aspect, the first feature map includes pixel values of multiple pixel points included in the damaged image.

[0009] In combination with the first aspect, in some implementation manners of the first aspect, the generation model further includes a convolution layer, and the convolution layer is used to perform a convolution operation on the input image to obtain the first feature map.

[0010] In combination with the first aspect, in some implementation manners of the first aspect, the generation model further includes a normalization module, and the normalization module is used to: perform normalization operations on the fourth feature map and the mask image respectively and sum them to obtain the summed data; perform a convolution operation on the fourth feature map to obtain a fifth feature map; and process the product of the fifth feature map and the summed data using a predefined activation function.

[0011] In combination with the first aspect, in some implementation manners of the first aspect, the normalization operation satisfies the following formula:

[0012]

[0013] where y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

[0014] Exemplarily, when performing a normalization operation on the fourth feature map, the above μ is the mean of the pixel values of all pixel points in the fourth feature map, and S is its corresponding standard deviation; when performing a normalization operation on the mask image, the above μ is the mean of the pixel values of all pixel points in the mask image, and S is its corresponding standard deviation.

[0015] In combination with the first aspect, in some implementation manners of the first aspect, the generation model includes multiple attention modules and multiple normalization modules. The multiple attention modules and the multiple normalization modules correspond one by one, and each normalization module is used to receive the fourth feature map from the corresponding attention module.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the generation model further includes a deconvolution layer for performing deconvolution operations on the received feature map.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the masked image includes a first value and a second value. The positions of the first value correspond to the pixel points of the undamaged area in the damaged image, and the positions of the second value correspond to the pixel points of the damaged area in the damaged image. The first value is 1, and the second value is a value close to 0.

[0018] In a second aspect, the present application provides an image generation method, which includes: obtaining a damaged image; inputting the damaged image into a pre-trained generation model to generate a lossless image of the damaged image through the generation model. The generation model is trained based on the deep convolutional generative adversarial network (DCGAN). The generation model includes an attention module, which is used to perform a convolution operation on the received first feature map to obtain a second feature map. After being processed by the masked image, the second feature map is used to perform a weighting operation on the first feature map to obtain a third feature map. The attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map; wherein, the first feature map includes the pixel values of multiple pixel points in the damaged image, or the first feature map is a feature map obtained by performing a convolution operation on the damaged image.

[0019] The present application can repair the damaged image to be repaired based on a pre-trained prediction model. The image generation method provided by the present application can alleviate the adverse visual experience caused by discontinuous pixels and is beneficial to obtaining good visual effects.

[0020] In a third aspect, the present application provides a model training device, including: a module for executing the method in any possible implementation of the first aspect above. Specifically, the device includes a module for executing the method in any possible implementation of the first aspect above.

[0021] In a fourth aspect, the present application provides an image generation device, including: a module for executing the method in any possible implementation of the second aspect above. Specifically, the device includes a module for executing the method in any possible implementation of the second aspect above.

[0022] In a fifth aspect, the present application provides a model training device, including a processor, which is coupled to a memory and can be used to execute instructions in the memory to implement the method in any possible implementation of the first aspect above. Optionally, the device further includes a memory. Optionally, the device further includes a communication interface, and the processor is coupled to the communication interface.

[0023] In a sixth aspect, the present application provides an image generation device, including a processor coupled to a memory, and the processor can be used to execute instructions in the memory to implement the method in any of the possible implementation manners in the above second aspect. Optionally, the device further includes a memory. Optionally, the device further includes a communication interface, and the processor is coupled to the communication interface.

[0024] In a seventh aspect, a processor is provided, including: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit the signal through the output circuit, so that the processor executes the method in any of the possible implementation manners in any of the above aspects.

[0025] In a specific implementation process, the above-mentioned processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be transistors, gate circuits, flip-flops, and various logic circuits, etc. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be output to, for example, but not limited to, a transmitter and transmitted by the transmitter, and the input circuit and the output circuit can be the same circuit, and this circuit is used as the input circuit and the output circuit at different times respectively. The embodiments of the present application do not limit the specific implementation manners of the processor and various circuits.

[0026] In an eighth aspect, a processing device is provided, including a processor and a memory. The processor is used to read instructions stored in the memory, and can receive a signal through a receiver and transmit the signal through a transmitter to execute the method in any of the possible implementation manners in any of the above aspects.

[0027] Optionally, there is one or more processors, and there is one or more memories.

[0028] Optionally, the memory can be integrated with the processor, or the memory is separately provided from the processor.

[0029] In a specific implementation process, the memory can be a non-transitory memory, such as a read-only memory, which can be integrated with the processor on the same chip, or can be separately provided on different chips. The embodiments of the present application do not limit the type of the memory and the setting manner of the memory and the processor.

[0030] The processing device in the above eighth aspect can be a chip, and the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, and is implemented by reading software code stored in the memory, and the memory can be integrated in the processor or can be located outside the processor and exist independently.

[0031] In a ninth aspect, a computer program product is provided, which includes: a computer program code, which, when executed, enables a computer to execute a method in any possible implementation of any of the above aspects.

[0032] In a tenth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed, the computer executes the method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 A schematic flow chart of a model training method provided in an embodiment of the present application;

[0035] Figure 2 A schematic flow chart of an image generation method provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the structure of the attention module provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of the structure of the normalization module provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of the generation model provided in the embodiment of the present application;

[0039] Figure 6 A schematic block diagram of a device provided in an embodiment of the present application;

[0040] Figure 7 A schematic block diagram of another device provided in an embodiment of the present application.

[0041] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] With the rapid development of deep learning, image inpainting has also made great progress and is widely used in fields such as object removal, restoration of damaged cultural relic images, post-production of films and television, and restoration of old photos. In order to eliminate or mitigate the effects caused by various factors and restore the original appearance of the image, it is necessary to use image restoration technology to repair the picture.

[0044] Currently, a large number of image inpainting techniques are implemented based on DCGAN. However, this DCGAN-based image inpainting method will have the problem of discontinuous pixels in the restored image, resulting in poor visual effects.

[0045] The data transmission method provided by the present application aims to solve the above technical problems in the prior art.

[0046] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0047] Figure 1 It is a schematic flowchart of a model training method 100 provided by an embodiment of the present application. The steps of method 100 can be executed by a model training device. Exemplarily, the model training device can be a processor or other devices with part or all of the functions of the model training device. The embodiments of the present application do not limit this. As Figure 1 shown, method 100 may include S101 and S102. Each step in method 100 will be described in detail below.

[0048] S101, obtain training samples, and the training samples include multiple groups of images.

[0049] Each group of images in the above multiple groups of images includes a lossless image and its corresponding damaged image, and the damaged image is obtained by processing the corresponding lossless image with a preset mask image.

[0050] Exemplarily, the above lossless images can be determined from existing publicly available image datasets, thereby obtaining a lossless image dataset. The above preset mask image can be an image in an artificially constructed mask dataset, which is used to simulate the randomness of image damage (for example, the construction of the mask image can be to mark the pixel values of the lossless region as 1, and the pixel values of the damaged region as values close to 0 or 0).

[0051] Combined with the above example, the model training device randomly combines each image in the obtained lossless image dataset with the mask images in the mask dataset to simulate the damage effect and obtain a damaged image dataset. A large number of damaged images are stored in this damaged image dataset. It should be noted that during the training process, each lossless image in the obtained lossless image dataset and its corresponding damaged image are used as a group of images, so that multiple groups of images can be obtained, and the multiple groups of images are used as training samples for model training.

[0052] It should be understood that the above lossless images are clear and complete images.

[0053] S102, input multiple groups of images into the generation model to train the generation model.

[0054] The above generation model is used to generate the corresponding lossless image based on the damaged image. The generation model is trained based on DCGAN. The generation model includes an attention module. The attention module is used to perform a convolution operation on the received first feature map to obtain a second feature map. After being processed by the mask image, the second feature map is used to perform a weighting operation on the first feature map to obtain a third feature map. The above attention module is also used to perform a convolution operation on the third feature map to obtain a fourth feature map.

[0055] It should be understood that the above mask image is the mask image used when generating the damaged image corresponding to the first feature map of the lossless image.

[0056] The model training method provided by the embodiments of this application inputs multiple groups of images in the obtained training samples into the generation model including an attention module in sequence, performs convolution operations (to obtain a second feature map) and mask image processing on the received first feature map respectively, uses the feature map after mask processing to perform a weighting operation on the first feature map, and then performs a convolution operation on the feature map after the weighting operation. This can solve the problem of pixel discontinuity after image repair, thereby alleviating the poor visual experience caused by pixel discontinuity and facilitating obtaining good visual effects.

[0057] Figure 2 It is a schematic structural diagram of the attention module provided by the embodiments of this application. As Figure 2As shown, the first feature map is divided into two paths and input into the attention module. The masked image is also input into the attention module. A 3×3 convolutional kernel is used to perform a convolution operation on one of the two paths of the first feature map to obtain a second feature map. The second feature map is multiplied by the input masked image (i.e., the process of the second feature map being processed by the masked image), and the obtained result is multiplied by the other path of the first feature map to obtain a third feature map. Then, a 3×3 convolutional kernel is used to perform a convolution operation on the third feature map, and a rectified linear unit (RELU) activation function is used to increase the non-linearity to obtain a fourth feature map.

[0058] The definition of the above RELU function is: RELU(x) = max(0, x). This function suppresses all values less than 0 to 0; for positive numbers, it directly outputs. As an activation function, this function has the following advantages: 1. It enables the training to converge quickly and solves the problem of gradient dispersion. 2. The input of this function can only transmit information when it is greater than 0, and this sparsity can improve the performance of the network.

[0059] It should be noted that the result obtained after the second feature map is processed by the masked image is equivalent to a weighting parameter; multiplying the obtained result by the other path of the first feature map to obtain a third feature map is equivalent to performing a weighting process on the first feature map.

[0060] It should be understood that Figure 2 The number of times of performing the convolution operation on the first feature map as shown can also be multiple times. If there are N (N in this application is an integer greater than 1) times of convolution operations, it means that the first feature map needs to be divided into N + 1 paths, and the convolution operation is performed on each of the N paths of the first feature map to obtain N second feature maps. Then, multiplying the N second feature maps by the masked image respectively can obtain N weighting parameters for performing a weighting operation on the remaining one path of the first feature map. It should be noted that the convolutional kernels used for the N times of convolution operations can be the same or different, such as (1×1), (3×3), or other sizes. This application does not make any limitations in this regard.

[0061] As an optional embodiment, the above first feature map includes the pixel values of multiple pixel points included in the damaged image. That is to say, the first feature map can be the input damaged image.

[0062] As an optional embodiment, the above generation model further includes a convolutional layer, which is used to perform a convolution operation on the input image to obtain a first feature map.

[0063] It should be understood that the generation model may include multiple convolutional layers. Each convolutional layer may include at least one convolutional kernel, and the size of the convolutional kernels included in each layer is the same. The size of the convolutional kernels included in different convolutional layers may be the same or different, and the present application does not limit this.

[0064] It should also be understood that the above input image may be a damaged image input to the generation model, or a feature map input to the convolutional layer.

[0065] As an optional embodiment, the above generation model further includes a normalization module, and this normalization module is used for: respectively performing normalization operations on the fourth feature map and the mask image and summing them to obtain the summed data; performing a convolutional operation on the fourth feature map to obtain a fifth feature map; using a predefined activation function to process the product of the fifth feature map and the summed data.

[0066] It should be understood that the above mask image is the mask image used when generating the damaged image corresponding to the fourth feature map from a lossless image.

[0067] As an optional embodiment, the above normalization operation satisfies the following formula:

[0068]

[0069] Where, y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

[0070] Combined with the above example, when performing the normalization operation on the fourth feature map, the above μ is the mean of the pixel values of all pixel points in the fourth feature map, and S is its corresponding standard deviation; when performing the normalization operation on the mask image, the above μ is the mean of the pixel values of all pixel points in the mask image, and S is its corresponding standard deviation.

[0071] The model training method provided by the embodiments of the present application respectively performs normalization operations on the received fourth feature map and mask image and sums them, and at the same time performs a convolutional operation on the fourth feature map, and then uses a predefined activation function to process the product of the feature map after the convolutional operation and the summed data, which can solve the problem of the offset of the mean and variance generated during the normalization operation of the damaged image, and is beneficial to improving the accuracy of damaged image repair.

[0072] Figure 3 is a schematic structural diagram of the normalization module provided by the embodiments of the present application. As Figure 3As shown, the fourth feature map is divided into two paths and input into the normalization module. A 3×3 convolutional layer is used to perform a convolution operation on one of the two paths of the fourth feature map to obtain a fifth feature map. The mask image is also input into the normalization module to perform normalization operations on the mask image and the other path of the fourth feature map respectively. The two obtained normalization results are added and multiplied by the fifth feature map to obtain a processed feature map. Then, a predefined activation function (e.g., sigmoid function) is used to process the processed feature map, and a sixth feature map is output.

[0073] The value range of the above sigmoid function is (0, 1), which can map a real number to the interval (0, 1), and it can directly use the gradient descent algorithm to optimize network parameters. As an activation function, this function has the advantages of being smooth and easy to differentiate.

[0074] It should be understood that Figure 3 The number of convolution operations performed on the shown fourth feature map can also be multiple times. If M (M in this application is an integer greater than 1) operations are performed, it means that the fourth feature map needs to be divided into M + 1 paths, and a convolution operation is performed on each of the M paths of the fourth feature map to obtain M fifth feature maps. Then, the M fifth feature maps are sequentially multiplied by the data after normalization operations to obtain a processed feature map. Then, a predefined activation function (e.g., sigmod function) is used to process the processed feature map. It should be noted that the convolution kernels used for the M convolution operations can be the same or different, such as (1×1), (3×3), or other sizes. This application does not make any limitations on this.

[0075] As an optional embodiment, the generation model includes multiple attention modules and multiple normalization modules. The multiple attention modules can correspond to the multiple normalization modules one by one. Each normalization module is used to receive the fourth feature map from the corresponding attention module to perform normalization processing on the received fourth feature map.

[0076] Exemplarily, the number of multiple attention modules and multiple normalization modules included in the generation model can be different. When the number of attention modules is greater than the number of normalization modules, each normalization module can be used to receive the fourth feature map from the corresponding attention module; or, when the number of attention modules is less than the number of normalization modules, for the normalization modules with corresponding attention modules, they can be used to receive the fourth feature map from the corresponding attention module, and for the normalization modules without corresponding attention modules, they can be used to receive feature maps from other modules or other convolutional layers of this generation model.

[0077] As an optional embodiment, the generation model further includes a deconvolution layer for performing a deconvolution operation on the received feature map.

[0078] It should be understood that the generation model may include multiple deconvolution layers. Each deconvolution layer may include at least one deconvolution kernel, and the sizes of the deconvolution kernels included in each layer are the same. The sizes of the deconvolution kernels included in different deconvolution layers may be the same or different, and the present application does not limit this. It should be noted that the number of deconvolution layers in the generation model is the same as that of the convolution layers.

[0079] As an optional embodiment, the above-mentioned masked image includes a first value and a second value. The positions where the first value is located correspond to the pixel points of the undamaged area in the damaged image, and the positions where the second value is located correspond to the pixel points of the damaged area in the damaged image. The first value is 1, and the second value is a value close to 0.

[0080] It should be understood that the first value and the second value in the embodiments of the present application are only used to distinguish different values, and the order thereof is not limited. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily limit being different.

[0081] Figure 4 It is a schematic flowchart of an image generation method 400 provided by an embodiment of the present application. The steps of the method 400 can be executed by an image generation device. Exemplarily, the image generation device may be a processor or other devices with some or all functions of the image generation device, and the embodiments of the present application do not limit this. As Figure 4 shown, the method 400 may include S401 and S402. The following will elaborate on each step in the method 400.

[0082] S401, obtain a damaged image.

[0083] S402, input the damaged image into a pre-trained generation model to generate a lossless image of the damaged image through the generation model.

[0084] The above-mentioned generation model is trained based on DCGAN. The generation model includes an attention module. The attention module is used to perform a convolution operation on the received first feature map to obtain a second feature map. After being processed by the masked image, the second feature map is used to perform a weighting operation on the first feature map to obtain a third feature map. The attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map; wherein, the first feature map includes the pixel values of multiple pixel points in the damaged image, or the first feature map is a feature map obtained by performing a convolution operation on the damaged image.

[0085] After the model training is completed, it can be applied to the image generation device. The image generation device can input the damaged image to be repaired into the pre-trained generation model to obtain the repaired image of the damaged image. Since the attention module is added to the generation model, the pixels in the repaired image are more continuous, thus alleviating the adverse visual experience caused by the discontinuity between pixels and facilitating the acquisition of good visual effects.

[0086] Figure 5 It is a schematic structural diagram of a generation model provided by an embodiment of the present application. As Figure 5 shown, the generation model mainly includes five parts (each dotted box represents a part). Among them, the structures of the first part, the second part, and the third part are exactly the same, specifically including: a convolutional layer, an attention module, a normalization module, a RELU activation function, and global pooling. The fourth part includes five residual modules, and the fifth part includes three deconvolution functions (corresponding to the three convolutional layers of the first three parts) and a RELU activation function.

[0087] The above-mentioned convolutional layer is used to extract features from the input image; the RELU activation function is used to perform non-linear mapping on the input image; global pooling is used to compress the input image, thereby reducing overfitting; the residual module is used to extract features from the input image; the deconvolution layer is used for the reverse operation of the convolutional layer, that is, the signal output by convolution is restored to the signal input by convolution through deconvolution.

[0088] As Figure 5 shown, C represents the number of channels, H represents the number of rows on this channel, W represents the number of columns on this channel, and C, H, and W are all positive integers. i×C (such as 2C, 4C, 8C) means that the number of channels becomes i times the original (i in the embodiment of the present application is an integer greater than 0), H / i (such as H / 2, H / 4, H / 8) means that the number of rows becomes 1 / i times the original, and W / i (such as W / 2, W / 4, W / 8) means that the number of columns becomes 1 / i times the original.

[0089] It should be understood that the generation model may include one or more convolutional layers, or may include one or more deconvolution layers. Or the generation model may include one or more residual modules. It should be noted that the number of convolutional layers should be the same as the number of deconvolution layers.

[0090] It should be understood that the attention module in the generation module can be placed before the convolutional layer, or after the activation function and before global pooling, or after global pooling, or after the residual module and before the deconvolution layer, or after the deconvolution layer. The embodiment of the present application does not limit the position of the attention module.

[0091] It should also be understood that the normalization module in the generation module can be placed before the convolutional layer, or after the activation function and before the global pooling, or after the global pooling, or after the residual module and before the deconvolution layer, or after the deconvolution layer. It should be noted that when both the normalization module and the attention module exist in the generation module, the normalization module should be located after the attention module.

[0092] In the embodiments of the present application, the images in the obtained damaged image dataset are sequentially input into the above generation model, and the repaired images corresponding to the damaged images are output. A loss function is constructed for the obtained repaired images and the lossless images corresponding to the damaged images, and the parameters in the generation model are updated to minimize the value of the loss function. The parameters corresponding to the minimum value are used as the parameters of the model, and the model training is completed to obtain the above pre-trained generation model.

[0093] It should be understood that the above Figure 1 and Figure 4 The method embodiments shown can be executed by the same device, for example, by modules for implementing different functions in the same device, or can be executed by different devices. The present application does not limit this.

[0094] It should also be understood that the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0095] In the above, in combination with Figures 1 to 4 , the model training method and the image generation method according to the embodiments of the present application are described in detail. Next, in combination with Figure 6 and Figure 7 The model training device and the image generation device according to the embodiments of the present application will be described in detail.

[0096] Figure 6 FIG. 600 is a schematic block diagram of a device 600 provided by an embodiment of the present application. The device 600 includes an acquisition module 610 and a processing module 620.

[0097] In a possible implementation manner, the device 600 can correspond to the model training device in the above method embodiments.

[0098] Among them, the acquisition module 610 is used to acquire training samples. The training samples include multiple groups of images, and each group of images includes a lossless image and its corresponding damaged image. The damaged image is obtained by processing the corresponding lossless image with a preset mask image. The processing module 620 is used to input the multiple groups of images into a generation model to train the generation model. Among them, the generation model is used to generate the corresponding lossless image based on the damaged image. The generation model is trained based on the deep convolutional generative adversarial network DCGAN. The generation model includes an attention module. The attention module is used to perform a convolution operation on the received first feature map to obtain a second feature map. After being processed by the mask image, the second feature map is used to perform a weighting operation on the first feature map to obtain a third feature map. The attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map.

[0099] Optionally, the first feature map includes pixel values of multiple pixels included in the damaged image.

[0100] Optionally, the generation model further includes a convolutional layer, which is used to perform a convolution operation on the input image to obtain the first feature map.

[0101] Optionally, the generation model further includes a normalization module, which is used to: perform a normalization operation on the fourth feature map and the mask image respectively and sum them to obtain the summed data; perform a convolution operation on the fourth feature map to obtain a fifth feature map; use a predefined activation function to process the product of the fifth feature map and the summed data.

[0102] Optionally, the normalization operation satisfies the following formula:

[0103]

[0104] Among them, y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

[0105] Optionally, the generation model includes multiple attention modules and multiple normalization modules. The multiple attention modules correspond to the multiple normalization modules one by one. Each normalization module is used to receive the fourth feature map from the corresponding attention module.

[0106] Optionally, the generation model further includes a deconvolution layer, which is used to perform a deconvolution operation on the received feature map.

[0107] Optionally, the masked image includes a first value and a second value. The positions where the first value is located correspond to the pixel points of the undamaged area in the damaged image, and the positions where the second value is located correspond to the pixel points of the damaged area in the damaged image. The first value is 1, and the second value is a value close to 0. In another possible implementation, the apparatus 600 may correspond to the image generation apparatus in the above method embodiments.

[0108] Among them, the obtaining module 610 is configured to obtain a damaged image; the processing module 620 is configured to input the damaged image into a pre-trained generation model to generate a lossless image of the damaged image through the generation model. The generation model is trained based on the deep convolutional generative adversarial network DCGAN. The generation model includes an attention module. The attention module is configured to perform a convolution operation on the received first feature map to obtain a second feature map. After being processed by the masked image, the second feature map is used to perform a weighting operation on the first feature map to obtain a third feature map. The attention module is further configured to perform a convolution operation on the third feature map to obtain a fourth feature map. Among them, the first feature map includes the pixel values of multiple pixel points in the damaged image, or the first feature map is a feature map obtained by performing a convolution operation on the damaged image.

[0109] In an optional example, those skilled in the art can understand that the above functions of the apparatus 600 can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The apparatus 600 can be used to execute each process and / or step corresponding to the model training apparatus or the image generation apparatus in the above method embodiments.

[0110] It should be understood that the apparatus 600 is embodied in the form of functional modules here. The term "module" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a combined logic circuit and / or other suitable components supporting the described functions.

[0111] In the embodiments of the present application, the apparatus 600 may also be a chip or a chip system, for example: a system on chip (SoC).

[0112] Figure 77 is a schematic block diagram of another apparatus 700 provided in an embodiment of the present application. The apparatus 700 includes a processor 710, a communication interface 720, and a memory 730. The processor 710, the communication interface 720, and the memory 730 communicate with each other via an internal connection path. The memory 730 is used to store instructions, and the processor 710 is used to execute the instructions stored in the memory 730 to control the communication interface 720 to send and / or receive signals.

[0113] In one possible implementation, the device 700 may be a model training device, or the functions of the model training device in the above embodiment may be integrated into the device 700, and the device 700 may be used to execute the various steps and / or processes corresponding to the model training device in the above method embodiment.

[0114] In another possible implementation, the device 700 may be an image generating device, or the functions of the image generating device in the above embodiment may be integrated into the device 700, and the device 700 may be used to execute the various steps and / or processes corresponding to the image generating device in the above method embodiment.

[0115] Optionally, the memory 730 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store device type information. The processor 710 may be configured to execute instructions stored in the memory, and when the processor executes the instructions, the processor may execute the various steps and / or processes corresponding to the model training device or image generation device in the above-described method embodiment.

[0116] It should be understood that in the embodiment of the present application, the processor 710 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0117] In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware processor, or can be executed and completed by the combination of the hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor executes the instructions in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0118] The present application also provides a computer program product, which includes: a computer program (which can also be referred to as code or instructions). When the computer program is run, it causes the computer to execute the steps shown in Method 100 and Method 400.

[0119] The present application also provides a computer-readable storage medium, which stores a computer program (which can also be referred to as code or instructions). When the computer program is run, it causes the computer to execute the steps shown in Method 100 and Method 400.

[0120] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by the combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0121] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0122] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0123] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, in each embodiment of the present application, each functional unit may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit.

[0125] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0126] Those skilled in the art will readily think of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application aims to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0127] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire training samples, where the training samples include multiple groups of images, each group of images includes a lossless image and its corresponding damaged image, and the damaged image is obtained by processing the corresponding lossless image with a preset mask image; Inputting the multiple sets of images into a generative model to train the generative model; wherein the generative model is used to generate corresponding lossless images based on the damaged images, the generative model is trained based on a deep convolutional generative adversarial network (DCGAN), and the generative model includes an attention module, the attention module is used to perform a convolution operation on the received first feature map to obtain a second feature map, the second feature map is processed by the mask image, and is used to perform a weighted operation on the first feature map to obtain a third feature map, and the attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map; The generation model further includes a normalization module, which is used to: performing normalization operations on the fourth feature map and the mask image respectively and summing the results to obtain summed data; Performing a convolution operation on the fourth feature map to obtain a fifth feature map; Processing the product of the fifth feature map and the summed data using a predefined activation function; The normalization operation satisfies the following formula: ， Wherein, y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

2. The method according to claim 1, characterized in that The first feature map includes pixel values of a plurality of pixel points contained in the damaged image.

3. The method according to claim 1, characterized in that The generation model also includes a convolution layer, which is used to perform a convolution operation on the input image to obtain the first feature map.

4. The method according to claim 1, wherein The generation model includes multiple attention modules and multiple normalization modules, the multiple attention modules correspond one-to-one to the multiple normalization modules, and each normalization module is used to receive the fourth feature map from the corresponding attention module.

5. The method according to claim 1, wherein The generation model also includes a deconvolution layer for performing a deconvolution operation on the received feature map.

6. The method according to claim 1, characterized in that The mask image includes a first value and a second value, the position of the first value corresponds to each pixel point in the undamaged area of the damaged image, and the position of the second value corresponds to each pixel point in the damaged area of the damaged image. The first value is 1 and the second value is a value close to 0.

7. An image generation method, characterized in that: include: Acquire a damaged image, where the damaged image is obtained by processing a corresponding lossless image with a preset mask image; The damaged image is input into a pre-trained generative model to generate a lossless image of the damaged image through the generative model, wherein the generative model is trained based on a deep convolutional generative adversarial network (DCGAN), and the generative model includes an attention module, which is used to perform a convolution operation on the received first feature map to obtain a second feature map. The second feature map is processed by a mask image and then used to perform a weighted operation on the first feature map to obtain a third feature map. The attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map; wherein the first feature map includes pixel values of multiple pixels in the damaged image, or the first feature map is a feature map obtained by convolution of the damaged image; The generation model further includes a normalization module, which is used to: performing normalization operations on the fourth feature map and the mask image respectively and summing the results to obtain summed data; Performing a convolution operation on the fourth feature map to obtain a fifth feature map; Processing the product of the fifth feature map and the summed data using a predefined activation function; The normalization operation satisfies the following formula: ， Wherein, y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

8. A model training device, characterized in that: include: An acquisition module is used to acquire training samples, wherein the training samples include multiple groups of images, each group of images includes a lossless image and its corresponding damaged image, and the damaged image is obtained by processing the corresponding lossless image with a preset mask image; a processing module, configured to input the multiple sets of images into a generative model to train the generative model; wherein the generative model is configured to generate corresponding lossless images based on damaged images, the generative model is trained based on a deep convolutional generative adversarial network (DCGAN), and the generative model includes an attention module, the attention module is configured to perform a convolution operation on a received first feature map to obtain a second feature map, the second feature map is processed by the mask image, and then used to perform a weighted operation on the first feature map to obtain a third feature map, and the attention module is further configured to perform a convolution operation on the third feature map to obtain a fourth feature map; The processing module includes a normalization module, the normalization module being configured to perform normalization operations on the fourth feature map and the mask image respectively and sum them to obtain summed data; perform a convolution operation on the fourth feature map to obtain a fifth feature map; and process the product of the fifth feature map and the summed data using a predefined activation function; The normalization operation satisfies the following formula: ， Wherein, y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

9. An image generating device, characterized in that: include: Acquisition module: used to acquire a damaged image, wherein the damaged image is obtained by processing the corresponding lossless image with a preset mask image; Processing module: used to input the damaged image into a pre-trained generative model to generate a lossless image of the damaged image through the generative model, the generative model is trained based on a deep convolutional generative adversarial network (DCGAN), and the generative model includes an attention module, the attention module is used to perform a convolution operation on the received first feature map to obtain a second feature map, the second feature map is processed by a mask image, and then used to perform a weighted operation on the first feature map to obtain a third feature map, the attention module is further used to perform a convolution operation on the third feature map to obtain a fourth feature map; wherein, the first feature map includes pixel values of multiple pixels in the damaged image, or the first feature map is a feature map obtained by convolution operation on the damaged image; The processing module includes a normalization module, which is configured to: perform normalization operations on the fourth feature map and the mask image respectively and sum them to obtain summed data; perform a convolution operation on the fourth feature map to obtain a fifth feature map; and process the product of the fifth feature map and the summed data using a predefined activation function; The normalization operation satisfies the following formula: ， Wherein, y is the normalized pixel value, x is the pixel value before normalization, μ is the mean of the pixel values before normalization, and S is the standard deviation of the pixel values before normalization.

10. A model training device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

11. An image generating device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to claim 7.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

13. A computer program product, characterized in that include: A computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images