Image processing method, image processing model and electronic equipment

Abstract

The invention provides an image processing method, an image processing model and electronic equipment, the image processing model is deployed in the electronic equipment, and according to the scheme, the image processing model is utilized to carry out image restoration processing on a first image according to target restoration area information corresponding to the first image. And obtaining a second image and displaying the second image in the electronic equipment. A part of convolution layers in the image processing model are gated convolution layers, the gated convolution layers comprise a plurality of depth separable convolution layers which are arranged in parallel and a first product layer, and the first product layer is used for multiplying outputs of the plurality of depth separable convolution layers to serve as an output result of the gated convolution layers. Due to the fact that the parameter quantity and the operand of the depth separable convolution layer are greatly reduced, the image restoration effect of the model can be guaranteed, the model parameters can be effectively reduced, the reasoning speed is increased, and the model is very friendly to deploy on the mobile terminal side such as a smart phone and the like in order to construct the gating convolution in the model through the depth separable convolution.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing model and an electronic device. Background Art

[0002] The main purpose of image restoration is to repair damaged or incomplete parts of an image, which is usually achieved using a corresponding image restoration model. However, existing image restoration models can only output low-resolution image restoration results, have a large number of parameters, and the algorithms used are difficult to deploy in mobile terminals such as smartphones. Therefore, there is an urgent need for an image processing technology solution that can both guarantee the image restoration effect and be easily implemented in mobile terminals. Summary of the invention

[0003] Multiple aspects of the present application provide an image processing method, an image processing model, a training method for an image processing model, and an electronic device, so as to achieve the purpose of being implemented in electronic devices such as mobile terminals and improving the image restoration effect.

[0004] In a first aspect, the present application provides an image processing method applicable to electronic devices. The method comprises:

[0005] Determine target repair area information corresponding to the first image;

[0006] Obtaining an image processing model deployed on an electronic device;

[0007] Inputting the target repair area information and the first image into the image processing model to perform image repair processing to obtain a second image;

[0008] displaying the second image in the electronic device;

[0009] Among them, some convolutional layers in the image processing model are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

[0010] In a second aspect, the present application provides another image processing method, which is applicable to electronic devices. The method comprises:

[0011] In response to an interface background change event, determining a selected first image;

[0012] Performing dimension conversion on the first image to obtain a converted first image;

[0013] Determining target restoration area information corresponding to the first image after conversion;

[0014] Inputting the target repair area information and the converted first image into an image processing module deployed on the electronic device to perform image repair processing to obtain a second image;

[0015] Displaying the second image as an interface background picture in the electronic device;

[0016] Among them, some convolutional layers in the image processing model are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of the multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

[0017] In a third aspect, the present application provides a model for image processing, the image processing model comprising:

[0018] A first restoration network is used to perform image restoration processing on the first image according to target restoration area information corresponding to the first image, and output a first restoration result;

[0019] A second restoration network, connected to an output end of the first restoration network, configured to process the first restoration result and output a second restoration result containing a second image;

[0020] Among them, some convolutional layers in the first repair network are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of the multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

[0021] In a fourth aspect, the present application provides another image processing model, the image processing model comprising: a repair combination network, wherein the repair combination network includes a self-attention module; wherein,

[0022] The inpainting combination network is used to perform image inpainting processing on the first image according to information of a target inpainting region corresponding to the first image; wherein an intermediate result generated during the processing is the input of the self-attention module;

[0023] The self-attention module includes a spatial attention combination layer, a channel attention combination layer and a first summation layer;

[0024] The spatial attention combination layer is used to perform feature extraction processing on the intermediate result and output spatial attention features;

[0025] The channel attention combination layer is used to perform feature extraction processing on the intermediate result and output channel attention features;

[0026] The first summation layer is used to add and fuse the spatial attention feature and the channel attention feature to output attention feature information;

[0027] Among them, the spatial attention combination layer includes three convolutional layers, a second convolutional layer, a third convolutional layer and a second summation layer arranged in parallel; the inputs of the three convolutional layers are all the intermediate results, and the output ends of the first convolutional layer and the second convolutional layer in the three convolutional layers are both connected to the input end of the second convolutional layer, the output end of the third convolutional layer and the output end of the second convolutional layer are both connected to the input end of the third convolutional layer; the input ends of each of the three convolutional layers and the output end of the third convolutional layer are all connected to the second summation layer.

[0028] In a fifth aspect, the present application provides a method for training an image processing model, wherein the image processing model includes a repair combination network and a discriminator network. The training method includes:

[0029] Using the discriminant network to discriminate the authenticity of the first sample image and the second sample image, and obtain discrimination result information; wherein the second sample image is generated by performing image restoration processing on the first sample image by the restoration combination network;

[0030] Determining a first loss function according to a difference between the first sample image and the second sample image;

[0031] Determine a second loss function according to an output discrimination value of the discrimination network for the second image included in the discrimination result information;

[0032] Determine a third loss function according to an output discrimination value of the discriminant network for the first image and an output discrimination value of the discriminant network for the second image included in the discrimination result information;

[0033] Determine a fourth loss function according to intermediate judgment values ​​of the first image and the second image respectively of each intermediate network layer in the discriminant network included in the discriminant result information;

[0034] Using a pre-trained perceptual loss network to perform content and style difference analysis on the first sample image and the second sample image, to obtain a fifth loss function and a sixth loss function;

[0035] determining a seventh loss function according to the image gradient information of the second sample image;

[0036] Based on the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, the sixth loss function and the seventh loss function, the network parameters of the repair combination network and the discriminator are optimized.

[0037] In a sixth aspect, the present application provides an electronic device. The memory and the processor; wherein the memory is used to store a program; wherein the program includes the program code of the image processing model provided in other aspects of the present application; the processor is coupled to the memory and is used to execute the program stored in the memory, so as to use the image processing model to implement the steps in each method provided in the present application.

[0038] In a seventh aspect, the present application provides a chip system, which is applied to the electronic device provided by the present application. The chip system includes a processor, and when the processor executes instructions, the steps in each method provided by the present application are implemented.

[0039] In an eighth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, the steps in the methods provided by the present application are implemented.

[0040] In a ninth aspect, the present application provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in each method provided by the present application can be implemented.

[0041] The technical solutions provided by each embodiment of this application are as follows:

[0042] Some convolutional layers in the image processing model are gated convolutions, and the gated convolutions are constructed using depthwise separable convolutions (multiple ones). Compared with conventional convolutions, depthwise separable convolutions have significantly lower parameters and computational complexity. Therefore, depthwise separable convolutions are used to construct gated convolutions in the model, which can not only ensure the image restoration effect of the model, but also effectively reduce model parameters and improve the inference speed, making the model very user-friendly for deployment on electronic devices, especially on mobile terminals such as smartphones.

[0043] In addition, the self-attention module in the image processing model includes a spatial attention combination layer and a channel attention combination layer. The spatial attention combination layer is constructed by three convolutional layers, a second convolutional layer, a third convolutional layer and a second summation layer arranged in parallel, and the inputs of the three convolutional layers are the same. The output ends of the first convolutional layer and the second convolutional layer in the three convolutional layers are connected to the input end of the second convolutional layer, the output end of the third convolutional layer and the output end of the second convolutional layer are connected to the input end of the third convolutional layer, and the input ends of the three convolutional layers and the output end of the third convolutional layer are connected to the second summation layer. The spatial attention features output by the spatial attention combination layer and the channel attention features output by the channel attention combination layer are added and fused through the second summation layer to output the attention feature information, which is the global feature mined by the self-attention module for the input data. The convolution kernel parameters of the convolutional layer in the above self-attention module are fixed and will not change dynamically with different input data, which further improves the friendliness of the model deployed on the electronic device (such as a mobile terminal) side, and can also achieve the effect of obtaining global feature information and improving the quality of image restoration.

[0044] In addition, the image processing model includes a two-level restoration network, namely a first restoration network and a second restoration network. The second restoration network can be used to improve the restoration quality of the restoration result output by the first restoration network, so that the final output restoration image (second image) is a high-resolution and clear image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1A A schematic diagram of an image restoration application scenario for restoring a damaged object main structure in an image provided by an exemplary embodiment of the present application;

[0047] Figure 1B A schematic diagram of an application scenario of image restoration for image erasure provided by an exemplary embodiment of the present application;

[0048] Figure 2A A schematic diagram of an application scenario of image restoration for restoring stretching artifacts that appear in an image during image dimension conversion provided by an exemplary embodiment of the present application;

[0049] FIG. 2B to FIG. 2D A schematic diagram of the principle of background image setting of an electronic device provided by an exemplary embodiment of the present application;

[0050] Figure 3A schematic diagram of the structure of an existing image restoration model provided for an exemplary embodiment of the present application;

[0051] Figure 4A A schematic diagram of the structure of another existing image restoration model provided for an exemplary embodiment of the present application;

[0052] Figure 4B for Figure 4A Schematic diagram of the structure of the context attention module in the image restoration model shown in;

[0053] Figure 5A and Figure 5B A schematic diagram of the structure of the image processing model provided for this application;

[0054] Fig. 6A A schematic diagram of the structure of the first sub-network in the image processing model provided in this application;

[0055] Figure 6B A schematic diagram of the structure of the downsampling module in the image processing model provided in this application;

[0056] Figure 6C A schematic diagram of the structure of the upsampling module in the image processing model provided in this application;

[0057] Fig.6D A specific structural example of the first sub-network in the image processing model provided in this application;

[0058] Fig. 7A and Figure 7B A schematic diagram of the structure of the second sub-network in the image processing model provided in this application;

[0059] Fig. 8A and Figure 8B for Fig. 7A and Figure 7B A schematic diagram of the structure of the self-attention module included in the second sub-network shown;

[0060] Fig. 9A and Fig. 9B A schematic diagram of the structure of the gated convolutional layer in the image processing model provided in this application;

[0061] Fig. 10A and Fig. 10B A schematic diagram of the structure of the second restoration network in the image processing model provided in this application;

[0062] Fig.11 A schematic diagram of the structure of the discriminant network in the image processing model provided in this application;

[0063] Fig.12 A schematic diagram of the principle of determining the loss information provided for this application;

[0064] Fig.13 and Fig.14 A flowchart of the image processing method provided in this application;

[0065] Fig.15 A schematic diagram of a specific application scenario of the image processing method provided in this application;

[0066] Fig.16A and Fig. 16B A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0067] Image restoration is an important issue in the field of computer vision. Its main function is to fill in damaged or missing image information through relevant image restoration algorithms. Specifically, the image restoration algorithm can be used to construct a corresponding image restoration model to restore the damaged main structure of the object in the image. Of course, it can also be used to fill in the background of the main body of the object, eliminate unwanted parts of the image, etc. The following are several commonly used image restoration application scenarios:

[0068] Application scenario 1: Restoring the main structure of the destroyed object in the image

[0069] like Figure 1A As shown in (a), the structure of the main objects such as the tiger and the scenery in the image is destroyed, but it can be restored after image restoration. The restored image is as follows Figure 1A As shown in (b) in .

[0070] Application scenario 2: Image erasure

[0071] like Figure 1B As shown in (a), the user specifies the area to be erased in the image, such as the area where object 01 (including people, objects, etc.) is located. The image restoration model will use the background information to fill in object 01 in the image to achieve the purpose of erasing object 01.

[0072] In addition to the two application scenarios described above, there are other application scenarios, such as the following application scenario 3:

[0073] Application scenario 3: Repair stretching artifacts that appear in images during image dimension conversion

[0074] Electronic devices such as mobile terminals such as smart phones generally have a wallpaper function to facilitate users to set corresponding background images (wallpapers) for the device screen according to their own preferences. The wallpaper function can be implemented through corresponding applications on it, such as Figure 2B The subject application shown in, or Figure 2C the Settings application shown in the Figure 2DThe gallery application shown in FIG. In the wallpaper function, there is a conversion module for 3D wallpaper, through which the 2D (two-dimensional) image selected by the user can be converted into a 3D image to be displayed as a background image in the device. However, suppose the user selects Figure 2A In the process of converting the 2D image into a 3D image and rendering it in the new viewing angle, the background part that was originally blocked by the foreground will be exposed, so that the image obtained after the conversion will have artifacts, such as Figure 2A The black area 02 in the 3D image shown in (b) is a stretching artifact. Therefore, the image restoration model needs to fill and repair the black area in the 3D image using the background information in the image to eliminate the artifact.

[0075] The above application scenario three is the main application scenario of each embodiment provided in the following application.

[0076] At present, image restoration models are usually constructed using the following two categories of algorithms: traditional image restoration algorithms (non-machine learning algorithms) and machine learning algorithms. Traditional image restoration algorithms are based on technologies such as interpolation and texture synthesis, which often make it difficult to generate realistic, detailed images when restoring images. Machine learning algorithms (such as deep learning, etc.) are intelligent algorithms that have emerged in recent years with the development of artificial intelligence technology. They have been widely used in the field of image restoration and have achieved remarkable results, but there are also certain problems. For example, the current image restoration models based on machine learning can only output low-resolution image results, have large parameters, and have problems such as unfriendly operators deployed on the mobile terminal side. The following two examples are given to analyze in detail the problems of existing image restoration models based on machine learning.

[0077] Example 1: There is an existing image restoration model based on Fourier convolution (Large Mask Inpainting, LaMa), the network structure of which is as follows: Figure 3 As shown in the figure, the mask image m is used to cover the original image x to obtain a four-channel image x' as the input of the feedforward inpainting network. After the four-channel image x' is input into the feedforward inpainting network, it is first downsampled to a low resolution, then passes through several fast Fourier convolutions (FCC), and finally the output is upsampled to generate a high-resolution image.

[0078] Among them, FCC is mainly used for feature extraction, so FCC can also be called feature extraction module. Each FCC includes two branches: local branch and global branch. The local branch includes convolution operation, which mines local information through a smaller perception field. The global branch includes Fourier transform operation, which is used to mine global information. The two branches fuse the information through the concatenation operation (Concat), realizing the mining and fusion of local and global information. Specifically, Figure 3 When data is input into FCC, a part of the channel is processed by the local branch to obtain local features; the other part of the channel is processed by the global branch to obtain global features; the local features and the global features are fused and output, so that the output channel is consistent with the input channel. Among them, the local branch includes the first convolution module and the second convolution module, the global branch includes the third convolution module and the Fourier spectrum transform module, and the Fourier spectrum transform module includes the first combination layer (Con-BN-ReLU, including the convolution operation Con, the forward planning operation BN and the activation operation ReLu), the Fourier transform layer (Real FFT2d, referring to the real part two-dimensional fast Fourier transform), the second combination layer (Con-BN-ReLU), the inverse Fourier transform (Inv Real FFT2d) and the convolution layer (Conv 1x1).

[0079] Using the above Figure 3 The image restoration model shown has a good restoration effect and can process large-resolution images, but the network parameters are large and Fourier transform convolution (FFC) is difficult to deploy on the mobile terminal side.

[0080] Example 2: There is also an image restoration model based on gated convolution. The network structure of this model is as follows: Figure 4A As shown, it includes a generator network and a discriminator network composed of a coarse repair network and a fine repair network. The input color channel (RGB Channel), mask channel (also called mask channel) and hand-painted channel (Sketch Channel) and other information are first processed by the coarse repair network to output a coarse repair image, which is processed by the fine repair network to become a fine repair image, and then processed by the discriminant network (also called the discriminator network) for true and false discrimination. Among them, the convolution contained in the above-mentioned coarse repair network and the fine repair network is a gated convolution. Specifically, the above-mentioned fine repair network includes an encoding module and a decoding module. The encoding module includes two encoding branches: an upper encoding branch and a lower encoding branch. The upper encoding branch contains a contextual attention module (contextual attention, CA).

[0081] Figure 4AAlthough the image restoration model shown is relatively lightweight, the context attention module therein is difficult to deploy on the mobile terminal side because it contains dynamic convolutions, and the model can only output low-resolution restoration results, which are relatively blurry when upsampled to high resolution.

[0082] In order to solve the above problems existing in the existing image restoration model, the basic design idea of ​​the image restoration model of this application is:

[0083] 1) All gated convolutions in the generator network used for initial image restoration are implemented using depthwise separable convolutions, which can effectively reduce the number of convolution parameters and the amount of computation, and is also more user-friendly for deployment on mobile terminals;

[0084] 2) Use the self-attention module to replace the context attention module, so as to achieve the same restoration effect of obtaining global information and improving the filling quality;

[0085] 3) The generator network and the super-resolution network are combined into one model. The low-resolution restoration results are sampled on the super-resolution network and can directly output a filling result that is close to the clarity of the high-resolution image. This can solve the problem that the existing model has blurred restoration effects on the input high-resolution image due to limited computing power and memory.

[0086] Based on the above basic design ideas, this application provides an image processing method, an image processing model, an electronic device, etc. through multiple embodiments. Note: It can be understood that the image processing model described here is also an image restoration model, but different expressions are used in different narrative scenarios.

[0087] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0088] In the embodiments of the present application, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar phases with substantially the same functions and effects. For example, the first signal generating circuit and the second signal generating circuit are only used to distinguish different signal generating circuits, and the order thereof is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0089] It should be noted that the words "exemplary" or "for example" in this application are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In addition, "at least one" in this application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, B exists alone, wherein A and B can be singular or plural, etc. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b and c can represent: a, b, c, a, b and c, a and b, a and c, b and c.

[0090] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0091] To facilitate understanding of the present application, the image processing model provided by the present application is first described in detail. Before describing in detail, some of the proprietary names involved are first introduced and explained.

[0092] Downsampling operation: It is used to reduce the dimension and size of an image. Specifically, it means reducing the number of sampling points from a high-resolution image according to certain rules, thereby reducing the resolution of the image. More specifically, downsampling can achieve dimensionality reduction by filtering out pixel feature information that has little effect and redundant information in the image, thereby retaining key feature information.

[0093] Upsampling operation: refers to the process of interpolating the image and increasing the number of sampling points based on the existing pixel feature information in the image through a certain algorithm, thereby improving the image resolution. Common upsampling algorithms include but are not limited to: nearest neighbor interpolation, bilinear interpolation, cubic spline interpolation, etc. Through upsampling, a low-resolution image can be restored to a size similar to or the same as the original high-resolution image.

[0094] The image processing model provided in this application is mainly used for image restoration, such as the image restoration in the aforementioned application scenario three. In the following introduction to the various convolutional layers involved in the image processing model provided in this application, except for the convolutional layers specifically described, the step size S of the other convolutional layers not specifically described defaults to the first step size a1. The first step size a1 can be, but is not limited to, 1. In this embodiment, the first step size a1=1 is selected.

[0095] Figure 5A and Figure 5B The network structure diagram of the image processing model provided in the embodiment of the present application is shown. The image processing model is deployed in an electronic device, which may be a mobile terminal device, such as a personal computer terminal device (such as a laptop computer, a desktop computer, an all-in-one computer, etc.), a smart phone, a tablet, a car computer, a game console terminal device, a smart wearable device (such as a smart watch), etc. Figure 5A and Figure 5B As shown in Figure 1, the image processing model includes a repair combination network and a discriminant network.

[0096] The inpainting combination network is used to perform image inpainting processing on the first image to be inpainted according to the target inpainting region information to obtain a second image. The target inpainting region information is mask information corresponding to the first image, and the mask information is a binary mask image m (such as Figure 5A As shown), the acquisition and implementation of the target repair area information will be described in detail in the following method embodiment and will not be specifically introduced here.

[0097] In a specific implementation, the repair combination network includes a first repair network 11 and a second repair network 12. The first repair network 11 and the second repair network 12 are described in detail below.

[0098] 1. First Repair Network 11

[0099] The first restoration network 11 is a generation network (also referred to as a generator network), and its input includes the downsampled first image I and target restoration area information. The downsampled first image I is obtained by downsampling the first image to be restored using the downsampling network 14 in the model. The downsampling network 14 may be composed of at least one fourth downsampling module. The structure of the fourth downsampling module can be seen in Figure 6B or Fig.6DThe downsampling module shown in . The first image to be repaired is often a high-resolution image input by the user. In order to reduce the subsequent network calculation amount, improve the efficiency of image repair, etc., it is necessary to perform a downsampling operation on the first image. After the downsampling operation, the first image will be reduced in dimension and size. Therefore, the downsampled first image is a low-resolution image. After the target repair area information and the downsampled first image are input into the first repair network 11, the execution of the first repair network 11 will eventually obtain an output result Out1 (the first repair result). Since the first repair network 11 can only process low-resolution images, the output result Out1 is a low-resolution repair result. It is necessary to further use the second repair network 12 to upsample the low-resolution repair result to a high resolution. Therefore, the output result Out1 of the first repair network 11 is the input of the second repair network 12, and the output result Out2 of the second repair network 12 is the target image obtained by the final repair (such as Figure 5A The second image shown in ), the second restoration network 12 can also be called a super-resolution network.

[0100] The first repair network 11 includes a first sub-network 111 and a second sub-network 112. The network structure of the first sub-network 111 can be, but is not limited to, a common codec network structure (encoder-decoder structure), where a common codec network includes an encoder and a decoder. The second sub-network 112 includes two parallel encoders and a decoder. The target repair area information and the downsampled first image are first input to the first sub-network 111 for processing, and the first sub-network 111 will output a processing result (such as Figure 5B The processing result is a coarse-grained output result. It can be understood that the coarse-grained output result is a blurred image filling and repairing result. The processing result will be used as the input of the second sub-network 112. The execution of the second sub-network 112 will output the first repairing result (such as Figure 5B The output result Out1) shown in the figure is obtained by adding more detailed features on the basis of the processing result, and is a fine-grained output result.

[0101] From the above content, it can be understood that the first sub-network 111 is a coarse-grained generator, and the second sub-network 112 is a coarse-grained generator.

[0102] Fig. 6A FIG. 1 shows a schematic diagram of the structure of the first sub-network 111. Fig. 6A, the first subnetwork 111 includes: multiple seventh convolutional layers for convolution operations, a third downsampling module group for downsampling operations, and a second upsampling module group for upsampling operations. According to the connection position of each seventh convolutional layer, the multiple seventh convolutional layers are divided into the following three partial seventh convolutional layers: the first partial seventh convolutional layer, the second partial seventh convolutional layer, and the third partial seventh convolutional layer. Each partial seventh convolutional layer contains at least one seventh convolutional layer connected in sequence, and if the number of at least one convolutional layer connected in sequence is multiple (two or more), then the output of the previous convolutional layer in the at least one convolutional layer connected in sequence is the input of the next convolutional layer. Specifically, the third downsampling module group is connected between the first partial seventh convolutional layer and the second partial seventh convolutional layer, and the second upsampling module group is connected between the second partial seventh convolutional layer and the third partial seventh convolutional layer.

[0103] In addition, each of the multiple seventh convolutional layers is followed by an activation layer (not shown in the figure), and the activation layer is implemented using an activation function, which may be, but not limited to, an ELU (Elu activation) function, a ReLU (Rectified Linear Unit) function, a Sigmoid function (also called a Logistic function), a tanh function (also called a hyperbolic tangent activation function), a Selu function, a PreLU (Parametric ReLU) function, etc. The reason for connecting an activation layer after a convolutional layer (such as the seventh convolutional layer) is that the operation performed by the convolutional layer on the input image is linear, but the information of the input image is not all linearly separable. By performing nonlinear operations through the activation layer, the features can be better mapped to remove redundancy in the data and improve the network expression ability.

[0104] In specific implementation, except for the seventh convolutional layer arranged at the end of the seventh convolutional layer in the third part, the activation layers connected to the output ends of other seventh convolutional layers can be implemented using functions such as ELU. For the seventh convolutional layer arranged at the end of the seventh convolutional layer in the third part, the connected activation layer can be implemented using the tanh function, which is used to limit the output of the seventh convolutional layer arranged at the end to the interval [-1,1]. Therefore, the activation layer (such as the tanh function) is called a limit layer in this embodiment, and the activation function corresponding to the limit layer is called a limit function. Therefore, the first sub-network 111 also includes a second limit layer, which is connected to the final output end of the seventh convolutional layer in the third part, that is, connected to the output end of the seventh convolutional layer arranged at the end of the seventh convolutional layer in the third part. The second limit layer can be implemented using, but not limited to, the tanh function.

[0105] All seventh convolutional layers in the first sub-network 111 can be gated convolutional layers, and the structure of the gated convolutional layers will be described in detail below and will not be repeated here. Also, the third down-sampling module group includes at least one down-sampling module, and the second up-sampling module group includes at least one up-sampling module.

[0106] Figure 6B Figure 2 shows a schematic diagram of the structure of the downsampling module. Figure 6B The downsampling module includes multiple convolutional layers, and the step size S of some of the multiple convolutional layers is the first step size a1, and the step size S of another part of the convolutional layers is the second step size a2, and the second step size a2 is greater than the first step size a1.

[0107] Figure 6C Figure 2 shows a schematic diagram of the structure of the upsampling module. Figure 6C The upsampling module includes an interpolation layer and multiple convolutional layers connected in sequence; wherein the step size of the multiple convolutional layers is the first step size a1, and the interpolation layer is connected to the input end of the first convolutional layer in the multiple convolutional layers, and is used to amplify or scale the input of the upsampling module. In this embodiment, the function of the interpolation layer is to amplify the input of the upsampling module, and the corresponding magnification can be but is not limited to 2 times. The interpolation layer can be implemented using a corresponding interpolation algorithm, such as nearest neighbor interpolation, bilinear interpolation, etc.

[0108] It should be noted that the convolution layers in the above-mentioned upsampling module and downsampling module can also be gated convolution layers.

[0109] Fig.6D A specific structural example of the first sub-network 111 is shown, and the first sub-network 111 includes 9 gated convolutional layers (i.e., 9 seventh convolutional layers), the step size S of the 9 gated convolutional layers is 1, and 7 of the gated convolutional layers are sequentially connected between the third down-sampling module group and the second up-sampling module group, one of the remaining two gated convolutional layers is connected to the input end of the third down-sampling module group, and the other gated convolutional layer is connected to the output end of the second up-sampling module group. The third down-sampling module group includes 4 down-sampling modules, each of which includes a gated convolutional layer with a step size S of 2 and a gated convolutional layer with a step size S of 1. The second up-sampling module group includes 4 up-sampling modules, each of which includes an interpolation layer and two gated convolutional layers, the magnification of the interpolation layer is 2 times, and the step size S of the two gated convolutions is 1.

[0110] Fig. 7A FIG. 1 shows a schematic diagram of the structure of the second sub-network 112. Fig. 7A, the second sub-network 112 includes: a first encoder (encoder E1 as shown in the figure), a second encoder (encoder E2 as shown in the figure), a concatenation layer and a decoder D1. The first encoder and the second encoder are two encoders arranged in parallel, and the inputs of both are the output result Out_0 of the first sub-network 111 mentioned above. The output end of the first encoder and the output end of the second encoder are both connected to the input end of the decoder D1 through the concatenation layer C. That is, it can be understood that through the concatenation layer C, the output of the first encoder and the output of the second encoder can be concatenated by channel to obtain a concatenation operation result, and the concatenation operation result is input into the decoder D1 for decoding processing, and a decoding processing result, that is, the output result Out1, will be output.

[0111] Specifically, the functions of the structures included in the second sub-network 112 are as follows:

[0112] The first encoder is used to obtain the processing result (i.e. Fig.6D The output result Out_0 shown in FIG. 1 ); encoding the processing result and outputting a first encoding result;

[0113] A second encoder, used for encoding the processing result and outputting a second encoding result;

[0114] a concatenation layer C connected to an output end of the first encoder and an output end of the second encoder, configured to concatenate the first encoding result and the second encoding result, and output a concatenation result;

[0115] A decoder is connected to the output end of the splicing layer, and is used to decode the splicing result and output the first repair result (i.e. Fig. 7A The output result Out1) is shown in FIG.

[0116] The first encoder includes a plurality of fourth convolutional layers for convolution operations, a first downsampling module group for downsampling operations, and the self-attention module. The first downsampling module group is connected between the first part of the fourth convolutional layers and the second part of the fourth convolutional layers in the plurality of fourth convolutional layers, the self-attention module is connected between the remaining third part of the fourth convolutional layers and the second part of the fourth convolutional layers, and the output end of the third part of the fourth convolutional layers is connected to the input end of the splicing layer.

[0117] The second encoder includes a plurality of fifth convolutional layers and a second downsampling module group for convolution operation. The second downsampling module group is connected between the first part of the fifth convolutional layers and the remaining second part of the fifth convolutional layers in the plurality of fifth convolutional layers, and the output end of the second part of the fifth convolutional layers is connected to the input end of the concatenation layer. The second downsampling module group includes a plurality of downsampling modules for downsampling operation.

[0118] The module structure of the downsampling module can be found in other embodiments. Figure 6B and Fig.6D Related content described.

[0119] The decoder D1 includes a plurality of sixth convolutional layers for convolution operations, a first upsampling module group and a first limiting layer. The first part of the sixth convolutional layers in the plurality of sixth convolutional layers is connected between the splicing layer and the first upsampling module group, and the remaining second part of the sixth convolutional layers is connected between the first upsampling module group and the first limiting layer. The first limiting layer can be, but is not limited to, implemented using a tanh function, and is used to limit the output of the second part of the sixth convolutional layer to a first interval. The first upsampling module group includes a plurality of upsampling modules for upsampling operations. For the module structure of the upsampling module, refer to other embodiments in conjunction with Figure 6C and Fig.6D Related content described.

[0120] Similarly, the fifth convolutional layer, the fifth convolutional layer and the sixth convolutional layer in the second sub-network 112 can also be gated convolutional layers. The structure of the gated convolutional layer will be described in detail below and will not be repeated here.

[0121] Figure 7B A specific network structure example of the second sub-network 112 is shown. The structures of the first encoder, the second encoder and the decoder D1 included in the second sub-network 112 are respectively: the first encoder includes 4 gated convolution layers, and 1 of the gated convolution layers is connected to the input end of the first down-sampling module group (including 4 sampling modules), and 1 gated convolution layer is connected between the first down-sampling module group and the self-attention module, and the remaining 2 gated convolution layers are connected to the output end of the self-attention module; the second encoder includes 6 gated convolution layers and the second down-sampling module group (including 4 down-sampling modules), and 1 of the gated convolution layers is connected to the input end of the second down-sampling module group, and the remaining 5 gated convolution layers are connected to the output end of the second down-sampling module group; the decoder D1 includes 3 gated convolution layers, the first up-sampling module group (including 4 up-sampling modules) and the first limit layer, 2 of the 3 gated convolution layers are connected between the first up-sampling module group and the splicing layer, and the remaining 1 gated convolution layer is connected between the first up-sampling module group and the first limit layer.

[0122] From the above content, an intermediate result F (the intermediate convolution result output by the fourth convolution layer of the second part) generated by the second sub-network 112 in the process of processing the output result Out_0 of the first sub-network 111 is the input of the self-attention module. And accordingly, the function of the self-attention module is to process the intermediate result F to mine the global features of the intermediate result F. Among them, when processing the intermediate result F, the self-attention module will focus on the important information that needs to be paid attention to in the intermediate result F and filter out other unimportant information.

[0123] The above intermediate result F is specifically the feature map output by the fourth convolutional layer of the second part. Usually, an image mainly contains two dimensional features: scale space (such as length and width) and channel. For this reason, it is necessary to use the spatial attention mechanism and the channel attention mechanism to process these two dimensional features to improve the feature representation ability. Among them, the channel attention mechanism assigns different weight coefficients to each channel according to the importance of each channel, thereby strengthening the important channel features and suppressing the unimportant channel features. The spatial attention mechanism assigns different weights to different areas of the image, so that the model pays more attention to important regional features and ignores irrelevant information.

[0124] Based on this, in this embodiment, in order to more comprehensively mine image features to improve the image restoration effect, the above-mentioned self-attention module includes combination layers corresponding to the spatial attention mechanism and the channel attention mechanism respectively.

[0125] Fig. 8A and Figure 8B The structural diagram of the self-attention module is shown in Figure . Fig. 8A As shown, the self-attention module includes: a spatial attention combination layer, a channel attention combination layer and a first summation layer St1. The spatial attention combination layer is used to mine the scale space features of the intermediate result F, and the channel attention combination layer is used to mine the channel features of the intermediate result F. The output end of the spatial attention combination layer and the output end of the channel attention combination layer are both connected to the input end of the first summation layer St1. The intermediate result F is input into the spatial attention combination layer and processed through the attention combination layer respectively, and then the spatial attention features output by the spatial attention combination layer and the channel attention features output by the channel attention combination layer are added and fused through the first summation layer St1. The result of the addition and fusion is the attention feature information finally output by the self-attention module, and the attention feature information will be input into the convolution layer located downstream of the self-attention module.

[0126] From the above content, that is, an intermediate result F generated by the second sub-network 112 in the process of processing the output result of the first sub-network 111 is the input of the self-attention module. And the functions of each structural layer included in the self-attention module are as follows:

[0127] A spatial attention combination layer, used for performing feature extraction processing on the intermediate results and outputting spatial attention features;

[0128] A channel attention combination layer is used to perform feature extraction processing on the intermediate results and output channel attention features;

[0129] The first summation layer is used to add and fuse the spatial attention feature and the channel attention feature to output attention feature information.

[0130] Furthermore, if Figure 8B As shown, the spatial attention combination layer includes three parallel convolutional layers and a computing module, and the output ends of the three convolutional layers are all connected to the input end of the computing module. The computing module includes a first transposition layer, a second multiplication layer Mu2 and a third multiplication layer Mu3 connected to each other, and a second summation layer St2 connected to the third multiplication layer. The first convolutional layer of the three convolutional layers is connected to the input end of the second multiplication layer Mu2 through the first transposition layer, the second convolutional layer is directly connected to the input end of the second multiplication layer Mu2, and the output end of the third convolutional layer is connected to the input end of the third multiplication layer Mu3. Among them, the convolution kernel size k of each of the three convolutional layers can be but is not limited to 1x1.

[0131] Specifically, the functions of the structural layers included in the spatial attention combination layer are as follows:

[0132] The first convolutional layer is used to extract features from the intermediate results and output the first type of features (query features Q);

[0133] The second convolutional layer is used to extract features from the intermediate results and output the second type of features (key features K);

[0134] The third convolutional layer is used to extract features from the intermediate results and output the third type of features (value features V);

[0135] A first transposition layer, connected to an output end of the first convolutional layer, configured to perform transposition deformation on the first type of features and output the first type of features after transposition deformation;

[0136] a second product layer, connected to the output ends of the first transposed layer and the second convolutional layer, and configured to multiply the first type of features after transposition deformation by the second type of features, and output a first multiplication result;

[0137] A third product layer, connected to the output ends of the second product layer and the third convolution layer, and configured to multiply and fuse the spatial attention map obtained according to the first multiplication result with the third type of feature to obtain a second multiplication result;

[0138] The second summation layer is connected to the output end of the third product layer, and is used to add and fuse the second multiplication result and the intermediate result to output the spatial attention feature.

[0139] The above-mentioned spatial attention map can be obtained by normalizing the first multiplication result, that is, a normalization layer can be connected between the second product layer and the third product layer. The normalization layer can be constructed using a Softmax function, and the first multiplication result can be normalized using the normalization layer to obtain a spatial attention map and input it into the third product layer Mu3.

[0140] Taking the aforementioned intermediate result F as a feature map of size C×H×W as an example, the process of the above spatial attention combination layer processing the input intermediate result F can be briefly described as follows:

[0141] S11. Input the feature map of size C×H×W into the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, and three types of features can be obtained, namely, query feature Q, key feature K, and value feature V; where C, H, and W in the above C×H×W represent the number of channels, height, and width of the feature map respectively, the query feature Q is output by the first convolutional layer, the key feature K is output by the second convolutional layer, and the value feature V is output by the third convolutional layer. The main function of the above three convolutional layers is to reduce the number of channels, but will not change the height and width of the feature map.

[0142] S12. Transpose the query feature Q through the first transposition layer and then multiply it with the key feature K, and use the corresponding normalization function to normalize the result of the multiplication to obtain a spatial attention map of size (H×W)×(H×W); wherein the normalization function can be but is not limited to a softmax function.

[0143] S13. Multiply the spatial attention map with the value feature V and add the feature map to obtain the final output spatial attention feature.

[0144] Continue to see Figure 8BAs shown, the channel attention combination layer includes: a second transposition layer, a fourth product layer Mu4 and a fifth product layer Mu5 connected to each other, and a third summation layer St3 connected to the fifth product layer Mu5. The output end of the second transposition layer is connected to the input end of the fourth product layer Mu4, and the input of the second transposition layer is the intermediate result F. The input of the fourth product layer Mu4 includes the intermediate result F and the output of the second transposition layer. The input of the fifth product layer Mu5 includes the intermediate result F and the output of the fourth product layer Mu4. The input of the third summation layer St3 includes the output of the fifth product layer Mu5 and the intermediate result F. In the above, the output of the fourth product layer Mu4 is normalized before being input to the fifth product layer Mu5.

[0145] Specifically, the functions of the structural layers included in the channel attention combination layer are as follows:

[0146] A second transposition layer, used to perform transposition deformation on the intermediate result and output the deformed intermediate result;

[0147] a fourth product layer Mu4, connected to the output end of the second transposed layer, for multiplying the deformed intermediate result by the intermediate result to output a channel attention map;

[0148] a fifth product layer Mu5, connected to the output end of the fourth product layer, configured to multiply the channel attention map by the intermediate result and output a third multiplication result;

[0149] The third summation layer St3 and the output end of the fifth product layer are used to add the third multiplication result and the intermediate result to output the channel attention feature.

[0150] The above-mentioned channel attention map can be obtained by normalizing the multiplication result obtained by multiplying the deformed intermediate result and the intermediate result, that is, a normalization layer (not shown in the figure) can also be connected between the fourth product layer Mu4 and the fifth product layer Mu5. The normalization layer can be constructed using a Softmax function. The normalization layer can be used to normalize the input multiplication result to obtain a channel attention map and input it into the fifth product layer Mu5.

[0151] Still taking the feature map of the intermediate result F with a size of C×H×W as an example, the process of the above channel attention combination layer processing the input intermediate result F can be briefly described as follows:

[0152] S21, after transposing the feature map, a transposed feature map is obtained; the transposed feature map is multiplied with the original feature map, and the result of the multiplication is normalized to obtain a channel attention map of size C×C;

[0153] S22. Multiply the channel attention map with the original feature map and add the original feature map to obtain the final channel attention feature to be output.

[0154] It is understandable that the convolutional layers included in the above self-attention module may be but are not limited to conventional convolutional layers.

[0155] Compared with the context attention module used in the existing image restoration model, the self-attention module used in the image processing model provided in this embodiment has the following advantages: on the basis of being able to obtain global features and improve the image restoration effect, it can be easily deployed on a mobile terminal (such as a smart phone). The reasons are as follows:

[0156] See Figure 4B The schematic diagram of the structure of the context attention module used in the existing image restoration model is shown. The existing context attention module processes the input feature map as follows:

[0157] Step 1: The input feature map is processed into blocks and deformed (transposed) to obtain a series of convolution kernels;

[0158] Step 2: Then use the convolution kernel obtained in step 1 to perform a convolution operation with the original feature map and use the corresponding normalization function (such as softmax function) to perform a normalization operation to obtain an attention score;

[0159] Step 3: Use the attention score to perform deconvolution on the block-based feature map to obtain the final output.

[0160] The problem with the currently used context attention module is that the parameters of the convolution kernel in step 2 are not fixed, but dynamically change according to the input feature map. Therefore, the convolution involved is dynamic convolution, which is difficult to deploy on the mobile terminal side. In addition, the deconvolution operation in step 3 is relatively time-consuming and is an operator that is not friendly to the mobile terminal side. The self-attention combination layer provided in this embodiment does not involve the above problems.

[0161] In order to make the entire image processing model more suitable for deployment on the mobile terminal side, the above-mentioned FIG. 6A to FIG. 7BWhen the corresponding convolutional layer in the described first repair network 11 (the convolutional layer that does not belong to the self-attention module, such as the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer, etc.) is a gated convolutional layer, this gated convolutional layer can be implemented by using a depthwise separable convolutional layer. The depthwise separable convolutional layer is composed of two parts: channel-wise convolution (Depthwise Convolution) and pointwise convolution (Pointwise Convolution). Among them, the number of convolution kernels of the channel-wise convolution is consistent with the number of input channels, and one channel is convolved by only one convolution kernel; pointwise convolution often uses 1×1 convolution, which is used to fuse the feature map obtained in the previous step in the channel dimension. The depthwise separable convolutional layer reduces the parameters required for convolution calculation and improves the running speed by splitting the correlation between the spatial dimension and the channel (depth) dimension, so it is more suitable for deployment on the mobile terminal side.

[0162] In a specific implementation, the gated convolution layer may include multiple depth-wise separable convolution layers and a first product layer arranged in parallel. The inputs of the multiple depth-wise separable convolution layers are the same, which are the outputs of the corresponding upstream layers / modules. The output ends of the multiple depth-wise separable convolution layers are all connected to the input end of the first product layer, and the first product layer is used to multiply and fuse the outputs of the multiple depth-wise separable convolution layers as the output result of the gated convolution layer.

[0163] Fig. 9A and Fig. 9B FIG. 4 shows a schematic diagram of a specific structure of a gated convolutional layer provided in this embodiment. Fig. 9A As shown in the figure, the gated convolution layer is a combination layer, which includes two depth-separable convolution layers and the first product layer Mu1. The output ends of the two depth-separable convolution layers are connected to the input end of the first product layer Mu1. The gated convolution layer processes the input X as follows:

[0164] S31. Input X into two depth-separable convolutional layers for convolution processing to obtain two depth features, namely, the first depth feature (mask feature) output by the first depth-separable convolutional layer and the second depth feature (image feature) output by the second depth-separable convolutional layer.

[0165] S32, performing a limit process on the first depth feature to limit the value of the first depth feature to the interval [0,1], where Fig. 9B , the limit processing can be implemented by the corresponding third limit layer, and the third limit layer can be implemented by using, but not limited to, the sigmoid function. And, the second depth feature is activated to obtain the second depth feature after activation; wherein, Fig. 9B , the activation processing can be implemented through the corresponding activation layer, and the activation layer can be implemented but is not limited to using the ELU function.

[0166] S33. The first depth feature after limiting and the second depth feature after activation processing are finally input into the first product layer Mu1 for multiplication and fusion, so as to obtain the final output Y.

[0167] Among them, the expression of the final output Y is as follows:

[0168] Output Y = sigmoid(DS_conv(X))*ELU(DS_conv(X)), where DS_conv represents a depth-wise separable convolutional layer.

[0169] The above gated convolution can adaptively learn which pixels in the features are effective for the repair task, and assign different weights to different pixels based on the learning results, thereby improving the repair effect. The more effective the pixel is for the repair task, the higher the weight is often assigned.

[0170] Based on the above content, in a specific example, the gated convolution provided in this embodiment includes: a first depthwise separable convolution layer, a second depthwise separable convolution layer, a third limit layer, an activation layer and a first product layer Mu1;

[0171] An output end of the first depthwise separable convolutional layer is connected to the third restriction layer, and an output end of the second depthwise separable convolutional layer is connected to the activation layer;

[0172] The output end of the third limiting layer and the output end of the activation layer are both connected to the first product layer Mu1;

[0173] The third limiting layer is used to limit the first depth convolution feature output by the first depth separable convolution layer, and output the first depth convolution feature after limiting; the value of the first depth convolution feature after limiting is within a third interval;

[0174] The activation layer is used to activate the second depth convolution feature output by the second depth separable convolution layer, and output the activated second depth convolution feature;

[0175] The first product layer Mu1 is used to multiply the first deep convolution feature after limiting and the second deep convolution feature after activation, and output a fourth multiplication result; the fourth multiplication result is the output result of the gated convolution.

[0176] 2. Second Repair Network 12

[0177] The second repair network is a super-resolution network. Due to the limitations of computing power and memory, the aforementioned first repair network 11 (generation network) can only process low-resolution images, and its output result is often fuzzy when upsampled to the original image resolution. In this embodiment of the application, the output result of the first repair network 11 (that is, the aforementioned first repair result, which is a low-resolution filling repair result) is further processed by the second repair network 12, and the output result of the first repair network can be upsampled to a high resolution. Among them, the second repair network 12 can be any common lightweight super-resolution network model that can be deployed on a mobile terminal.

[0178] Optionally, the structure of the second repair network can be as follows Fig. 10A As shown, the second repair network includes: multiple eighth convolutional layers, two upsampling module groups and a fourth summation layer St4. The two upsampling module groups include a third upsampling module group and a fifth upsampling module group. The third upsampling module group is connected between the first part of the eighth convolutional layers in the multiple eighth convolutional layers and the remaining second part of the eighth convolutional layers, and the output end of the second part of the eighth convolutional layers and the output end of the fourth upsampling module group are both connected to the input end of the fourth summation layer St4, and the inputs of the first part of the convolutional layers and the fourth upsampling module group are both the outputs of the previous level network (that is, the first repair result output by the first repair network 11). The above-mentioned third upsampling module group and the fifth upsampling module group each include at least one upsampling module, and the detailed description of the structure of the upsampling module can be found in other embodiments.

[0179] Based on this Fig. 10A From the structure shown, it can be seen that the processing process of the second restoration network 12 is: the input (i.e. the first restoration result finally output by the first restoration network 11 at the previous level) will be processed by the first part of the eighth convolution layer (such as 7 eighth convolution layers), the third upsampling module group and the second part of the eighth convolution layer (such as 1 convolution layer), and the input will also be processed by the fourth upsampling module group; then, the two output results obtained by the above processing are added and fused to obtain the corresponding final restoration result (which is the second restoration result containing the second image).

[0180] Fig. 10BA specific structural example of the second repair network is shown, in which the second repair network 12 includes 8 eighth convolutional layers, a third upsampling module group, a fourth upsampling module group and a fourth summation layer St4, and the 8 eighth convolutional layers include 1 eighth convolutional layer with a convolution kernel size of 5×5, two eighth convolutional layers with a convolution kernel size of 1×1, and 5 eighth convolutional layers with a convolution kernel size of 3×3. 1 eighth convolutional layer with a convolution kernel size of 5×5, 1 eighth convolutional layer with a convolution kernel size of 1×1, 4 eighth convolutional layers with a convolution kernel size of 3×3, 1 eighth convolutional layer with a convolution kernel size of 1×1, the third upsampling module group and 1 eighth convolutional layer with a convolution kernel size of 3×3 are sequentially connected to form a first branch, and the fourth upsampling module group forms a second branch. The outputs of the first branch and the second branch are connected to the fourth summation layer St4.

[0181] In specific implementation, the upsampling operations of the two upsampling module groups in the second repair network 12 can be implemented using a common interpolation algorithm, such as bilinear interpolation. In addition, all convolutional layers in the second repair network 12 can be implemented using conventional convolutional layers (i.e., ordinary convolutional layers), and except for the last convolutional layer (i.e., the eighth convolutional layer connected to the output end of the third upsampling module group, which is not connected to the activation layer), each of the remaining convolutional layers is connected to an activation layer (not shown in the figure), and the activation layer can be implemented using, but not limited to, a PReLU activation function.

[0182] Furthermore, the image processing model provided in the embodiment of the present application may include a discriminant network 13 (also called a discriminator network) in addition to the first inpainting network 11 and the second inpainting network 12. The discriminant network 13 is a binary classifier for discriminating the authenticity of the input image. Figure 5B , the input image of the discriminant network 13 includes the original first image I and the second image output by the second restoration network 12 is the restored image of the first image I.

[0183] The network structure of the above-mentioned discriminant network 13 is an encoder structure, which can be implemented using conventional multiple convolutional layers.

[0184] Fig.11 A specific structural example of the discriminant network is shown. In this specific example, the discriminant network 13 is implemented using 4 ninth convolutional layers with a step size S of 2 and a convolution kernel size of 5×5. The four ninth convolutional layers are sequentially connected end to end. The ninth convolutional layer can be a conventional convolutional layer. Each ninth convolutional layer is connected to an activation layer (not output in the figure). The activation layer can be implemented using but not limited to the following activation function: LeakyReLU activation function. The discriminant network 13 can downsample the input to 1 / 32 of the original input resolution.

[0185] Based on the discriminant network 13, the first image I and the second image The discrimination result information and the original first image I and the second image The loss information can be determined, and based on the loss information, the image processing model can be iteratively optimized until the iteration stop condition is met, so as to optimize the model parameters of the image restoration model. For example, the network parameters of the first restoration network 11 and the second restoration network 12 are optimized to improve the image restoration effect, and the network parameters of the discriminant network 13 are optimized to improve the ability of the discriminant network to distinguish true from false.

[0186] The above-mentioned discrimination result information may include: the output discrimination value of the discrimination network 13 for the first image I, the output discrimination value of the discrimination network 13 for the second image The output discriminant value of each intermediate network layer in the discriminant network 13 for the first image I, the intermediate discriminant value for the second image It can be understood that the output discriminant value is the output of the last network layer in the discriminant network 13, and the intermediate network layer refers to the other network layers in the discriminant network except the last network layer.

[0187] In this embodiment, the loss information used to optimize the image processing model includes:

[0188] 1) The first loss function Lrecon (reconstruction loss function) is based on the original first image I and the second image The first loss function Lrecon is expressed as follows:

[0189]

[0190] Among them, ‖.‖1 represents the L1 norm, also known as Manhattan distance, minimum absolute error, etc. The operation corresponding to the L1 norm is vector (such as ) is the sum of the absolute values ​​of the non-zero elements in Then ||ΔX1||1=∑|x p |, p represents the Pth element in the vector ΔX1.

[0191] 2) Model loss function, including the second loss function L G (to repair the combined network loss) and the third loss function L D (to determine network loss); among them,

[0192] The second loss function L G According to the discriminant network 13, the second image The output discrimination value is determined by

[0193] The third loss function LD According to the output discrimination value of the discriminant network 13 for the first image I and the output discrimination value of the second image The output judgment result is determined.

[0194] The second loss function L G And the third loss function L D The expressions are as follows:

[0195]

[0196]

[0197] Where D represents the discriminant network, E represents the calculated mean, and D(I) represents the output discriminant value of the discriminant network for the first image I. Represents the discriminant network for the second image The output discriminant value of .

[0198] 3) The fourth loss function L FM (is the feature matching loss function, i.e., FM loss function), which is based on the intermediate discriminant value of each intermediate network layer (convolutional layer) in the discriminant network for the first image I and the intermediate discriminant value of the second image The fourth loss function L FM The main function of is to stabilize the training of the image processing model (a GAN (Generative Adversarial Network) model), which is expressed as follows:

[0199]

[0200] Among them, k represents the kth intermediate network layer in the discriminant network, D k (I) indicates They represent the intermediate discriminant value of the kth intermediate network layer in the discriminant network for the first image I and the intermediate discriminant value of the second image I. The intermediate discriminant value of . ‖.‖2 represents the L2 norm. The operation corresponding to the L2 norm is the square root of the sum of the squares of the vector elements). but

[0201] 4) The perceptual loss function includes: the fifth loss function Lcontent (content loss function) and the sixth loss function Lstyle (style loss function); wherein,

[0202] The fifth loss function Lcontent and the sixth loss function Lstyle use the pre-trained perceptual loss network to calculate and analyze the original first image I and the second image The perceptual loss network can be a VGG network (also known as VGGNet, which is a classic convolutional neural network architecture). Specifically, for example, it can be a VGG19 network composed of 19 convolutional layers or a VGG16 network composed of 16 convolutional layers. In this embodiment, the VGG19 network is selected.

[0203] The expressions of the fifth loss function Lcontent and the sixth loss function Lstyle are as follows:

[0204]

[0205]

[0206] Among them, VGG i (*) represents the feature map output by the i-th intermediate network layer (convolutional layer) in the VGG network. j (*) represents the feature map output by the jth intermediate network layer (convolutional layer) in the VGG network. In this embodiment, VGG represents the pre-trained VGG19 network, and Gram represents the calculated Gram matrix. "*" represents the first image I or the second image

[0207] 5) Seventh loss function L TV (is the total variation loss function), which is based on the second image The seventh loss function L TV The expression is:

[0208]

[0209] in, Indicates the image gradient in the x direction. Indicates the image gradient in the y direction. The sixth loss function L TV By constraining the image gradient and in the pixel domain, the effect of removing noise and maintaining boundaries can be achieved.

[0210] From the loss information described in 1) to 5) above, we can know:

[0211] The total loss function is: L = L recon +α1L G +α2L content +α3L style +α4L TV +α5L FM

[0212] Among them, α1, α2, α3, α4, and α5 are the weights of the corresponding loss function items in the total loss function, which can be flexibly set according to the actual situation. For example, they can be set as follows: α1=0.01, α2=0.01, α3=20, α4=0.1, α5=0.3.

[0213] Based on the above loss information, a corresponding optimizer can be used to implement iterative optimization of the image processing model. The optimizer can be, but is not limited to, an Adam (Adaptive Moment Estimation) optimizer. The Adam optimizer is an adaptive optimization algorithm that can adjust the learning rate based on historical gradient information. Its main function is to update the model parameters based on the gradient information to minimize the loss function.

[0214] From the above content, see Fig.12 As shown, the image processing model provided in the embodiment of the present application may also include: a loss determination module 14 and an optimizer 15. The loss determination module 14 is used to determine the first image I and the second image based on the discriminant network 13. The discrimination result information of the first image I and the second image Determine loss information. For details of the loss information, please refer to the above descriptions 1) to 5). The optimizer 15 is used to optimize the model parameters of the image processing model according to the loss information. For example, the network parameters of the restoration network (such as the first restoration network and the second restoration network) in the image processing model are optimized.

[0215] In addition to the aforementioned networks, the image processing model may also include other networks in some embodiments, such as a pre-trained recognition network, which may be connected to the input end of the aforementioned first restoration network 11 to identify target restoration area information that needs to be restored in the first image I. The network structure of the recognition network is not specifically limited, as long as it can correctly identify the target restoration area information that needs to be restored in the first image I.

[0216] The present application also provides several image processing methods implemented by using the aforementioned image processing model. The execution subjects of the several image processing methods are all electronic devices, more specifically mobile terminal devices. The aforementioned image processing model is deployed on the electronic device. Fig.16A As shown, the electronic device 100 can be, but is not limited to: a smart phone (such as a foldable phone, a straight phone), a tablet, a smart wearable device (such as a smart bracelet). The structure of the electronic device 100 is described in detail in other embodiments below and will not be repeated here. In addition, the scene in which the image processing method is applied can be the above-mentioned combined FIG. 2A to FIG. 2CThe described application scenario three: repairing the stretching artifacts that appear in the image when the image dimension is converted, of course, can also be applied to other scenarios, such as the application scenario one and application scenario two described above, etc., which are not limited here. Application scenario three is the main application scenario of each image processing method provided in this application.

[0217] Fig.13 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. Fig.13 As shown, the image processing method comprises the following steps:

[0218] 201. Determine target restoration area information corresponding to a first image;

[0219] 202. Obtain an image processing model deployed on the electronic device;

[0220] 203. Input the target repair area information and the first image into the image processing model for image processing to obtain a second image;

[0221] 204. Display the second image in the electronic device;

[0222] Among them, some convolutional layers in the image processing model are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

[0223] In the above 201, the first image is acquired in different ways in different application scenarios.

[0224] For example, when setting an interface background scene for a mobile terminal such as a smart phone, the first image may be determined according to an interface background change event triggered by a user. The interface background change event triggering may be implemented as follows but is not limited to: Figure 2B and Figure 2C When the user wants to change the current interface background of the electronic device 100, he can open the theme application on the mobile terminal or the "Desktop and Personalization" function in the setting application. At this time, multiple wallpaper images / theme images will be displayed on the application interface. The user can select an image from the multiple wallpaper images / theme images and trigger the setting of the selected image as wallpaper. Figure 2D , the user can also select a picture from multiple pictures provided in the gallery application of the mobile terminal, and trigger the "Set as" function provided by the gallery application to set the selected picture as wallpaper. In response to the interface background change event triggered by the user, the mobile terminal determines the image selected by the user as the first image.

[0225] Generally, the image initially selected by the user is a 2D image. In order to present a better wallpaper effect, when the user activates the 3D wallpaper function provided by the mobile terminal, the 2D image selected by the user can be converted into a 3D image for presentation. Based on this, the first image can also be an image obtained by performing a dimensional conversion (upward dimensional conversion (such as conversion from 2D to 3D) or downward dimensional conversion (such as conversion from 3D to 2D)) on the image selected by the user. After the dimensional conversion, after obtaining the first image, it can be identified whether the first image needs to be repaired, and when necessary, the above steps S201 to S203 are triggered to be executed.

[0226] From the above content, that is, in a specific example, the method provided by this embodiment may also include the following steps:

[0227] 200a, in response to an interface background change event triggered by a user through an interactive interface provided by an electronic device, determining a first image selected by a user;

[0228] 200b. Determine whether it is necessary to perform dimensional conversion on the first image based on the dimensional attribute of the first image and the interface background function information activated by the electronic device; for example, if the first image is a 2D image and the 3D wallpaper function of the mobile terminal is in an activated state, it is determined that it is necessary to perform dimensional conversion on the first image to convert the first image from 2D to 3D.

[0229] 200c. When necessary, perform dimension conversion on the first image to obtain the converted first image;

[0230] 200b. Trigger execution of step 201 based on the converted first image.

[0231] The region to be repaired in the converted first image can be identified by using, but is not limited to, a pre-trained recognition network, so as to obtain target repair region information. The obtained target repair region information can be a binary mask image, such as Figure 5A The binary mask image m shown in FIG. 1 , the white area in the binary mask image m represents the position of the area to be repaired in the first image I.

[0232] For another example, in a shooting scene, the first image may be obtained from a camera. When a user uses a camera on a mobile terminal to take a photo, due to certain factors, the main structure of the object in the photo is often destroyed. In this case, if the user activates the shooting image repair function on the mobile terminal, the image captured by the camera can be first input into the image processing model for repair processing, and then the repaired image is presented to the user. Based on this, in another specific example, the method provided in this embodiment may also include the following steps:

[0233] 200a', acquiring a first image captured by a camera on the electronic device;

[0234] 200b', identifying whether the first image needs to be repaired;

[0235] For example, the first image captured by the camera can be input into a pre-trained recognition network to determine whether to repair the first image based on the output recognition result. For example, if the binary mask image contained in the recognition result does not contain a white area, it means that the first image does not need to be repaired; conversely, if a white area exists, it means that the first image needs to be repaired.

[0236] 200c', when necessary, trigger the execution of the above step 201 according to the recognition result.

[0237] For another example, in a scenario where the user actively wants to erase the image content, the first image may be actively input by the user. For example, the user opens the first image in the gallery through an image beautification application on an electronic device, and uses the tool provided by the image beautification application to mark the area in the first image that he wants to erase. After marking, the image beautification application may determine the target repair area information according to the marking information of the first image, and based on the target repair area information, call the image processing model deployed on the electronic device to repair the first image.

[0238] For detailed description of the image processing model in 202, please refer to the relevant content in other aforementioned embodiments.

[0239] See Figure 5A and Figure 5B As shown, the above image processing model includes a first restoration network and a second restoration network for image restoration. And accordingly, the above step 203 "inputting the target restoration area information and the first image into the image processing model for image restoration processing to obtain a second image" may specifically include:

[0240] 2031. Input the target restoration area information and the first image into the first restoration network, and output a first restoration result;

[0241] 2032. Input the first restoration result into the second restoration network, and output a second restoration result containing the second image.

[0242] Before the first image is fed into the first restoration network, such as combining Figure 5B As shown, the first image may be downsampled using the corresponding downsampling network 14 in the image processing model before being input into the first restoration network 11. The specific reasons for downsampling the first image first are described in detail here, and reference may be made to the relevant contents in other embodiments of the present application.

[0243] In the above 2031, the first repair network includes a first sub-network and a second sub-network, and the second sub-network is connected to the output end of the first sub-network. For the network structure of the first sub-network and the second sub-network, please refer to FIG. 6A to FIG. 8B shown.

[0244] In a specific implementation scheme, the implementation process of the above step 2031 is as follows:

[0245] 20311. Use the first sub-network to process the target repair area and the first image to obtain a processing result.

[0246] As combined Figures 6A to 6D The network structure diagram of the first sub-network 111 is shown, and the specific processing process of this step 20311 is:

[0247] A11, performing a convolution operation on the target repair area and the first image to obtain a sixth convolution result;

[0248] This convolution operation is implemented using the seventh convolution layer of the first part. If the seventh convolution layer of the first part includes one gated convolution layer, the sixth convolution result is obtained by convolving the target repair area and the first image once using the one gated convolution layer.

[0249] A12. Perform a downsampling operation on the sixth convolution result to obtain a third downsampling result;

[0250] This downsampling operation is implemented using the third downsampling module group. If the third downsampling module group includes 4 downsampling modules, the third downsampling result is obtained by continuously downsampling the sixth convolution result four times.

[0251] A13, performing a convolution operation on the third down-sampling result to obtain a seventh convolution result;

[0252] This convolution operation is implemented using the seventh convolution layer of the second part. If the seventh convolution layer of the second part includes 7 gated convolution layers, the seventh convolution result is obtained by continuously convolving the third down-sampling result seven times using the 7 gated convolution layers.

[0253] A14, performing an upsampling operation on the seventh convolution result to obtain a second upsampling result;

[0254] The up-sampling operation is implemented by using the second down-sampling module group. If the second up-sampling module group includes 4 up-sampling modules, the second up-sampling result is obtained by up-sampling the seventh convolution result four times in a row.

[0255] A15. Perform a convolution operation on the second up-sampling result to obtain an eighth convolution result;

[0256] This convolution operation is implemented using the seventh convolution layer of the third part. If the seventh convolution layer of the third part includes one gated convolution layer, the eighth convolution result is obtained by convolving the second up-sampling result once using the one gated convolution layer.

[0257] A16. Perform limit processing on the eighth convolution result to obtain the processing result whose value is limited within the second interval.

[0258] This restriction process is implemented using the second limit layer. The second interval range can be, for example, the interval range of [-1, 1]. The processing result is as follows: Figure 6B The output result Out_0 is shown in FIG.

[0259] 20312. Input the processing results into the first encoder and the second encoder in the second sub-network respectively for encoding, and perform a splicing operation on the first encoding result output by the first encoder and the second encoding result output by the second encoder to obtain a splicing result.

[0260] As combined Figures 7A to 7B The network structure diagram of the second sub-network 112 is shown, and the specific implementation process of "inputting the processing result into the first encoder (encoder E1) for encoding to obtain the first encoding result" in this step 20312 is as follows:

[0261] A21, performing a convolution operation on the processing result to obtain a first convolution result;

[0262] This convolution operation is implemented using the fourth convolution layer of the first part. If the fourth convolution layer of the first part includes one gated convolution layer, the first convolution result is obtained by convolving the processing result (output result Out_0) once using the one gated convolution layer.

[0263] A22. Perform a downsampling operation on the first convolution result to obtain a first downsampling result;

[0264] The downsampling operation is implemented by using the first downsampling module group. If the first downsampling module group includes 4 downsampling modules, the first downsampling result is obtained by downsampling the first convolution result four times in a row by using the 4 downsampling modules.

[0265] A23, performing a convolution operation on the first down-sampling result to obtain a second convolution result;

[0266] This convolution operation is implemented using the fourth convolution layer of the second part. If the fourth convolution layer of the first part includes one gated convolution layer, the second convolution result is obtained by convolving the first down-sampling result once using one gated convolution layer.

[0267] The second convolution result is an intermediate result produced by the second sub-network and will be input into its downstream self-attention module.

[0268] A23, inputting the second convolution result into the self-attention module, and outputting the attention feature information;

[0269] Combination Fig. 8A and Figure 8B The structural diagram of the self-attention module is shown in FIG. 1 , and the specific implementation of this step A23 is as follows:

[0270] A231, use the spatial attention combination layer to perform feature extraction on the second convolution result and output the spatial attention feature;

[0271] A232, using the channel attention combination layer to perform feature extraction processing on the second convolution result, and output the channel attention feature;

[0272] A232. Use the first summation layer St1 to add and fuse the spatial attention feature and the channel attention feature to obtain attention feature information.

[0273] Furthermore, the specific implementation of the above step A231 is as follows:

[0274] A2311, performing a convolution operation on the second convolution result to obtain first-category features, second-category features, and third-category features;

[0275] This step is achieved by using the first convolution layer, the second convolution layer, and the third convolution layer to convolve the second convolution result respectively.

[0276] A2312. Obtain a spatial attention map according to the first type of features and the second type of features;

[0277] This step is achieved by first transposing and deforming the first type of features using the first transposition layer, and then using the second product layer Mu2 to multiply them with the second type of features.

[0278] A2313, according to the spatial attention map, the third type of features and the second convolution result, the spatial attention features are obtained. This step is achieved by first multiplying the spatial attention map with the third type of features using the third convolution layer, and then using the second summation layer St2 to add and fuse with the second convolution.

[0279] And, the specific implementation of the above step A232 is as follows:

[0280] A2321, transpose and transform the second convolution result to obtain a transformed second convolution;

[0281] The transposed deformation in this step is achieved using the second transposed layer.

[0282] A2322, multiplying the deformed second convolution by the second convolution to obtain a channel attention map;

[0283] The multiplication operation in this step is implemented using the fourth product layer Mu4.

[0284] A2323, multiplying the channel attention map by the second convolution result to obtain a third multiplication result;

[0285] The multiplication operation in this step is implemented using the fifth product layer Mu5 A2324, and the third multiplication result is added to the second convolution to obtain the channel attention feature.

[0286] The addition operation in this step is implemented using the third summation layer St3.

[0287] A24. Perform a convolution operation on the attention feature information obtained through the above step A23 to obtain a first encoding result.

[0288] This convolution operation is implemented using the fourth convolution layer of the third part. If the fourth convolution layer of the third part includes two gated convolution layers, the first encoding result is obtained by continuously convolving the attention feature information twice using the two gated convolution layers.

[0289] Continue to combine Figures 7A to 7B As shown, the specific implementation process of "inputting the processing result into the second encoder (encoder E2) for encoding to obtain the second encoding result" in the aforementioned step 20312 is as follows:

[0290] A31, performing a convolution operation on the processing result (output result Out_0) to obtain a third convolution result;

[0291] This convolution operation is implemented using the fifth convolution layer of the first part. If the fifth convolution layer of the first part includes one gated convolution layer, the first convolution result is obtained by convolving the processing result once using the one gated convolution layer.

[0292] A32, performing a downsampling operation on the third convolution result to obtain a second downsampling result;

[0293] This downsampling operation is implemented using the second downsampling module group. If the second downsampling module group includes 4 downsampling modules, the second downsampling result is obtained by downsampling the third convolution result four times in a row using the 4 downsampling modules.

[0294] A33. Perform a convolution operation on the second down-sampling result to obtain the second encoding result.

[0295] This convolution operation is implemented using the fifth convolution layer of the second part. If the fifth convolution layer of the second part includes 5 gated convolution layers, the second encoding result is obtained by continuously convolving the second down-sampling result five times using the 5 gated convolution layers.

[0296] Continue to combine Figures 7A to 7B As shown, the "concatenating the first encoding result and the second encoding result to obtain a concatenated result" in the aforementioned step 20312 can be implemented using the concatenating layer C, and the function of the concatenating layer C is to concatenate the input data according to the channels.

[0297] 20313. Input the splicing result into the decoder in the second sub-network for decoding to obtain a first repair result.

[0298] Still continue to combine Fig. 7A As shown in Figure B, the specific implementation process of this step 20313 is as follows:

[0299] A41, performing a convolution operation on the splicing result to obtain a fourth convolution result;

[0300] This convolution operation is implemented using the sixth convolution layer of the first part. If the sixth convolution layer of the first part includes two gated convolution layers, the fourth convolution result is obtained by performing two consecutive convolutions on the spliced ​​results using the two gated convolution layers.

[0301] A42. Perform an upsampling operation on the fourth convolution result to obtain a first upsampling result;

[0302] The upsampling operation is implemented by using the first upsampling module group. If the first upsampling module group includes 4 upsampling modules, the first upsampling result is obtained by upsampling the fourth convolution result four times in a row by using the 4 upsampling modules.

[0303] A43. Perform a convolution operation on the first up-sampling result to obtain a fifth convolution result;

[0304] This convolution operation is implemented using the sixth convolution layer of the second part. If the sixth convolution layer of the second part includes one gated convolution layer, the fifth convolution result is obtained by convolving the first up-sampling result once using the one gated convolution layer.

[0305] A44, performing a limit processing on the fifth convolution result to obtain the first repair result whose value is limited to a first interval. The limit processing is implemented by using a first limit layer. The first interval range is, for example, a [-1, 1] interval range.

[0306] It should be noted that, in a specific example, all gated convolutions in the first repair network include gated convolutions including two depthwise separable convolution layers arranged in parallel, such as Fig. 9A Or as shown in 9B.

[0307] In a specific implementation, if combined with Fig. 10A and Fig. 10B The structural diagram of the second restoration network shown in FIG. 1 is a schematic diagram of the structure of the second restoration network. The implementation process of the aforementioned step 2032 "inputting the first restoration result into the second restoration network and outputting the second restoration result containing the second image" is as follows:

[0308] 20321. Perform a convolution operation on the first repair result to obtain a ninth convolution result;

[0309] This convolution operation is implemented using the eighth convolution layer of the first part. If the eighth convolution layer of the first part includes 7 convolution layers, the ninth convolution result is obtained by performing seven consecutive convolutions on the first restoration result using 7 convolution layers.

[0310] 20322. Perform an upsampling operation on the ninth convolution result to obtain a third upsampling result;

[0311] The upsampling operation is implemented by using a third upsampling module group. The third upsampling module group includes at least one upsampling module. If the third upsampling module group includes one upsampling module, the third upsampling result is obtained by performing an upsampling operation on the third upsampling result using one upsampling module.

[0312] 20323. Perform a convolution operation on the third upsampling result to obtain a tenth convolution result;

[0313] This convolution operation is implemented using the eighth convolution layer of the second part. If the eighth convolution layer of the second part contains one convolution layer, the tenth convolution result is obtained by convolving the third upsampling result once using one convolution layer.

[0314] 20324. Perform an upsampling operation on the first repair result to obtain a fourth upsampling result;

[0315] The upsampling operation is implemented by using a fourth upsampling module group. The fourth upsampling module group includes at least one upsampling module. If the fourth upsampling module group includes one upsampling module, the first upsampling result is obtained by performing an upsampling operation on the first repair result using one upsampling module.

[0316] 20325. Add and fuse the fourth upsampling result and the tenth convolution result to obtain a second restoration result.

[0317] This addition and fusion operation is implemented using the fourth summation layer St4.

[0318] In the above 204, the second image is displayed differently in different application scenarios. For example, based on the description of the first image obtained in step 201, the second image can be displayed as follows but is not limited to: displaying the second image as an interface background image in the electronic device, or directly displaying it normally.

[0319] Furthermore, the method provided in this embodiment may also include the following steps:

[0320] 205. Input the first image and the second image into the discrimination network of the processing model to discriminate authenticity, and obtain discrimination result information;

[0321] 206. Determine loss information according to the discrimination result information, the first image, and the second image;

[0322] 207. Optimize model parameters of the image processing model based on the loss information.

[0323] For detailed descriptions of the specific contents contained in the above-mentioned discrimination result information, the determination of loss information, and the specific implementation of the optimized image processing model, etc., please refer to the relevant contents in the aforementioned other embodiments, and no further details will be given here.

[0324] Fig.14 This is a flow chart of another image processing method provided in an embodiment of the present application. The application scenario of this method is the aforementioned application scenario three. Fig.14 As shown, the image processing method comprises the following steps:

[0325] 301. In response to an interface background change event, determine a selected first image;

[0326] 302. Perform dimension conversion on the first image to obtain a converted first image;

[0327] 303. Determine target restoration area information corresponding to the first image after conversion;

[0328] 304. Input the target repair area information and the converted first image into an image processing module deployed on the electronic device for image repair processing to obtain a second image.

[0329] 305. Display the second image as an interface background image in the electronic device;

[0330] Among them, some convolutional layers in the image processing model are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of the multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

[0331] For the specific implementation description of the above steps, please refer to the relevant content in other embodiments. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other steps. For other steps that may be included and the specific implementation description, please refer to the relevant content in other embodiments, which will not be repeated here.

[0332] To facilitate understanding of the image processing method provided above, Fig.15 Give an example of a specific application scenario.

[0333] like Fig.15 , the user has a smart phone, and the 3D wallpaper provided by the smart phone has a dimension conversion module that can convert the dimension of the image. The current interface wallpaper of the display screen of the smart phone is an image with flowers (as shown in the smart phone on the right). The user triggers the wallpaper change operation, and the selected wallpaper is a 2D image pict1 with animals on the smart phone. In response to the wallpaper change operation, the smart phone inputs the 2D image pict1 into the dimension conversion module. The dimension conversion module performs a dimensional conversion on the 2D image pict1 and outputs a 3D image pict2 generated by the conversion. There are stretching artifacts (such as black areas) in the 3D image pict2 that need to be repaired. At this time, the image processing model deployed on the smart phone will be called, and the 3D image pict2 will be input into the image processing model. For example, the recognition network in the image processing model (not shown in the figure) can recognize the 3D image pict2 and output the corresponding binary mask image m2. The white area in the binary mask image m2 represents the target area to be repaired in the 3D image pict2. Further, the 3D image pict2 and the binary mask image m2 are input into the repair combination network in the image processing model. The repair combination network will be executed to obtain the repaired image pict2' of the 3D image pict2, so that the repaired image pict2' is displayed on the display interface of the smart phone as the interface wallpaper (i.e., the background picture). At this point, the interface wallpaper of the smart phone is changed from the original 2D image pict1 with animals to the repaired image pict2'. The changed interface wallpaper is displayed as shown in the smart phone on the left.

[0334] In summary, the technical method provided in the embodiments of the present application has the following beneficial effects:

[0335] 1. The existing image restoration model has a large number of parameters and a slow inference speed. This application uses depthwise separable convolution to implement all gated convolutions in the image processing model generation network (the first restoration network). Because depthwise separable convolution greatly reduces the number of parameters and computational complexity compared to conventional convolutions, this can effectively reduce model parameters and improve inference speed while ensuring the image restoration effect, which is more friendly to deployment on the mobile terminal side.

[0336] 2. The contextual attention module commonly used in existing models contains dynamic convolution, which is difficult to deploy on the mobile terminal side. This application uses a self-attention module to replace it. While also achieving the purpose of obtaining global information, it can also improve the image restoration effect (such as image filling quality effect).

[0337] 3. Existing models are limited by computing power and memory and can only output low-resolution filling results, while user input is usually high-resolution images. This application combines the generation network (first restoration network) and the super-resolution network (second restoration network) into one model, which can sample the low-resolution restoration results on the super-resolution network and directly output the filling results with a clarity close to that of the high-resolution image, alleviating the problem of the original model's blurred restoration effect on high-resolution images.

[0338] The present application also provides a training method for an image processing model, wherein the image processing model includes a repair combination network and a discriminator network. The method is also applied to electronic devices. The training method includes the following steps:

[0339] 401. Using the discriminant network to discriminate the authenticity of a first sample image and a second sample image, and obtain discrimination result information; the second sample image is generated by performing image restoration processing on the first sample image by the restoration combination network;

[0340] 402. Determine a first loss function according to a difference between the first sample image and the second sample image;

[0341] 403. Determine a second loss function according to an output discrimination value of the discrimination network for the second image included in the discrimination result information;

[0342] 404. Determine a third loss function according to the output discrimination value of the discrimination network for the first image and the output discrimination value of the second image included in the discrimination result information;

[0343] 405. Determine a fourth loss function according to intermediate judgment values ​​of the first image and the second image respectively of each intermediate network layer in the discrimination network included in the discrimination result information;

[0344] 406. Use a pre-trained perceptual loss network to perform content and style difference analysis on the first sample image and the second sample image to obtain a fifth loss function and a sixth loss function;

[0345] 407. Determine a seventh loss function according to the image gradient information of the second sample image;

[0346] 408. Optimize network parameters of the repair combination network and the discriminator based on the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, the sixth loss function and the seventh loss function.

[0347] The first sample image is a real image (an original image without any processing), and the second sample image is a generated image generated by repairing the first image.

[0348] For detailed description of the above steps 401 to 408, please refer to the relevant content in other embodiments. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other steps. For other steps that may be included and specific implementation descriptions, please refer to the relevant content in other embodiments, which will not be repeated here.

[0349] The present application also provides an electronic device. For the specific form of the electronic device, please refer to the relevant content in the above other embodiments. For example, Fig.16A and Fig. 16B 1 shows a schematic diagram of the structure of the electronic device 100. Fig.16A As shown, the electronic device 100 includes: a memory 121 and a processor 110; wherein,

[0350] The memory 121 is used to store programs; wherein the program includes the program code of the image processing model provided in the present application; the processor 110 is coupled to the memory, and is used to execute the program stored in the memory, so as to use the image processing model to implement the steps in the various image processing methods provided in the present application.

[0351] The memory 121 (which is a memory storage) can be used to store computer executable program codes, and the executable program codes include instructions. The memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 121, and / or instructions stored in a memory provided in the processor.

[0352] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0353] The processor can generate operation control signals according to the instruction operation code and timing signals to complete the control of instruction fetching and execution.

[0354] In addition, the processor 110 may be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0355] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0356] Furthermore, in addition to the functional components described above, Fig.16A and Fig. 16B As shown, the electronic device 100 also includes: an external memory interface 120, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0357] Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0358] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0359] Furthermore, it can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0360] The electronic device 100 implements the display function through a GPU (Graphics Processing Unit), a display screen 194, and an application processor. The GPU connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0361] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0362] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0363] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0364] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0365] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0366] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0367] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, can implement one or more steps in any of the above-mentioned visual media search methods.

[0368] The computer-readable storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0369] Another embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed by a computer, one or more steps in any of the above methods can be implemented.

[0370] Among them, the electronic device, computer-readable storage medium, and computer program product provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0371] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0372] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0373] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0374] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0375] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applicable to electronic devices, the method comprises: Determine target repair area information corresponding to the first image; Obtaining an image processing model deployed on an electronic device; Inputting the target repair area information and the first image into the image processing model to perform image repair processing to obtain a second image; displaying the second image in the electronic device; Among them, some convolutional layers in the image processing model are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

2. The method according to claim 1, characterized in that The image processing model includes a first restoration network and a second restoration network; as well as, Inputting the target repair area information and the first image into the image processing model to perform image repair processing to obtain a second image, including: Inputting the target restoration area information and the first image into the first restoration network, and outputting a first restoration result; The first restoration result is input into the second restoration network, and a second restoration result containing the second image is output.

3. The method according to claim 2, characterized in that The first repair network includes a first sub-network and a second sub-network having a self-attention module, wherein the second sub-network is connected to an output end of the first sub-network; The self-attention module includes a spatial attention combination layer and a channel attention combination layer; an intermediate result generated by the second sub-network in the process of processing the output of the first sub-network is the input of the self-attention module; The spatial attention combination layer performs feature extraction processing on the intermediate result and outputs spatial attention features; The channel attention combination layer performs feature extraction processing on the intermediate result and outputs channel attention features; The attention feature information output by the self-attention module is obtained by adding and fusing the spatial attention feature and the channel attention feature.

4. The method according to claim 3, characterized in that The spatial attention combination layer performs feature extraction processing on the intermediate result and outputs spatial attention features, including: Performing a convolution operation on the intermediate result to obtain first-category features, second-category features, and third-category features; Obtaining a spatial attention map according to the first type of features and the second type of features; The spatial attention feature is obtained according to the spatial attention map, the third type of feature and the intermediate result.

5. The method according to claim 4, characterized in that According to the first type of features and the second type of features, a spatial attention map is obtained, including: Transpose and transform the first type of feature and multiply it with the second type of feature to obtain a first multiplication result; Obtaining the spatial attention map according to the first multiplication result; And, according to the spatial attention map, the third type of features and the intermediate results, the spatial attention features are obtained, including: Multiplying the spatial attention map by the third type of feature to obtain a second multiplication result; Add the second multiplication result to the intermediate result to obtain the spatial attention feature.

6. The method according to claim 3, characterized in that The channel attention combination layer performs feature extraction processing on the intermediate result and outputs channel attention features, including: Transpose and transform the intermediate result to obtain a transformed intermediate result; Multiplying the deformed intermediate result by the intermediate result to obtain a channel attention map; Multiplying the channel attention map by the intermediate result to obtain a third multiplication result; The third multiplication result is added to the intermediate result to obtain the channel attention feature.

7. The method according to any one of claims 3 to 6, characterized in that Inputting the target repair area information and the first image into the first repair network, and outputting a first repair result, including: Using the first sub-network to process the target repair area and the first image to obtain a processing result; Inputting the processing results into the first encoder and the second encoder in the second sub-network for encoding respectively, and performing a splicing operation on the first encoding result output by the first encoder and the second encoding result output by the second encoder to obtain a splicing result; wherein the first encoder includes the self-attention module; The splicing result is input into a decoder in the second sub-network for decoding to obtain the first repair result.

8. The method according to claim 7, characterized in that Inputting the processing result into the first encoder for encoding to obtain the first encoding result includes: Performing a convolution operation on the processing result to obtain a first convolution result; Performing a downsampling operation on the first convolution result to obtain a first downsampling result; Performing a convolution operation on the first downsampling result to obtain a second convolution result; the second convolution result is the intermediate result input to the self-attention module; Inputting the second convolution result into the self-attention module, and outputting the attention feature information; A convolution operation is performed on the attention feature information to obtain the first encoding result.

9. The method according to claim 7, characterized in that: Inputting the processing result into the second encoder for encoding to obtain the second encoding result includes: Performing a convolution operation on the processing result to obtain a third convolution result; Performing a downsampling operation on the third convolution result to obtain a second downsampling result; A convolution operation is performed on the second down-sampling result to obtain the second encoding result.

10. The method according to claim 7, characterized in that Inputting the splicing result into a decoder in the second sub-network for decoding to obtain the first repair result includes: Performing a convolution operation on the concatenated result to obtain a fourth convolution result; Performing an upsampling operation on the fourth convolution result to obtain a first upsampling result; Performing a convolution operation on the first upsampling result to obtain a fifth convolution result; The fifth convolution result is subjected to a limit processing to obtain the first repair result whose value is limited to a first interval.

11. The method according to claim 7, characterized in that The target repair area and the first image are processed by using the first sub-network to obtain a processing result, including: Performing a convolution operation on the target repair area and the first image to obtain a sixth convolution result; Performing a downsampling operation on the sixth convolution result to obtain a third downsampling result; Performing a convolution operation on the third down-sampling result to obtain a seventh convolution result; Performing an upsampling operation on the seventh convolution result to obtain a second upsampling result; Performing a convolution operation on the second up-sampling result to obtain an eighth convolution result; The eighth convolution result is subjected to a limit processing to obtain the processing result whose value is limited to within a second interval.

12. The method according to any one of claims 2 to 6, characterized in that Some convolutional layers in the first repair network are gated convolutions, and the gated convolutions include two depth-wise separable convolutional layers arranged in parallel.

13. The method according to any one of claims 2 to 6, characterized in that Inputting the first restoration result into the second restoration network, and outputting a second restoration result containing the second image, comprises: Performing a convolution operation on the first repair result to obtain a ninth convolution result; Performing an upsampling operation on the ninth convolution result to obtain a third upsampling result; Performing a convolution operation on the third upsampling result to obtain a tenth convolution result; Performing an upsampling operation on the first repair result to obtain a fourth upsampling result; The fourth upsampling result and the tenth convolution result are added and fused to obtain the second restoration result.

14. An image processing method, characterized in that: Applicable to electronic equipment, the method comprises: In response to an interface background change event, determining a selected first image; Performing dimension conversion on the first image to obtain a converted first image; Determining target restoration area information corresponding to the first image after conversion; Inputting the target repair area information and the converted first image into an image processing module deployed on the electronic device for image repair processing to obtain a second image; Displaying the second image as an interface background picture in the electronic device; Among them, some convolutional layers in the image processing model are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of the multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

15. An image processing model, characterized in that: include: A first restoration network is used to perform image restoration processing on the first image according to target restoration area information corresponding to the first image, and output a first restoration result; A second restoration network, connected to an output end of the first restoration network, configured to process the first restoration result and output a second restoration result containing a second image; Among them, some convolutional layers in the first repair network are gated convolutional layers, and the gated convolutional layers include multiple depth-separable convolutional layers arranged in parallel and a first product layer, and the first product layer is used to multiply the outputs of the multiple depth-separable convolutional layers as the output result of the gated convolutional layer.

16. The image processing model according to claim 15, characterized in that: The first repair network includes a first sub-network and a second sub-network having a self-attention module, wherein the second sub-network is connected to an output end of the first sub-network; The convolutional layers in the first repair network that do not belong to the self-attention module are all gated convolutional layers; The self-attention module includes a spatial attention combination layer, a channel attention combination layer and a first summation layer; an intermediate result generated by the second sub-network in the process of processing the output of the first sub-network is the input of the self-attention module; The spatial attention combination layer is used to perform feature extraction processing on the intermediate result and output spatial attention features; The channel attention combination layer is used to perform feature extraction processing on the intermediate result and output channel attention features; The first summation layer is used to add and fuse the spatial attention feature and the channel attention feature to output attention feature information.

17. The image processing model according to claim 16, characterized in that The spatial attention combination layer includes: The first convolutional layer is used to extract features from the intermediate results and output first-category features; A second convolutional layer is used to extract features from the intermediate results and output second-category features; A third convolutional layer is used to extract features from the intermediate results and output third-category features; A first transposition layer, connected to an output end of the first convolutional layer, configured to perform transposition deformation on the first type of features and output the first type of features after transposition deformation; a second product layer, connected to the output ends of the first transposed layer and the second convolutional layer, and configured to multiply the first type of features after transposition deformation by the second type of features, and output a first multiplication result; A third product layer, connected to output ends of the second product layer and the third convolution layer, and configured to multiply the spatial attention map obtained according to the first multiplication result by the third type of feature to obtain a second multiplication result; A second summation layer is connected to the output end of the third product layer, and is used to add the second multiplication result and the intermediate result to output the spatial attention feature.

18. The image processing model according to claim 16, characterized in that The channel attention combination layer includes: A second transposition layer, used to perform transposition deformation on the intermediate result and output the deformed intermediate result; a fourth product layer, connected to an output end of the second transposed layer, and configured to multiply the deformed intermediate result by the intermediate result to output a channel attention map; a fifth product layer, connected to an output end of the fourth product layer, configured to multiply the channel attention map by the intermediate result and output a third multiplication result; The third summation layer and the output end of the fifth product layer are used to add the third multiplication result and the intermediate result to output the channel attention feature.

19. The image processing model according to any one of claims 16 to 18, characterized in that: The second sub-network includes: A first encoder, used to encode the processing result output by the first sub-network, and output a first encoding result; A second encoder, used to encode the processing result output by the first sub-network, and output a second encoding result; A splicing layer, connected to the output ends of the first encoder and the second encoder, configured to perform a splicing operation on the first encoding result and the second encoding result, and output a splicing result; A decoder is connected to the output end of the splicing layer, and is used to decode the splicing result and output the first repair result.

20. The image processing model according to claim 19, characterized in that The first encoder comprises: a plurality of fourth convolutional layers, a first downsampling module group and the self-attention module; wherein, The first down-sampling module group is connected between the first part of the fourth convolutional layers and the second part of the fourth convolutional layers in the plurality of fourth convolutional layers, and the self-attention module is connected between the remaining third part of the fourth convolutional layers and the second part of the fourth convolutional layers; the output end of the third part of the fourth convolutional layers is connected to the input end of the splicing layer; The first down-sampling module group includes a plurality of down-sampling modules that perform down-sampling operations in sequence.

21. The image processing model according to claim 19, characterized in that The second encoder comprises: a plurality of fifth convolutional layers and a second downsampling module group; wherein, The second down-sampling module group is connected between the first part of the fifth convolutional layers and the remaining second part of the fifth convolutional layers in the plurality of fifth convolutional layers; the output end of the second part of the fifth convolutional layers is connected to the input end of the splicing layer; The second down-sampling module group includes a plurality of down-sampling modules that perform down-sampling operations in sequence.

22. The image processing model according to claim 19, characterized in that The decoder comprises: a plurality of sixth convolutional layers, a first upsampling module group and a first limit layer; wherein, A first part of the sixth convolutional layers among the plurality of sixth convolutional layers are connected between the concatenated layer and the first up-sampling module group, and a remaining second part of the sixth convolutional layers are connected between the first up-sampling module group and the first restriction layer; The first limiting layer is used to limit the output of the sixth convolutional layer of the second part to a first interval range; The first up-sampling module group includes a plurality of up-sampling modules that perform up-sampling operations in sequence.

23. The image processing model according to any one of claims 16 to 18, characterized in that The first sub-network includes: a plurality of seventh convolutional layers, a third down-sampling module group, a second up-sampling module group and a second limit layer; wherein, The third down-sampling module group is connected between the first part of the seventh convolutional layers and the second part of the seventh convolutional layers in the plurality of seventh convolutional layers, the second up-sampling module group is connected between the remaining third part of the seventh convolutional layers and the second part of the seventh convolutional layers, and the output end of the third part of the seventh convolutional layers is connected to the second limit layer; The second limit layer is used to limit the output of the seventh convolutional layer of the third part to a second interval range; The third downsampling module group includes a plurality of downsampling modules that perform downsampling operations in sequence; The second up-sampling module group includes a plurality of up-sampling modules that perform up-sampling operations in sequence.

24. The image processing model according to any one of claims 15 to 18, characterized in that: The gated convolution includes a first depth-separable convolution layer, a second depth-separable convolution layer, a third limit layer, an activation layer and the first product layer; An output end of the first depthwise separable convolutional layer is connected to the third threshold layer, and an output end of the second depthwise separable convolutional layer is connected to the activation layer; The output end of the third limiting layer and the output end of the activation layer are both connected to the first product layer; The third limiting layer is used to limit the first depth convolution feature output by the first depth separable convolution layer, and output the first depth convolution feature after limiting; the value of the first depth convolution feature after limiting is within a third interval; The activation layer is used to activate the second depth convolution feature output by the second depth separable convolution layer, and output the activated second depth convolution feature; The first product layer is used to multiply the first deep convolution feature after limiting and the second deep convolution feature after activation, and output a fourth multiplication result; the fourth multiplication result is the output result of the gated convolution.

25. The image processing model according to any one of claims 15 to 18, characterized in that The second restoration network includes: a plurality of eighth convolutional layers, a third upsampling module group, a fourth upsampling module group and a fourth summation layer; The third up-sampling module group is connected between a first part of the eighth convolutional layers of the plurality of eighth convolutional layers and a remaining second part of the eighth convolutional layers; The output end of the eighth convolutional layer of the second part and the output end of the fourth up-sampling module are both connected to the fourth summation layer; the input end of the fourth up-sampling module is connected to the input end of the eighth convolutional layer of the first part; The third up-sampling module group and the fourth up-sampling module group include at least one up-sampling module.

26. An image processing model, characterized in that include: A repair combination network, wherein the repair combination network includes a self-attention module; wherein, The inpainting combination network is used to perform image inpainting processing on the first image according to the target inpainting region information corresponding to the first image; wherein an intermediate result generated during the processing is the input of the self-attention module; The self-attention module includes a spatial attention combination layer, a channel attention combination layer and a first summation layer; The spatial attention combination layer is used to perform feature extraction processing on the intermediate result and output spatial attention features; The channel attention combination layer is used to perform feature extraction processing on the intermediate result and output channel attention features; The first summation layer is used to add and fuse the spatial attention feature and the channel attention feature to output attention feature information; Among them, the spatial attention combination layer includes three convolutional layers, a second multiplication layer, a third multiplication layer and a second summation layer arranged in parallel; the inputs of the three convolutional layers are all the intermediate results, and the output ends of the first convolutional layer and the second convolutional layer in the three convolutional layers are both connected to the input end of the second multiplication layer, the output end of the third convolutional layer and the output end of the second multiplication layer are both connected to the input end of the third convolutional layer; the input ends of each of the three convolutional layers and the output end of the third convolutional layer are all connected to the second summation layer.

27. The image processing model according to claim 26, characterized in that The spatial attention combination layer also includes a first transposition layer, which is connected between the first convolutional layer and the second multiplication layer, and is used to transpose the convolution result output by the first convolutional layer.

28. An electronic device, characterized in that: include: Memory and processor; wherein, The memory is used to store a program; wherein the program includes the program code of the image processing model described in any one of claims 15 to 25, or the program code of the image processing model described in claim 26 or 27; The processor is coupled to the memory, and is used to execute the program stored in the memory, so as to use the image processing model to implement the steps in the image processing method described in any one of claims 1 to 13, or to implement the steps in the image processing method described in claim 14.

Images