Image processing methods, image processing models, and electronic devices

Abstract

This application provides an image processing method, an image processing model, and an electronic device. The electronic device deploys an image processing model. This solution uses the image processing model to perform image restoration processing on the first image based on the target restoration region information corresponding to the first image, thereby obtaining a second image and displaying the second image on the electronic device. Some convolutional layers in the aforementioned image processing model are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer. Because the number of parameters and computational cost of depthwise separable convolutional layers is significantly reduced, using depthwise separable convolution to construct the gated convolution in the model can both ensure the image restoration effect of the model and effectively reduce model parameters and improve inference speed, making the model very user-friendly for deployment on mobile terminals such as smartphones.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, an image processing model, and an electronic device. Background Technology

[0002] Image inpainting primarily aims to repair damaged or missing parts of an image, typically achieved using appropriate image inpainting models. However, existing image inpainting models suffer from limitations such as only outputting low-resolution image restoration results, having a large number of parameters, and employing algorithms that are difficult to deploy on mobile devices like smartphones. Therefore, there is an urgent need for an image processing technology solution that can both guarantee image restoration quality and be easily implemented on mobile devices. Summary of the Invention

[0003] This application provides an image processing method, an image processing model, a training method for the image processing model, and an electronic device, aiming to achieve both implementation in mobile terminals and other electronic devices and improved image restoration effects. Therefore,

[0004] Firstly, this application provides an image processing method suitable for electronic devices. The method includes:

[0005] Determine the target restoration area information corresponding to the first image;

[0006] Acquire the image processing model deployed on the electronic device;

[0007] The target restoration area information and the first image are input into the image processing model for image restoration processing to obtain the second image;

[0008] The second image is displayed in the electronic device;

[0009] In the image processing model, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer.

[0010] Secondly, this application provides another image processing method suitable for electronic devices. The method includes:

[0011] In response to a background change event, determine the first image selected;

[0012] Perform dimensional transformation on the first image to obtain the transformed first image;

[0013] Determine the target restoration region information corresponding to the converted first image;

[0014] The target restoration area information and the converted first image are input into the image processing module deployed on the electronic device for image restoration processing to obtain the second image;

[0015] The second image is displayed as the interface background image in the electronic device;

[0016] In the image processing model, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer.

[0017] Thirdly, this application provides an image processing model, which includes:

[0018] The first repair network is used to perform image repair processing on the first image based on the target repair region information corresponding to the first image, and output the first repair result;

[0019] The second repair network is connected to the output of the first repair network and is used to process the first repair result and output a second repair result containing the second image.

[0020] In the first repair network, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer.

[0021] Fourthly, this application provides another image processing model, which includes: a repair ensemble network, wherein the repair ensemble network includes a self-attention module; wherein,

[0022] The repair combination network is used to perform image repair processing on the first image based on the target repair region information corresponding to the first image; wherein, an intermediate result generated during the processing is the input of the self-attention module;

[0023] The self-attention module includes a spatial attention combination layer, a channel attention combination layer, and a first summation layer;

[0024] The spatial attention combination layer is used to perform feature extraction processing on the intermediate results and output spatial attention features;

[0025] The channel attention combination layer is used to perform feature extraction processing on the intermediate results and output channel attention features;

[0026] The first summing layer is used to add and fuse the spatial attention features and the channel attention features to output attention feature information;

[0027] The spatial attention combination layer includes three convolutional layers, a second multiplication layer, a third multiplication layer, and a second summing layer arranged in parallel. The inputs of the three convolutional layers are the intermediate results, and the outputs of the first and second convolutional layers are connected to the input of the second multiplication layer, and the outputs of the third convolutional layer and the second multiplication layer are connected to the input of the third convolutional layer. The inputs of the three convolutional layers and the output of the third convolutional layer are all connected to the second summing layer.

[0028] Fifthly, this application provides a training method for an image processing model, which includes a repair ensemble network and a discriminator network. The training method includes:

[0029] The discriminant network is used to distinguish between the authenticity of the first sample image and the second sample image to obtain the discrimination result information; wherein, the second sample image is generated by the inpainting and combination network performing image inpainting processing on the first sample image;

[0030] The first loss function is determined based on the difference between the first sample image and the second sample image;

[0031] The second loss function is determined based on the output discrimination value of the discrimination network for the second image contained in the discrimination result information;

[0032] Based on the discrimination result information containing the discrimination network's output discrimination value for the first image and the output discrimination value for the second image, a third loss function is determined;

[0033] Based on the intermediate judgment values ​​of each intermediate network layer in the discrimination network for the first image and the second image contained in the discrimination result information, the fourth loss function is determined.

[0034] A pre-trained perceptual loss network is used to perform content and style difference analysis on the first sample image and the second sample image to obtain a fifth loss function and a sixth loss function.

[0035] The seventh loss function is determined based on the image gradient information of the second sample image;

[0036] Based on the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, the sixth loss function, and the seventh loss function, the network parameters of the repair combination network and the discriminator are optimized.

[0037] Sixthly, this application provides an electronic device. The device includes a memory and a processor; wherein the memory is used to store a program; wherein the program includes program code of an image processing model provided in other aspects of this application; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the methods provided in this application using the image processing model.

[0038] In a seventh aspect, this application provides a chip system applied to the electronic device provided in this application. The chip system includes a processor, which, when executing instructions, implements the steps of the methods provided in this application.

[0039] Eighthly, this application provides a computer-readable storage medium. The computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the steps in the methods provided in this application described above.

[0040] Ninthly, this application provides a computer program product. The computer program product includes a computer program that, when executed by a processor, can implement the steps of the methods provided in this application described above.

[0041] The technical solutions provided in the embodiments of this application are as follows:

[0042] Some convolutional layers in the image processing model are gated convolutions, and these gated convolutions are constructed using multiple depthwise separable convolutions. Since depthwise separable convolutions significantly reduce the number of parameters and computational cost compared to regular convolutions, using depthwise separable convolutions to construct the gated convolutions in the model can ensure the image restoration effect of the model while effectively reducing model parameters and improving inference speed. This makes the model very user-friendly for deployment on electronic devices, especially mobile terminals such as smartphones.

[0043] Furthermore, the self-attention module in the image processing model includes a spatial attention combination layer and a channel attention combination layer. The spatial attention combination layer is constructed from three convolutional layers, a second multiplication layer, a third multiplication layer, and a second summing layer arranged in parallel. The inputs to the three convolutional layers are identical. The outputs of the first and second convolutional layers are connected to the input of the second multiplication layer, and the outputs of the third and second convolutional layers are connected to the input of the third convolutional layer. The inputs of each of the three convolutional layers and the output of the third convolutional layer are all connected to the second summing layer. The spatial attention features output from the spatial attention combination layer and the channel attention features output from the channel attention combination layer are fused through the second summing layer to obtain the global features mined from the input data by the self-attention module. The convolutional kernel parameters in the above self-attention module are fixed and do not dynamically change with different input data. This further improves the model's usability on electronic devices (such as mobile terminals) while still achieving the same effect of acquiring global feature information and improving image restoration quality.

[0044] Furthermore, the image processing model contains two-level restoration networks, namely the first restoration network and the second restoration network. The second restoration network can be used to improve the restoration quality of the restoration result output by the first restoration network, so that the final output restored image (the second image) is a high-resolution and clear image. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1A A schematic diagram illustrating an image restoration application scenario for restoring the damaged main structure of an object in an image, provided as an exemplary embodiment of this application;

[0047] Figure 1B A schematic diagram illustrating an image restoration application scenario for image erasure provided in an exemplary embodiment of this application;

[0048] Figure 2A A schematic diagram illustrating an image restoration application scenario for repairing stretching artifacts in an image during dimensional transformation, as provided in an exemplary embodiment of this application.

[0049] Figures 2B to 2D A schematic diagram illustrating the principle of setting a background image for an electronic device as provided in an exemplary embodiment of this application;

[0050] Figure 3A schematic diagram of the structure of an existing image restoration model provided as an exemplary embodiment of this application;

[0051] Figure 4A A schematic diagram of the structure of another existing image restoration model provided as an exemplary embodiment of this application;

[0052] Figure 4B for Figure 4A The diagram shows the structure of the context attention module in the image inpainting model.

[0053] Figure 5A and Figure 5B A schematic diagram of the structure of the image processing model provided in this application;

[0054] Figure 6A A schematic diagram of the structure of the first sub-network in the image processing model provided in this application;

[0055] Figure 6B A schematic diagram of the downsampling module in the image processing model provided in this application;

[0056] Figure 6C A schematic diagram of the upsampling module in the image processing model provided in this application;

[0057] Figure 6D This is a specific structural example of the first sub-network in the image processing model provided in this application;

[0058] Figure 7A and Figure 7B A schematic diagram of the structure of the second sub-network in the image processing model provided in this application;

[0059] Figure 8A and Figure 8B for Figure 7A and Figure 7B The diagram shows the structure of the self-attention module included in the second sub-network;

[0060] Figure 9A and Figure 9B A schematic diagram of the structure of the gated convolutional layer in the image processing model provided in this application;

[0061] Figure 10A and Figure 10B A schematic diagram of the structure of the second repair network in the image processing model provided in this application;

[0062] Figure 11 This is a schematic diagram of the discriminant network structure in the image processing model provided in this application;

[0063] Figure 12 A schematic diagram illustrating the principle for determining the loss information provided in this application;

[0064] Figure 13 and Figure 14 A flowchart illustrating the image processing method provided in this application;

[0065] Figure 15 A schematic diagram illustrating a specific application scenario of the image processing method provided in this application;

[0066] Figure 16A and Figure 16B A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0067] Image inpainting is an important problem in the field of computer vision. Its main function is to fill in damaged or missing information in images using relevant image inpainting algorithms. Specifically, image inpainting algorithms can be used to build corresponding image inpainting models to restore the main structure of damaged objects in images. They can also be used to fill in the background of objects, remove unwanted parts from images, and so on. Below are some commonly used image inpainting application scenarios:

[0068] Application Scenario 1: Restoring the main structure of a damaged object in an image

[0069] like Figure 1A As shown in (a), the structure of the main objects such as the tiger and scenery in the image is damaged. It can be restored after image restoration. The restored image is as follows. Figure 1A As shown in (b) of the diagram.

[0070] Application Scenario 2: Image Erasure

[0071] like Figure 1B As shown in (a), the user specifies the area in the image that they want to erase, such as the area where object 01 (including people, objects, etc.) is located. The image restoration model will use background information to fill object 01 in the image to achieve the purpose of erasing object 01.

[0072] In addition to the two application scenarios mentioned above, there are other application scenarios, such as the third application scenario described below:

[0073] Application Scenario 3: Repairing stretching artifacts in images during dimensionality transformation

[0074] Electronic devices such as smartphones and other mobile devices generally have a wallpaper function, allowing users to set a background image (wallpaper) for their device screen according to their own preferences. This wallpaper function can be accessed through corresponding applications, such as... Figure 2B The theme application shown in the image, or Figure 2C The settings application shown in the image, or Figure 2DThe image gallery application shown here includes a conversion module for 3D wallpapers. This module allows users to convert their selected 2D images into 3D images for display as background pictures on their devices. However, assuming the user selects an image like... Figure 2A In the 2D image shown in (a), during the process of converting the 2D image to a 3D image and rendering it in the new viewpoint dimension, the background part that was originally occluded by the foreground will be exposed, resulting in artifacts in the converted image, such as... Figure 2A The black area 02 in the 3D image shown in (b) is a stretching artifact. Therefore, the image inpainting model needs to fill and repair the black area in the 3D image using the background information in the image to eliminate the artifact.

[0075] The above-mentioned application scenario three is the main application scenario of the various embodiments provided in the following text application.

[0076] Currently, image restoration models are typically constructed using two main categories of algorithms: traditional image restoration algorithms (non-machine learning algorithms) and machine learning algorithms. Traditional image restoration algorithms, based on techniques such as interpolation and texture synthesis, often struggle to generate realistic, detailed images. Machine learning algorithms (such as deep learning) are intelligent algorithms that have emerged in recent years alongside the development of artificial intelligence. They have been widely applied in the field of image restoration and have achieved remarkable results, but they also have certain problems. For example, current machine learning-based image restoration models suffer from limitations such as only being able to output low-resolution images, having large parameters, and being unsuitable for deployment on mobile devices. The following two examples illustrate the problems of existing machine learning-based image restoration models.

[0077] Example 1: There is an existing image inpainting model based on Fourier convolution (Large Mask Inpainting, LaMa), whose network structure is as follows: Figure 3 As shown, the four-channel image x' obtained by overlaying the original image x with the mask image m is used as the input to the feedforward inpainting network. After the four-channel image x' is input into the feedforward inpainting network, it is first downscaled to a low resolution, then goes through several Fast Fourier Convolutions (FCCs), and finally the output is upscaled to generate a high-resolution image.

[0078] The FCC (Feature Extraction Control) module is primarily used for feature extraction. Each FCC consists of two branches: a local branch and a global branch. The local branch includes convolution operations to extract local information using a smaller receptive field. The global branch includes Fourier transform operations to extract global information. The two branches are then fused together using a concatenation operation, achieving the extraction and fusion of local and global information. Specifically, for example... Figure 3 When data is input to the FCC, some channels undergo local branch processing to obtain local features, while other channels undergo global branch processing to obtain global features. The local and global features are then fused for output, ensuring consistency between the output and input channels. The local branches include a first and second convolutional module, while the global branches include a third convolutional module and a Fourier spectrum transform module. The Fourier spectrum transform module includes a first combined layer (Con-BN-ReLU, containing convolution operation Con, forward programming operation BN, and activation operation ReLU), a Fourier transform layer (Real FFT2d, referring to the real-part two-dimensional fast Fourier transform), a second combined layer (Con-BN-ReLU), an inverse Fourier transform (Inv Real FFT2d), and a convolutional layer (Conv 1x1).

[0079] Using the above Figure 3 The image restoration model shown has good restoration effect and can handle high-resolution images, but it has a large number of network parameters and Fourier transform convolution (FFC) is difficult to deploy on mobile terminals.

[0080] Example 2: There is also an image inpainting model based on gated convolution, whose network structure is as follows: Figure 4A As shown, the diagram includes a generator network and a discriminator network, consisting of a coarse restoration network and a fine restoration network. The input information, such as the color channel (RGB channel), mask channel, and sketch channel, is first processed by the coarse restoration network to output a coarse restored image. This coarse restored image is then processed by the fine restoration network to become a fine restored image, which is then passed to the discriminator network for real / fake detection. The convolutions in both the coarse and fine restoration networks are gated convolutions. Specifically, the fine restoration network includes an encoding module and a decoding module. The encoding module includes two encoding branches: an upper encoding branch and a lower encoding branch. The upper encoding branch contains a contextual attention (CA) module.

[0081] Figure 4AAlthough the image restoration model shown is relatively lightweight, its context attention module contains dynamic convolution, making it difficult to deploy on mobile devices. Furthermore, the model can only output low-resolution restoration results, and upsampling to high resolution results in a blurry image.

[0082] To address the aforementioned problems with existing image restoration models, the basic design concept of the image restoration model in this application is as follows:

[0083] 1) All gated convolutions in the generator network used for initial image restoration are implemented using depthwise separable convolutions, which can effectively reduce the number of convolution parameters and the amount of computation, and is also more friendly to deployment on mobile terminals.

[0084] 2) Replace the context attention module with a self-attention module to achieve the same effect of obtaining global information and improving fill quality;

[0085] 3) By combining the generator network and the super-resolution network into one model, the low-resolution restoration results are sampled by the super-resolution network, and the filling results with a clarity close to that of the high-resolution image can be directly output. This can solve the problem that the existing model has a blurry restoration effect on the input high-resolution image due to the limitation of computing power and memory.

[0086] Based on the above design principles, this application provides an image processing method, an image processing model, and an electronic device through multiple embodiments. Note: It is understood that the image processing model described herein is also an image restoration model, only expressed differently in different descriptive contexts.

[0087] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0088] In the embodiments of this application, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first signal generation circuit and the second signal generation circuit are only used to distinguish different signal generation circuits and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0089] It should be noted that the words "exemplary" or "for example" in this application are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. In addition, "at least one" in this application means one or more, and "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural, etc. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plural item. For example, at least one of a, b, and c can be represented as: a, b, c, a, b, and c, a and b, a and c, b and c.

[0090] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0091] To facilitate understanding of the solution provided in this application, the image processing model will be described in detail first. Before proceeding with the detailed description, some proprietary terms involved will be introduced and explained.

[0092] Downsampling is an operation that reduces the dimensionality and size of an image. Specifically, it involves reducing the number of sampling points from a high-resolution image according to certain rules, thereby lowering the image resolution. More specifically, downsampling can achieve dimensionality reduction by filtering out pixel features that are less important or redundant, thus retaining key features.

[0093] Upsampling is an operation that increases the number of sampling points in an image by interpolating the image based on existing pixel features using a specific algorithm, thereby improving the image's resolution. Common upsampling algorithms include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, and cubic spline interpolation. Upsampling can restore a low-resolution image to a size similar to or the same as the original high-resolution image.

[0094] The image processing model provided in this application is mainly used for image restoration, such as the image restoration in the aforementioned application scenario three. In the following description of the image processing model provided in this application, the stride S of each convolutional layer, unless otherwise specified, is assumed to be the first step length a1. The first step length a1 can be, but is not limited to, 1; in this embodiment, the first step length a1 = 1 is selected.

[0095] Figure 5A and Figure 5B This diagram illustrates the network structure of an image processing model provided in an embodiment of this application. The image processing model is deployed on an electronic device, which can be a mobile terminal device, such as a personal computer (e.g., laptop, desktop, all-in-one), smartphone, tablet, in-vehicle computer, game console, or smart wearable device (e.g., smartwatch). See also... Figure 5A and Figure 5B As shown, this image processing model includes a repair ensemble network and a discriminant network. Among them,

[0096] The inpainting network is used to perform image inpainting processing on a first image to be inpainted based on target inpainting region information to obtain a second image. The target inpainting region information is the mask information corresponding to the first image, and this mask information is a binary mask image m (e.g., ...). Figure 5A As shown in the figure, the acquisition of information about the target repair area will be described in detail in the method embodiments below, and will not be described in detail here.

[0097] In specific implementation, the repair combination network includes a first repair network 11 and a second repair network 12. The first repair network 11 and the second repair network 12 are described in detail below.

[0098] I. First Repair Network 11

[0099] The first restoration network 11 is a generator network (which may be referred to as a generator network), and its input includes the downsampled first image I and the target restoration region information. The downsampled first image I is obtained by downsampling the first image to be restored using the downsampling network 14 in the model. The downsampling network 14 may be composed of at least one fourth downsampling module. The structure of this fourth downsampling module can be found in [reference needed]. Figure 6B or Figure 6DThe downsampling module is shown in the diagram. The first image to be repaired is often a high-resolution image input by the user. To reduce the computational load of subsequent networks and improve image repair efficiency, it is necessary to downsample the first image. After downsampling, the first image will be reduced in dimensionality and size, thus becoming a low-resolution image. The target repair region information and the downsampled first image are input into the first repair network 11. Executing the first repair network 11 will ultimately yield an output result Out1 (the first repair result). Since the first repair network 11 can only process low-resolution images, this output result Out1 is a low-resolution repair result. It needs to be further upsampled to high resolution using the second repair network 12. Therefore, the output result Out1 of the first repair network 11 is the input of the second repair network 12, and the output result Out2 of the second repair network 12 is the final repaired target image (e.g., ...). Figure 5A The second image shown The second repair network 12 can also be called a super-resolution network.

[0100] The first restoration network 11 includes a first sub-network 111 and a second sub-network 112. The network structure of the first sub-network 111 can be, but is not limited to, common encoder-decoder network structures (encoder-decoder structures), which typically include one encoder and one decoder. The second sub-network 112 includes two encoders arranged in parallel and one decoder. The target restoration region information and the downsampled first image are first input into the first sub-network 111 for processing. The first sub-network 111 will output a processing result (such as...). Figure 5B The output result shown is Out_0, which is a coarse-grained output result. This coarse-grained output result can be understood as a blurred image filling and repair result. The processing result is used as input to the second sub-network 112, which outputs the first repair result (e.g., ...). Figure 5B The output result shown in the figure (Out1) is a first repair result obtained by adding more detailed features to the processing result, and is a fine-grained output result.

[0101] From the above, it can be understood that the first sub-network 111 is a coarse-grained generator, and the second sub-network 112 is a coarse-grained generator.

[0102] Figure 6A A schematic diagram of the structure of the first sub-network 111 is shown. Figure 6AThe aforementioned first sub-network 111 includes: multiple seventh convolutional layers for convolution operations, a third downsampling module group for downsampling operations, and a second upsampling module group for upsampling operations. Based on the connection positions of each seventh convolutional layer, the multiple seventh convolutional layers are divided into three parts: a first part seventh convolutional layer, a second part seventh convolutional layer, and a third part seventh convolutional layer. Each part seventh convolutional layer contains at least one seventh convolutional layer connected sequentially, and if the number of such sequentially connected at least one convolutional layer is multiple (two or more), then the output of the previous convolutional layer in such sequentially connected at least one convolutional layer is the input of the next convolutional layer. Specifically, the third downsampling module group is connected between the first part seventh convolutional layer and the second part seventh convolutional layer, and the second upsampling module group is connected between the second part seventh convolutional layer and the third part seventh convolutional layer.

[0103] Furthermore, each of the multiple seventh convolutional layers is immediately followed by an activation layer (not shown in the diagram). This activation layer is implemented using an activation function, which can be, but is not limited to, ELU (Elu activation), ReLU (Rectified Linear Unit), Sigmoid (also called Logistic function), tanh (also called Hyperbolic Tangent activation function), Selu, PreLU (Parametric ReLU), etc. The reason for connecting an activation layer after a convolutional layer (such as the seventh convolutional layer) is that while the operations performed by the convolutional layer on the input image are linear, the information in the input image is not always linearly separable. The activation layer performs non-linear operations, which better maps features, removes redundancy in the data, and enhances the network's expressive power.

[0104] In specific implementation, except for the last seventh convolutional layer in the third part, the activation layers connected to the outputs of the other seventh convolutional layers can all be implemented using functions such as ELU. For the last seventh convolutional layer in the third part, the connected activation layer can be implemented using the tanh function, used to restrict the output of this last seventh convolutional layer to the [-1, 1] interval. Therefore, this activation layer (such as the tanh function) is called a limiting layer in this embodiment, and the activation function corresponding to the limiting layer is called a limiting function. Thus, the first sub-network 111 also includes a second limiting layer, which is connected to the final output of the seventh convolutional layer in the third part, i.e., to the output of the last seventh convolutional layer in the third part. The second limiting layer can be implemented using, but is not limited to, the tanh function.

[0105] All seventh convolutional layers in the first sub-network 111 described above can be gated convolutional layers. The structure of gated convolutional layers will be described in detail below and will not be repeated here. Furthermore, the third downsampling module group contains at least one downsampling module, and the second upsampling module group contains at least one upsampling module.

[0106] Figure 6B A schematic diagram of the downsampling module is shown. Figure 6B The downsampling module includes multiple convolutional layers, and the stride S of a portion of the convolutional layers is the first stride a1, while the stride S of another portion of the convolutional layers is the second stride a2, where the second stride a2 is greater than the first stride a1.

[0107] Figure 6C A schematic diagram of the upsampling module is shown. Figure 6C The upsampling module includes an interpolation layer and multiple convolutional layers connected sequentially. The stride of these convolutional layers is a first stride a1. The interpolation layer is connected to the input of the first convolutional layer in the sequence and is used to amplify or scale the input of the upsampling module. In this embodiment, the interpolation layer amplifies the input of the upsampling module, and the amplification factor can be, but is not limited to, 2 times. The interpolation layer can be implemented using appropriate interpolation algorithms, such as nearest neighbor interpolation or bilinear interpolation.

[0108] It should be noted that the convolutional layers in the above-mentioned upsampling and downsampling modules can also be gated convolutional layers.

[0109] Figure 6D A specific structural example of the first sub-network 111 is shown. This first sub-network 111 includes nine gated convolutional layers (i.e., nine seventh convolutional layers). The stride S of each of these nine gated convolutional layers is 1. Seven of these gated convolutional layers are sequentially connected between the third downsampling module group and the second upsampling module group. One of the remaining two gated convolutional layers is connected to the input of the third downsampling module group, and the other is connected to the output of the second upsampling module group. The third downsampling module group includes four downsampling modules, each including one gated convolutional layer with a stride S of 2 and one gated convolutional layer with a stride S of 1. The second upsampling module group includes four upsampling modules, each including one interpolation layer and two gated convolutional layers. The interpolation layer has a magnification factor of 2, and the stride S of both gated convolutions is 1.

[0110] Figure 7A A schematic diagram of the structure of the second sub-network 112 is shown. Figure 7AThe second sub-network 112 includes: a first encoder (encoder E1 as shown in the figure), a second encoder (encoder E2 as shown in the figure), a splicing layer, and a decoder D1. The first encoder and the second encoder are two encoders arranged in parallel, and the input of both is the output result Out_0 of the first sub-network 111. The output terminals of the first encoder and the second encoder are both connected to the input terminal of the decoder D1 through the splicing layer C. That is to say, through the splicing layer C, the output of the first encoder and the output of the second encoder can be spliced ​​by channel to obtain the splicing operation result. The splicing operation result is input into the decoder D1 for decoding processing, and a decoding processing result, namely the output result Out1, will be output.

[0111] Specifically, the functions of each structure included in the second sub-network 112 are as follows:

[0112] The first encoder is used to obtain the processing result (i.e., the result obtained by the first sub-network 111 processing the target restoration region and the first image) Figure 6D The output result Out_0 shown in the figure is used to encode the processing result and output the first encoded result.

[0113] The second encoder is used to encode the processing result and output the second encoded result;

[0114] The splicing layer C is connected to the output terminals of the first encoder and the second encoder, and is used to perform a splicing operation on the first encoding result and the second encoding result, and output the splicing result.

[0115] The decoder, connected to the output of the splicing layer, is used to decode the splicing result and output the first repair result (i.e., Figure 7A The output result Out1 is shown in the figure.

[0116] The aforementioned first encoder includes multiple fourth convolutional layers for convolution operations, a first downsampling module group for downsampling operations, and the self-attention module. The first downsampling module group connects the first portion of the multiple fourth convolutional layers to the second portion, the self-attention module connects the remaining third portion of the fourth convolutional layers to the second portion, and the output of the third portion of the fourth convolutional layers is connected to the input of the splicing layer.

[0117] The second encoder described above includes multiple fifth convolutional layers for convolution operations and a second downsampling module group. The second downsampling module group is connected between a first portion of the fifth convolutional layers and the remaining second portion of the fifth convolutional layers, and the output of the second portion of the fifth convolutional layers is connected to the input of the splicing layer. The second downsampling module group includes multiple downsampling modules for downsampling operations.

[0118] For details on the module structure of the downsampling module, please refer to other embodiments. Figure 6B and Figure 6D The relevant content described.

[0119] The decoder D1 described above includes multiple sixth convolutional layers for convolution operations, a first upsampling module group, and a first limiting layer. A first portion of the multiple sixth convolutional layers is connected between the splicing layer and the first upsampling module group, and the remaining second portion of the sixth convolutional layers is connected between the first upsampling module group and the first limiting layer. The first limiting layer, which may be implemented using, but is not limited to, a tanh function, is used to limit the output of the second portion of the sixth convolutional layers to a first interval range. The first upsampling module group includes multiple upsampling modules for upsampling operations. The module structure of the upsampling modules can be found in other embodiments. Figure 6C and Figure 6D The relevant content described.

[0120] Similarly, the fifth, sixth, and third convolutional layers in the second sub-network 112 can also be gated convolutional layers. The structure of gated convolutional layers will be described in detail below and will not be repeated here.

[0121] Figure 7B A specific network structure example of the second sub-network 112 is shown. The structures of the first encoder, the second encoder, and the decoder D1 included in the second sub-network 112 are as follows: The first encoder includes 4 gated convolutional layers, one of which is connected to the input of the first downsampling module group (including 4 sampling modules), another gated convolutional layer is connected between the first downsampling module group and the self-attention module, and the remaining 2 gated convolutional layers are connected to the output of the self-attention module; The second encoder includes 6 gated convolutional layers and the second downsampling module group (including 4 downsampling modules), one of which is connected to the input of the second downsampling module group, and the remaining 5 gated convolutional layers are connected to the output of the second downsampling module group; The decoder D1 includes 3 gated convolutional layers, a first upsampling module group (including 4 upsampling modules), and a first limiting layer. Two of the 3 gated convolutional layers are connected between the first upsampling module group and the splicing layer, and the remaining 1 gated convolutional layer is connected between the first upsampling module group and the first limiting layer.

[0122] Based on the above, an intermediate result F (the intermediate convolution result output by the fourth convolutional layer of the second part) generated by the second sub-network 112 during the processing of the output result Out_0 of the first sub-network 111 is the input of the self-attention module. Correspondingly, the function of the self-attention module is to process the intermediate result F to extract its global features. Specifically, when processing the intermediate result F, the self-attention module focuses its attention on the important information within F, filtering out other unimportant information.

[0123] The intermediate result F mentioned above is specifically the feature map output by the fourth convolutional layer in the second part. Typically, an image mainly contains two dimensions of features: scale space (e.g., length and width) and channels. Therefore, spatial attention and channel attention mechanisms are needed to process these two dimensions and improve feature representation capabilities. Channel attention assigns different weights to each channel based on its importance, thereby strengthening important channel features and suppressing less important ones. Spatial attention assigns different weights to different regions of the image, making the model focus more on important regional features and ignore irrelevant information.

[0124] Therefore, in this embodiment, in order to more comprehensively mine image features and improve the image restoration effect, the self-attention module includes combined layers corresponding to spatial attention mechanism and channel attention mechanism respectively.

[0125] Figure 8A and Figure 8B The diagram shows the structure of the self-attention module. Figure 8A As shown, the self-attention module includes a spatial attention combination layer, a channel attention combination layer, and a first summing layer St1. The spatial attention combination layer is used to mine the scale-space features of the intermediate result F, and the channel attention combination layer is used to mine the channel features of the intermediate result F. The outputs of both the spatial attention combination layer and the channel attention combination layer are connected to the input of the first summing layer St1. The intermediate result F is input into the spatial attention combination layer and processed by the attention combination layer, respectively. Then, the spatial attention features output by the spatial attention combination layer and the channel attention features output by the channel attention combination layer are added and fused by the first summing layer St1. The result of the addition and fusion is the final attention feature information output by the self-attention module, which is then input to the convolutional layer located downstream of the self-attention module.

[0126] Based on the above, that is, an intermediate result F generated by the second sub-network 112 during the processing of the output of the first sub-network 111 serves as the input to the self-attention module. Furthermore, the functions of each structural layer within the self-attention module are as follows:

[0127] A spatial attention combination layer is used to perform feature extraction processing on the intermediate results and output spatial attention features;

[0128] The channel attention combination layer is used to perform feature extraction processing on the intermediate results and output channel attention features;

[0129] The first summation layer is used to add and fuse the spatial attention features and the channel attention features to output attention feature information.

[0130] Furthermore, such as Figure 8B As shown, the spatial attention layer comprises three convolutional layers and a computation module arranged in parallel. The outputs of the three convolutional layers are all connected to the inputs of the computation module. The computation module includes a first transposed layer, two connected second product layers Mu2 and Mu3, and a second summing layer St2 connected to the third product layer. The first convolutional layer is connected to the input of the second product layer Mu2 via the first transposed layer; the second convolutional layer is directly connected to the input of the second product layer Mu2; and the output of the third convolutional layer is connected to the input of the third product layer Mu3. The kernel size k of each of the three convolutional layers can be, but is not limited to, 1x1.

[0131] Specifically, the functions of each structural layer included in the spatial attention combination layer are as follows:

[0132] The first convolutional layer is used to extract features from the intermediate results and output the first type of features (which is the query feature Q).

[0133] The second convolutional layer is used to extract features from the intermediate results and output the second type of features (key features K);

[0134] The third convolutional layer is used to extract features from the intermediate results and output the third type of features (value features V);

[0135] The first transpose layer is connected to the output of the first convolutional layer and is used to transpose and deform the first type of features and output the transposed and deformed first type of features.

[0136] The second product layer is connected to the outputs of the first transposed layer and the second convolutional layer, and is used to multiply the transposed first type of feature with the second type of feature to output the first multiplication result;

[0137] The third product layer, connected to the outputs of the second product layer and the third convolutional layer, is used to multiply and fuse the spatial attention map obtained based on the first product result with the third type of feature to obtain the second product result.

[0138] The second summation layer, connected to the output of the third product layer, is used to add and fuse the second multiplication result with the intermediate result to output the spatial attention feature.

[0139] The aforementioned spatial attention map can be obtained by normalizing the result of the first multiplication. That is, a normalization layer can be connected between the second and third multiplication layers. The normalization layer can be constructed using a function such as Softmax. The spatial attention map can be obtained by normalizing the result of the first multiplication layer using this normalization layer and then input into the third multiplication layer Mu3.

[0140] Taking the aforementioned intermediate result F as a feature map of size C×H×W as an example, the process of the spatial attention combination layer processing the input intermediate result F can be briefly described as follows:

[0141] S11. Input the feature map of size C×H×W into the first, second, and third convolutional layers respectively to obtain three types of features: query feature Q, key feature K, and value feature V. Here, C, H, and W in C×H×W represent the number of channels, height, and width of the feature map, respectively. The query feature Q is output by the first convolutional layer, the key feature K by the second convolutional layer, and the value feature V by the third convolutional layer. The main function of these three convolutional layers is to reduce the number of channels, but they do not change the height and width of the feature map.

[0142] S12. The query feature Q is transposed and transformed through the first transpose layer, and then multiplied with the key feature K. The result of the multiplication is normalized using the corresponding normalization function to obtain a spatial attention map of size (H×W)×(H×W). The normalization function may be, but is not limited to, the softmax function.

[0143] S13. Multiply the spatial attention map with the value feature V and add the feature map to obtain the final output spatial attention feature.

[0144] See also Figure 8BAs shown, the aforementioned channel attention combination layer includes: a second transposed layer, a fourth product layer Mu4 and a fifth product layer Mu5 connected together, and a third summing layer St3 connected to the fifth product layer Mu5. The output of the second transposed layer is connected to the input of the fourth product layer Mu4, and the input of the second transposed layer is the intermediate result F. The input of the fourth product layer Mu4 includes the intermediate result F and the output of the second transposed layer. The input of the fifth product layer Mu5 includes the intermediate result F and the output of the fourth product layer Mu4. The input of the third summing layer St3 includes the output of the fifth product layer Mu5 and the intermediate result F. In the above, the output of the fourth product layer Mu4 is normalized before being input to the fifth product layer Mu5.

[0145] Specifically, the functions of each structural layer included in the channel attention combination layer are as follows:

[0146] The second transpose layer is used to transpose and deform the intermediate result, and output the deformed intermediate result.

[0147] The fourth product layer Mu4 is connected to the output of the second transposed layer and is used to multiply the deformed intermediate result with the intermediate result to output a channel attention map;

[0148] The fifth product layer Mu5 is connected to the output of the fourth product layer and is used to multiply the channel attention map with the intermediate result to output the third product result.

[0149] The third summing layer St3, along with the output of the fifth product layer, is used to add the third multiplication result to the intermediate result and output the channel attention feature.

[0150] The aforementioned channel attention map can be obtained by normalizing the multiplication result obtained by multiplying the deformed intermediate result with the intermediate result. That is, a normalization layer (not shown in the figure) can be connected between the fourth product layer Mu4 and the fifth product layer Mu5. The normalization layer can be constructed using a function such as Softmax. The input multiplication result can be normalized using this normalization layer to obtain the channel attention map and input it to the fifth product layer Mu5.

[0151] Continuing with the example of an intermediate result F being a feature map of size C×H×W, the processing procedure of the aforementioned channel attention combination layer on the input intermediate result F can be summarized as follows:

[0152] S21. After transposing the feature map, we obtain the transposed feature map. The transposed feature map is then multiplied by the original feature map. The result of the multiplication is normalized to obtain a channel attention map of size C×C.

[0153] S22. Multiply the channel attention map by the original feature map and add the original feature map to obtain the final channel attention feature to be output.

[0154] Understandably, the convolutional layers included in the self-attention module described above are not limited to regular convolutional layers.

[0155] The self-attention module used in the image processing model provided in this embodiment has the following advantages over the context attention module used in existing image restoration models: While achieving the same goal of acquiring global features and improving image restoration results, it can be easily deployed on mobile terminals (such as smartphones). The reasons are as follows:

[0156] See also Figure 4B The diagram shown illustrates the structure of the context attention module used in existing image restoration models. The processing flow of the input feature map by the existing context attention module is as follows:

[0157] Step 1: Divide the input feature map into blocks and deform it (by transpose) to obtain a series of convolution kernels;

[0158] Step 2: Then, use the convolution kernel obtained in Step 1 to perform a convolution operation with the original feature map and perform a normalization operation using the corresponding normalization function (such as the softmax function) to obtain the attention score;

[0159] Step 3: Use attention scores to perform deconvolution on the segmented feature maps to obtain the final output.

[0160] The problem with the currently used context attention module is that the parameters of the convolutional kernel in step 2 are not fixed but dynamically change according to the input feature map, resulting in dynamic convolutions, which are difficult to deploy on mobile devices. Furthermore, the deconvolution operation in step 3 is time-consuming and unfriendly to mobile devices. The self-attention combination layer provided in this embodiment does not address these issues.

[0161] To make the entire image processing model more suitable for deployment on mobile terminals, the aforementioned combination Figures 6A to 7BWhen the corresponding convolutional layers (convolutional layers not belonging to the self-attention module, such as the fourth, fifth, sixth, and seventh convolutional layers, etc.) in the first repair network 11 described are gated convolutional layers, these gated convolutional layers can be constructed using depthwise separable convolutional layers. Depthwise separable convolutional layers consist of two parts: depthwise convolution and pointwise convolution. In depthwise convolution, the number of kernels is the same as the number of input channels, and each channel is convolved by only one kernel. Pointwise convolution often uses 1×1 convolutions to fuse the feature maps obtained in the previous step along the channel dimension. By splitting the correlation between the spatial dimension and the channel (depth) dimension, depthwise separable convolutional layers reduce the parameters required for convolution computation, improving running speed and making them more suitable for deployment on mobile terminals.

[0162] In practice, the gated convolutional layer may include multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The multiple depthwise separable convolutional layers share the same input, which is the output of the corresponding upstream layer / module. The outputs of the multiple depthwise separable convolutional layers are all connected to the input of the first multiplication layer, which multiplies and fuses the outputs of the multiple depthwise separable convolutional layers to obtain the output of the gated convolutional layer.

[0163] Figure 9A and Figure 9B A schematic diagram of a specific structure of the gated convolutional layer provided in this embodiment is shown. For example... Figure 9A As shown, the gated convolutional layer is a combined layer, comprising two depthwise separable convolutional layers and a first multiplication layer Mu1. The outputs of the two depthwise separable convolutional layers are both connected to the input of the first multiplication layer Mu1. The gated convolutional layer processes the input X as follows:

[0164] S31. Input X into two depthwise separable convolutional layers for convolution processing to obtain two depth features: the first depth feature (mask feature) output by the first depthwise separable convolutional layer and the second depth feature (image feature) output by the second depthwise separable convolutional layer.

[0165] S32. Perform limit processing on the first depth feature to restrict the value of the first depth feature to the interval [0,1], where, for example... Figure 9B Limiting can be implemented through a corresponding third limiting layer, which can be implemented using, but is not limited to, the sigmoid function. Furthermore, activation processing is applied to the second deep features to obtain the activated second deep features; where, for example... Figure 9B Activation processing can be achieved through a corresponding activation layer, which can be implemented using, but is not limited to, the ELU function.

[0166] S33. The first depth feature after limiting and the second depth feature after activation are finally input into the first product layer Mu1 for multiplication and fusion to obtain the final output Y.

[0167] The final output Y is expressed as follows:

[0168] The output Y = sigmoid(DS_conv(X)) * ELU(DS_conv(X)), where DS_conv represents a depthwise separable convolutional layer.

[0169] The gated convolution described above can adaptively learn which pixels in the features are effective for the restoration task, and assign different weights to different pixels based on the learning results, thereby improving the restoration effect. Pixels that are more effective for the restoration task are typically assigned higher weights.

[0170] Based on the above, in a specific example, the gated convolution provided in this embodiment includes: a first depthwise separable convolutional layer, a second depthwise separable convolutional layer, a third limit layer, an activation layer, and a first multiplication layer Mu1;

[0171] The output of the first depth-separable convolutional layer is connected to the third confinement layer, and the output of the second depth-separable convolutional layer is connected to the activation layer;

[0172] The output of the third limiting layer and the output of the activation layer are both connected to the first product layer Mu1;

[0173] The third limiting layer is used to limit the first depth convolutional feature output by the first depth separable convolutional layer, and output the first depth convolutional feature after limiting; the value of the first depth convolutional feature after limiting is within the third interval range.

[0174] The activation layer is used to activate the second depth convolutional features output by the second depth separable convolutional layer, and output the activated second depth convolutional features.

[0175] The first multiplication layer Mu1 is used to multiply the first depthwise convolutional feature after limiting by the second depthwise convolutional feature after activation, and output the fourth multiplication result; the fourth multiplication result is the output result of the gated convolution.

[0176] II. Second Repair Network 12

[0177] The second repair network is a super-resolution network. Due to limitations in computing power and memory, the aforementioned first repair network 11 (generator network) can only process low-resolution images, and its output, when upsampled to the original image resolution, is often blurry. In this embodiment, the second repair network 12 is further used to process the output of the first repair network 11 (i.e., the aforementioned first repair result, which is a low-resolution fill repair result), thereby upsampling the output of the first repair network to a high resolution. The second repair network 12 can be any common lightweight super-resolution network model that can be deployed on mobile terminals.

[0178] Alternatively, the structure of the second repair network can be as follows: Figure 10A As shown, the second repair network includes: multiple eighth convolutional layers, two upsampling module groups, and a fourth summing layer St4. The two upsampling module groups include a third upsampling module group and a fifth upsampling module group. The third upsampling module group connects the first part of the eighth convolutional layers to the remaining second part of the eighth convolutional layers, and the outputs of both the second part of the eighth convolutional layers and the fourth upsampling module group are connected to the input of the fourth summing layer St4. The inputs of the first part of the convolutional layers and the fourth upsampling module group are the outputs of the previous level network (i.e., the first repair result output by the first repair network 11). Both the third and fifth upsampling module groups include at least one upsampling module; detailed descriptions of the upsampling module structure can be found in other embodiments.

[0179] Based on this Figure 10A As can be seen from the structure shown, the processing of the second repair network 12 is as follows: the input (i.e., the first repair result finally output by the first repair network 11) will be processed through the first part of the eighth convolutional layer (e.g., 7 eighth convolutional layers), the third upsampling module group and the second part of the eighth convolutional layer (e.g., 1 convolutional layer), and the input will also be processed through the fourth upsampling module group; then, the two output results obtained from the above processing are added and fused to obtain the corresponding final repair result (the second repair result containing the second image).

[0180] Figure 10BA specific structural example of the second repair network is shown. In this example, the second repair network 12 includes eight eighth convolutional layers, a third upsampling module group, a fourth upsampling module group, and a fourth summing layer St4. The eight eighth convolutional layers include one eighth convolutional layer with a kernel size of 5×5, two eighth convolutional layers with a kernel size of 1×1, and five eighth convolutional layers with a kernel size of 3×3. The eighth convolutional layer with a kernel size of 5×5, the eighth convolutional layer with a kernel size of 1×1, the four eighth convolutional layers with a kernel size of 3×3, the eighth convolutional layer with a kernel size of 1×1, the third upsampling module group, and the eighth convolutional layer with a kernel size of 3×3 are sequentially connected to form a first branch. The fourth upsampling module group forms a second branch. The outputs of the first branch and the second branch are connected to the fourth summing layer St4.

[0181] In specific implementation, the upsampling operations of the two upsampling module groups in the second repair network 12 can be implemented using common interpolation algorithms, such as bilinear interpolation. Furthermore, all convolutional layers in the second repair network 12 can be implemented using regular convolutional layers (i.e., ordinary convolutional layers), and except for the last convolutional layer (i.e., the eighth convolutional layer connected to the output of the third upsampling module group, which is not followed by an activation layer), each remaining convolutional layer is followed by an activation layer (not shown in the figure). This activation layer can be, but is not limited to, using the PReLU activation function.

[0182] Furthermore, the image processing model provided in this application embodiment, in addition to the first repair network 11 and the second repair network 12, may also include a discriminant network 13 (also called a discriminator network). This discriminant network 13 is a binary classifier used to determine the authenticity of the input image. See also... Figure 5B The input images to the discrimination network 13 include the original first image I and the second image output by the second repair network 12. This is the repaired image of the first image I.

[0183] The network structure of the aforementioned discriminant network 13 is an encoder structure, which can be implemented using conventional multiple convolutional layers.

[0184] Figure 11 A specific structural example of a discriminant network is shown. In this example, discriminant network 13 is implemented using four ninth convolutional layers with a stride S of 2 and a kernel size of 5×5. These four ninth convolutional layers are sequentially connected end-to-end. The ninth convolutional layers can be regular convolutional layers. Each ninth convolutional layer is followed by an activation layer (not shown in the figure). This activation layer can be implemented using, but is not limited to, the LeakyReLU activation function. This discriminant network 13 can downsample the input to 1 / 32 of the original input resolution.

[0185] Based on the discriminant network 13, the first image I and the second image I are compared. The discrimination result information and the original first image I and second image Loss information can be determined, and the image processing model can be iteratively optimized based on this loss information until the iteration stopping condition is met, thereby optimizing the model parameters of the image restoration model. For example, the network parameters of the first restoration network 11 and the second restoration network 12 can be optimized to improve the image restoration effect, and the network parameters of the discriminant network 13 can be optimized to improve the discriminant network's ability to distinguish between real and fake images.

[0186] The aforementioned discrimination result information may include: the discrimination value output by the discrimination network 13 for the first image I, and the discrimination value for the second image. The output discriminant value; and the intermediate discriminant value of each intermediate network layer in the discriminant network 13 for the first image I, and for the second image. The intermediate discriminant value. Understandably, the output discriminant value is the output of the last network layer in discriminant network 13, and the intermediate network layers refer to all network layers in the discriminant network except the last one.

[0187] In this embodiment, the loss information used to optimize the image processing model includes:

[0188] 1) The first loss function, Lrecon (the reconstruction loss function), is based on the original first image I and the second image. The absolute value of the difference between them is determined. The first loss function, Lrecon, is expressed as follows:

[0189]

[0190] Here, ||.||1 represents the L1 norm, also known as Manhattan distance, minimum absolute error, etc. The operation corresponding to the L1 norm is vector (e.g., ... The sum of the absolute values ​​of the non-zero elements in a , denoted as . Then ||ΔX1||1=∑|x p |, p represents the P-th element in vector ΔX1.

[0191] 2) Model loss function, including the second loss function L G (To repair the combined network loss) and the third loss function L D (To determine network losses); among them,

[0192] Second loss function L G It is based on the discrimination network 13 on the second image The output discriminant value is determined by;

[0193] Third loss function LD Based on the output discrimination value of the discrimination network 13 for the first image I and the second image I The output discrimination result is determined.

[0194] Second loss function L G and the third loss function L D The expressions are as follows:

[0195]

[0196]

[0197] Where D represents the discriminant network, E represents the calculation of the mean, and D(I) represents the output discriminant value of the discriminant network for the first image I. This indicates that the discriminant network has a different view of the second image. The output discriminant value.

[0198] 3) Fourth loss function L FM (This is the feature matching loss function, i.e., the FM loss function), which is based on the intermediate discriminant values ​​of each intermediate network layer (convolutional layer) in the discriminant network for the first image I and for the second image. The intermediate discriminant value is determined. The fourth loss function L... FM Its main function is to stabilize the training of image processing models (specifically, GAN (Generative Adversarial Network) models), and it is expressed as follows:

[0199]

[0200] Where k represents the k-th intermediate network layer in the discriminant network, D k (I) indicates These represent the intermediate discriminant values ​​of the k-th intermediate network layer in the discriminant network for the first image I and for the second image, respectively. The intermediate discriminant value. ||.|2 represents the L2 norm. The operation corresponding to the L2 norm is the square root of the sum of the squares of the vector elements, as denoted as... but

[0201] 4) Perceptual loss functions include: the fifth loss function Lcontent (content loss function) and the sixth loss function Lstyle (style loss function); among them,

[0202] The fifth loss function, Lcontent, and the sixth loss function, Lstyle, utilize a pre-trained perceptual loss network to calculate and analyze the original first image I and the second image. The perceptual loss network is determined by the differences in content and style between feature maps. The perceptual loss network can be a VGG network (also known as VGGNet, a classic convolutional neural network architecture), specifically, for example, a VGG19 network consisting of 19 convolutional layers or a VGG16 network consisting of 16 convolutional layers. In this embodiment, the VGG19 network is selected.

[0203] The expressions for the fifth loss function Lcontent and the sixth loss function Lstyle are as follows:

[0204]

[0205]

[0206] Among them, VGG i (*) represents the feature map output by the i-th intermediate network layer (convolutional layer) in the VGG network. j (*) represents the feature map output by the j-th intermediate network layer (convolutional layer) in the VGG network. In this embodiment, VGG refers to the pre-trained VGG19 network, and Gram represents the calculated Gram matrix. “*” represents either the first image I or the second image.

[0207] 5) Seventh loss function L TV (This is the total variation loss function), which is based on the second image. The image gradient information is used to determine this. The seventh loss function L... TV The expression is:

[0208]

[0209] in, This indicates calculating the image gradient in the x-direction. This represents calculating the image gradient in the y-direction. The sixth loss function L... TV By constraining the image gradient over the pixel domain, noise can be removed while preserving the boundaries.

[0210] From the loss information described in 1) to 5) above, it can be concluded that:

[0211] The total loss function is: L = L recon +α1L G +α2L content +α3L style +α4L TV +α5L FM

[0212] Wherein, α1, α2, α3, α4, and α5 are the weights of the corresponding loss function terms in the total loss function, which can be flexibly set according to the actual situation. For example, they can be set as follows: α1 = 0.01, α2 = 0.01, α3 = 20, α4 = 0.1, and α5 = 0.3.

[0213] Based on the aforementioned loss information, an appropriate optimizer can be used to iteratively optimize the image processing model. This optimizer can be, but is not limited to, the Adam (Adaptive Moment Estimation) optimizer. The Adam optimizer is an adaptive optimization algorithm that adjusts the learning rate based on historical gradient information. Its main function is to update model parameters based on gradient information, thereby minimizing the loss function.

[0214] From the above content, see also Figure 12 As shown, the image processing model provided in this embodiment may further include: a loss determination module 14 and an optimizer 15. The loss determination module 14 is used to perform loss determination on the first image I and the second image I based on the discriminant network 13. The discrimination result information and the first image I and the second image Determine the loss information. For details regarding the loss information, please refer to the descriptions in sections 1) to 5) above. Optimizer 15 is used to optimize the model parameters of the image processing model based on the loss information. For example, optimizing the network parameters of the repair network (such as the first repair network and the second repair network) in the image processing model.

[0215] In addition to the networks described above, the image processing model may include other networks in some embodiments, such as a pre-trained recognition network. This recognition network can be connected to the input of the first restoration network 11 to identify the target restoration area information that needs to be restored in the first image I. The network structure of the recognition network is not specifically limited, as long as it can correctly identify the target restoration area information that needs to be restored in the first image I.

[0216] This application also provides several image processing methods implemented using the aforementioned image processing model. The execution subject of these image processing methods is an electronic device, more specifically a mobile terminal device. The aforementioned image processing model is deployed on the electronic device. For example... Figure 16A As shown, the electronic device 100 can be, but is not limited to, a smartphone (such as a flip phone or a candybar phone), a tablet, or a smart wearable device (such as a smart bracelet). The structure of the electronic device 100 is described in detail in other embodiments below and will not be repeated here. Furthermore, the scenario in which the image processing method is applied can be one that combines the above-mentioned... Figures 2A to 2CThe described application scenario three is to repair stretching artifacts that appear in images during image dimensionality transformation. Of course, it can also be applied to other scenarios, such as application scenarios one and two mentioned above, and is not limited here. Application scenario three is the main application scenario for the image processing methods provided in this application.

[0217] Figure 13 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 13 As shown, the image processing method includes the following steps:

[0218] 201. Determine the target restoration area information corresponding to the first image;

[0219] 202. Obtain the image processing model deployed on the electronic device;

[0220] 203. Input the target restoration area information and the first image into the image processing model for image processing to obtain the second image;

[0221] 204. Display the second image in the electronic device;

[0222] In the image processing model, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer.

[0223] In the above 201, the method of acquiring the first image is different in different application scenarios.

[0224] For example, in setting the interface background for mobile terminals such as smartphones, the first image can be determined based on an interface background change event triggered by the user. The implementation of the interface background change event triggering can be, but is not limited to, the following: such as combining... Figure 2B and Figure 2C When a user wants to change the current background of the electronic device 100, they can open the "Desktop and Personalization" function in the theme application or settings application on the mobile terminal. At this time, multiple wallpaper images / theme images will be displayed on the application interface. The user can select one image from these multiple wallpaper images / theme images and set it as the wallpaper. For example, combined with... Figure 2D Users can also select an image from a variety of pictures provided by the mobile device's gallery application and set it as wallpaper using the "Set as" function provided by the gallery application. In response to the user-triggered background change event, the mobile device will designate the user-selected image as the first image.

[0225] Generally, the image initially selected by the user is a 2D image. To present a better wallpaper effect, if the user activates the 3D wallpaper function provided by the mobile terminal, the selected 2D image can be converted into a 3D image for presentation. Based on this, the first image can also be an image obtained by performing a dimensionality transformation (upward transformation (e.g., from 2D to 3D) or downward transformation (e.g., from 3D to 2D)) on the image selected by the user. After obtaining the first image through dimensionality transformation, it can be identified whether the first image needs repair. If so, the above steps S201 to S203 are triggered.

[0226] Based on the above, in a specific instance, the method provided in this implementation may further include the following steps:

[0227] 200a. In response to an interface background change event triggered by the user through an interactive interface provided by an electronic device, determine the first image selected by the user;

[0228] 200b. Based on the dimensional attributes of the first image and the background function information of the interface launched by the electronic device, determine whether it is necessary to perform dimensional transformation on the first image; for example, if the first image is a 2D image and the 3D wallpaper function of the mobile terminal is in the launched state, then it is determined that it is necessary to perform dimensional transformation on the first image to convert the first image from 2D to 3D.

[0229] 200c. When necessary, the first image is dimensionally transformed to obtain the transformed first image;

[0230] 200b. Based on the converted first image, trigger the execution of step 201 above.

[0231] It is possible, but not limited to, using a pre-trained recognition network to identify the regions in the converted first image that need to be repaired, thereby obtaining the target repair region information. This target repair region information can be a binary mask image, such as... Figure 5A The binary mask image m shown in the figure represents the location of the area to be repaired in the first image I.

[0232] For example, in a shooting scenario, the first image can be obtained from a camera. When a user takes a photo using the camera on a mobile terminal, the main structure of the object in the photo is often damaged due to various factors. In this case, if the user activates the image restoration function on the mobile terminal, the image captured by the camera can be first input into an image processing model for restoration processing, and then the restored image can be presented to the user. Based on this, in another specific example, the method provided in this implementation may also include the following steps:

[0233] 200a', Acquire the first image captured by the camera on the electronic device;

[0234] 200b', Identify whether the first image needs repair;

[0235] For example, the first image captured by the camera can be input into a pre-trained recognition network, and the output recognition result can be used to determine whether the first image needs to be repaired. For example, if the binary mask image contained in the recognition result does not contain white areas, it means that the first image does not need to be repaired; otherwise, if white areas exist, it means that the first image needs to be repaired.

[0236] 200c' When necessary, the above step 201 will be triggered based on the recognition result.

[0237] For example, in a scenario where a user actively wants to erase image content, the first image can be manually entered by the user. For instance, the user opens the first image in their gallery using an image enhancement app on their electronic device, and uses the tools provided by the app to mark the areas they want to erase. After marking, the image enhancement app can determine the target restoration area information based on the marked information of the first image, and based on this target restoration area information, call the image processing model deployed on the electronic device to perform restoration processing on the first image.

[0238] For a detailed description of the image processing model in section 202 above, please refer to the relevant content in the other embodiments mentioned above.

[0239] See also Figure 5A and Figure 5B As shown, the image processing model described above includes a first restoration network and a second restoration network for image inpainting. Correspondingly, step 203, "inputting the target restoration region information and the first image into the image processing model for image inpainting processing to obtain a second image," may specifically include:

[0240] 2031. Input the target restoration area information and the first image into the first restoration network, and output the first restoration result;

[0241] 2032. Input the first repair result into the second repair network, and output the second repair result containing the second image.

[0242] Before inputting the first image into the first repair network, such as combining Figure 5B As shown, the first image can be downsampled using the corresponding downsampling network 14 in the image processing model before being input into the first repair network 11. The specific reasons for downsampling the first image first can be detailed in other embodiments of this application.

[0243] In the aforementioned 2031, the first repair network includes a first subnetwork and a second subnetwork, with the second subnetwork connected to the output of the first subnetwork. For details regarding the network structure of the first and second subnetworks, please refer to [link to relevant documentation]. Figures 6A to 8B As shown.

[0244] In a specific feasible solution, the implementation process of step 2031 above is as follows:

[0245] 20311. The target repair area and the first image are processed using the first sub-network to obtain the processing result.

[0246] Such as combination Figures 6A to 6D The diagram shows the network structure of the first sub-network 111. The specific processing procedure of step 20311 is as follows:

[0247] A11. Perform a convolution operation on the target repair region and the first image to obtain the sixth convolution result;

[0248] This convolution operation is implemented using the seventh convolutional layer of the first part. If the seventh convolutional layer of the first part contains a gated convolutional layer, the sixth convolution result is obtained by performing a convolution on the target restoration region and the first image using that gated convolutional layer.

[0249] A12. Perform a downsampling operation on the sixth convolution result to obtain the third downsampling result;

[0250] This downsampling operation is implemented using the third downsampling module group. If the third downsampling module group contains four downsampling modules, then the third downsampling result is obtained by downsampling the sixth convolution result four times consecutively.

[0251] A13. Perform a convolution operation on the third downsampling result to obtain the seventh convolution result;

[0252] This convolution operation is implemented using the seventh convolutional layer in the second part. If the seventh convolutional layer in the second part contains seven gated convolutional layers, the result of the seventh convolution is obtained by performing seven consecutive convolutions on the third downsampling result using these seven gated convolutional layers.

[0253] A14. Perform an upsampling operation on the seventh convolution result to obtain a second upsampling result;

[0254] This upsampling operation is implemented using the second downsampling module group. If the second upsampling module group contains four upsampling modules, then the second upsampling result is obtained by upsampling the seventh convolution result four times consecutively.

[0255] A15. Perform a convolution operation on the second upsampling result to obtain the eighth convolution result;

[0256] This convolution operation is implemented using the seventh convolutional layer in the third part. If the seventh convolutional layer in the third part contains a gated convolutional layer, then the eighth convolution result is obtained by performing a convolution on the second upsampling result using that gated convolutional layer.

[0257] A16. Perform value limiting processing on the eighth convolution result to obtain the processing result whose value is limited to the range of the second interval.

[0258] This constraint processing is implemented using a second constraint layer. The second interval range can be, for example, [-1, 1]. The processing result is as follows: Figure 6B The output result shown is Out_0.

[0259] 20312. The processing results are respectively input into the first encoder and the second encoder in the second sub-network for encoding, and the first encoding result output by the first encoder and the second encoding result output by the second encoder are spliced ​​together to obtain the spliced ​​result.

[0260] Such as combination Figures 7A to 7B The network structure diagram of the second sub-network 112 shown below illustrates the specific implementation process of "inputting the processing result into the first encoder (encoder E1) for encoding to obtain the first encoding result" in step 20312 as follows:

[0261] A21. Perform a convolution operation on the processing result to obtain the first convolution result;

[0262] This convolution operation is implemented using the fourth convolutional layer of the first part. If the fourth convolutional layer of the first part contains a gated convolutional layer, the first convolution result is obtained by performing a convolution on the processing result (output result Out_0) using that gated convolutional layer.

[0263] A22. Perform a downsampling operation on the first convolution result to obtain the first downsampling result;

[0264] This downsampling operation is implemented using the first downsampling module group. If the first downsampling module group contains 4 downsampling modules, then the first downsampling result is obtained by downsampling the first convolution result four times consecutively using the 4 downsampling modules.

[0265] A23. Perform a convolution operation on the first downsampling result to obtain the second convolution result;

[0266] This convolution operation is implemented using the fourth convolutional layer in the second part. If the fourth convolutional layer in the first part contains one gated convolutional layer, the second convolution result is obtained by performing a convolution on the first downsampling result using one gated convolutional layer.

[0267] The second convolution result is an intermediate result produced by the second sub-network and will be input into its downstream self-attention module.

[0268] A23. Input the second convolution result into the self-attention module and output the attention feature information;

[0269] Combination Figure 8A and Figure 8B The schematic diagram of the self-attention module is shown. The specific implementation of step A23 is as follows:

[0270] A231. Use a spatial attention combination layer to extract features from the second convolution result and output spatial attention features.

[0271] A232. Use a channel attention combination layer to extract features from the second convolution result and output channel attention features.

[0272] A232. Using the first summing layer St1, the spatial attention features and the channel attention features are added and fused to obtain attention feature information.

[0273] Furthermore, the specific implementation of step A231 above is as follows:

[0274] A2311. Perform a convolution operation on the second convolution result to obtain the first type of feature, the second type of feature, and the third type of feature;

[0275] This step involves using the first, second, and third convolutional layers to perform convolution on the result of the second convolution, respectively.

[0276] A2312. Based on the first type of features and the second type of features, obtain a spatial attention map;

[0277] This step involves first transposing the first type of feature using the first transpose layer, and then multiplying it with the second type of feature using the second product layer Mu2.

[0278] A2313. Based on the spatial attention map, the third type of feature, and the second convolution result, the spatial attention feature is obtained. This step is achieved by first multiplying the spatial attention map and the third type of feature using the third convolution layer, and then adding and fusing them using the second summing layer St2 and the second convolution.

[0279] Furthermore, the specific implementation of step A232 above is as follows:

[0280] A2321. Transpose the second convolution result to obtain the deformed second convolution.

[0281] The transpose deformation in this step is achieved using a second transpose layer.

[0282] A2322. Multiply the deformed second convolution with the second convolution to obtain the channel attention map;

[0283] The multiplication operation in this step is implemented using the fourth product layer, Mu4.

[0284] A2323. Multiply the channel attention map with the second convolution result to obtain the third multiplication result;

[0285] The multiplication operation in this step is implemented using the A2324 function of the fifth product layer Mu5. The result of the third multiplication is added to the second convolution to obtain the channel attention feature.

[0286] The addition operation in this step is implemented using the third summation layer, St3.

[0287] A24. Perform a convolution operation on the attention feature information obtained through step A23 above to obtain the first encoding result.

[0288] This convolution operation is implemented using the fourth convolutional layer in the third part. If the fourth convolutional layer in the third part contains two gated convolutional layers, the first encoding result is obtained by performing two consecutive convolutions on the attention feature information using these two gated convolutional layers.

[0289] Continue to combine Figures 7A to 7B As shown, the specific implementation process of "inputting the processing result into the second encoder (encoder E2) for encoding to obtain the second encoding result" in step 20312 above is as follows:

[0290] A31. Perform a convolution operation on the processing result (output result Out_0) to obtain the third convolution result;

[0291] This convolution operation is implemented using the fifth convolutional layer of the first part. If the fifth convolutional layer of the first part contains a gated convolutional layer, the first convolution result is obtained by performing a convolution on the processing result using that gated convolutional layer.

[0292] A32. Perform a downsampling operation on the third convolution result to obtain a second downsampling result;

[0293] This downsampling operation is implemented using the second downsampling module group. If the second downsampling module group contains 4 downsampling modules, then the second downsampling result is obtained by downsampling the third convolution result four times consecutively using the 4 downsampling modules.

[0294] A33. Perform a convolution operation on the second downsampling result to obtain the second encoding result.

[0295] This convolution operation is implemented using the fifth convolutional layer of the second part. If the fifth convolutional layer of the second part contains 5 gated convolutional layers, the second encoding result is obtained by performing five consecutive convolutions on the second downsampling result using these 5 gated convolutional layers.

[0296] Continue to combine Figures 7A to 7B As shown, the "concatenation operation of the first encoding result and the second encoding result to obtain the concatenation result" in step 20312 can be achieved using the concatenation layer C. The function of the concatenation layer C is to concatenate the input data according to the channels.

[0297] 20313. Input the splicing result into the decoder in the second sub-network for decoding to obtain the first repair result.

[0298] Continue to combine Figure 7A As shown in Figure B, the specific implementation process of step 20313 is as follows:

[0299] A41. Perform a convolution operation on the splicing result to obtain the fourth convolution result;

[0300] This convolution operation is implemented using the sixth convolutional layer of the first part. If the sixth convolutional layer of the first part contains two gated convolutional layers, the fourth convolution result is obtained by performing two consecutive convolutions on the concatenated result using these two gated convolutional layers.

[0301] A42. Perform an upsampling operation on the fourth convolution result to obtain the first upsampling result;

[0302] This upsampling operation is implemented using the first upsampling module group. If the first upsampling module group contains 4 upsampling modules, then the first upsampling result is obtained by upsampling the fourth convolution result four times consecutively using the 4 upsampling modules.

[0303] A43. Perform a convolution operation on the first upsampling result to obtain the fifth convolution result;

[0304] This convolution operation is implemented using the sixth convolutional layer of the second part. If the sixth convolutional layer of the second part contains a gated convolutional layer, the fifth convolution result is obtained by performing a convolution on the first upsampling result using that gated convolutional layer.

[0305] A44. The fifth convolution result is subjected to a limiting process to obtain the first repair result whose value is limited to a first interval. This limiting process is implemented using a first limiting layer. The first interval range is, for example, the range [-1, 1].

[0306] It should be noted that, in a specific example, all gated convolutions in the first repair network described above include two depthwise separable convolutional layers arranged in parallel, such as... Figure 9A Or as shown in 9B.

[0307] In a specific feasible solution, such as combining Figure 10A and Figure 10B The schematic diagram of the second repair network shown illustrates the implementation process of step 2032, "inputting the first repair result into the second repair network and outputting a second repair result containing the second image," as follows:

[0308] 20321. Perform a convolution operation on the first repair result to obtain the ninth convolution result;

[0309] This convolution operation is implemented using the eighth convolutional layer of the first part. If the eighth convolutional layer of the first part contains 7 convolutional layers, the ninth convolution result is obtained by performing seven consecutive convolutions on the first repair result using the 7 convolutional layers.

[0310] 20322. Perform an upsampling operation on the ninth convolution result to obtain the third upsampling result;

[0311] This upsampling operation is implemented using a third upsampling module group. This third upsampling module group contains at least one upsampling module. If the third upsampling module group contains one upsampling module, then the third upsampling result is obtained by performing an upsampling operation on the third upsampling result using one upsampling module.

[0312] 20323. Perform a convolution operation on the third upsampling result to obtain the tenth convolution result;

[0313] This convolution operation is implemented using the eighth convolutional layer in the second part. If the eighth convolutional layer in the second part contains one convolutional layer, the tenth convolution result is obtained by performing a convolution on the third upsampling result using one convolutional layer.

[0314] 20324. Perform an upsampling operation on the first repair result to obtain a fourth upsampling result;

[0315] This upsampling operation is implemented using a fourth upsampling module group. This fourth upsampling module group contains at least one upsampling module. If the fourth upsampling module group contains one upsampling module, then the first upsampling result is obtained by performing an upsampling operation on the first repair result using one upsampling module.

[0316] 20325. The fourth upsampling result and the tenth convolution result are added and fused to obtain the second repair result.

[0317] This addition and merging operation is implemented using the fourth summing layer, St4.

[0318] In step 204 above, the display method of the second image varies depending on the application scenario. For example, based on the description-related content obtained from the first image in step 201, the display of the second image may, but is not limited to, the following: displaying the second image as an interface background image on the electronic device, or displaying it directly in a conventional manner.

[0319] Furthermore, the method provided in this embodiment may also include the following steps:

[0320] 205. Input the first image and the second image into the discrimination network of the processing model to perform authenticity discrimination and obtain discrimination result information;

[0321] 206. Based on the discrimination result information, the first image, and the second image, determine the loss information;

[0322] 207. Based on the loss information, optimize the model parameters of the image processing model.

[0323] For a detailed description of the specific content included in the above-mentioned discrimination result information, the determination of loss information, and the specific implementation of the optimized image processing model, please refer to the relevant content in the other embodiments mentioned above, which will not be elaborated here.

[0324] Figure 14 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. The application scenario for this method is the aforementioned application scenario three. For example... Figure 14 As shown, the image processing method includes the following steps:

[0325] 301. In response to the interface background change event, determine the first selected image;

[0326] 302. Perform dimensional transformation on the first image to obtain the transformed first image;

[0327] 303. Determine the target restoration region information corresponding to the converted first image;

[0328] 304. Input the target restoration area information and the converted first image into the image processing module deployed on the electronic device for image restoration processing to obtain the second image;

[0329] 305. Display the second image as an interface background image in the electronic device;

[0330] In the image processing model, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer.

[0331] For a detailed description of the implementation of each of the above steps, please refer to the relevant content in other embodiments. Furthermore, in addition to the steps described above, the method provided in this application embodiment may also include other steps. For details of these other steps and their specific implementation, please refer to the relevant content in other embodiments, which will not be repeated here.

[0332] To facilitate understanding of the image processing methods provided above, the following will further combine... Figure 15 Here is a specific application scenario example.

[0333] like Figure 15 The user has a smartphone, and the smartphone's 3D wallpaper includes a dimension conversion module that can transform the dimensions of an image. The smartphone's current wallpaper is an image with a flower (as shown on the right). The user triggers a wallpaper change operation, selecting a 2D image (pict1) with an animal on the smartphone. In response to this wallpaper change operation, the smartphone inputs the 2D image (pict1) to the dimension conversion module. The module performs a dimension upscaling transformation on the 2D image (pict1) and outputs a transformed 3D image (pict2). This 3D image (pict2) contains stretching artifacts (such as black areas) that need to be repaired. At this point, the image processing model deployed on the smartphone is invoked, and the 3D image pict2 is input into the image processing model. If the recognition network in the image processing model (not shown in the figure) can recognize the 3D image pict2, it can output the corresponding binary mask image m2. The white area in the binary mask image m2 represents the target area to be repaired in the 3D image pict2. Further, the 3D image pict2 and the binary mask image m2 are input into the repair combination network in the image processing model. Executing the repair combination network will produce the repaired image pict2' of the 3D image pict2. The repaired image pict2' is then displayed as the interface wallpaper (i.e., background image) on the smartphone screen. Thus, the smartphone interface wallpaper is changed from the original 2D image pict1 with animals to the repaired image pict2'. The changed interface wallpaper is shown as shown on the left side of the smartphone.

[0334] In summary, the technical method provided in this application has the following beneficial effects:

[0335] 1. Existing image restoration models have a large number of parameters and slow inference speed. This application implements all gated convolutions in the image processing model generator network (first restoration network) using depthwise separable convolutions. Since depthwise separable convolutions significantly reduce the number of parameters and computational cost compared to conventional convolutions, this can effectively reduce model parameters and improve inference speed while ensuring image restoration effect, making it more friendly to deployment on mobile terminals.

[0336] 2. The contextual attention module commonly used in existing models contains dynamic convolution, which is difficult to deploy on mobile terminals. This application uses a self-attention module to replace it, which can improve the image restoration effect (such as the image filling quality effect) while achieving the same goal of obtaining global information.

[0337] 3. Existing models, limited by computing power and memory, can only output low-resolution inpainting results, while user input is usually high-resolution images. This application combines a generator network (first inpainting network) and a super-resolution network (second inpainting network) into a single model. This allows the low-resolution inpainting results to be sampled through the super-resolution network, directly outputting inpainting results with a clarity close to that of the high-resolution image, thus alleviating the problem of blurry inpainting effects on high-resolution images caused by the original model.

[0338] This application also provides a training method for an image processing model, which includes a repair ensemble network and a discriminator network. The method is also applied to electronic devices. The training method includes the following steps:

[0339] 401. The discriminant network is used to distinguish between the authenticity of the first sample image and the second sample image to obtain the discrimination result information; the second sample image is generated by the inpainting and combining network through image inpainting processing of the first sample image;

[0340] 402. Determine the first loss function based on the difference between the first sample image and the second sample image;

[0341] 403. Determine the second loss function based on the output discrimination value of the discrimination network for the second image contained in the discrimination result information;

[0342] 404. Based on the discriminant values ​​of the discriminant network for the first image and the output discriminant values ​​for the second image contained in the discriminant result information, determine the third loss function;

[0343] 405. Based on the intermediate judgment values ​​of each intermediate network layer in the discrimination network for the first image and the second image contained in the discrimination result information, determine the fourth loss function;

[0344] 406. Using a pre-trained perceptual loss network, perform content and style difference analysis on the first sample image and the second sample image to obtain the fifth loss function and the sixth loss function;

[0345] 407. Determine the seventh loss function based on the image gradient information of the second sample image;

[0346] 408. Based on the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, the sixth loss function, and the seventh loss function, optimize the network parameters of the repair combination network and the discriminator.

[0347] The first sample image is a real image (the original image without any processing), and the second sample image is a generated image that has been repaired from the first image.

[0348] For a detailed description of steps 401 to 408 above, please refer to the relevant content in other embodiments. Furthermore, in addition to the steps described above, the method provided in this application embodiment may also include other steps. For details of these other steps and their specific implementation, please refer to the relevant content in other embodiments, which will not be repeated here.

[0349] This application also provides an electronic device. For details regarding the specific form of the electronic device, please refer to the relevant content in the other foregoing embodiments. For example, Figure 16A and Figure 16B A schematic diagram of the structure of the electronic device 100 is shown. For example... Figure 16A As shown, the electronic device 100 includes: a memory 121 and a processor 110; wherein,

[0350] The memory 121 is used to store a program; wherein the program includes the program code of the image processing model provided in the present application; the processor 110 is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the various image processing methods provided in the present application using the image processing model.

[0351] Memory 121 (a main memory) can be used to store computer executable program code, including instructions. Memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in memory 121 and / or instructions stored in memory disposed in the processor.

[0352] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0353] The processor can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0354] In addition, the processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0355] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0356] Furthermore, in addition to the functional components described above, such as Figure 16A and Figure 16B As shown, the electronic device 100 also includes: an external memory interface 120, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0357] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0358] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0359] Furthermore, it should be understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0360] Electronic device 100 implements display functions through a GPU (Graphics Processing Unit), a display screen 194, and an application processor. The GPU connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0361] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0362] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0363] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0364] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0365] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0366] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0367] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, can implement one or more steps of any of the above-described visual media search methods.

[0368] Computer-readable storage media can be non-transitory computer-readable storage media, such as ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

[0369] Another embodiment of this application provides a computer program product containing instructions. When the computer program product is executed by a computer, it can perform one or more steps of any of the methods described above.

[0370] In this embodiment, the electronic device, computer-readable storage medium, and computer program product are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0371] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0372] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0373] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0374] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0375] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Applicable to electronic devices, the method includes: Determine the target restoration area information corresponding to the first image; Acquire the image processing model deployed on the electronic device; The target restoration area information and the first image are input into the image processing model for image restoration processing to obtain the second image; The second image is displayed in the electronic device; In the image processing model, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer. Furthermore, if, in response to an interface background change event triggered by the user on the electronic device, the image selected by the user is determined to be the first image, then: the target repair area information is determined by identifying the area to be repaired in the first image after dimensional transformation, and the second image is displayed on the electronic device as the interface background image.

2. The method according to claim 1, characterized in that, The image processing model includes a first repair network and a second repair network; as well as, The target restoration area information and the first image are input into the image processing model for image restoration processing to obtain a second image, including: The target restoration area information and the first image are input into the first restoration network, and the first restoration result is output. The first repair result is input into the second repair network, and a second repair result containing the second image is output.

3. The method according to claim 2, characterized in that, The first repair network includes a first sub-network and a second sub-network with a self-attention module, wherein the second sub-network is connected to the output of the first sub-network; The self-attention module includes a spatial attention combination layer and a channel attention combination layer; an intermediate result generated by the second sub-network during the processing of the output of the first sub-network serves as the input of the self-attention module. The spatial attention combination layer performs feature extraction processing on the intermediate results and outputs spatial attention features. The channel attention combination layer performs feature extraction processing on the intermediate results and outputs channel attention features; The attention feature information output by the self-attention module is obtained by adding and fusing the spatial attention features and the channel attention features.

4. The method according to claim 3, characterized in that, The spatial attention combination layer performs feature extraction processing on the intermediate results and outputs spatial attention features, including: The intermediate results are convolutionally processed to obtain the first type of features, the second type of features, and the third type of features. Based on the first type of features and the second type of features, a spatial attention map is obtained; The spatial attention features are obtained based on the spatial attention map, the third type of features, and the intermediate results.

5. The method according to claim 4, characterized in that, Based on the first type of features and the second type of features, a spatial attention map is obtained, including: The first type of feature is transposed and transformed, and then multiplied with the second type of feature to obtain the first multiplication result; Based on the first multiplication result, the spatial attention map is obtained; And, based on the spatial attention map, the third type of feature, and the intermediate results, the spatial attention features are obtained, including: Multiply the spatial attention map by the third type of feature to obtain the second multiplication result; The spatial attention feature is obtained by adding the second multiplication result to the intermediate result.

6. The method according to claim 3, characterized in that, The channel attention combination layer performs feature extraction processing on the intermediate results and outputs channel attention features, including: The intermediate result is transposed and deformed to obtain the deformed intermediate result; Multiply the deformed intermediate result by the intermediate result to obtain the channel attention map; Multiply the channel attention map by the intermediate result to obtain the third multiplication result; The third multiplication result is added to the intermediate result to obtain the channel attention feature.

7. The method according to any one of claims 3 to 6, characterized in that, The target restoration region information and the first image are input into the first restoration network, and a first restoration result is output, including: The first sub-network is used to process the target repair area and the first image to obtain the processing result; The processing results are respectively input into the first encoder and the second encoder in the second sub-network for encoding, and the first encoding result output by the first encoder and the second encoding result output by the second encoder are concatenated to obtain the concatenated result; wherein, the first encoder includes the self-attention module; The splicing result is input into the decoder in the second sub-network for decoding to obtain the first repair result.

8. The method according to claim 7, characterized in that, The processing result is input into the first encoder for encoding to obtain the first encoded result, including: Perform a convolution operation on the processing result to obtain the first convolution result; Perform a downsampling operation on the first convolution result to obtain the first downsampling result; The first downsampling result is convolved to obtain a second convolution result; the second convolution result is the intermediate result input to the self-attention module. The second convolution result is input into the self-attention module, and the attention feature information is output. The attention feature information is convolved to obtain the first encoding result.

9. The method according to claim 7, characterized in that, The processing result is input into the second encoder for encoding to obtain the second encoded result, including: Perform a convolution operation on the processing result to obtain a third convolution result; The third convolution result is downsampled to obtain the second downsampled result; The second downsampling result is convolved to obtain the second encoding result.

10. The method according to claim 7, characterized in that, The splicing result is input into the decoder in the second sub-network for decoding to obtain the first repair result, including: Perform a convolution operation on the splicing result to obtain the fourth convolution result; An upsampling operation is performed on the fourth convolution result to obtain the first upsampling result; Perform a convolution operation on the first upsampling result to obtain the fifth convolution result; The fifth convolution result is subjected to a limit processing to obtain the first repair result whose value is limited to the first interval range.

11. The method according to claim 7, characterized in that, The first sub-network is used to process the target restoration region and the first image to obtain a processing result, including: A convolution operation is performed on the target repair region and the first image to obtain the sixth convolution result; The sixth convolution result is downsampled to obtain the third downsampled result; Perform a convolution operation on the third downsampling result to obtain the seventh convolution result; An upsampling operation is performed on the seventh convolution result to obtain a second upsampling result; Perform a convolution operation on the second upsampling result to obtain the eighth convolution result; The eighth convolution result is subjected to a limit processing to obtain the processing result whose value is limited to the range of the second interval.

12. The method according to any one of claims 2 to 6, characterized in that, In the first repair network, some convolutional layers are gated convolutions, which include two depth-separable convolutional layers arranged in parallel.

13. The method according to any one of claims 2 to 6, characterized in that, The first restoration result is input into the second restoration network, and a second restoration result containing the second image is output, including: Perform a convolution operation on the first repair result to obtain the ninth convolution result; An upsampling operation is performed on the ninth convolution result to obtain the third upsampling result; Perform a convolution operation on the third upsampling result to obtain the tenth convolution result; An upsampling operation is performed on the first repair result to obtain a fourth upsampling result; The fourth upsampling result is added to and fused with the tenth convolution result to obtain the second repair result.

14. An image processing method, characterized in that, Applicable to electronic devices, the method includes: In response to a background change event, determine the first image selected; Perform dimensional transformation on the first image to obtain the transformed first image; Determine the target restoration region information corresponding to the converted first image; The target restoration area information and the converted first image are input into the image processing model deployed on the electronic device for image restoration processing to obtain the second image; The second image is displayed as the interface background image in the electronic device; In the image processing model, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer.

15. An image processing model, characterized in that, include: The first repair network is used to perform image repair processing on the first image based on the target repair region information corresponding to the first image, and output the first repair result; The second repair network is connected to the output of the first repair network and is used to process the first repair result and output a second repair result containing the second image. In the first repair network, some convolutional layers are gated convolutional layers. The gated convolutional layer includes multiple depthwise separable convolutional layers arranged in parallel and a first multiplication layer. The first multiplication layer is used to multiply the outputs of the multiple depthwise separable convolutional layers to obtain the output result of the gated convolutional layer. The image processing model is deployed on an electronic device; and if, in response to an interface background change event triggered by the user on the electronic device, the image selected by the user is determined to be the first image, then: the target repair area information is determined by identifying the area to be repaired in the first image after dimensional transformation, and the second image is displayed as the interface background image on the electronic device.

16. The image processing model according to claim 15, characterized in that, The first repair network includes a first sub-network and a second sub-network with a self-attention module, wherein the second sub-network is connected to the output of the first sub-network; In the first repair network, all convolutional layers that do not belong to the self-attention module are gated convolutional layers. The self-attention module includes a spatial attention combination layer, a channel attention combination layer, and a first summation layer; an intermediate result generated by the second sub-network during the processing of the output of the first sub-network serves as the input to the self-attention module. The spatial attention combination layer is used to perform feature extraction processing on the intermediate results and output spatial attention features; The channel attention combination layer is used to perform feature extraction processing on the intermediate results and output channel attention features; The first summing layer is used to add and fuse the spatial attention features and the channel attention features to output attention feature information.

17. The image processing model according to claim 16, characterized in that, The spatial attention combination layer includes: The first convolutional layer is used to extract features from the intermediate results and output the first type of features. The second convolutional layer is used to extract features from the intermediate results and output the second type of features; The third convolutional layer is used to extract features from the intermediate results and output the third type of features; The first transpose layer is connected to the output of the first convolutional layer and is used to transpose and deform the first type of features and output the transposed and deformed first type of features. The second product layer is connected to the outputs of the first transposed layer and the second convolutional layer, and is used to multiply the transposed first type of feature with the second type of feature to output the first multiplication result; The third product layer, connected to the outputs of the second product layer and the third convolutional layer, is used to multiply the spatial attention map obtained based on the first product result with the third type of feature to obtain the second product result; The second summing layer, connected to the output of the third product layer, is used to add the second multiplication result to the intermediate result and output the spatial attention feature.

18. The image processing model according to claim 16, characterized in that, The channel attention combination layer includes: The second transpose layer is used to transpose and deform the intermediate result, and output the deformed intermediate result. The fourth product layer, connected to the output of the second transposed layer, is used to multiply the deformed intermediate result with the intermediate result to output a channel attention map; The fifth product layer, connected to the output of the fourth product layer, is used to multiply the channel attention map with the intermediate result and output the third product result. The third summation layer, along with the output of the fifth product layer, is used to add the third multiplication result to the intermediate result and output the channel attention feature.

19. The image processing model according to any one of claims 16 to 18, characterized in that, The second sub-network includes: The first encoder is used to encode the processing result output by the first sub-network and output the first encoded result; The second encoder is used to encode the processing result output by the first sub-network and output the second encoded result. The splicing layer is connected to the output terminals of the first encoder and the second encoder, and is used to splice the first encoding result and the second encoding result, and output the splicing result; The decoder is connected to the output of the splicing layer and is used to decode the splicing result and output the first repair result.

20. The image processing model according to claim 19, characterized in that, The first encoder includes: multiple fourth convolutional layers, a first downsampling module group, and the self-attention module; wherein, The first part of the fourth convolutional layer and the second part of the fourth convolutional layer are connected by the first downsampling module group, and the remaining third part of the fourth convolutional layer and the second part of the fourth convolutional layer are connected by the self-attention module; the output of the third part of the fourth convolutional layer is connected to the input of the splicing layer. The first downsampling module group contains multiple downsampling modules that perform downsampling operations in sequence.

21. The image processing model according to claim 19, characterized in that, The second encoder includes: multiple fifth convolutional layers and a second downsampling module group; wherein, The second downsampling module group connects the first part of the fifth convolutional layer and the remaining second part of the fifth convolutional layer; the output of the second part of the fifth convolutional layer is connected to the input of the splicing layer. The second downsampling module group contains multiple downsampling modules that perform downsampling operations in sequence.

22. The image processing model according to claim 19, characterized in that, The decoder includes: multiple sixth convolutional layers, a first upsampling module group, and a first limiting layer; wherein... The first part of the sixth convolutional layers is connected between the splicing layer and the first upsampling module group, and the remaining second part of the sixth convolutional layers is connected between the first upsampling module group and the first limit layer. The first limiting layer is used to limit the output of the sixth convolutional layer in the second part to a first interval range; The first upsampling module group contains multiple upsampling modules that perform upsampling operations in sequence.

23. The image processing model according to any one of claims 16 to 18, characterized in that, The first sub-network includes: multiple seventh convolutional layers, a third downsampling module group, a second upsampling module group, and a second limit layer; wherein, The third downsampling module group is connected between the first part of the seventh convolutional layer and the second part of the seventh convolutional layer, and the second upsampling module group is connected between the remaining third part of the seventh convolutional layer and the second part of the seventh convolutional layer. The output of the third part of the seventh convolutional layer is connected to the second limit layer. The second limiting layer is used to limit the output of the seventh convolutional layer in the third part to a second interval range; The third downsampling module group contains multiple downsampling modules that perform downsampling operations in sequence; The second upsampling module group contains multiple upsampling modules that perform upsampling operations in sequence.

24. The image processing model according to any one of claims 15 to 18, characterized in that, The gated convolution includes a first depthwise separable convolutional layer, a second depthwise separable convolutional layer, a third limit layer, an activation layer, and a first multiplication layer; The output of the first depth-separable convolutional layer is connected to the third limiting layer, and the output of the second depth-separable convolutional layer is connected to the activation layer; The output of the third limiting layer and the output of the activation layer are both connected to the first product layer; The third limiting layer is used to limit the first depth convolutional feature output by the first depth separable convolutional layer, and output the first depth convolutional feature after limiting; the value of the first depth convolutional feature after limiting is within the third interval range. The activation layer is used to activate the second depth convolutional features output by the second depth separable convolutional layer, and output the activated second depth convolutional features. The first multiplication layer is used to multiply the first depthwise convolutional feature after being limited by the limit by the second depthwise convolutional feature after activation, and output a fourth multiplication result; the fourth multiplication result is the output result of the gated convolution.

25. The image processing model according to any one of claims 15 to 18, characterized in that, The second repair network includes: multiple eighth convolutional layers, a third upsampling module group, a fourth upsampling module group, and a fourth summing layer; The third upsampling module group is connected between the first part of the eighth convolutional layer and the remaining second part of the eighth convolutional layer; The output of the eighth convolutional layer in the second part and the output of the fourth upsampling module are both connected to the fourth summing layer; the input of the fourth upsampling module is connected to the input of the eighth convolutional layer in the first part. The third upsampling module group and the fourth upsampling module group each contain at least one upsampling module.

26. An image processing model, characterized in that, include: A repair combinatorial network is provided, wherein the repair combinatorial network includes a self-attention module; wherein... The insulation combination network is used to perform image insulation processing on the first image based on the target insulation region information corresponding to the first image, and output a second image; wherein, an intermediate result generated during the processing is the input of the self-attention module; The self-attention module includes a spatial attention combination layer, a channel attention combination layer, and a first summation layer; The spatial attention combination layer is used to perform feature extraction processing on the intermediate results and output spatial attention features; The channel attention combination layer is used to perform feature extraction processing on the intermediate results and output channel attention features; The first summing layer is used to add and fuse the spatial attention features and the channel attention features to output attention feature information; The spatial attention combination layer comprises three convolutional layers arranged in parallel, a second multiplication layer, a third multiplication layer, and a second summing layer. The inputs to the three convolutional layers are the intermediate results, and the outputs of the first and second convolutional layers are connected to the input of the second multiplication layer, and the outputs of the third convolutional layer and the second multiplication layer are connected to the input of the third convolutional layer. The inputs of each of the three convolutional layers and the output of the third convolutional layer are all connected to the second summing layer. Furthermore, the image processing model is deployed on an electronic device; if, in response to an interface background change event triggered by the user on the electronic device, the image selected by the user is determined as the first image, then: the target repair area information is determined by identifying the area to be repaired in the first image after dimensional transformation, and the second image will be displayed as the interface background image on the electronic device.

27. The image processing model according to claim 26, characterized in that, The spatial attention combination layer further includes a first transpose layer connected between the first convolutional layer and the second product layer, used to transpose and deform the convolution result output by the first convolutional layer.

28. An electronic device, characterized in that, include: Memory and processor; among which, The memory is used to store a program; wherein the program includes program code of the image processing model according to any one of claims 15 to 25, or program code of the image processing model according to claim 26 or 27. The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the image processing method according to any one of claims 1 to 13, or the steps of the image processing method according to claim 14, using the image processing model.

Images