Image restoration method and device, electronic equipment and computer readable storage medium

Abstract

The application provides an image restoration method and device based on artificial intelligence, electronic equipment and computer readable storage medium; the method comprises the following steps: obtaining an image to be restored; performing degradation removal processing on the image to be restored to obtain a latent code feature corresponding to the image to be restored and at least two spatial features of different scales; performing mapping processing on the latent code feature based on at least two latent code networks to obtain at least two latent codes corresponding to the at least two latent code networks one by one; and based on the at least two spatial features of different scales, modulating the prior feature corresponding to the at least two latent codes to obtain a restored image corresponding to the image to be restored. Through the application, the authenticity and credibility of image restoration can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an image restoration method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0003] The purpose of image restoration in related technologies is to recover high-quality images from low-quality images. Low-quality images can be images that have undergone unknown degradation processing, such as low resolution, noise, blurring, compression artifacts, etc. When applied to real-world scenarios, the restoration process becomes more complicated because the low-quality images to be restored are obtained through complex degradation processing and contain diverse content. Currently, there is no effective method in related technologies to recover high-quality images with authenticity and credibility from low-quality images. Summary of the Invention

[0004] This application provides an image restoration method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence, which can improve the realism and reliability of image restoration.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image restoration method based on artificial intelligence, including:

[0007] Obtain the image to be recovered;

[0008] The image to be restored is subjected to degradation removal processing to obtain the hidden code features corresponding to the image to be restored and at least two spatial features of different scales;

[0009] The implicit code features are mapped based on at least two implicit code networks to obtain at least two implicit codes that correspond one-to-one with the at least two implicit code networks.

[0010] Based on the spatial features at least two different scales, the prior features corresponding to the at least two hidden codes are modulated to obtain the restored image corresponding to the image to be restored.

[0011] This application provides an image restoration device based on artificial intelligence, including:

[0012] The acquisition module is used to acquire the image to be recovered;

[0013] The degradation removal module is used to perform degradation removal processing on the image to be restored to obtain the hidden code features corresponding to the image to be restored and at least two spatial features of different scales.

[0014] The mapping module is used to perform mapping processing on the hidden code features based on at least two hidden code networks to obtain at least two hidden codes that correspond one-to-one with the at least two hidden code networks.

[0015] The generation module is used to modulate the prior features corresponding to the at least two hidden codes based on the spatial features at the at least two different scales, so as to obtain the restored image corresponding to the image to be restored.

[0016] In the above scheme, obtaining the first loss function for loss determination of the at least two second image samples includes:

[0017] Determine a first difference between the real image sample corresponding to the first pattern sample and each of the at least two second image samples;

[0018] The first difference corresponding to each of the second image samples is summed to obtain the first loss function.

[0019] In the above scheme, obtaining the second loss function for loss determination of the third image sample includes:

[0020] Determine a second difference between the real image sample corresponding to the first pattern sample and the third image sample;

[0021] The trained image recognition model is used to extract features from the real image samples to obtain a first real feature map, and the third image samples are used to extract features to obtain a first predicted feature map.

[0022] Determine the third difference between the first true feature map and the first predicted feature map;

[0023] The third image sample is discriminated by a discriminant network to obtain a discrimination result corresponding to the third image sample, and the discrimination result is substituted into a first logic function to obtain a first logic processing result;

[0024] The image restoration network is coupled with a trained generative network, and the discriminative network is trained alongside the generative network.

[0025] The local discriminant network is used to discriminate the corresponding local region of the third image sample to obtain the local discriminant result of the third image sample. The local discriminant result is then substituted into the second logic function to obtain the second logic processing result.

[0026] The local discrimination network is trained for the corresponding local region.

[0027] The local discriminant network is used to extract features from the real image samples to obtain a second real feature map, and the local discriminant network is used to extract features from the third image samples to obtain a second predicted feature map.

[0028] Determine the fourth difference between the second true feature map and the second predicted feature map;

[0029] The real image samples are processed by a trained type recognition network to obtain the real category features corresponding to the real image samples, and the third image sample is processed by feature extraction to obtain the predicted category features.

[0030] Determine the fifth difference between the true category feature and the predicted category feature;

[0031] The second difference, the third difference, the fourth difference, the fifth difference, the first logical processing result, and the second logical processing result are fused together to obtain the second loss function.

[0032] In the above scheme, the degradation removal process is performed through the degradation removal network; the degradation removal network includes N cascaded downsampling networks, one convolutional network, and M cascaded upsampling networks, where N is an integer greater than or equal to 2, and M is equal to N; the degradation removal module is further configured to: perform downsampling processing on the image to be restored through the N cascaded downsampling networks to obtain a downsampling processing result; perform convolution processing on the downsampling processing result through the convolutional network to obtain the hidden code features corresponding to the image to be restored; and perform upsampling processing on the hidden code features through the M cascaded upsampling networks to obtain M spatial features of different scales corresponding to the image to be restored.

[0033] In the above solution, the degradation removal module is further configured to: perform downsampling on the input of the n-th downsampling network through the n-th downsampling network in N cascaded downsampling networks, and transmit the n-th downsampling result output by the n-th downsampling network to the (n + 1)-th downsampling network to continue downsampling, so as to obtain the corresponding (n + 1)-th downsampling result; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th downsampling network is the image to be restored. When n takes the value of 2 ≤ n < N, the input of the n-th downsampling network is the (n - 1)-th downsampling result output by the (n - 1)-th downsampling network.

[0034] In the above solution, the degradation removal module is further configured to: perform upsampling on the input of the m-th upsampling network through the m-th upsampling network in M cascaded upsampling networks, so as to obtain the spatial feature corresponding to the m-th upsampling network, and transmit the spatial feature corresponding to the m-th upsampling network to the (m + 1)-th upsampling network to continue upsampling, so as to obtain the spatial feature corresponding to the (m + 1)-th upsampling network; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the m-th sampling network is the latent code feature. When m takes the value of 2 ≤ m < M, the input of the m-th upsampling network is the (m - 1)-th upsampling result output by the (m - 1)-th upsampling network.

[0035] In the above solution, before performing upsampling on the input of the m-th upsampling network through the m-th upsampling network in M cascaded upsampling networks to obtain the spatial feature corresponding to the n-th upsampling network, the degradation removal module is further configured to: when m takes the value of 1, splice the output of the M-th downsampling network and the output of the convolutional network, and use the splicing result as the input of the m-th upsampling network to replace using the output of the convolutional network as the input of the m-th upsampling network; when m takes the value of 2 ≤ m < M, splice the output of the (M + 1 - m)-th downsampling network and the output of the (m - 1)-th upsampling network, and use the splicing result as the input of the m-th upsampling network to replace using the spatial feature corresponding to the (m - 1)-th upsampling network as the input of the m-th upsampling network.

[0036] In the above scheme, the hidden code network includes shared convolutional layers, shared linear layers, and non-shared linear layers; the mapping module is further configured to: perform feature extraction processing on the hidden code features through the shared convolutional layers of the hidden code network to obtain a shared feature extraction result corresponding to the hidden code features; perform linear transformation processing on the shared feature extraction result through the shared linear layers of the hidden code network to obtain a shared processing result corresponding to the hidden code features; and perform linear transformation processing on the shared processing result through the non-shared linear layers of the hidden code network to obtain the hidden code corresponding to the hidden code network.

[0037] In the above scheme, the modulation processing is implemented through an image restoration network, which includes R intermediate networks and a tail network; the R intermediate networks are located on the input side of the image restoration network, and the tail network is located on the output side of the image restoration network; wherein R is an integer greater than or equal to 1; the generation module is further configured to: perform modulation processing based on corresponding spatial features on the prior features of the corresponding hidden codes of the R intermediate networks through the R intermediate networks of the image restoration network to obtain R intermediate features that correspond one-to-one with the R intermediate networks; and perform modulation processing on the intermediate features output by adjacent intermediate networks through the tail network of the image restoration network on the corresponding hidden codes and corresponding spatial features to obtain the restored image corresponding to the image to be restored.

[0038] In the above scheme, the generation module is further configured to: modulate the prior features of the corresponding hidden code of the r-th intermediate network based on the spatial features of the r-th intermediate network through the r-th intermediate network in the R cascaded intermediate networks to obtain the r-th intermediate feature corresponding to the r-th intermediate network, and transmit the intermediate feature output by the r-th intermediate network to the (r+1)-th intermediate network for further modulation processing; wherein, r is an integer variable whose value increases from 1. When r is 1, the prior features of the corresponding hidden code of the r-th intermediate network are obtained based on the r-th hidden code of the r-th intermediate network. When r is 2≤r≤R-1, the prior features of the corresponding hidden code of the r-th intermediate network are obtained based on the intermediate features output by the (r-1)-th intermediate network and the r-th hidden code corresponding to the r-th intermediate network; and modulate the R-th intermediate feature output by the R-th intermediate network and the tail hidden code corresponding to the tail network based on the spatial features of the tail network through the tail generation network.

[0039] In the above scheme, the generation module is further configured to: generate tail prior features corresponding to the tail hidden code and the Rth intermediate feature through the tail generation network of the tail network; perform spatial modulation processing on the tail prior features through the spatial features corresponding to the tail network; wherein the modulation processing result is the restored image corresponding to the image to be restored.

[0040] In the above scheme, the generation module is further configured to: when r is 1, perform the following processing: generate the r-th prior feature corresponding to the r-th hidden code through the intermediate generation network of the r-th intermediate network; perform spatial modulation processing on the r-th prior feature through the spatial features corresponding to the r-th intermediate network, and use the modulation processing result as the r-th intermediate feature corresponding to the r-th intermediate network; when r takes the value 2≤r≤R-1, perform the following processing: generate the r-th prior feature corresponding to the r-th hidden code and the r-th intermediate feature through the generation network of the r-th intermediate network; perform spatial modulation processing on the r-th prior feature through the spatial features corresponding to the r-th intermediate network, and use the modulation processing result as the r-th intermediate feature corresponding to the r-th intermediate network.

[0041] In the above scheme, the generation module is further configured to: perform convolution processing on the spatial features corresponding to the r-th intermediate network to obtain the spatial modulation affine coefficients and spatial modulation bias coefficients corresponding to the spatial features; randomly select all channels of the r-th prior feature, select half of the total number of channels as the hold channels, and use the remaining channels as modulation channels; multiply the channel features of the modulation channels of the r-th prior feature with the spatial modulation affine coefficients, and add the multiplication result with the spatial modulation bias coefficients; and concatenate the addition result with the channel features of the hold channels of the r-th prior feature.

[0042] In the above scheme, the image restoration method is implemented through a neural network model, which includes a degradation removal network and an image restoration network. The device further includes a training module, configured to: before performing degradation removal processing on the image to be restored to obtain the hidden code features and at least two spatial features of different scales corresponding to the image to be restored, perform degradation removal processing on a first image sample through the degradation removal network to obtain at least two second image samples corresponding to the first image sample; wherein, the at least two second image samples are output one-to-one through at least two upsampling networks in the degradation removal network; perform image restoration processing on the first image sample through the degradation removal network and the image restoration network to obtain a third image sample corresponding to the first image sample; obtain a first loss function for loss judgment on the at least two second image samples and a second loss function for loss judgment on the third image sample; and train the neural network model based on the first loss function and the second loss function.

[0043] In the above scheme, the training module is further configured to: determine the first difference between the real image sample corresponding to the first image sample and each of the at least two second image samples, and sum the first difference corresponding to each second image sample to obtain the first loss function.

[0044] In the above scheme, the training module is further configured to: determine a second difference between the real image sample corresponding to the first image sample and the third image sample; perform feature extraction processing on the real image sample using a trained image recognition model to obtain a first real feature map, and perform feature extraction processing on the third image sample to obtain a first predicted feature map; determine a third difference between the first real feature map and the first predicted feature map; perform discrimination processing on the third image sample using a discriminant network to obtain a discrimination result corresponding to the third image sample, and substitute the discrimination result into a first logical function to obtain a first logical processing result; wherein, the image restoration network is coupled with a trained generator network, and the discriminant network is trained alongside the generator network; perform discrimination processing on the corresponding local region of the third image sample using a local discriminant network to obtain a local discrimination result corresponding to the third image sample, and substitute the local discrimination result into a first logical function to obtain a first logical processing result; wherein, the image restoration network is coupled with a trained generator network, and the discriminant network is trained alongside the generator network; and the local region of the third image sample is discriminated using a local discriminant network to obtain a local discrimination result corresponding to the third image sample, and the local region of the third image sample is discriminated using a local discriminant network. The partial discrimination result is substituted into the second logic function to obtain the second logic processing result; wherein, the local discrimination network is trained for the corresponding local region; the local discrimination network is used to perform feature extraction processing on the real image sample to obtain the second real feature map, and the third image sample is used to perform feature extraction processing to obtain the second predicted feature map; a fourth difference between the second real feature map and the second predicted feature map is determined; the trained type recognition network is used to perform feature extraction processing on the real image sample to obtain the real category feature corresponding to the real image sample, and the third image sample is used to perform feature extraction processing to obtain the predicted category feature; a fifth difference between the real category feature and the predicted category feature is determined; the second difference, the third difference, the fourth difference, the fifth difference, the first logic processing result, and the second logic processing result are fused to obtain the second loss function.

[0045] This application provides an electronic device, including:

[0046] Memory, used to store executable instructions;

[0047] The processor, when executing executable instructions stored in the memory, implements the AI-based image restoration method provided in the embodiments of this application.

[0048] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the artificial intelligence-based image restoration method provided in this application.

[0049] The embodiments of this application have the following beneficial effects:

[0050] Using the prior features of the image to be restored as a reference, which correspond to the image to be restored, the restoration is highly targeted, and the prior features encapsulate rich prior knowledge. By combining the prior features with multi-scale spatial features from the image to be restored, the restored image corresponding to the image to be restored is obtained, which greatly improves the credibility and authenticity of the image to be restored. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the structure of the artificial intelligence-based image restoration system provided in the embodiments of this application;

[0052] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0053] Figure 3A This is a schematic diagram of the neural network model of the image restoration method based on artificial intelligence provided in the embodiments of this application;

[0054] Figure 3B This is a schematic diagram of the degradation removal network model of the image restoration method based on artificial intelligence provided in this application embodiment;

[0055] Figure 3C This is a schematic diagram illustrating the inference of the intermediate modulation network and the tail modulation network of the AI-based image restoration method provided in this application embodiment;

[0056] Figure 3D This is a schematic diagram of the intermediate generation network and the tail generation network of the image restoration method based on artificial intelligence provided in the embodiments of this application;

[0057] Figures 4A-4E This is a flowchart illustrating the image restoration method based on artificial intelligence provided in an embodiment of this application;

[0058] Figure 5 This is a modulation schematic diagram of the image restoration method based on artificial intelligence provided in the embodiments of this application;

[0059] Figure 6 These are comparison images showing the effects of the AI-based image restoration method provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0062] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0065] 1) Generative Adversarial Networks (GANs): GANs learn by having two neural networks compete against each other. A GAN consists of a generator network and a discriminator network. The generator network randomly samples from the latent space as input, and its output should try to imitate the real samples in the training set as much as possible. The discriminator network takes real samples or the output of the generator network as input, and its goal is to distinguish the output of the generator network from the real samples as much as possible. The generator network, on the other hand, tries to deceive the discriminator network as much as possible. The two networks compete against each other and constantly adjust their parameters. The ultimate goal is to make it impossible for the discriminator network to determine whether the output of the generator network is real.

[0066] 2) Image to be restored: Image obtained through arbitrary degradation processing, such as yellowed old photos, noisy images, low-resolution images, etc. The image to be restored can be a face image, such as a human face image and an animal face image. The image to be restored can also be a non-face image, such as a landscape image, a building image, etc.

[0067] 3) Restored image: This is an image with authenticity and credibility obtained by repairing the image to be restored. The content of the restored image is the same as that of the image to be restored, and the image quality of the restored image is higher than that of the image to be restored.

[0068] 4) Generative Facial Prior (GFP): refers to the distributed prior information learned by the generative model, such as the facial prior information learned by the face generation model during training.

[0069] The goal of image inpainting in related technologies is to recover high-quality images from low-quality images. Low-quality images can be those that have undergone unknown degradation processes, such as low resolution, noise, blurring, compression artifacts, etc. When applied to real-world scenarios, taking a face image as an example, the low-quality image to be restored becomes even more complex because it has undergone complex degradation processes and contains diverse poses and expressions. Related technologies utilize specific geometric facial prior information for face restoration, such as facial landmarks, face segmentation maps, or face component heatmaps. This geometric facial prior information is crucial for recovering accurate face shapes and details. However, this geometric facial prior information is usually derived from low-quality images... The estimated geometric facial prior information from the image to be restored is difficult to obtain accurately, thus limiting the image restoration effect. In related technologies, image restoration can also be performed by finding reference facial prior information, such as using high-quality reference facial images or high-definition facial component dictionaries as reference facial prior information. However, these reference facial prior information often need to belong to the same person, which greatly limits the application scenarios of image restoration. In many cases, high-definition reference facial prior information for the same person is difficult to obtain, and facial component dictionaries need to be constructed in advance with high-definition facial component prior information. The limited dictionary size limits the diversity and richness of the image restoration results. In summary, the image restoration results of related technologies are not satisfactory in practical application scenarios.

[0070] This application provides an image restoration method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence. It improves the realism and reliability of image restoration without relying on estimating geometric prior information from a low-quality image to be restored, nor requiring the acquisition of other difficult-to-obtain reference prior information. The following describes exemplary applications of the electronic device provided in this application. The electronic device can be various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, smart home devices like smart TVs, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or it can be a server. The following describes exemplary applications when the electronic device is implemented as a server.

[0071] See Figure 1 , Figure 1This is a schematic diagram of the architecture of an AI-based image restoration system provided in this application embodiment. To support an image processing application, terminal 400 connects to server 200 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Terminal 400 receives the image to be restored input by the user and sends the received image to be restored to server 200. Server 200 calls an image restoration network to perform image restoration processing on the image to be restored, obtains the restored image, and returns it to terminal 400. Terminal 400 then presents the restored image.

[0072] exist Figure 1 The example below illustrates how terminal 400 and server 200 work together to restore an image. Understandably, as an alternative, terminal 400 can rely on its own capabilities to restore the image.

[0073] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0074] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, as... Figure 1 As an alternative, the terminal 400 can rely on its own capabilities to complete image restoration. Figure 2 The terminal 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0075] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0076] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0077] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0078] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0079] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0080] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0081] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0082] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0083] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0084] In some embodiments, the AI-based image restoration device provided in this application can be implemented in software. Figure 2 An AI-based image restoration device 455 stored in memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: acquisition module 4551, degradation removal module 4552, mapping module 4553, generation module 4554, and training module 4555. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0085] The image restoration method based on artificial intelligence provided in this application will be described in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application. The image restoration method based on artificial intelligence provided in the embodiments of this application can be completed independently by the terminal 400 mentioned above or jointly by the terminal 400 and the application server 200 mentioned above.

[0086] The structure of the neural network model used in the AI-based image restoration method provided in this application embodiment is described in detail below.

[0087] See Figure 3A , Figure 3AThis is a schematic diagram of the neural network model of the image restoration method based on artificial intelligence provided in this application embodiment. The image restoration method based on artificial intelligence provided in this application embodiment is implemented based on a neural network model. The neural network model consists of a degradation removal network and an image restoration network. The image restoration network includes R intermediate networks and a tail network, where R is an integer greater than or equal to 1. Each intermediate network includes a corresponding intermediate generator network and a corresponding intermediate modulation network. The tail network includes a tail generator network and a corresponding tail modulation network. The generator network consisting only of the intermediate generator network and the tail generator network is a pre-trained generator network. If the image restoration task is a face image restoration task, the generator network is a trained face generation model, such as the generator network in Style Generative Adversarial Networks (GAN). The intermediate generation network and the tail generation network are both components of the generation network, together forming the generation network. The intermediate modulation network and the tail generation network can be understood as the connection between each intermediate generation network, the tail generation network and the degradation removal network in the generation network. In addition to this connection, each intermediate generation network and the tail generation network in the generation network are also connected to the degradation removal network through the hidden code network.

[0088] See Figure 3B , Figure 3B This is a schematic diagram of the degradation removal network model of the AI-based image restoration method provided in this application embodiment. The degradation removal network can be a U-net network structure, i.e., a U-shaped network structure, composed of N downsampling networks, M upsampling networks, and one convolutional network. M and N are both integers greater than or equal to 1, and M and N have the same value, thus forming a U-shaped network structure. Each downsampling network is a residual network. The input undergoes two processes and is then fused. One process is convolution and activation, and the other process is convolution. Each upsampling network is a residual network. The input undergoes two processes and is then fused. One process is convolution and activation, and the other process is convolution. One process is convolution. The input is processed by two methods and then fused. After passing through multiple downsampling networks and convolutional networks, the output is the latent code feature. The latent code feature is mapped through different linear layers to obtain different latent codes. Multiple upsampling networks can output multiple spatial features respectively. Each spatial feature is processed by convolutional layers, activation functions, and convolutional layers to obtain spatial modulation coefficients (spatial modulation affine coefficients α and spatial modulation bias coefficients β). Furthermore, each upsampling network input can also output an intermediate image. Each output intermediate image can be compared with the labeled real image for loss comparison, thereby supervising the learning process of the entire upsampling network through clean features of the real image.

[0089] See Figure 3C , Figure 3C This is a schematic diagram of the inference of the intermediate modulation network and the tail modulation network of the image restoration method based on artificial intelligence provided in this application embodiment. In the intermediate modulation network and the tail modulation network, the feature F is segmented based on the channel dimension. The features in half of the randomly selected channels are retained and output. The features in the other half of the channels are modulated based on the spatial modulation coefficient, which is equivalent to performing spatial feature transformation. The features are multiplied by the spatial modulation affine coefficient α and then added to the spatial modulation bias coefficient β. Then, the features of all channels are concatenated to obtain the modulation result. The modulation result is then passed to the style convolutional layer in the generator network.

[0090] See Figure 3D , Figure 3D This is a schematic diagram of the intermediate generation network and the tail generation network of the image restoration method based on artificial intelligence provided in this application embodiment. The intermediate generation network is obtained by combining the style convolutional layer and the style upsampling layer. The intermediate generation network is connected to the intermediate modulation network. The intermediate modulation network of the last intermediate network is connected to the tail generation network (a combination of style convolutional layer and style upsampling layer) of the tail network. The tail generation network is connected to the tail modulation network. The output of the tail modulation network is input into the style convolutional layer and the image transformation layer to output the final restored image.

[0091] The training of the neural network model used in the AI-based image restoration method provided in the embodiments of this application is described in detail below.

[0092] See Figure 4A , Figure 4A This is a flowchart illustrating the artificial intelligence-based image restoration method provided in this application embodiment. The following is a summary of the process. Figure 4A Steps 101-104 shown are explained.

[0093] In step 101, the first image sample is subjected to degradation removal processing by a degradation removal network to obtain at least two second image samples corresponding to the first image sample.

[0094] As an example, see Figure 3A At least two second image samples are output one-to-one through at least two upsampling networks in the degradation removal network; the image restoration method is implemented by a neural network model, which includes a degradation removal network, a set number of hidden code networks, and an image restoration network; the degradation removal network includes several upsampling networks, convolutional networks, and several downsampling networks, and the image restoration network includes several intermediate networks and a tail network, with the total number of intermediate networks and tail networks being a set number.

[0095] In step 102, the first image sample is processed by a degradation removal network and an image restoration network to obtain a third image sample corresponding to the first image sample.

[0096] As an example, the third image sample is the restoration result corresponding to the first image sample. For example, the first image sample is an image that has been degraded, which can be a face image, a building image, an animal image, etc. After the first image sample is processed by a neural network model to restore the image, the third image sample is obtained, which is the restored image corresponding to the first image sample.

[0097] In step 103, a first loss function for loss determination of at least two second image samples and a second loss function for loss determination of a third image sample are obtained.

[0098] In some embodiments, obtaining the first loss function for loss judgment of at least two second image samples in step 103 can be achieved by the following technical solution: determining the first difference between the real image sample corresponding to the first image sample and each of the at least two second image samples; summing the first difference corresponding to each second image sample to obtain the first loss function.

[0099] As an example, each upsampling network outputs an intermediate image, which is the second image sample. Different upsampling networks will output different second image samples. For each output second image sample, it is compared with the real image sample to obtain multiple first differences. The sum of multiple first differences is used as the first loss function, so that multiple intermediate layers of the degradation removal network can learn clean features from the real image. The real image sample is the original image of the first image sample. The first image sample can only be obtained by performing arbitrary degradation processing on the original image.

[0100] In some embodiments, obtaining the second loss function for loss judgment on the third image sample in step 103 can be achieved through the following technical solution: determining the second difference between the real image sample corresponding to the first image sample and the third image sample; performing feature extraction processing on the real image sample using a trained image recognition model to obtain a first real feature map, and performing feature extraction processing on the third image sample to obtain a first predicted feature map; determining the third difference between the first real feature map and the first predicted feature map; performing discrimination processing on the third image sample using a discriminant network to obtain the discrimination result corresponding to the third image sample, and substituting the discrimination result into the first logic function to obtain the first logic processing result; wherein, the image restoration network is coupled with a trained generator network, and the discriminant network is trained alongside the generator network; performing discrimination processing on the corresponding local region of the third image sample using a local discriminant network to obtain the first logic processing result. The local discrimination result corresponding to the third image sample is substituted into the second logical function to obtain the second logical processing result; wherein, the local discrimination network is trained for the corresponding local region; the local discrimination network is used to perform feature extraction processing on the real image sample to obtain the second real feature map, and the third image sample is used to perform feature extraction processing to obtain the second predicted feature map; the fourth difference between the second real feature map and the second predicted feature map is determined; the trained type recognition network is used to perform feature extraction processing on the real image sample to obtain the real category feature corresponding to the real image sample, and the third image sample is used to perform feature extraction processing to obtain the predicted category feature; the fifth difference between the real category feature and the predicted category feature is determined; the second difference, the third difference, the fourth difference, the fifth difference, the first logical processing result, and the second logical processing result are fused to obtain the second loss function.

[0101] As an example, the learning objectives for training a neural network model include the following four points: 1) Reconstruction loss, used to constrain the output restored image to be close to the real image of the image to be restored; 2) Adversarial loss, used to restore real texture; 3) Component loss, used to further enhance facial details; 4) Type preservation loss.

[0102] As an example, for the reconstruction loss, L1 loss and perceptual loss can be used as the reconstruction loss Lr ec, defined as in formula (1):

[0103]

[0104] Where φ is the feature map output of the pre-trained VGG-19 network consisting of convolutional layers 1 to 5, and the perceptual loss can be obtained based on the feature map, λ. l1 and λ perHere, y1 represents the weights of the L1 loss function and the perceptual loss, y2 represents the output restored image (the third image sample of the first image sample), y2 represents the labeled ground truth image (the ground truth image sample of the first image sample), and ||y1-y2||1 represents the sum of the second differences corresponding to at least one first image sample. It is the sum of the third differences corresponding to at least one first image sample.

[0105] As an example, for adversarial loss, the adversarial loss function L is adopted. adv To encourage the neural network model to generate natural images and realistic textures, a logistic loss function is used, similar to that of generative networks (such as the StyleGAN2 model), as shown in Equation (2):

[0106]

[0107] Where D represents the discriminant network corresponding to the generator network, and λ adv Let represent the loss weights for adversarial loss, `softplus` represent the activation function, and y1 be the third image sample of the first image sample. The third image sample is processed by a discriminative network to obtain the discrimination result D(y1) for the corresponding third image sample, and this discrimination result is then substituted into the first logical function. After (softplus()), the first logical processing result L is obtained. adv .

[0108] As an example, for component loss, in order to further perceive important components of the image, such as important components of the face image, face component loss is introduced, such as component loss of the left eye local discriminant network, component loss of the right eye local discriminant network, and component loss of the mouth local discriminant network. First, the region of interest is pruned. For each region, an independent local discriminant network is trained to identify whether the restored layout image (e.g., eyes, nose) is real, so as to further encourage the local image to approach the natural face components. Furthermore, feature style loss based on the discriminant network is incorporated. Unlike feature matching loss, feature style loss attempts to match the Gram matrix between the real local image and the restored local image. The Gram matrix is ​​used to calculate feature correlation and can effectively capture texture information. Features are extracted from multiple layers of the already trained discriminant network and the Gram matrix of the intermediate features of the real local image and the restored local image is matched. Through experiments, it can be found that feature style loss can play a better role than other feature matching loss in generating realistic facial details and reducing artifacts. The definition of face component loss is as shown in formula (3):

[0109]

[0110] Here, ROI is the region of interest in the component set, i.e., a local area. If it is for face restoration, the component set includes the left eye, the right eye, and the mouth. ROI It is a local discriminant network for each region of interest. The local discriminant network is trained on the corresponding local region. φ represents the multi-resolution features of the local region extracted from the discriminant network, and λ is the local discriminant network. local λ represents the loss weight of the local discriminative loss. fs The weights represent the feature style loss. The first term is the local discriminant network loss, and the second term is the feature style loss, which is the loss of the local discriminant network over the local region y of the third image sample. ROI1 The discrimination process is performed to obtain the local discrimination result D corresponding to the third image sample. ROI (y ROI1 The local discrimination result is then substituted into the second logic function. The second logical processing result, namely the first term (local discriminant network loss), is obtained by processing the local region y of the real image sample through the local discriminant network. ROI2 Feature extraction is performed to obtain a second true feature map, and the local region y of the third image sample is then processed. ROI1 Feature extraction is performed to obtain the second predicted feature map; the fourth difference between the second true feature map and the second predicted feature map, ||Gramφy, is determined. ROI1 -Gram(φy ROI2 )||1 is the sum of the fourth differences between the second ground truth feature map and the second predicted feature map corresponding to at least one first image sample.

[0111] Regarding category preservation loss, similar to perceptual loss, it is applied in the neural network model. The identity loss is defined based on the embedded features of the input image. A pre-trained type recognition model is used to capture the most prominent features for type identification. Category preservation loss emphasizes minimizing the distance between the recovered image and the real image in the deep feature space.

[0112] L id =λ id ‖η(y1)-η(y2)‖1 (4)

[0113] Where η represents the feature extractor for type recognition, and for face restoration, it is the feature extractor for identity recognition. idThe loss weights represent the category retention loss. The trained type recognition network performs feature extraction on the real image sample y2 to obtain the real category feature η(y2) of the corresponding real image sample. The third image sample y1 is then subjected to feature extraction to obtain the predicted category feature η(y1). ‖η(y1)-η(y2)‖1 is the sum of the fifth difference between the real category feature and the predicted category feature corresponding to at least one first image sample.

[0114] As an example, the overall goal of the neural network model is the sum of the above losses, see Equation (5):

[0115] L total =L rec +L adv +L comp +L id (5)

[0116] Among them, L total It is the second loss function, L rec The loss is obtained based on the sum of the second and third differences, L. adv The loss is obtained based on the result of the first logical processing, L comp The loss is obtained by summing the results of the fourth interpolation and the second logical processing, L. id The loss is based on the fifth difference.

[0117] In step 104, a neural network model is trained based on the first loss function and the second loss function.

[0118] As an example, the first loss function and the second loss function are backpropagated in the neural network model to determine the parameter change value of the neural network model when the sum of the first loss function and the second loss function reaches the minimum value based on the learning rate of the neural network model, and the parameters of the neural network model are updated based on the parameter change value.

[0119] The image restoration method based on artificial intelligence provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0120] See Figure 4B , Figure 4B This is a flowchart illustrating the artificial intelligence-based image restoration method provided in this application embodiment. The following is a summary of the process. Figure 4A Steps 201-204 are shown below for explanation.

[0121] In step 201, the image to be restored is acquired.

[0122] As an example, the image to be restored can be an image input by a user, a degraded image obtained from the Internet, that is, there is no corresponding real image (i.e., the image before degradation), or an image obtained by arbitrarily degrading a given real image. The image to be restored can be a face image, a building image, or an animal image.

[0123] In step 202, a degradation removal process is performed on the image to be restored to obtain the hidden code feature corresponding to the image to be restored and at least two spatial features of different scales.

[0124] In some embodiments, referring to Figure 4C , Figure 4C is a schematic flowchart of an image restoration method based on artificial intelligence provided by an embodiment of the present application. Figure 4C The steps shown in step 202 can be implemented through steps 2021 to 2023, and will be described in combination with each step.

[0125] In step 2021, the image to be restored is downsampled through N cascaded downsampling networks to obtain a downsampling result.

[0126] As an example, the degradation removal process is executed by a degradation removal network; the degradation removal network includes N cascaded downsampling networks, a convolutional network, and M cascaded upsampling networks, where N is an integer greater than or equal to 2, M is equal to N, and each downsampling network is a residual network. The input is processed twice and then fused. One process is convolution processing and activation processing, and the other process is convolution processing.

[0127] In some embodiments, the above-mentioned downsampling of the image to be restored through N cascaded downsampling networks to obtain a downsampling result can be achieved through the following technical solution: through the nth downsampling network in the N cascaded downsampling networks, the input of the nth downsampling network is downsampled, and the nth downsampling result output by the nth downsampling network is transmitted to the (n + 1)th downsampling network to continue the downsampling process to obtain the corresponding (n + 1)th downsampling result; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the nth downsampling network is the image to be restored. When n takes the value of 2 ≤ n < N, the input of the nth downsampling network is the (n - 1)th downsampling result output by the (n - 1)th downsampling network.

[0128] As an example, assume that N is 3, that is, there are three cascaded downsampling networks. Through the first downsampling network in the three cascaded downsampling networks, the input of the first downsampling network is downsampled. The input of the first downsampling network is the image to be restored or the initial convolutional features of the image to be restored. The first downsampling result output by the first downsampling network is transmitted to the second downsampling network for continuous downsampling to obtain the corresponding second downsampling result, until the downsampling is completed through the third downsampling network, and the downsampling result corresponding to the three cascaded downsampling networks is obtained.

[0129] In step 2022, the downsampling result is convolved through a convolutional network to obtain the latent code features corresponding to the image to be restored.

[0130] As an example, the convolutional network is located between N downsampling networks and M upsampling networks. The convolutional network can be a single convolutional layer or multiple convolutional layers.

[0131] In step 2023, the latent code features are upsampled through M cascaded upsampling networks to obtain M spatial features of different scales corresponding to the image to be restored.

[0132] In some embodiments, the above-mentioned upsampling of the latent code features through M cascaded upsampling networks to obtain M spatial features of different scales corresponding to the image to be restored can be achieved through the following technical solution: through the m-th upsampling network in the M cascaded upsampling networks, the input of the m-th upsampling network is upsampled to obtain the spatial features corresponding to the m-th upsampling network, and the spatial features corresponding to the m-th upsampling network are transmitted to the (m + 1)-th upsampling network for continuous upsampling to obtain the spatial features corresponding to the (m + 1)-th upsampling network; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the m-th sampling network is the latent code features. When m takes the value of 2 ≤ m < M, the input of the m-th upsampling network is the (m - 1)-th upsampling result output by the (m - 1)-th upsampling network.

[0133] As an example, when M is also 3, the input of the first upsampling network in the three cascaded upsampling networks is upsampled by the first upsampling network. The input of the first sampling network is the latent code feature, and the spatial feature corresponding to the first upsampling network is obtained. The spatial feature corresponding to the first upsampling network is transmitted to the second upsampling network for continuous upsampling processing to obtain the spatial feature corresponding to the second upsampling network until the upsampling processing is completed through the third upsampling network. Thus, the spatial feature corresponding to the first upsampling network, the spatial feature corresponding to the second upsampling network, and the spatial feature corresponding to the third upsampling network are obtained respectively. The spatial features output by these different upsampling networks correspond to different resolution scales.

[0134] In some embodiments, before obtaining the spatial feature corresponding to the m-th upsampling network by upsampling the input of the m-th upsampling network in the M cascaded upsampling networks, the following technical solutions may also be executed: when m takes the value of 1, the output of the M-th downsampling network is concatenated with the output of the convolutional network, and the concatenated result is used as the input of the m-th upsampling network to replace using the output of the convolutional network as the input of the m-th upsampling network; when m takes the value of 2≤m<M, the output of the (M + 1 - m)-th downsampling network is concatenated with the output of the (m - 1)-th upsampling network, and the concatenated result is used as the input of the m-th upsampling network to replace using the spatial feature corresponding to the (m - 1)-th upsampling network as the input of the m-th upsampling network.

[0135] As an example, the degradation removal network can be improved based on the cascaded structure of the downsampling network, the convolutional network, and the upsampling network. That is, the downsampling network and the upsampling network can be correspondingly connected. For example, there are three downsampling networks and three upsampling networks. The output of the third downsampling network is connected to the input of the first upsampling network. Therefore, the output of the third downsampling network is concatenated with the output of the convolutional network, and the concatenated result is used as the input of the first upsampling network to replace directly using the output of the convolutional network as the input of the first upsampling network; the output of the second downsampling network is concatenated with the output of the first upsampling network, and the concatenated result is used as the input of the second upsampling network to replace using the spatial feature corresponding to the first upsampling network as the input of the second upsampling network. The output of the first downsampling network is concatenated with the output of the second upsampling network, and the concatenated result is used as the input of the third upsampling network to replace using the spatial feature corresponding to the second upsampling network as the input of the third upsampling network. After a large number of experiments, it is found that the structure of correspondingly connecting the downsampling network and the upsampling network can output a restored image with higher authenticity and credibility.

[0136] In step 203, the hidden code features are mapped based on at least two hidden code networks to obtain at least two hidden codes that correspond one-to-one with the at least two hidden code networks.

[0137] In some embodiments, see Figure 4D , Figure 4D This is a schematic flowchart of the image restoration method based on artificial intelligence provided in the embodiments of this application. Figure 4D Step 203 shown can be implemented through steps 2031 to 2033, and will be explained in conjunction with each step.

[0138] In step 2031, the hidden code features are processed by the shared convolutional layer of the hidden code network to obtain the shared feature extraction result of the corresponding hidden code features.

[0139] As an example, hidden code networks include shared convolutional layers, shared linear layers, and non-shared linear layers;

[0140] In step 2032, the shared feature extraction result is linearly transformed through the shared linear layer of the hidden code network to obtain the shared processing result of the corresponding hidden code feature.

[0141] In step 2033, the shared processing result is linearly transformed through the non-shared linear layer of the occult code network to obtain the occult code of the corresponding occult code network.

[0142] As an example, feature extraction is performed on the implicit code features through the shared convolutional layers of the implicit code network. The number of implicit code networks is the same as the number of upsampling networks, and the number of implicit code networks is the same as the number of generator networks. The implicit code features are mapped to three different implicit codes through three different implicit code networks. The three implicit code networks have the same shared convolutional layers and shared linear layers, and each of the three implicit code networks has a different non-shared linear layer, thus obtaining different implicit codes. Then, different prior features generated by different generator networks are obtained using the different implicit codes.

[0143] In step 204, based on spatial features at at least two different scales, the prior features corresponding to at least two hidden codes are modulated to obtain the restored image of the corresponding image to be restored.

[0144] In some embodiments, see Figure 4E , Figure 4E This is a schematic flowchart of the image restoration method based on artificial intelligence provided in the embodiments of this application. Figure 4E Step 204 shown can be implemented through steps 2041 to 2042, which will be explained in conjunction with each step.

[0145] In step 2041, the prior features of the corresponding hidden codes of the R intermediate networks of the image recovery network are modulated based on the corresponding spatial features to obtain R intermediate features that correspond one-to-one with the R intermediate networks.

[0146] As an example, modulation processing is implemented through an image restoration network, which includes R intermediate networks and one tail network, where R is an integer greater than or equal to 1. Each intermediate network includes an intermediate generator network and a corresponding intermediate modulation network, and the tail network includes a tail generator network and a corresponding tail modulation network. Modulation processing is implemented through the intermediate modulation network and the tail modulation network. The R intermediate networks are located on the input side of the image restoration network, and the tail network is located on the output side of the image restoration network.

[0147] In some embodiments, step 2041, through the R intermediate networks of the image restoration network, modulates the prior features of the corresponding hidden codes of the R intermediate networks based on the corresponding spatial features to obtain R intermediate features corresponding one-to-one with the R intermediate networks. This can be achieved through the following technical solution: through the r-th intermediate network in the R cascaded intermediate networks, modulates the prior features of the corresponding hidden codes of the r-th intermediate network based on the spatial features of the corresponding r-th intermediate network to obtain the r-th intermediate feature of the corresponding r-th intermediate network, and transmits the intermediate feature output by the r-th intermediate network to the (r+1)-th intermediate network for further modulation processing; where r is an integer variable that increments from 1. When r is 1, the prior features of the corresponding hidden codes of the r-th intermediate network are obtained based on the r-th hidden code of the r-th intermediate network. When r is 2≤r≤R-1, the prior features of the corresponding hidden codes of the r-th intermediate network are obtained based on the intermediate features output by the (r-1)-th intermediate network and the r-th hidden code of the corresponding r-th intermediate network.

[0148] As an example, continuing with the above embodiment, since the image restoration network must include the same number of generator networks, when there are 3 downsampling networks, 3 upsampling networks, and 3 hidden code features, it is necessary to include 2 intermediate networks and 1 tail network, for a total of 3 networks. Through the 2 intermediate networks of the image restoration network, the prior features of the corresponding hidden codes of the 2 intermediate networks are modulated based on the corresponding spatial features to obtain 2 intermediate features that correspond one-to-one with the 2 intermediate networks. Specifically, through the first intermediate network of the 2 cascaded intermediate networks, the prior features of the corresponding hidden codes of the first intermediate network are modulated based on the spatial features of the corresponding first intermediate network. The prior features of the corresponding hidden codes of the first intermediate network are obtained based on the first hidden code of the first intermediate network, thus obtaining the first intermediate feature corresponding to the first intermediate network. The intermediate feature output by the first intermediate network is then transmitted to the second intermediate network for further modulation processing. The prior features of the corresponding hidden codes of the second intermediate network are obtained based on the intermediate features output by the first intermediate network and the second hidden code corresponding to the second intermediate network. At this point, the processing of the intermediate networks ends.

[0149] In some embodiments, the above-mentioned modulation processing based on the spatial features of the corresponding r-th intermediate network is performed on the prior features of the r-th intermediate network of the r-th intermediate network to obtain the r-th intermediate feature of the corresponding r-th intermediate network. This can be achieved by the following technical solution: When r is 1, the following processing is performed: the r-th prior features of the corresponding r-th implicit code are generated by the intermediate generator network of the r-th intermediate network; the r-th prior features are spatially modulated by the spatial features of the corresponding r-th intermediate network, and the modulation result is used as the r-th intermediate feature of the corresponding r-th intermediate network; when r takes the value 2≤r≤R-1, the following processing is performed: the r-th prior features of the corresponding r-th implicit code and the r-th intermediate feature are generated by the generator network of the r-th intermediate network; the r-th prior features are spatially modulated by the spatial features of the corresponding r-th intermediate network, and the modulation result is used as the r-th intermediate feature of the corresponding r-th intermediate network.

[0150] As an example, let's take the second intermediate network as an illustration (2 is greater than or equal to 2 and less than or equal to 3-1). The second prior feature corresponding to the second hidden code and the second intermediate feature is generated through the generator network of the second intermediate network. The second prior feature is spatially modulated through the spatial feature corresponding to the second intermediate network, and the modulation result is used as the second intermediate feature of the corresponding second intermediate network. The spatial feature of the corresponding second intermediate network is the spatial feature output by the second upsampling network. That is, the following correspondence exists: the spatial feature output by the m-th upsampling network is the spatial feature of the r-th intermediate network, and the values ​​of m and r are equal.

[0151] As an example, the first intermediate network is used to illustrate the process. The first prior feature corresponding to the first hidden code is generated by the intermediate generator network of the first intermediate network. The first prior feature is spatially modulated by the spatial feature corresponding to the first intermediate network. The modulation result is used as the first intermediate feature of the corresponding first intermediate network. The spatial feature of the corresponding first intermediate network is the spatial feature output by the first upsampling network. That is, the following correspondence exists: the spatial feature output by the m-th upsampling network is the spatial feature of the r-th intermediate network, and the values ​​of m and r are equal.

[0152] In some embodiments, the above-mentioned spatial modulation processing of the r-th prior feature using the spatial features of the corresponding r-th intermediate network can be achieved through the following technical solution: performing convolution processing on the spatial features of the corresponding r-th intermediate network to obtain the spatial modulation affine coefficients and spatial modulation bias coefficients of the corresponding spatial features; randomly selecting all channels of the r-th prior feature, using half the total number of channels as the hold channels, and using the remaining channels as the modulation channels; multiplying the channel features of the modulation channels of the r-th prior feature with the spatial modulation affine coefficients, and adding the result of the multiplication to the spatial modulation bias coefficients; and concatenating the result of the addition with the channel features of the hold channels of the r-th prior feature.

[0153] As an example, the coefficients obtained through modulation processing based on spatial features are spatial modulation coefficients, including spatial modulation affine coefficients and spatial modulation bias coefficients. Convolution processing is performed on the spatial features of the corresponding r-th intermediate network, which is equivalent to convolution processing on the spatial features output by the m-th upsampled network, where m and r are equal. This yields the spatial modulation affine coefficients and spatial modulation bias coefficients for the corresponding spatial features. Each spatial feature, after passing through a convolutional layer, activation function, and further processing, yields spatial modulation coefficients (spatial modulation affine coefficient α and spatial modulation bias coefficient β). All channels of the r-th prior feature are randomly selected. Assuming there are 64 channels, 32 channels are randomly selected as hold channels, and the remaining 32 channels are used as modulation channels. The channel features of the modulation channels of the r-th prior feature are multiplied with the spatial modulation affine coefficient α. The result of the multiplication is added to the spatial modulation bias coefficient β. The result of the addition is concatenated with the channel features of the hold channels of the r-th prior feature. The channel features of the hold channels are not processed in any way (they are processed through an identity mapping) and are directly concatenated with the results of the multiplication and addition.

[0154] In step 2042, the tail network of the image restoration network modulates the intermediate features output by the adjacent intermediate networks with corresponding hidden codes and corresponding spatial features to obtain the restored image corresponding to the image to be restored.

[0155] In some embodiments, the modulation processing of the intermediate features output by the adjacent intermediate networks through the tail network of the image restoration network in step 2042, which modulates the corresponding hidden codes and corresponding spatial features, can be achieved by the following technical solution: through the tail generation network, the Rth intermediate features output by the Rth intermediate network and the tail hidden codes of the corresponding tail network are modulated based on the spatial features of the corresponding tail network.

[0156] In some embodiments, the above-mentioned modulation processing of the Rth intermediate feature output by the Rth intermediate network and the tail hidden code of the corresponding tail network based on the spatial features of the corresponding tail network can be achieved by the following technical solution: generating tail prior features of the corresponding tail hidden code and the Rth intermediate feature through the tail generation network of the tail network; performing spatial modulation processing on the tail prior features through the spatial features of the corresponding tail network; wherein, the modulation processing result is the restored image corresponding to the image to be restored.

[0157] As an example, the Rth intermediate network is the last intermediate network and also the network adjacent to the tail network. Continuing with the above embodiment, the 3rd intermediate network is the last intermediate network. The 3rd intermediate feature output by the 3rd intermediate network and the tail hidden code of the corresponding tail network are modulated based on the spatial features of the corresponding tail network. Convolution processing is performed on the spatial features of the corresponding 3rd intermediate network to obtain the spatial modulation affine coefficients and spatial modulation bias coefficients of the corresponding spatial features. The spatial features of the corresponding 3rd intermediate network are the spatial features output by the 3rd upsampling network. Randomization is performed on all channels of the 3rd prior feature. The selection process involves choosing half the total number of channels as the hold channels and using the remaining channels as modulation channels. The channel features of the modulation channels from the third prior feature are multiplied by the spatial modulation affine coefficients. The result of this multiplication is then added to the spatial modulation bias coefficients. The summation result is then concatenated with the channel features of the hold channels from the third prior feature. The resulting image can be directly converted and processed by the image output layer to output the restored image, or it can be convolved by the style convolutional layer of the tail generator network before being converted and processed by the image output layer to output the restored image.

[0158] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0159] The embodiments of this application can have the following application scenarios. For example, they can provide a repair function for degraded photos in electronic photo albums. In response to receiving a user's repair operation for degraded photos, the degraded photos can be restored using the image restoration method provided in the embodiments of this application to obtain a restored image, and the restored image can be displayed on the terminal's display interface. Alternatively, they can repair video frames for older videos or repair damaged image frames in video files. The following will take face restoration in image inpainting as an example for detailed explanation. Face restoration usually relies on facial prior information to restore realistic and believable facial details, such as facial geometric prior information or reference prior information. However, low-quality input images cannot provide accurate geometric priors, and high-quality reference priors are difficult to obtain, thus limiting their applicability in real-world scenarios. The neural network model provided in this application embodiment, such as Generative Facial Prior-Generative Adversarial Networks (GFP-GAN), can make full use of the rich and diverse facial prior information learned in the pre-training stage of the face generative network (hereinafter referred to as face GAN or GAN) for face restoration. This generated facial prior information is integrated into the face restoration process through a spatial feature transformation layer based on channel segmentation, thereby achieving a good balance between realism and believability. Due to the rich and diverse facial prior information and special network design, the neural network model provided in this application embodiment can restore facial details and enhance colors in a single inference process.

[0160] In the neural network model provided in this application embodiment, real-world face restoration is performed using generated facial prior information. The prior information is implicitly encapsulated in a pre-trained face generation model, such as the StyleGAN model. These face generation models can generate realistic and believable faces with high variability, thus providing rich and diverse prior information, such as geometric information, facial texture information, and color information, enabling simultaneous restoration of facial details and enhancement of colors. Previous attempts typically used face generation models for inversion, first converting the degraded image into the implicit code of a pre-trained face generation model, and then performing complex image reconstruction operations. Although visually realistic, this usually produces low-fidelity images because the low-dimensional implicit code is insufficient to guide accurate restoration.

[0161] To address the aforementioned technical issues, the neural network model provided in this application can simultaneously restore facial details and enhance color during a single inference process. The neural network model consists of a degradation removal network and a pre-trained face generation model. The face generation model provides prior information about the face. The two are connected through a latent code mapping (latent code network) and a newly proposed Channel Split-Spatial Feature Transform (CS-SFT) layer. The CS-SFT layer is used to spatially modulate a portion of the features and allow the remaining features to pass directly through, thereby better preserving effective information. Furthermore, the facial component loss of the local discriminant network further enhances the perception of facial details, while the category preservation loss is used to further improve the realism of the face restoration.

[0162] The neural network model is described in detail below. This model utilizes prior facial information encapsulated in a pre-trained face generation network for face restoration. Given an input face image x, which has undergone arbitrary degradation processing and is the image to be restored, face restoration involves estimating a high-quality image to make it as similar in realism and credibility as possible to the real face image x. The neural network model consists of a degradation removal network and a pre-trained face generation network, connected by a latent code mapping and a newly proposed Channel Split-Spatial Feature Transform (CS-SFT) layer. Specifically, the degradation removal network is used to eliminate complex degradations in the input x and extract latent code features F. latent and spatial features at multiple resolution scales F spatial Used for subsequent operations, the hidden code feature F latent Mapping to the intermediate hidden code W to obtain facial-like features (prior features F) prior Multi-resolution scale spatial features F spatial Used to process prior features F using CS-SFT layers prior Spatial modulation is performed to restore the realism of the image while maintaining high credibility.

[0163] When training a neural network model, in addition to considering the loss of the overall discriminant network, it is also necessary to consider the loss of the local discriminant network to enhance the perception of facial details, such as the eyes and mouth, and also to consider identity loss for identity training.

[0164] This application provides a degradation removal network in a neural network model that is designed to explicitly remove degradation and extract clean spatial features F with intermediate guidance information.spatial This alleviates the processing burden of subsequent networks. The degradation removal network adopts the Unet network structure, which can increase the receptive field to eliminate large blurs and generate spatial features at multiple resolution scales for subsequent processing. The Unet network structure also uses the pyramid restoration loss function to supervise the learning of the intermediate layers of the Unet network structure. The loss supervision at each level of the pyramid can not only enhance the image restoration ability, but also provide clean features as learning targets for spatial features at each resolution scale.

[0165] The implicit code mapping provided in this application embodiment is only a coarse mapping feedforward, and the burden of achieving realism is left to the subsequent CS-SFT layer. The input image is converted into implicit code W, which is used to "retrieve" features that are close to the face. After this operation, the prior features of the face generation model can be obtained, and the prior features carry the prior information of the face.

[0166] To combine restoration and color enhancement, face generation models learn a rich variety of prior information, including realistic detail information and vivid texture information. For example, face generation models also encapsulate color prior information, which can be used to combine restoration and color enhancement. In real-world applications, face images are often old photos, which can be black and white, retro yellow, or other muted colors. The vivid color prior information encapsulated in the face generation model can be used for color enhancement.

[0167] Given the prior features F of a pre-trained GAN prior and the clean spatial features F extracted from the input image spatial The neural network model effectively combines these two features to maintain both realism and credibility. When combining these features, affine transformation parameters are generated for spatial feature modulation, transforming the spatial features F of the input image... spatial The affine transformation parameters are obtained through several convolutional layers, and the prior features F are then analyzed based on these parameters. prior Perform scaling and offset operations (modulation):

[0168] F output =SFT(F prior |α,β)=α⊙F prior +β (6);

[0169] Although it can effectively integrate facial information from the input image, it cannot achieve a balance between realism and credibility because all prior features (contributing to realism) are affected by the spatial features F of the input image. spatial The impact (contribution credibility) of the spatial feature F is that if a very low-quality image is used as input, the "blurred" spatial feature F spatialThis will exert an influence during the modulation process, causing the modulated output features to tend to be blurry, thus failing to achieve realism and rich facial details.

[0170] Based on this, spatial modulation is further performed on some features through a CS-SFT layer, while leaving some features unmodulated, in order to better preserve image information. See [link to relevant documentation]. Figure 5 , Figure 5 This is a modulation diagram provided in the embodiment of this application. The prior feature F is segmented based on the channel dimension. The features in half of the randomly selected channels are retained and output. The features in the other half of the channels are modulated based on the spatial modulation coefficient, which is equivalent to performing spatial feature transformation. The features are multiplied by the spatial modulation affine coefficient α and then added to the spatial modulation bias coefficient β. Then, the features of all channels are concatenated to obtain the modulation result. The modulation result is then passed to the subsequent structure of the generator network, see formula (7):

[0171]

[0172] in, and For prior features F prior Features obtained by segmentation along the channel dimension; Concat represents the connection operation.

[0173] The CS-SFT layer can achieve the following beneficial effects: it can improve the generation of facial prior features F prior and the multi-scale spatial features F extracted from the input image to be restored spatial By combining these elements more effectively, the resulting restoration achieves a high level of balance between texture realism and facial fidelity. It directly merges prior information and effective modulation of the input image, thereby achieving a balance between texture fidelity and fidelity. In addition to performance, CS-SFT also reduces the computational complexity of modulation because it requires fewer modulation channels.

[0174] The learning objectives of training a neural network model include the following four points: 1) Reconstruction loss, which is used to constrain the output restored image to be close to the real image of the image to be restored; 2) Adversarial loss, which is used to restore the real texture; 3) Component loss, which is used to further enhance facial details; 4) Identity preservation loss (a type of category preservation loss).

[0175] For the reconstruction loss, L1 loss and perceptual loss can be used as the reconstruction loss Lrec, defined as follows:

[0176]

[0177] Where φ is the feature map output by convolutional layers 1 to 5 in the pre-trained VGG-19 network, and λl1 and λ per y1 represents the loss weights of the L1 loss function and the weights of the perceptual loss function, y2 represents the output restored image, and y3 represents the labeled real image.

[0178] To address adversarial losses, an adversarial loss function L is employed. adv This is used to encourage the neural network model to generate natural images and realistic textures. Similar to the StyleGAN2 model's loss function, it employs the logistic loss function.

[0179]

[0180] Where D represents the discriminant network, λ adv This indicates the weight used to counteract losses.

[0181] To address component loss and further perceive important facial components, facial component loss is introduced, such as component loss from the left eye local discriminant network, the right eye local discriminant network, and the mouth local discriminant network. First, regions of interest are pruned. For each region, independent small local discriminant networks are trained to determine whether the reconstructed layout image (e.g., eyes, nose) is realistic, further encouraging local images to approximate a natural facial component distribution. Furthermore, a feature-style loss based on the discriminant network is incorporated. Unlike feature matching loss, feature-style loss attempts to match the Gram matrix between the real and reconstructed local images. The Gram matrix is ​​used to calculate feature correlation and can effectively capture texture information. Features are extracted from multiple layers of the trained discriminant network, and the Gram matrix of intermediate features between the real and reconstructed local images is matched. Experiments show that feature-style loss outperforms other feature matching losses in generating realistic facial details and reducing artifacts. The definition of facial component loss is as follows:

[0182]

[0183] Here, ROI is the region of interest in the component set, which includes the left eye, the eye itself, and the mouth. ROI It is a local discriminant network for each region of interest, where φ represents the multi-resolution features extracted from the learned discriminant network, and λ... local λ represents the loss weight of the local discriminative loss. fs The weights represent the feature style loss. The first term is the local discriminant network loss, and the second term is the feature style loss.

[0184] To address the identity preservation loss, we apply it to our neural network model. Similar to perceptual loss, the identity loss is defined based on the embedded features of the input image. We use a pre-trained face recognition model to capture the most prominent features for identity verification. The identity preservation loss emphasizes that the recovered image should have only a small distance from the real image in the deep feature space.

[0185] L id =λ id ‖η(y1)-η(y2)‖1 (11);

[0186] Where η represents the facial feature extractor, λ id The loss weights represent the loss for identity preservation. The overall goal of the neural network model is the sum of the above losses:

[0187] L total =L rec +L adv +L comp +L id (12);

[0188] The neural network model provided in this application adopts generated facial prior features. Based on the network structure of the neural network model, it effectively realizes facial restoration through channel splitting and spatial feature modulation, achieving a balance between texture realism and facial credibility, and achieving better technical results than related technologies. Furthermore, the neural network model provided in this application can realize facial restoration and color enhancement.

[0189] See Figure 6 , Figure 6 This is a comparison of the effects of the AI-based image restoration method provided in this application embodiment. The neural network model provided in this application embodiment utilizes rich and diverse facial prior information for face restoration. These priors contain sufficient facial texture and color enhancement information, enabling simultaneous face restoration and color enhancement. The model structure and loss of the neural network model effectively integrate facial prior information into the face restoration process. The neural network model with CS-SFT layers can simultaneously restore facial details and enhance colors in a single inference process, achieving a balance between facial realism and credibility. Extensive experiments demonstrate that the neural network model achieves better performance than related technologies on both test sets. See Table 1 for examples of testing on different datasets. FID measures the similarity between two sets of images from a statistical perspective of the computer vision features of the original image; it is a measure of the distance between the feature vectors of the real image and the generated image (the smaller the better). NIQE is also an image quality evaluation parameter (the smaller the better). It can be seen that the neural network model provided in this application embodiment can achieve the best image restoration effect.

[0190]

[0191] Table 1

[0192] Next, the exemplary structure of the software module implementation of the artificial intelligence-based image restoration device 455 provided in the embodiments of the present application will be continued. In some embodiments, as Figure 2 shown, the software module stored in the artificial intelligence-based image restoration device 455 in the memory 450 may include: an acquisition module 4551 for acquiring an image to be restored; a degradation removal module 4552 for performing degradation removal processing on the image to be restored to obtain the hidden code features corresponding to the image to be restored and at least two spatial features of different scales; a mapping module 4553 for performing mapping processing on the hidden code features based on at least two hidden code networks to obtain at least two hidden codes corresponding one-to-one to the at least two hidden code networks; a generation module 4554 for modulating the prior features corresponding to the at least two hidden codes based on the at least two spatial features of different scales to obtain the restored image corresponding to the image to be restored.

[0193] In some embodiments, the degradation removal processing is performed by a degradation removal network; the degradation removal network includes N cascaded downsampling networks, a convolutional network, and M cascaded upsampling networks, where N is an integer greater than or equal to 2, and M is equal to N; the degradation removal module 4552 is further configured to: perform downsampling processing on the image to be restored through N cascaded downsampling networks to obtain a downsampling processing result; perform convolutional processing on the downsampling processing result through the convolutional network to obtain the hidden code features corresponding to the image to be restored; perform upsampling processing on the hidden code features through M cascaded upsampling networks to obtain M spatial features of different scales corresponding to the image to be restored.

[0194] In some embodiments, the degradation removal module 4552 is further configured to: perform downsampling processing on the input of the n-th downsampling network through the n-th downsampling network in the N cascaded downsampling networks, and transmit the n-th downsampling processing result output by the n-th downsampling network to the (n + 1)-th downsampling network to continue the downsampling processing to obtain the corresponding (n + 1)-th downsampling processing result; where n is an integer variable starting from 1, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th downsampling network is the image to be restored. When 2 ≤ n < N, the input of the n-th downsampling network is the (n - 1)-th downsampling processing result output by the (n - 1)-th downsampling network.

[0195] In some embodiments, the degradation removal module 4552 is further configured to: perform upsampling on the input of the m-th upsampling network through the m-th upsampling network in the M cascaded upsampling networks to obtain the spatial feature corresponding to the m-th upsampling network, and transmit the spatial feature corresponding to the m-th upsampling network to the (m + 1)-th upsampling network to continue the upsampling process to obtain the spatial feature corresponding to the (m + 1)-th upsampling network; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m = 1, the input of the m-th sampling network is the latent code feature. When 2 ≤ m < M, the input of the m-th upsampling network is the m - 1 upsampling processing result output by the (m - 1)-th upsampling network.

[0196] In some embodiments, before performing upsampling on the input of the m-th upsampling network through the m-th upsampling network in the M cascaded upsampling networks to obtain the spatial feature corresponding to the n-th upsampling network, the degradation removal module 4552 is further configured to: when m = 1, splice the output of the M-th downsampling network and the output of the convolutional network, and use the splicing result as the input of the m-th upsampling network to replace using the output of the convolutional network as the input of the m-th upsampling network; when 2 ≤ m < M, splice the output of the (M + 1 - m)-th downsampling network and the output of the (m - 1)-th upsampling network, and use the splicing result as the input of the m-th upsampling network to replace using the spatial feature corresponding to the (m - 1)-th upsampling network as the input of the m-th upsampling network.

[0197] In some embodiments, the latent code network includes a shared convolutional layer, a shared linear layer, and a non-shared linear layer; the mapping module 4553 is further configured to: perform feature extraction on the latent code feature through the shared convolutional layer of the latent code network to obtain the shared feature extraction result corresponding to the latent code feature; perform linear transformation on the shared feature extraction result through the shared linear layer of the latent code network to obtain the shared processing result corresponding to the latent code feature; perform linear transformation on the shared processing result through the non-shared linear layer of the latent code network to obtain the latent code corresponding to the latent code network.

[0198] In some embodiments, the modulation processing is implemented through an image restoration network, which includes R intermediate networks and a tail network; the R intermediate networks are located on the input side of the image restoration network, and the tail network is located on the output side of the image restoration network; wherein R is an integer greater than or equal to 1; the generation module 4554 is further configured to: perform modulation processing based on corresponding spatial features on the prior features of the corresponding implicit codes of the R intermediate networks through the R intermediate networks of the image restoration network to obtain R intermediate features that correspond one-to-one with the R intermediate networks; and perform modulation processing on the intermediate features output by adjacent intermediate networks through the tail network of the image restoration network on the corresponding implicit codes and corresponding spatial features to obtain the restored image corresponding to the image to be restored.

[0199] In some embodiments, the generation module 4554 is further configured to: perform modulation processing on the prior features of the corresponding hidden code of the r-th intermediate network based on the spatial features of the corresponding r-th intermediate network through the r-th intermediate network in the R cascaded intermediate networks to obtain the r-th intermediate feature of the corresponding r-th intermediate network, and transmit the intermediate feature output by the r-th intermediate network to the (r+1)-th intermediate network for further modulation processing; wherein, r is an integer variable whose value increases from 1. When r is 1, the prior features of the corresponding hidden code of the r-th intermediate network are obtained based on the r-th hidden code of the r-th intermediate network. When r is 2≤r≤R-1, the prior features of the corresponding hidden code of the r-th intermediate network are obtained based on the intermediate features output by the (r-1)-th intermediate network and the r-th hidden code of the corresponding r-th intermediate network; and perform modulation processing on the R-th intermediate feature output by the R-th intermediate network and the tail hidden code of the corresponding tail network based on the spatial features of the corresponding tail network through the tail generation network.

[0200] In some embodiments, the generation module 4554 is further configured to: generate tail prior features corresponding to the tail hidden code and the Rth intermediate feature through the tail generation network of the tail network; perform spatial modulation processing on the tail prior features through the spatial features of the corresponding tail network; wherein the modulation processing result is the restored image corresponding to the image to be restored.

[0201] In some embodiments, the generation module 4554 is further configured to: when r is 1, perform the following processing: generate the r-th prior feature corresponding to the r-th hidden code through the intermediate generation network of the r-th intermediate network; perform spatial modulation processing on the r-th prior feature through the spatial features of the corresponding r-th intermediate network, and use the modulation processing result as the r-th intermediate feature of the corresponding r-th intermediate network; when r takes the value 2≤r≤R-1, perform the following processing: generate the r-th prior feature corresponding to the r-th hidden code and the r-th intermediate feature through the generation network of the r-th intermediate network; perform spatial modulation processing on the r-th prior feature through the spatial features of the corresponding r-th intermediate network, and use the modulation processing result as the r-th intermediate feature of the corresponding r-th intermediate network.

[0202] In some embodiments, the generation module 4554 is further configured to: perform convolution processing on the spatial features of the corresponding r-th intermediate network to obtain the spatial modulation affine coefficients and spatial modulation bias coefficients of the corresponding spatial features; randomly select all channels of the r-th prior feature, select half of the total number of channels as the hold channels, and use the remaining channels as the modulation channels; multiply the channel features of the modulation channels of the r-th prior feature with the spatial modulation affine coefficients, and add the result of the multiplication to the spatial modulation bias coefficients; and concatenate the result of the addition with the channel features of the hold channels of the r-th prior feature.

[0203] In some embodiments, the image restoration method is implemented through a neural network model, which includes a degradation removal network and an image restoration network. The apparatus 455 further includes a training module 4555, configured to: before performing degradation removal processing on the image to be restored to obtain the hidden code features and at least two spatial features of different scales corresponding to the image to be restored, perform degradation removal processing on a first image sample through the degradation removal network to obtain at least two second image samples corresponding to the first image sample; wherein, the at least two second image samples are output one-to-one through at least two upsampling networks in the degradation removal network; perform image restoration processing on the first image sample through the degradation removal network and the image restoration network to obtain a third image sample corresponding to the first image sample; obtain a first loss function for loss judgment on the at least two second image samples and a second loss function for loss judgment on the third image sample; and train the neural network model based on the first loss function and the second loss function.

[0204] In some embodiments, the training module 4555 is further configured to: determine the first difference between the real image sample corresponding to the first image sample and each of the at least two second image samples, and sum the first difference corresponding to each second image sample to obtain a first loss function.

[0205] In some embodiments, the training module 4555 is further configured to: determine a second difference between the real image sample corresponding to the first image sample and the third image sample; perform feature extraction processing on the real image sample using a trained image recognition model to obtain a first real feature map, and perform feature extraction processing on the third image sample to obtain a first predicted feature map; determine a third difference between the first real feature map and the first predicted feature map; perform discrimination processing on the third image sample using a discriminant network to obtain a discrimination result corresponding to the third image sample, and substitute the discrimination result into a first logic function to obtain a first logic processing result; wherein, the image restoration network is coupled with a trained generator network, and the discriminant network is trained alongside the generator network; and perform discrimination processing on the corresponding local region of the third image sample using a local discriminant network to obtain a local discrimination result corresponding to the third image sample. The local discrimination result is then substituted into the second logical function to obtain the second logical processing result. The local discrimination network is trained for the corresponding local region. Feature extraction is performed on real image samples using the local discrimination network to obtain the second real feature map, and feature extraction is performed on the third image sample to obtain the second predicted feature map. A fourth difference is determined between the second real feature map and the second predicted feature map. A trained identity recognition network is then used to extract features from real image samples to obtain the real identity features corresponding to the real image samples, and feature extraction is performed on the third image sample to obtain the predicted identity features. A fifth difference is determined between the real identity features and the predicted identity features. The second, third, fourth, and fifth differences, the first logical processing result, and the second logical processing result are then fused to obtain the second loss function.

[0206] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the artificial intelligence-based image restoration method described above in this application.

[0207] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the AI-based image restoration method provided in this application. For example... Figures 4A-4E The image restoration method based on artificial intelligence is shown.

[0208] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0209] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0210] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0211] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0212] In summary, through the embodiments of this application, prior features corresponding to the image to be restored are used as a reference. Since the reference corresponds to the image to be restored, the restoration is highly targeted. Moreover, the prior features encapsulate rich prior knowledge. By combining the prior features with multi-scale spatial features from the image to be restored, the restored image corresponding to the image to be restored is obtained, which greatly improves the credibility and authenticity of the image to be restored.

[0213] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image restoration method based on artificial intelligence, characterized in that, include: Obtain the image to be recovered; The image to be restored is downsampled by N cascaded downsampling networks in the degradation removal network to obtain the downsampling result; The downsampling result is convolved by the convolutional network in the degradation removal network to obtain the hidden code features corresponding to the image to be restored; the hidden code features are upsampled by M cascaded upsampling networks in the degradation removal network to obtain M spatial features of different scales corresponding to the image to be restored, where N is an integer greater than or equal to 2, and M is equal to N; The implicit code features are mapped based on at least two implicit code networks to obtain at least two implicit codes that correspond one-to-one with the at least two implicit code networks. Based on the spatial features at least two different scales, the prior features corresponding to the at least two hidden codes are modulated to obtain the restored image corresponding to the image to be restored.

2. The method according to claim 1, characterized in that, The image to be restored is downsampled by N cascaded downsampling networks in the degradation removal network to obtain the downsampling result, including: The input of the nth downsampling network is downsampled through the nth downsampling network in the N cascaded downsampling networks, and the output of the nth downsampling network is transmitted to the (n+1)th downsampling network for further downsampling, so as to obtain the corresponding (n+1)th downsampling result. Where n is an integer variable that increments from 1, and the range of n is... n When n is 1, the input of the nth downsampling network is the image to be restored; when n is 2... n At that time, the input of the nth downsampling network is the (n-1)th downsampling result output by the (n-1)th downsampling network.

3. The method according to claim 1, wherein the upsampling of the hidden code features through M cascaded upsampling networks in the degradation removal network to obtain M spatial features of different scales corresponding to the image to be restored includes: The m-th upsampling network in M ​​cascaded upsampling networks is used to upsample the input of the m-th upsampling network to obtain the spatial features corresponding to the m-th upsampling network. The spatial features corresponding to the m-th upsampling network are then transmitted to the (m+1)-th upsampling network for further upsampling to obtain the spatial features corresponding to the (m+1)-th upsampling network. Where m is an integer variable whose value starts from 1 and increments, and the range of m is... m When m is 1, the input of the m-th upsampling network is the hidden code feature; when m is 2... At that time, the input of the m-th upsampling network is the (m-1)-th upsampling processing result output by the (m-1)-th upsampling network.

4. The method according to claim 3, characterized in that, Before obtaining the spatial features corresponding to the m-th upsampling network by upsampling the input of the m-th upsampling network through M cascaded upsampling networks, the method further includes: When m takes the value The output of the Mth downsampling network is concatenated with the output of the convolutional network, and the concatenation result is used as the input of the mth upsampling network, instead of using the output of the convolutional network as the input of the mth upsampling network. When m takes the value When the output of the (M+1-m)th downsampling network is concatenated with the output of the (m-1)th upsampling network, the concatenation result is used as the input of the mth upsampling network, instead of using the spatial features of the corresponding (m-1)th upsampling network as the input of the mth upsampling network.

5. The method according to claim 1, characterized in that, The hidden code network includes shared convolutional layers, shared linear layers, and non-shared linear layers; The step of performing mapping processing on the hidden code features based on at least two hidden code networks to obtain at least two hidden codes that correspond one-to-one with the at least two hidden code networks includes: The hidden code features are extracted by the shared convolutional layer of the hidden code network to obtain the shared feature extraction result corresponding to the hidden code features; The shared feature extraction result is linearly transformed by the shared linear layer of the hidden code network to obtain the shared processing result corresponding to the hidden code feature; The shared processing result is linearly transformed by the non-shared linear layer of the hidden code network to obtain the hidden code corresponding to the hidden code network.

6. The method according to claim 1, characterized in that, The modulation processing is implemented through an image restoration network, which includes R intermediate networks and one tail network; the R intermediate networks are located on the input side of the image restoration network, and the tail network is located on the output side of the image restoration network; where R is an integer greater than or equal to 1. The step of modulating the prior features corresponding to the at least two hidden codes based on the spatial features at at least two different scales to obtain the restored image corresponding to the image to be restored includes: Through the R intermediate networks of the image recovery network, the prior features of the corresponding hidden codes of the R intermediate networks are modulated based on the corresponding spatial features to obtain R intermediate features that correspond one-to-one with the R intermediate networks. The tail network of the image restoration network modulates the intermediate features output by the adjacent intermediate networks with corresponding hidden codes and spatial features to obtain the restored image corresponding to the image to be restored.

7. The method according to claim 6, characterized in that, Through the R intermediate networks of the image recovery network, the prior features of the corresponding hidden codes of the R intermediate networks are modulated based on corresponding spatial features to obtain R intermediate features that correspond one-to-one with the R intermediate networks, including: Through the r-th intermediate network in R cascaded intermediate networks, the prior features of the corresponding hidden code of the r-th intermediate network are modulated based on the spatial features of the r-th intermediate network to obtain the r-th intermediate feature of the r-th intermediate network, and the intermediate feature output by the r-th intermediate network is transmitted to the (r+1)-th intermediate network for further modulation processing. Where r is an integer variable that increments from 1. When r is 1, the prior features of the hidden code corresponding to the r-th intermediate network are obtained based on the r-th hidden code of the r-th intermediate network. When r is 2... r At that time, the prior features of the corresponding hidden code of the r-th intermediate network are obtained based on the intermediate features output by the (r-1)-th intermediate network and the r-th hidden code corresponding to the r-th intermediate network. The tail network of the image restoration network modulates the intermediate features output by adjacent intermediate networks with corresponding hidden codes and corresponding spatial features, including: Through the tail network, the Rth intermediate feature output by the Rth intermediate network and the tail hidden code corresponding to the tail network are modulated based on the spatial features of the tail network.

8. The method according to claim 7, characterized in that, The step of performing modulation processing on the Rth intermediate feature output by the Rth intermediate network and the tail hidden code corresponding to the tail network based on the spatial features of the tail network includes: Tail prior features corresponding to the tail hidden code and the Rth intermediate feature are generated by the tail generation network of the tail network. The tail prior features are spatially modulated based on the spatial features corresponding to the tail network. The modulation processing result is the restored image corresponding to the image to be restored.

9. The method according to claim 7, characterized in that, The process involves using the r-th intermediate network among R cascaded intermediate networks to perform modulation processing on the prior features of the corresponding hidden code of the r-th intermediate network based on the spatial features of the corresponding r-th intermediate network, to obtain the r-th intermediate feature corresponding to the r-th intermediate network, including: When r is 1, the following processing is performed: The intermediate generator network of the r-th intermediate network generates the r-th prior feature corresponding to the r-th hidden code; The r-th prior feature is spatially modulated using the spatial features corresponding to the r-th intermediate network, and the modulation result is used as the r-th intermediate feature corresponding to the r-th intermediate network. When r takes the value 2 r When this happens, perform the following processing: The r-th prior feature corresponding to the r-th hidden code and the r-th intermediate feature is generated by the generator network of the r-th intermediate network; The r-th prior feature is spatially modulated using the spatial features corresponding to the r-th intermediate network, and the modulation result is used as the r-th intermediate feature corresponding to the r-th intermediate network.

10. The method according to claim 9, characterized in that, The step of spatially modulating the r-th prior feature using the spatial features corresponding to the r-th intermediate network includes: Convolution processing is performed on the spatial features corresponding to the r-th intermediate network to obtain the spatial modulation affine coefficients and spatial modulation bias coefficients corresponding to the spatial features; For all channels of the r-th prior feature, a random selection process is performed, and half of the total number of channels are selected as the hold channels, while the remaining channels are used as the modulation channels. The channel feature of the modulation channel of the r-th prior feature is multiplied with the spatial modulation affine coefficient, and the result of the multiplication is added to the spatial modulation bias coefficient. The summation result is then concatenated with the channel feature of the preserved channel of the r-th prior feature.

11. The method according to any one of claims 1 to 10, characterized in that, The image restoration method is implemented through a neural network model, which includes a degradation removal network and an image restoration network. Before performing degradation removal processing on the image to be restored to obtain the hidden code features corresponding to the image to be restored and at least two spatial features at different scales, the method further includes: The degradation removal network is used to perform degradation removal processing on the first image sample to obtain at least two second image samples corresponding to the first image sample; Wherein, the at least two second image samples are output one-to-one through at least two upsampling networks in the degradation removal network; The first image sample is processed by the degradation removal network and the image restoration network to obtain a third image sample corresponding to the first image sample; Obtain a first loss function for loss determination on the at least two second image samples, and a second loss function for loss determination on the third image sample; The neural network model is trained based on the first loss function and the second loss function.

12. An image restoration device based on artificial intelligence, characterized in that, include: The acquisition module is used to acquire the image to be recovered; The degradation removal module is used to downsample the image to be restored through N cascaded downsampling networks in the degradation removal network to obtain a downsampling result; to convolve the downsampling result through a convolutional network in the degradation removal network to obtain hidden code features corresponding to the image to be restored; and to upsample the hidden code features through M cascaded upsampling networks in the degradation removal network to obtain M spatial features of different scales corresponding to the image to be restored, where N is an integer greater than or equal to 2, and M is equal to N; The mapping module is used to perform mapping processing on the hidden code features based on at least two hidden code networks to obtain at least two hidden codes that correspond one-to-one with the at least two hidden code networks. The generation module is used to modulate the prior features corresponding to the at least two hidden codes based on the spatial features at the at least two different scales, so as to obtain the restored image corresponding to the image to be restored.

13. The apparatus according to claim 12, characterized in that, The degradation removal module is further configured to downsample the input of the nth downsampling network through the nth downsampling network in an N-level cascaded downsampling network, and transmit the nth downsampling result output by the nth downsampling network to the (n+1)th downsampling network for further downsampling processing, thereby obtaining the (n+1)th downsampling result; where n is an integer variable that increments from 1, and the range of n is... n When n is 1, the input of the nth downsampling network is the image to be restored; when n is 2... n At that time, the input of the nth downsampling network is the (n-1)th downsampling result output by the (n-1)th downsampling network.

14. The apparatus according to claim 12, characterized in that, The degradation removal module is further configured to upsample the input of the m-th upsampling network through the m-th upsampling network in a cascaded network of M networks to obtain the spatial features corresponding to the m-th upsampling network, and then transmit the spatial features corresponding to the m-th upsampling network to the (m+1)-th upsampling network for further upsampling processing to obtain the spatial features corresponding to the (m+1)-th upsampling network; where m is an integer variable whose value starts from 1 and increments by 1, and the range of m is 1 / 2. m When m is 1, the input of the m-th upsampling network is the hidden code feature; when m is 2... At that time, the input of the m-th upsampling network is the (m-1)-th upsampling processing result output by the (m-1)-th upsampling network.

15. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the artificial intelligence-based image restoration method according to any one of claims 1 to 11.

16. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the AI-based image restoration method according to any one of claims 1 to 11 when executed by a processor.

17. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the artificial intelligence-based image restoration method according to any one of claims 1 to 11.

Images