Image enhancement method and device, equipment, storage medium and program product

Abstract

The invention provides an image enhancement method, device and equipment, a storage medium and a program product, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving. The method comprises the steps that a hidden variable of a face image to be enhanced is acquired, noise is added to the hidden variable, a noise-added hidden variable of the face image is obtained, and the face image is an image of the face of a target object; extracting facial features of the face in the face image; in combination with the facial features, performing denoising processing on the noise-added hidden variable to obtain a denoised hidden variable of the facial image; performing image reconstruction on the de-noising hidden variables to obtain a face enhanced image of the face image; according to the invention, the image enhancement effect of the face image can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image enhancement method, device, equipment, storage medium and program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines are given the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] Image enhancement is also an important application direction of artificial intelligence. In related technologies, for facial image enhancement, noise is often gradually added to the facial image until it becomes a completely random noise image; then, starting from the random noise image, noise is gradually removed to obtain the final facial enhanced image. However, since the denoising intensity is difficult to control during the denoising process, the final facial enhanced image is prone to lose a large number of original features, resulting in poor enhancement effect of the facial enhanced image. Summary of the invention

[0004] The embodiments of the present application provide an image enhancement method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the image enhancement effect of facial images.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an image enhancement method, including:

[0007] Acquire a latent variable of a facial image to be enhanced, and add noise to the latent variable to obtain a noisy latent variable of the facial image, wherein the facial image is an image of the face of a target object;

[0008] Extracting facial features of the face in the facial image;

[0009] Combining the facial features, denoising the noisy latent variables to obtain denoised latent variables of the facial image;

[0010] Performing image reconstruction on the denoising latent variables to obtain a face-enhanced image of the face image.

[0011] The present application also provides an image enhancement device, including:

[0012] An acquisition module, used for acquiring latent variables of a facial image to be enhanced, and adding noise to the latent variables to obtain noisy latent variables of the facial image, wherein the facial image is an image of the face of a target object;

[0013] An extraction module, used for extracting facial features of the face in the facial image;

[0014] A denoising module, used to denoise the noisy latent variable in combination with the facial features to obtain a denoised latent variable of the facial image;

[0015] The reconstruction module is used to perform image reconstruction on the denoising latent variables to obtain a face-enhanced image of the face image.

[0016] The present application also provides an electronic device, including:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the image enhancement method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] The embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the image enhancement method provided in the embodiment of the present application is implemented.

[0020] The embodiment of the present application also provides a computer program product, including computer executable instructions or a computer program, which, when executed by a processor, implements the image enhancement method provided in the embodiment of the present application.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By applying the above-mentioned embodiment of the present application, the latent variables of the facial image to be enhanced are first obtained, and noise is added to the latent variables to obtain the noisy latent variables of the facial image; then the facial features of the face in the facial image are extracted; thereby, the noisy latent variables are denoised in combination with the facial features to obtain the denoised latent variables of the facial image; the denoised latent variables are image reconstructed to obtain the facial enhanced image of the facial image. Here, in the image enhancement process of the facial image, the denoising process of the noisy latent variables is increased by controlling the facial features of the facial image, that is, the noisy latent variables are denoised in combination with the facial features, thereby reducing the loss of facial features of the facial image, so that the facial enhanced image finally obtained can retain more facial features of the facial image and improve the image enhancement effect of the facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of the architecture of the image enhancement system provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3 is a flow chart of an image enhancement method provided in an embodiment of the present application;

[0026] Figure 4 is a flow chart of an image enhancement method provided in an embodiment of the present application;

[0027] Figure 5 is a flowchart of processing facial fusion information provided by an embodiment of the present application;

[0028] Figure 6 is a structural schematic diagram of an image enhancement model provided in an embodiment of the present application;

[0029] Figure 7 is a structural schematic diagram of an image enhancement model provided in an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of generating an image mask provided in an embodiment of the present application;

[0031] Fig. 9 is a structural schematic diagram of the image generation diffusion model provided in an embodiment of the present application;

[0032] Fig.10 is a flow chart of an image enhancement method provided in an embodiment of the present application;

[0033] Fig.11 is a schematic diagram of self-attention processing provided by an embodiment of the present application;

[0034] Fig.12 is a structural schematic diagram of an image denoising model provided in an embodiment of the present application;

[0035] Fig.13 is a schematic diagram of the stitching and repairing result of the face thinning image provided in an embodiment of the present application;

[0036] Fig.14 is a schematic diagram of a face enhancement image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0041] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0042] 1) Client: An application running in a terminal to provide various services, such as a client that supports image enhancement.

[0043] 2) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0044] 3) Convolutional Neural Network (CNN): It is a feedforward neural network whose artificial neurons can respond to surrounding units within a certain coverage area, and has excellent performance in large image processing. Convolutional neural network consists of one or more convolutional layers and a fully connected layer at the top (corresponding to the classic neural network), as well as associated weights and pooling layers.

[0045] 4) U-Net model: It is an algorithm that uses a fully convolutional network for semantic segmentation.

[0046] 5) Attention mechanism: A solution to the problem that imitates human attention, which refers to quickly filtering out high-value information from a large amount of information. It is mainly used to solve the problem that it is difficult to obtain a reasonable vector representation when the input sequence of the time series model is long. The approach is to retain the intermediate results of the time series model, learn it with a new model, and associate it with the output to achieve the purpose of information screening.

[0047] Based on the above description of the nouns and terms involved in the embodiments of the present application, the embodiments of the present application are described in detail below. The embodiments of the present application provide an image enhancement method, device, electronic device, computer readable storage medium and computer program product, which can improve the image enhancement effect of facial images.

[0048] It should be noted that the collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0049] The following describes the image enhancement system provided by the embodiment of the present application. Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the image enhancement system provided in the embodiment of the present application. To support an exemplary application, the image enhancement system 100 includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, using a wireless or wired link to achieve data transmission.

[0050] Here, the terminal 400 (for example, a client running image enhancement support) sends a facial image and an image enhancement request for the facial image to the server 200 in response to an image enhancement instruction for the facial image to be enhanced, where the facial image is an image of the face of the target object; the server 200 receives the facial image and the image enhancement request; in response to the image enhancement request, obtains latent variables of the facial image, and adds noise to the latent variables to obtain noisy latent variables of the facial image; extracts facial features of the face in the facial image; denoises the noisy latent variables in combination with the facial features to obtain denoised latent variables of the facial image; reconstructs the denoised latent variables to obtain a facial enhanced image of the facial image; returns the facial enhanced image to the terminal 400; the terminal 400 receives and displays the facial enhanced image.

[0051] In some embodiments, the image enhancement method provided in the embodiments of the present application is implemented by an electronic device, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, video, games, metaverse, image generation, etc.

[0052] In some embodiments, the electronic device for implementing the image enhancement method provided in the embodiments of the present application may be various types of terminals or servers. Among them, the server (such as server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal (such as terminal 400) may be a laptop, a tablet computer, a desktop computer, a smart phone, an intelligent voice interaction device (such as a smart speaker), a smart home appliance (such as a smart TV), a smart watch, a vehicle-mounted terminal, a wearable device, a virtual reality (VR) device, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited to this.

[0053] In some embodiments, the image enhancement method provided in the embodiments of the present application can be implemented with the aid of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model application, which can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing resources and storage resources. As an example, a server (such as server 200) can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0054] In some embodiments, the image enhancement method provided in the embodiments of the present application can be implemented with the aid of blockchain technology. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. As an example, multiple servers can be composed of a blockchain, and the server is a node on the blockchain. There can be an information connection between each node in the blockchain, and information can be transmitted between nodes through the information connection. Among them, the data related to the image enhancement method provided in the embodiments of the present application (such as image denoising model, image enhancement model, and face enhanced image, etc.) can be stored on the blockchain.

[0055] In some embodiments, the terminal or server can implement the image enhancement method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0056] The electronic device for implementing the image enhancement method provided by the embodiment of the present application is described below. Figure 2 , Figure 2 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. The electronic device 500 provided in the embodiment of the present application can be a terminal or a server. Figure 2 As shown, the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not used in the following examples. Figure 2 Various buses are labeled as bus system 540 .

[0057] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0058] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0059] The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may include one or more storage devices physically located away from the processor 510. The memory 550 includes a volatile memory or a non-volatile memory, and may also include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0060] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0061] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0062] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0063] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0064] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0065] In some embodiments, the image enhancement device provided in the embodiments of the present application can be implemented in a software manner. Figure 2An image enhancement device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, including the following software modules: an acquisition module 5551, an extraction module 5552, a denoising module 5553, and a reconstruction module 5554. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0066] The following describes the image enhancement method provided by the embodiment of the present application. As mentioned above, the image enhancement method provided by the embodiment of the present application can be implemented by the server or the terminal alone, or by the server and the terminal in collaboration. Therefore, the execution subject of each step will not be repeatedly described below. Figure 3 , Figure 3 : is a flow chart of an image enhancement method provided in an embodiment of the present application. The image enhancement method provided in an embodiment of the present application includes:

[0067] Step 101: Obtain latent variables of the facial image to be enhanced, and add noise to the latent variables to obtain noisy latent variables of the facial image.

[0068] The facial image is an image of the face of the target object.

[0069] In step 101, when performing image enhancement on a facial image, the latent variables of the facial image may be first obtained. For example, the facial image may be encoded to obtain the latent variables of the facial image; here, the encoding process may be implemented by an encoder, and the encoding process is a downsampling process. After the latent variables of the facial image are obtained, noise is added to the latent variables to obtain the noisy latent variables of the facial image. For example, the noise may be Gaussian white noise, etc., and the noisy latent variables are obtained by adding Gaussian white noise to the latent variables.

[0070] It should be noted that the facial image is an image of the face of the target object. The target object may be a virtual object, such as a virtual person, a virtual animal, a virtual cartoon character, a virtual game character, etc., or a real object, such as a real person or object. The facial image may be obtained by performing facial recognition on an object image of the target object, such as performing facial detection on the object image, determining the facial region in the object image, and cutting out the facial region from the object image to obtain the facial image. The facial image is an image that needs to be enhanced, and the process of image enhancement is the process of refining the content of the image. For example, the facial image may be a facial sketch (such as a facial sketch of a game character designed by art design), and the facial sketch may be refined by performing image enhancement on the facial sketch.

[0071] Step 102: extract facial features of the face in the facial image.

[0072] Step 103: De-noising the noisy latent variables in combination with the facial features to obtain the de-noised latent variables of the facial image.

[0073] In step 102, facial features of the face may be extracted from the facial image. In step 103, the noisy latent variables may be denoised in combination with the facial features to obtain denoised latent variables of the facial image.

[0074] In some embodiments, facial features may be extracted in the following manner: performing facial key point extraction on a facial image to obtain facial key point information of the facial image, and performing a second feature extraction on the facial key point information to obtain facial features of the face; or performing edge detection on the facial image to obtain facial contour line information of the facial image, and performing a third feature extraction on the facial contour line information to obtain facial features of the face; or performing facial element extraction on the facial image to obtain facial element information of the facial image, and performing a fourth feature extraction on the facial element information to obtain facial features of the face.

[0075] It should be noted that the facial features may be at least one of the following features: facial key point features extracted from facial key point information, facial contour line features extracted from facial contour line information, and facial element features extracted from facial element information. Here, facial element information includes but is not limited to skin, hair, color, style, texture, light and shadow, etc.

[0076] In practical applications, the extraction of facial key point information can be achieved through facial key point detection technology (also known as facial key point positioning or facial alignment), that is, given a facial image, the key areas of the face are located, including eyebrows, eyes, nose, mouth, facial contours, etc. For example, a 68-point annotation scheme can be used to divide facial key points into internal key points and contour key points. The internal key points include eyebrows, eyes, nose, mouth, etc., a total of 51 key points, and the contour key points include 17 key points. Facial contour line information can be extracted using edge detection algorithms such as Canny. Facial element features can be extracted using a facial element extraction model; or the facial image can be directly encoded to obtain an encoding result, and the encoding result can be used as the facial element feature.

[0077] In some embodiments, the extraction of facial features can be achieved by cascading M layers of first coding layers, where M is an integer greater than 0. The facial feature extraction process includes: calling the first layer of the M layers of first coding layers to encode the facial image and obtain the facial features output by the first layer of the first coding layer; calling the i-th layer of the M layers of first coding layers to encode the facial features output by the (i-1)-th layer of the first coding layer to obtain the facial features output by the i-th layer of the first coding layer; traversing i to obtain the facial features output by each layer of the first coding layer of the M layers of first coding layers, where i is an integer greater than 0 and not greater than M.

[0078] Based on this, the above denoising process can be implemented by a first image denoising model; the first image denoising model includes a first encoder and a first decoder, the first decoder includes a cascade of M layers of first decoding layers, and the first decoding layer corresponds to the first encoding layer one by one. The denoising process includes: calling the first encoder to encode the noisy latent variable to obtain the encoded latent variable; calling the Mth first decoding layer of the M-layer first decoding layer to decode the encoded latent variable and the facial features output by the Mth first encoding layer to obtain the decoded latent variable output by the Mth first decoding layer; calling the i-th first decoding layer of the M-layer first decoding layer to decode the decoded latent variable output by the (i+1)th first decoding layer and the facial features output by the i-th first encoding layer to obtain the decoded latent variable output by the i-th first decoding layer; traversing i to obtain the decoded latent variable output by the 1st first decoding layer of the M-layer first decoding layer, and using the decoded latent variable output by the 1st first decoding layer as the denoised latent variable obtained by denoising the noisy latent variable.

[0079] In some embodiments, the facial features can be extracted from the facial fusion information of the facial key point information and the facial contour line information, so as to ensure that the facial features can represent both the facial key point features and the facial contour line features. Based on this, the facial features can also be called facial structural features. Figure 4 , Figure 3 Step 102 shown may be performed by Figure 4 Steps 1021 to 1023 shown are implemented as follows: Step 1021, extracting facial key points of the facial image to obtain facial key point information of the facial image, and performing edge detection on the facial image to obtain facial contour line information of the facial image; Step 1022, fusing the facial key point information and the facial contour line information to obtain facial fusion information of the facial image; Step 1023, performing first feature extraction on the facial fusion information to obtain facial features of the face.

[0080] It should be noted that the extraction of facial key point information and facial contour line information in step 1021 can adopt the above-described extraction method of facial key point information and facial contour line information, which will not be repeated here. In step 1022, the facial key point information and facial contour line information are fused to obtain facial fusion information. In some embodiments, step 1022 can be implemented by the following steps: performing convolution processing of multiple different scales on the facial key point information to obtain first convolution features at each scale, and merging the first convolution features at each scale to obtain a first merged feature; performing convolution processing of multiple different scales on the facial contour line information to obtain second convolution features at each scale, and merging the second convolution features at each scale to obtain a second merged feature; splicing the first merged feature and the second merged feature to obtain a first spliced ​​feature, and using the first spliced ​​feature as the facial fusion information of the facial image.

[0081] In the process of implementing step 1022, the scale of the convolution processing of the facial key point information and the scale of the convolution processing of the facial contour line information may be the same or different. Figure 5 , the scales of the convolution processing are 3x3, 5x5, and 7x7 respectively. Here, the facial key point information is convolved at each scale to obtain the first convolution feature at each scale, and the first convolution features at each scale are merged to obtain the first merged feature; the facial contour line information is convolved at each scale to obtain the second convolution feature at each scale, and the second convolution features at each scale are merged to obtain the second merged feature; finally, the first merged feature and the second merged feature are spliced ​​to obtain the first spliced ​​feature, which is the facial fusion information obtained by fusing the facial key point information and the facial contour line information. In practical applications, the first convolution feature and the second convolution feature can both be in the form of feature maps, and can be merged according to the feature channel dimension of the feature map when merging, and can also be spliced ​​according to the feature channel dimension of the feature map when splicing.

[0082] In step 1023, the process of first feature extraction can be implemented by cascading M layers of first coding layers, where M is an integer greater than 0. Based on this, the process of first feature extraction includes: calling the first layer of the first coding layer of the M layers of first coding layers, encoding the facial fusion information, and obtaining the facial features of the face output by the first layer of the first coding layer; calling the i-th layer of the first coding layer of the M layers of first coding layers, encoding the facial features output by the (i-1)-th layer of the first coding layer, and obtaining the facial features of the face output by the i-th layer of the first coding layer; traversing i, obtaining the facial features of the face output by each layer of the first coding layer of the M layers of first coding layers, where i is an integer greater than 0 and not greater than M. It should be noted that i here can start from 1 until traversing to M, and i is incremented by 1 each time during the traversal process.

[0083] Here, the first coding layer plays a downsampling role, and the encoding process is the downsampling process. The first coding layer may include a convolution layer and a self-attention layer. For example, when the first coding layer of the first layer encodes the facial fusion information, the facial fusion information may be firstly self-attention processed to obtain the self-attention result, and then the self-attention result may be convolution processed to obtain the facial features of the face output by the first coding layer of the first layer. For another example, when the first coding layer of the i-th layer encodes, the facial features output by the first coding layer of the (i-1)th layer may be firstly self-attention processed to obtain the self-attention result, and then the self-attention result may be convolution processed to obtain the facial features of the face output by the first coding layer of the i-th layer. In actual implementation, the cascaded M-layer first coding layer constitutes a feature extraction model, for example, the feature extraction model may be a controlnet model.

[0084] In some embodiments, the denoising process is implemented by a first image denoising model, the first image denoising model includes a first encoder and a first decoder, the first decoder includes a cascade of M layers of first decoding layers, and the first decoding layer corresponds to the first encoding layer one by one. Based on this, the denoising process includes: calling the first encoder to encode the noisy latent variable to obtain the encoded latent variable; calling the Mth first decoding layer of the M-layer first decoding layer to decode the encoded latent variable and the facial features output by the Mth first encoding layer to obtain the decoded latent variable output by the Mth first decoding layer; calling the i-th first decoding layer of the M-layer first decoding layer to decode the decoded latent variable output by the (i+1)th first decoding layer and the facial features output by the i-th first encoding layer to obtain the decoded latent variable output by the i-th first decoding layer; traversing i to obtain the decoded latent variable output by the 1st first decoding layer of the M-layer first decoding layer, and using the decoded latent variable output by the 1st first decoding layer as the denoised latent variable obtained by denoising the noisy latent variable. It should be noted that i here can start from M and traverse to 1. During the traversal process, i decreases by 1 each time.

[0085] In some embodiments, the encoded latent variables and the facial features output by the first encoding layer of the Mth layer can be decoded in the following manner to obtain the decoded latent variables output by the first decoding layer of the Mth layer: based on the first weight value of the encoded latent variables and the second weight value of the facial features output by the first encoding layer of the Mth layer, the encoded latent variables and the facial features output by the first encoding layer of the Mth layer are weighted summed to obtain the second spliced ​​features; the second spliced ​​features are decoded to obtain the decoded latent variables output by the first decoding layer of the Mth layer; accordingly, the (i+1)th layer can be decoded in the following manner The decoding latent variables output by the first decoding layer and the facial features output by the first encoding layer of the i-th layer are decoded to obtain the decoding latent variables output by the first decoding layer of the i-th layer: based on the third weight value of the decoding latent variables output by the first decoding layer of the (i+1)-th layer and the fourth weight value of the facial features output by the first encoding layer of the i-th layer, the decoding latent variables output by the first decoding layer of the (i+1)-th layer and the facial features output by the first encoding layer of the i-th layer are weighted summed to obtain the third splicing feature; the third splicing feature is decoded to obtain the decoding latent variables output by the first decoding layer of the i-th layer.

[0086] It should be noted that the first weight value, the second weight value, the third weight value and the fourth weight value may be preset, and may be adjusted according to actual conditions during actual implementation.

[0087] As an example, see Figure 6 The image enhancement model includes a feature extraction model (such as a controlnet model), a first image denoising model, a denoising model, a face information fusion model, an image encoder and an image decoder; wherein the first image denoising model is actually a Unet network model, including a first encoder of Unet and a first decoder of Unet, the feature extraction model includes a cascaded M-layer first encoding layer, the first decoder includes a cascaded M-layer first decoding layer, and the first decoding layer corresponds to the first encoding layer one by one.

[0088] So, based on Figure 6The image enhancement model shown, 1) calls the image encoder to encode the facial image to be enhanced to obtain a hidden variable. 2) calls the noise addition model to add noise to the hidden variable to obtain the noisy hidden variable. 3) calls the facial information fusion model to fuse the facial key point information and the facial contour line information to obtain the facial fusion information. 4) calls the feature extraction model (including the cascaded M-layer first encoding layer) to extract facial structural features: calls the first layer of the M-layer first encoding layer to encode the facial fusion information to obtain the facial structural features output by the first layer of the first encoding layer; calls the i-th layer of the M-layer first encoding layer to encode the facial structural features output by the (i-1)-th layer of the first encoding layer to obtain the facial structural features output by the i-th layer of the first encoding layer; traverses i to obtain the facial structural features output by each layer of the first encoding layer of the M-layer first encoding layer. 5) Calls the first encoder to encode the noisy hidden variable to obtain the encoded hidden variable. 6) Call the Mth first decoding layer of the Mth first decoding layer to decode the encoded latent variables and the facial features output by the Mth first encoding layer to obtain the decoded latent variables output by the Mth first decoding layer; call the ith first decoding layer of the Mth first decoding layer to decode the decoded latent variables output by the (i+1)th first decoding layer and the facial features output by the ith first encoding layer to obtain the decoded latent variables output by the ith first decoding layer; traverse i to obtain the decoded latent variables output by the 1st first decoding layer of the Mth first decoding layer, and use the decoded latent variables output by the 1st first decoding layer as the denoised latent variables obtained by denoising the noisy latent variables. 7) Call the image decoder to decode the denoised latent variables to obtain the face enhanced image of the face image.

[0089] By applying the above embodiment, in the image enhancement process of the facial image, the denoising process of the noisy latent variables adds the control of the facial features of the facial image (i.e., the facial structural features extracted from the facial fusion information of the facial key point information and the facial contour information), that is, the denoising process of the noisy latent variables is performed in combination with the facial structural features, thereby reducing the loss of the facial structural features of the facial image, so that the finally obtained facial enhanced image can maintain more facial structural features of the facial image, thereby improving the image enhancement effect of the facial image.

[0090] In some embodiments, the facial features extracted by the first feature extraction operation are facial structural features; and after extracting the facial structural features of the face in the facial image, facial element features of the face in the facial image may also be extracted. Based on this, the noisy latent variables may be encoded in the following manner to obtain the encoded latent variables: the noisy latent variables may be encoded in combination with the facial element features to obtain the encoded latent variables.

[0091] It should be noted that encoding the noisy latent variable to obtain the encoded latent variable is achieved by the first encoder. Figure 6 , the first encoder includes a cascade of P third encoding layers, where P is an integer greater than 0; each third encoding layer includes a convolution layer and a self-attention layer; based on this, the process of calling the first encoder to encode the noisy latent variable includes: splicing the noisy latent variable and the facial element feature to obtain the sixth splicing feature; calling the first third encoding layer of the P third encoding layers, performing self-attention processing on the sixth splicing feature to obtain the first self-attention result, and encoding the first self-attention result to obtain the encoding result output by the first third encoding layer; splicing the encoding result output by the (p-1)th third encoding layer and the facial element feature to obtain the seventh splicing feature; calling the pth third encoding layer of the P third encoding layers, performing self-attention processing on the seventh splicing feature to obtain the pth self-attention result, and encoding the pth self-attention result to obtain the encoding result output by the pth third encoding layer; traversing p to obtain the encoding result output by the pth third encoding layer, where p is an integer greater than 0 and not greater than P.

[0092] It should be noted that the facial element features can be obtained by encoding the facial image through the first encoder, and the facial element features are used to characterize the skin, hair, color, style, texture, light and shadow, texture and other features of the facial image. P can be equal to M, for example Figure 6 The P and M shown are both 4. The first coding layer and the first decoding layer include a convolution layer and a self-attention layer, respectively. It can be understood that there are P first coding layers, each of which includes a corresponding self-attention layer, and self-attention processing needs to be performed. In practical applications, only some of the N first coding layers can be added to control the facial element features (that is, the facial element features need to be spliced ​​before self-attention processing) to reduce the amount of calculation of self-attention processing, improve calculation efficiency and reduce computing resource usage; of course, if the computing resources are sufficient, the facial element features can also be added to each of the P second coding layers (that is, the facial element features need to be spliced ​​before self-attention processing) to improve the image enhancement effect.

[0093] By applying the above embodiment, in the image enhancement process of the facial image, in addition to adding the control of the facial features of the facial image (i.e., the facial structural features extracted from the facial fusion information of the facial key point information and the facial contour information), the facial element features (used to characterize the skin, hair, color, style, texture, light and shadow and other features of the facial image) are also added to the denoising process of the noisy latent variables. That is, the facial structural features and the facial element features are combined to perform denoising on the noisy latent variables, thereby reducing the loss of the facial structural features and the facial element features of the facial image, so that the finally obtained facial enhanced image can not only maintain more facial structural features of the facial image, but also maintain rich facial element features, so that the elements such as skin, hair, color, style, texture, light and shadow are more realistic and natural, and the image enhancement effect of the facial image is further improved.

[0094] In some embodiments, the denoising process is implemented by a second image denoising model, the second image denoising model includes a second encoder and a second decoder, the second encoder includes a cascaded N-layer second encoding layer, N is an integer greater than 0; based on this, combined with facial features, the noisy latent variable is denoised to obtain a denoised latent variable, including: splicing the noisy latent variable and the facial features to obtain a fourth spliced ​​feature; calling the first second encoding layer of the N-layer second encoding layer, performing self-attention processing on the fourth spliced ​​feature to obtain a first self-attention result, and encoding the first self-attention result to obtain The encoding result output by the second encoding layer of the 1st layer; the encoding result output by the second encoding layer of the (j-1)th layer and the facial features are spliced ​​to obtain the fifth spliced ​​feature; the second encoding layer of the jth layer of the Nth layer of the second encoding layer is called to perform self-attention processing on the fifth spliced ​​feature to obtain the jth self-attention result, and the jth self-attention result is encoded to obtain the encoding result output by the second encoding layer of the jth layer; traverse j to obtain the encoding result output by the second encoding layer of the Nth layer, where j is an integer greater than 0 and not greater than N; the encoding result output by the second encoding layer of the Nth layer is decoded to obtain a denoising latent variable.

[0095] It should be noted that the facial features may include at least one of the following features: facial key point features extracted from facial key point information, facial contour line features extracted from facial contour line information, and facial element features extracted from facial element information. It is understandable that there are N second coding layers, each of which includes a corresponding self-attention layer, and self-attention processing needs to be performed. In practical applications, the control of the facial features can be added to only some of the N second coding layers (i.e., the facial features need to be spliced ​​before self-attention processing) to reduce the amount of calculation of self-attention processing, improve calculation efficiency, and reduce the occupation of computing resources; of course, if the computing resources are sufficient, the control of the facial features can also be added to each of the N second coding layers (i.e., the facial features need to be spliced ​​before self-attention processing).

[0096] As an example, see Figure 7 The image enhancement model includes a feature extraction model, a second image denoising model, a denoising model, an image encoder and an image decoder; wherein the second image denoising model is actually a Unet network model, including a second Unet encoder and a second Unet decoder, and the second encoder includes a cascaded N-layer second encoding layer.

[0097] So, based on Figure 7 The image enhancement model shown, 1) calling the image encoder to encode the face image to be enhanced to obtain hidden variables. 2) calling the noise addition model to add noise to the hidden variables to obtain the noisy hidden variables. 3) calling the feature extraction model to extract features from the face image to obtain facial features. 4) splicing the noisy hidden variables and facial features to obtain the fourth splicing feature; calling the first second encoding layer of the N-layer second encoding layer to perform self-attention processing on the fourth splicing feature to obtain the first self-attention result, and encoding the first self-attention result to obtain the encoding result output by the first second encoding layer; splicing the encoding result output by the (j-1)-th second encoding layer and the facial features to obtain the fifth splicing feature; calling the j-th second encoding layer of the N-layer second encoding layer to perform self-attention processing on the fifth splicing feature to obtain the j-th self-attention result, and encoding the j-th self-attention result to obtain the encoding result output by the j-th second encoding layer; traversing j to obtain the encoding result output by the N-th second encoding layer. 5) Decode the encoding result output by the second encoding layer of the Nth layer to obtain a denoised latent variable. 6) Call the image decoder to decode the denoised latent variable to obtain a face enhanced image of the face image. It should be noted that j here can start from 1 until it traverses to N, and j increases by 1 each time during the traversal process.

[0098] By applying the above embodiment, in the image enhancement process of the facial image, the denoising process of the noisy latent variables adds the control of the facial features of the facial image (for example, the facial structural features extracted from the facial fusion information of the facial key point information and the facial contour information, and the facial element features of the skin, hair, color, style, texture, light and shadow, etc., which are used to characterize the facial image), that is, the denoising process is performed on the noisy latent variables in combination with the facial features, thereby reducing the loss of the facial features of the facial image, so that the facial enhanced image finally obtained can maintain more facial features of the facial image, and improve the image enhancement effect of the facial image.

[0099] In practical applications, the image enhancement model can be based on an image generation diffusion model, such as a stable diffusion model, a deep float model, etc.; the image denoising model can be a Denoising Unet model based on a Unet network.

[0100] In some embodiments, the above denoising process may include T denoising processes, where T is an integer greater than 0; based on this, combined with facial features, the denoising process can be performed on the noisy latent variable in the following manner to obtain the denoised latent variable of the facial image: combined with facial features, the denoising process is performed on the noisy latent variable for the first time to obtain the intermediate denoising latent variable output by the first denoising process; combined with facial features, the intermediate denoising latent variable output by the (t-1)th denoising process is performed for the tth time to obtain the intermediate denoising latent variable output by the tth denoising process; traverse t to obtain the intermediate denoising latent variable output by the Tth denoising process, and use the intermediate denoising latent variable output by the Tth denoising process as the denoising latent variable of the facial image. It should be noted that t here can start from 1 until it is traversed to T, and t increases by 1 each time during the traversal process.

[0101] It should be noted that the denoising process in the embodiment of the present application is a forward diffusion process, that is, noise is gradually added to the latent variable until the noisy latent variable is obtained; and the denoising process is a reverse diffusion process, that is, the noisy latent variable is subjected to T denoising processes to gradually remove the noise and obtain the denoised latent variable. The T denoising processes can be understood as denoising processes of T time steps, and a denoising process is completed once in each time step. Each denoising process can be implemented using the denoising logic provided in the above embodiment.

[0102] Step 104: Perform image reconstruction on the denoised latent variables to obtain a face-enhanced image of the face image.

[0103] In step 104, the image reconstruction process is to decode the denoising latent variables to obtain a face-enhanced image of the face image. The decoding process can be implemented by a decoder, and the decoding process is an upsampling process.

[0104] In some embodiments, the facial image belongs to the object image of the target object, that is, the facial image is obtained by performing facial recognition on the object image of the target object; therefore, after performing image reconstruction on the denoised latent variables to obtain a face-enhanced image of the facial image, the face-enhanced image can also be added to the object-enhanced image of the object image to improve the object-enhanced image. Specifically, this can be achieved through the following process: obtaining the object-enhanced image of the object image, and determining the target area in the object-enhanced image corresponding to the facial image; replacing the target area with the face-enhanced image to obtain a first enhanced image.

[0105] Since the image enhancement process of the face image and the object image is performed separately, when the face enhancement image is added to the object enhancement image, there is distortion and misalignment at the connection of the two images, so it is necessary to repair the distortion and misalignment at the image connection. The face enhancement image has multiple image edges, so in some embodiments, the first enhanced image can be repaired in the following way: generate a rectangular target gradient map corresponding to each image edge, the length of the target gradient map is the same as the length of the image edge, the brightness of the target gradient map decreases from the center line to both sides, and the length of the center line is the length of the target gradient map; add each target gradient map to the target image to obtain an image mask, the pixel value of each pixel point of the target image is 0, and the center line of each target gradient map in the image mask coincides with the corresponding image edge in the face enhancement image; based on the image mask, repair the first enhanced image to obtain a second enhanced image.

[0106] It should be noted that see Figure 8 When generating the target gradient map, 1) an initial gradient map with the same length as the image edge of the face enhancement image is generated, and the brightness of the initial gradient map starts from the first image edge and gradually decreases toward the second image edge, and the first image edge and the second image edge are in a relative relationship. Figure 8 As shown in (1), the length of the image edge of the face-enhanced image is 256, then a 256*256 initial gradient image is generated, and the brightness of the initial gradient image starts from the first image edge at the bottom and gradually decreases toward the second image edge at the top until the brightness of the second image edge is 0. 2) Generate a completely black intermediate image, the length of the intermediate image can be the length of a certain edge of the face-enhanced image, and the width of the intermediate image can be the width of an image tile, such as Figure 8As shown in (2), a 256*64 completely black intermediate image is generated. 3) From the first image side with the highest brightness of the initial gradient image, a rectangular gradient image with a width of tile / 2 and a length equal to the length of the first image side is cut out, as shown in Figure 8 As shown in (3), a 256*32 rectangular gradient image is obtained. 4) Paste the rectangular gradient image to the first area of ​​the intermediate image, as shown in Figure 8 5) Rotate the rectangular gradient image 180 degrees, and paste the rotated rectangular gradient image to the second area of ​​the intermediate image to obtain the target gradient image, as shown in (4). Figure 8 As shown in (5), the first area and the second area constitute the intermediate image. Thus, a target gradient map of an image edge is obtained. The length of the target gradient map is the same as the length of the image edge. The brightness of the target gradient map decreases from the center line to both sides. The length of the center line is the length of the target gradient map.

[0107] After obtaining the target gradient map of each image edge, each target gradient map is added to the target image to obtain an image mask, such as Figure 8 As shown in (6). Here, since the face enhanced image is a square, it is only necessary to calculate the target gradient map of one image edge, and apply the target gradient map to each image edge. For the image mask, the center line of each target gradient map in the image mask coincides with the corresponding image edge in the face enhanced image. In this way, the first enhanced image can be repaired based on the image mask to obtain the second enhanced image.

[0108] By applying the above-mentioned embodiment of the present application, the latent variables of the facial image to be enhanced are first obtained, and noise is added to the latent variables to obtain the noisy latent variables of the facial image; then the facial features of the face in the facial image are extracted; thereby, the noisy latent variables are denoised in combination with the facial features to obtain the denoised latent variables of the facial image; the denoised latent variables are image reconstructed to obtain the facial enhanced image of the facial image. Here, in the image enhancement process of the facial image, the denoising process of the noisy latent variables is increased by controlling the facial features of the facial image, that is, the noisy latent variables are denoised in combination with the facial features, thereby reducing the loss of facial features of the facial image, so that the facial enhanced image finally obtained can retain more facial features of the facial image and improve the image enhancement effect of the facial image.

[0109] The following takes image enhancement (or image refinement) of a face sketch (ie, a face image) as an example to illustrate an exemplary application of an embodiment of the present application in an actual application scenario.

[0110] In art design, the design of an object (such as a person or an animal) is usually divided into three stages, including concept setting, sketch design, and sketch refinement. Among them, (1) Concept setting stage: designers conduct preliminary design and conception based on the basic requirements and characteristics of the object image discussed with customers or team members (such as the gender, age, personality, etc. of the object) to obtain a preliminary concept setting. (2) Sketch design stage: designers usually use tools such as sketches or hand-drawing to further design and conceive the concept setting to obtain the corresponding object sketch. (3) Sketch refinement stage: designers usually further polish and refine the object sketch to obtain a more detailed and realistic object image. Designers will make more detailed designs and concepts for each part of the object, including facial expressions, clothing details, muscle lines, etc. Since each step in the above object image design process requires manual participation of designers, the production efficiency of original painting design is very low. Therefore, the production efficiency of original painting design can be improved by automating the process, such as automating the sketch refinement stage.

[0111] Sketch refinement refers to the process in which the original artist adds details, clothing, light and shadow effects to a sketch that is 30%-50% complete to obtain a refined image. In order to achieve a better refinement effect for the object sketch, the face sketch can be refined separately. In related technologies, noise is often gradually added to the face image until it becomes a completely random noise image; then, starting from the random noise image, the noise is gradually removed to obtain the final face enhancement image. However, since the denoising intensity is difficult to control during the denoising process, the final face enhancement image is prone to lose a large number of original features, resulting in poor enhancement effect of the face enhancement image.

[0112] Based on this, an embodiment of the present application provides an image enhancement method to at least solve the above-mentioned problems. An embodiment of the present application provides an image enhancement model (i.e., a face refinement model), which is based on an image generative diffusion model. In order to enhance the feature retention of the face sketch, the embodiment of the present application 1) adds facial structural features including facial key points and facial contour lines on the basis of the image generative diffusion model; 2) adds facial element features of the face sketch in the self-attention processing of the image generative diffusion model. In this way, after obtaining the face refinement image of the face sketch, the face refinement image and the character refinement image (obtained by refining the character sketch corresponding to the face sketch) can be seamlessly spliced ​​by stitching and repairing with an image gradient mask.

[0113] Next, we will explain the image generation diffusion model. The image generation diffusion model is a generation model based on the diffusion process, which is used to generate high-quality images with rich textures and details. The image generation process of the image generation diffusion model is a diffusion process in which noise gradually fades away. In this diffusion process, the model starts from an original image containing random noise, gradually removes the noise of the initial image, and obtains the generated image. The generation process of the image generation diffusion model is divided into a forward diffusion process and a reverse diffusion process. Among them, the forward diffusion process is that the model gradually adds noise to the original image until it completely becomes a random noise image. The reverse diffusion process is that the model starts from a completely random noise image and gradually removes the noise to obtain the generated image. In the reverse diffusion process, the model needs to learn how to remove noise at each time step. Therefore, the model usually adopts a neural network structure, such as a convolutional neural network (CNN) or a variational autoencoder (VAE).

[0114] As an example, see Fig. 9 , Fig. 9 : is a structural diagram of the image generation diffusion model provided in the embodiment of the present application. Here, the image generation diffusion model is implemented based on a stable diffusion network, including an encoder (Encoder), a decoder (Decoder), a denoising module (Diffusion Process), and an image denoising model (based on a Unet network, such as Denoising Unet). Among them, the encoder is used to convert the input image x into a latent variable (latent representation) z. The latent variable can capture the important features of the input image and help the model remove noise during the reverse diffusion process. The denoising module is used to perform denoising processing on the latent variable (such as adding Gaussian white noise) to obtain the noisy latent variable z T The image denoising model is used to add noise to the latent variable z T Perform denoising to obtain the denoising latent variable z0. The denoising process is divided into multiple time steps. The image denoising model can predict the noise item that needs to be removed at each time step, and remove the corresponding noise at each time step to obtain z in turn. T-1 、z T-2 , ......z1, z0. The decoder is used to reconstruct the image based on the denoising latent variable z0 to obtain the generated image During model training, it learns to generate images by minimizing the reconstruction error, which is calculated based on the difference between the generated image and the original image. By optimizing the reconstruction error, the model can learn to gradually remove noise during the back-diffusion process, ultimately generating high-quality images.

[0115] As an example, see Fig.10 Based on the image enhancement method provided in the embodiment of the present application, the sketch to be refined can be processed as follows: (1) The sketch to be refined is subjected to face detection through a face detection model (such as a facexlib model) to obtain a face sketch; the face sketch can be adjusted to a preset resolution, such as 1024*1024. (2) The face sketch is subjected to face refinement processing through a face refinement model (i.e., a face enhancement model) to obtain a face refinement image; in this way, the face in the face sketch can be automatically modified and polished, and rich texture and light and shadow information can be added to obtain a face refinement image. (3) Through the stitching repair model, the face refinement image and the refinement results of other parts of the sketch to be refined are spliced ​​by using a mask repair method to obtain a refined image of the sketch to be refined.

[0116] Next, the image enhancement method provided in the embodiment of the present application is described in detail. (1) The image enhancement method provided in the embodiment of the present application can be used to refine or enhance the content of a face sketch. The face sketch contains facial structural features, such as facial features and face shape, but lacks details and light and shadow effects. The face refinement model can enrich the details of the facial skin, hair, light and shadow, so that the face after refinement and polishing can be spliced ​​back into the character refinement image. (2) In practical applications, face refinement and facial feature retention are a contradiction. Art designers hope to enrich the content of the face sketch as much as possible, so that elements such as skin, light and shadow, and hair are as realistic as possible, and hope to retain the facial features of the original face in the face sketch as much as possible, that is, face refinement needs to enrich the details without losing key information. Therefore, a) the face refinement model is required to generate as many new elements as possible, and b) the face refinement model is required to retain specific elements in the face sketch as much as possible, such as facial features and face shape. (3) The face refinement model is based on an image generation diffusion model. For example, the image generation diffusion model may be a stable diffusion model, a deep float_if model, etc.

[0117] As an example, see Figure 6 , Figure 6 Schematic diagram of the structure of the image enhancement model provided in the embodiment of the present application. The image enhancement model includes a face information fusion model, a feature extraction model (such as controlnet), an image generation diffusion model (such as Fig. 9 Here, Figure 6The image denoising model in the image generation diffusion model is shown. The image denoising model is actually a Unet network model, including the first encoder of Unet and the first decoder of Unet. The first encoder includes multiple encoding layers, each encoding layer is composed of a convolutional network and a self-attention network; the first decoder includes multiple decoding layers, each decoding layer is also composed of a convolutional network and a self-attention network; the feature extraction model includes multiple feature extraction layers, each feature extraction layer is also composed of a convolutional network and a self-attention network.

[0118] Specifically, (1) in order to increase the structural control in the image refinement process, a facial key point extraction algorithm is used to extract facial key point information from the facial sketch x; and an edge detection algorithm (such as the Canny edge detection algorithm) is used to extract facial contour line information from the facial sketch x. The facial key point information and the facial contour line information are fused by a facial information fusion model to obtain facial fusion information. The facial fusion information is subjected to feature extraction by a feature extraction model to obtain facial fusion features (i.e., the above-mentioned facial structural features). The facial fusion features are input into the first decoder of Unet, so that the first decoder of Unet can combine the facial fusion features and the first encoder of Unet from the noise-added latent variable x. t The multi-level image features extracted (obtained by adding noise to the latent variables of the face sketch x) generate denoised latent variables x0, thereby generating a face refinement image based on the denoised latent variables x0 In this way, the facial structural features of the face sketch can be used to influence the generated face refinement image. (2) The facial element features (including color, style, texture, light and shadow, etc.) of the face sketch are added to the self-attention network of Unet, so that the model also pays attention to the facial element features when performing self-attention processing, thereby ensuring that the facial style information in addition to the contour lines and key points can also be retained in the face refinement image generated by denoising. The following are explained separately.

[0119] (1) Structural control module based on facial key point information and facial contour line information. The basic requirement of face refinement is to ensure that the face refinement image retains the facial structural features (including key points and contour lines) in the face sketch and supplements it with richer texture, skin, light and shadow information. In order to keep the facial structural features of the face sketch as much as possible, the facial key point information and facial contour line information of the face sketch are extracted separately through the pre-trained model; the facial fusion information of the facial key point information and facial contour line information is input into the feature extraction model (such as controlnet) to extract the facial fusion features; through a multi-level connection method, the facial fusion features are fused with the image features of the noisy face sketch extracted by the first encoder of Unet. Since the facial element features of the input face sketch are always affected during the image generation process, the generated face refinement image can retain more facial element features of the face sketch.

[0120] In practical applications, facial key point detection is also called facial key point positioning or facial alignment. It means locating the key areas of the face, including eyebrows, eyes, nose, mouth, facial contour, etc., given a facial image. For example, a 68-point annotation scheme can be used to divide facial key points into internal key points and contour key points. The internal key points include eyebrows, eyes, nose, mouth, etc., a total of 51 key points, and the contour key points include 17 key points. This can be achieved through a facial key point detection model. Since facial key points alone cannot control the face shape, facial contour lines are added as additional constraints. Facial contour lines can be extracted using edge detection algorithms such as Canny.

[0121] In practical applications, since facial key point information and facial contour line information belong to information with different semantics, the embodiment of the present application additionally designs a facial information fusion model for processing. Figure 5 , Figure 5 : is a structural diagram of the facial information fusion model provided in the embodiment of the present application. Here, the facial information fusion model includes three groups of convolutional layers of different scales, which are 3x3, 5x5, and 7x7 respectively. The facial information fusion model is used to perform feature extraction processing on the facial key point information and the facial contour line information respectively to obtain their respective feature maps; then the two feature maps are spliced ​​along the feature channel dimension to obtain facial fusion information. The facial fusion information is used as the input of the feature extraction model (such as ControlNet). Here, the multi-scale feature extraction and fusion method can obtain richer feature information and enhance the fusion effect.

[0122] (2) Image supervision self-attention module. On the basis of the structural control of key points and contour lines, supervision of facial element features such as color, style, texture, light and shadow of the face sketch is further added. Specifically, in the self-attention network of Unet, the key-value is expanded to the concatenation of the current feature itself and the facial element features. After the noisy latent variables of the face sketch are processed by the image encoder of Unet, a latent variable with a lower dimension is obtained, and the matrix transformation weights of the key-value are shared with the self-attention network. In an embodiment of the present application, the self-attention network includes both attention to the face sketch itself and attention to the face in the face sketch, so that the generation process better maintains the features of the face in the face sketch.

[0123] The backbone network Unet (i.e., image denoising model) is mainly composed of a stack of convolutional networks and self-attention networks. Among them, the self-attention network searches for useful information in the image after weight changes, because each self-attention process allows each image block in the image to interact. In some embodiments, the self-attention network linearly projects the input feature v into Q, K, V:

[0124] Q=W Q v, K = W K v,V=W V v;

[0125] Among them, W Q , W K , W V is a learnable linear transformation matrix; the self-attention processing calculation formula is:

[0126]

[0127] In the embodiment of the present application, the facial element features of the face sketch are additionally added to the self-attention network as a supervision signal, such as Fig.11 Specifically, assuming that the face sketch is x sup , the first encoder of Unet obtains the hidden variable z sup , add the corresponding linear transformation matrix For linear projection. In this way, the features obtained by the linear projection of the self-attention network are:

[0128]

[0129] Correspondingly, the self-attention processing calculation formula is:

[0130]

[0131] It should be noted that since the output dimension of the self-attention network is the same as the dimension of Q, the output feature dimension of the self-attention network remains unchanged before and after the facial element features are added to the self-attention network, which can maintain the network connection relationship of Unet and facilitate the modification of Unet.

[0132] In practical applications, the self-attention calculation of facial element features can be added only to part of the self-attention network of Unet to reduce the amount of calculation and improve the calculation efficiency. As an example, see Fig.12 , Fig.12 is a schematic diagram of the structure of the image denoising model provided in the embodiment of the present application. Here, Fig.12 The Unet structure shown contains 3 downsampling. It can be understood that the computational complexity of self-attention processing in IN1 and IN2 layers is very high, reaching (64×64+64×64) 2 Therefore, the self-attention calculation of facial element features can be added to the IN1 and IN2 layers instead of the downsampled IN4, IN5, IN7, IN8, MID layers, and their corresponding output layers, thereby reducing the amount of self-attention processing calculations. Of course, if computing power allows, self-attention calculations of facial element features can also be added to each self-attention network of Unet to further improve the image refinement effect.

[0133] It should be noted that the facial information fusion model and feature extraction model in the face refinement model need to be trained, and the linear transformation matrix of the self-attention network and It is also a parameter that needs to be determined through training. Other parameters of the image generation diffusion model in the face refinement model (such as other layers of Unet, encoders, and decoders) can use pre-trained weights, and frozen parameters do not participate in training. In actual implementation, when initializing training, you can use Instead of random initialization, the convergence speed of model training is accelerated. During model training, 1) the training data is a large number of facial images. The key points and contours of the facial images are extracted during preprocessing. At the same time, based on image degradation processing (such as Gaussian blur), the corresponding "original image" data is generated. 2) Exemplarily, the training parameters include: batch size equal to 64, learning rate equal to 1e-4, training steps 50000, optimizer Adam; in actual implementation, the training parameters can be adjusted as needed. 3) The loss function can use the image reconstruction loss function L LDM , as shown below:

[0134]

[0135] Among them, ∈ represents Gaussian noise, θ represents the network parameters of Unet; z trepresents the noise added latent variable; t represents the sampling time point.

[0136] (3) Mask stitching repair module based on image gradient. Figure 1 It is usually part of the character sketch of a complete character, so in order to "stitch" the face refinement image into the character refinement image (that is, the refinement results of other parts of the character sketch, such as the character's body, background, etc.), it is necessary to deal with the distortion and dislocation at the connection of the images. This distortion and dislocation is caused by the face and the character's body coming from different stages of the refinement process. See Fig.13 , Fig.13 FIG. 1 is a schematic diagram of the stitching and repairing result of the face thinning image provided in the embodiment of the present application. Fig.13 As shown in (1), the face refinement image is directly pasted back into the character refinement image. The unrepaired image shows obvious stitching marks, while Fig.13 As shown in (2), the joints of the restored images are more natural. In this regard, the present application provides a mask restoration strategy based on image gradients to restore the joints of images, which can significantly reduce the distortion and dislocation of image splicing and make the transition of the joints of images more natural.

[0137] Specifically, an image mask is first generated, and then the inpainting function of stable diffusion is used to perform stitching repair based on the image mask. Figure 8 , Figure 8 : is a schematic diagram of generating an image mask provided in an embodiment of the present application. The process of generating the image mask includes:

[0138] (1) Generate a gradient map (i.e., the target gradient map mentioned above). Here, Figure 8 (5) shows the generated gradient map, which indicates that the repair intensity in the middle (white highlight) is large and gradually decreases from the middle to both sides, so that the transition effect at the joint of the repaired image is smoother. Figure 8 , in practical applications, the gradient map can be generated by the following code:

[0139] gradient = Image.linear_gradient("L"); / / Generate a 256*256 initial gradient map. The brightness of the initial gradient map starts from the first image edge at the bottom and gradually decreases toward the second image edge at the top until the brightness of the second image edge is 0. Figure 8 As shown in (1);

[0140] row_gradient = Image.new("L",(face.width,tile_height),"black"); / / Generate a 256*64 completely black intermediate image, such as Figure 8 As shown in (2);

[0141] row_gradient.paste(gradient.resize((face.width,tile_height / / 2),resample=Image.BICUBIC),(0,0)); / / Crop to get a 256*32 rectangular gradient image, and paste the rectangular gradient image to the first area of ​​the middle image, such as Figure 8 As shown in (3) and (4);

[0142] row_gradient.paste(gradient.rotate(180).resize((face.width,tile_height / / 2),resam ple=Image.BICUBIC),(0,tile_height / / 2)); / / Rotate the rectangular gradient image 180 degrees, and paste the rotated rectangular gradient image to the second area of ​​the intermediate image to obtain the target gradient image, such as Figure 8 As shown in (5).

[0143] Among them, face.width represents the width of the face sketch, and tile_height=64 represents the width of the gradient map.

[0144] (2) Apply the gradient map to the four edges of the face sketch to obtain an image mask for repairing the image connection, such as Figure 8 As shown in (6) in the figure, based on the image mask, the image inpainting function of the stable diffusion can be called to achieve image stitching and repair.

[0145] It should be noted that the embodiments of the present application can be applied to various scenarios (such as game original design scenarios, animation or comic design scenarios, advertising design scenarios, etc.) including the refinement of face sketches, and different model parameters can be designed according to the scene requirements. It should be noted that the faces appearing in the drawings provided in the embodiments of the present application are synthetic, not real faces.

[0146] The application of the above embodiments of the present application can not only improve the efficiency of face sketch refinement processing, but also improve the effect of face sketch refinement processing. Fig.14 , Fig.14 is a schematic diagram of a face enhancement image provided by an embodiment of the present application, where the face enhancement image is obtained by (1) super-resolution technology, (2) image generation diffusion model, and (3) refinement of the face sketch by an embodiment of the present application. Fig.14It can be seen that compared with the face enhanced images generated by other methods, the face enhanced images generated by the embodiments of the present application better maintain the facial features of the face in the face sketch and enrich the details of the face sketch, making the skin, light and shadow, hair, texture, style and other elements more realistic and natural.

[0147] The following is a description of an exemplary structure of the image enhancement device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the image enhancement device 555 of the memory 550 may include: an acquisition module 5551, used to acquire the latent variables of the facial image to be enhanced, and add noise to the latent variables to obtain the noisy latent variables of the facial image, wherein the facial image is an image of the face of the target object; an extraction module 5552, used to extract the facial features of the face in the facial image; a denoising module 5553, used to denoise the noisy latent variables in combination with the facial features to obtain the denoised latent variables of the facial image; and a reconstruction module 5554, used to reconstruct the denoised latent variables to obtain a face-enhanced image of the facial image.

[0148] In some embodiments, the extraction module 5552 is further used to extract facial key points of the facial image to obtain facial key point information of the facial image, and to perform edge detection on the facial image to obtain facial contour line information of the facial image; to fuse the facial key point information and the facial contour line information to obtain facial fusion information of the facial image; and to perform a first feature extraction on the facial fusion information to obtain facial features of the face.

[0149] In some embodiments, the first feature extraction process is implemented by cascading M layers of first coding layers, where M is an integer greater than 0; the extraction module 5552 is also used to call the 1st layer of the M layers of first coding layers to encode the facial fusion information to obtain the facial features of the face output by the 1st layer of the first coding layer; call the i-th layer of the M layers of first coding layers to encode the facial features output by the (i-1)th layer of the first coding layer to obtain the facial features of the face output by the i-th layer of the first coding layer; traverse the i to obtain the facial features of the face output by each layer of the first coding layer of the M layers of first coding layers, where i is an integer greater than 0 and not greater than the M.

[0150] In some embodiments, the denoising process is implemented by a first image denoising model, wherein the first image denoising model includes a first encoder and a first decoder, wherein the first decoder includes a cascaded M-layer first decoding layer, wherein the first decoding layer corresponds to the first encoding layer one by one; the denoising module 5553 is further used to call the first encoder to encode the noisy latent variable to obtain an encoded latent variable; call the Mth first decoding layer of the M-layer first decoding layer to decode the encoded latent variable and the facial features output by the Mth first encoding layer to obtain a decoded latent variable output by the Mth first decoding layer; call the i-th first decoding layer of the M-layer first decoding layer to decode the decoded latent variable output by the (i+1)th first decoding layer and the facial features output by the i-th first encoding layer to obtain a decoded latent variable output by the i-th first decoding layer; traverse the i to obtain a decoded latent variable output by the 1st first decoding layer of the M-layer first decoding layer, and use the decoded latent variable output by the 1st first decoding layer as the denoised latent variable obtained by denoising the noisy latent variable.

[0151] In some embodiments, the facial features are facial structural features; the extraction module 5552 is further used to extract facial element features of the face in the facial image after extracting the facial features of the face in the facial image; the denoising module 5553 is further used to encode the noisy latent variables in combination with the facial element features to obtain encoded latent variables.

[0152] In some embodiments, the denoising module 5553 is further used to perform weighted summation of the encoded latent variables and the facial features output by the M-th first encoding layer based on the first weight value of the encoded latent variables and the second weight value of the facial features output by the M-th first encoding layer to obtain a second splicing feature; decode the second splicing feature to obtain a decoded latent variable output by the M-th first decoding layer; the denoising module 5553 is further used to perform weighted summation of the decoded latent variables output by the (i+1)-th first decoding layer and the facial features output by the i-th first encoding layer based on the third weight value of the decoded latent variables output by the (i+1)-th first decoding layer and the fourth weight value of the facial features output by the i-th first encoding layer to obtain a third splicing feature; decode the third splicing feature to obtain a decoded latent variable output by the i-th first decoding layer.

[0153] In some embodiments, the extraction module 5552 is further used to perform convolution processing of multiple different scales on the facial key point information to obtain a first convolution feature at each scale, and merge the first convolution features at each scale to obtain a first merged feature; perform convolution processing of multiple different scales on the facial contour line information to obtain a second convolution feature at each scale, and merge the second convolution features at each scale to obtain a second merged feature; splice the first merged feature and the second merged feature to obtain a first spliced ​​feature, and use the first spliced ​​feature as the facial fusion information of the facial image.

[0154] In some embodiments, the extraction module 5552 is also used to extract facial key points of the facial image to obtain facial key point information of the facial image, and perform a second feature extraction on the facial key point information to obtain facial features of the face; or, perform edge detection on the facial image to obtain facial contour line information of the facial image, and perform a third feature extraction on the facial contour line information to obtain facial features of the face; or, perform facial element extraction on the facial image to obtain facial element information of the facial image, and perform a fourth feature extraction on the facial element information to obtain facial features of the face.

[0155] In some embodiments, the denoising process is implemented by a second image denoising model, the second image denoising model includes a second encoder and a second decoder, the second encoder includes a cascaded N-layer second encoding layer, and N is an integer greater than 0; the denoising module 5553 is also used to splice the noise-added latent variable and the facial feature to obtain a fourth spliced ​​feature; call the first second encoding layer of the N-layer second encoding layer, perform self-attention processing on the fourth spliced ​​feature, obtain the first self-attention result, and encode the first self-attention result to obtain the output of the first second encoding layer encoding result; splicing the encoding result output by the (j-1)th second encoding layer and the facial features to obtain a fifth splicing feature; calling the jth second encoding layer of the Nth second encoding layer, performing self-attention processing on the fifth splicing feature to obtain the jth self-attention result, and encoding the jth self-attention result to obtain the encoding result output by the jth second encoding layer; traversing the j to obtain the encoding result output by the Nth second encoding layer, where j is an integer greater than 0 and not greater than the N; decoding the encoding result output by the Nth second encoding layer to obtain the denoising latent variable.

[0156] In some embodiments, the facial image belongs to the object image of the target object; the reconstruction module 5554 is further used to obtain the object enhanced image of the object image after performing image reconstruction on the denoised latent variables to obtain the face enhanced image of the facial image, and determine the target area in the object enhanced image corresponding to the facial image; replace the target area with the face enhanced image to obtain a first enhanced image.

[0157] In some embodiments, the face-enhanced image has multiple image edges; the reconstruction module 5554 is further used to generate a rectangular target gradient map corresponding to each image edge after replacing the target area with the face-enhanced image to obtain a first enhanced image, the length of the target gradient map is the same as the length of the image edge, the brightness of the target gradient map decreases from the center line to both sides, and the length of the center line is the length of the target gradient map; each target gradient map is added to the target image to obtain an image mask, the pixel value of each pixel point of the target image is 0, and the center line of each target gradient map in the image mask coincides with the corresponding image edge in the face-enhanced image; based on the image mask, the first enhanced image is repaired to obtain a second enhanced image.

[0158] In some embodiments, the acquisition module 5551 is further used to encode the facial image to be enhanced to obtain the latent variable; the reconstruction module 5554 is further used to decode the denoising latent variable to obtain the facial enhanced image of the facial image.

[0159] In some embodiments, the denoising process includes T denoising processes, where T is an integer greater than 0; the denoising module 5553 is further used to perform a first denoising process on the noisy latent variable in combination with the facial features to obtain an intermediate denoising latent variable output by the first denoising process; perform a t-th denoising process on the intermediate denoising latent variable output by the (t-1)th denoising process in combination with the facial features to obtain an intermediate denoising latent variable output by the t-th denoising process; traverse the t to obtain an intermediate denoising latent variable output by the Tth denoising process, and use the intermediate denoising latent variable output by the Tth denoising process as the denoising latent variable of the facial image.

[0160] It should be noted that the description of the device embodiment in this application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, which will not be described in detail here. The unfinished technical details of the image enhancement device provided in the embodiment of this application can be understood based on the description of the technical details in the method embodiment above.

[0161] The embodiment of the present application also provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions or computer program from the computer readable storage medium, and the processor executes the computer executable instructions or computer program, so that the electronic device performs the image enhancement method provided in the embodiment of the present application.

[0162] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image enhancement method provided in the embodiment of the present application.

[0163] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0164] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0165] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0166] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0167] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image enhancement method, characterized in that: The method comprises: Acquire a latent variable of a facial image to be enhanced, and add noise to the latent variable to obtain a noisy latent variable of the facial image, wherein the facial image is an image of the face of a target object; Extracting facial features of the face in the facial image; Combining the facial features, denoising the noisy latent variables to obtain denoised latent variables of the facial image; Performing image reconstruction on the denoising latent variables to obtain a face-enhanced image of the face image.

2. The method according to claim 1, characterized in that The extracting facial features of the face in the facial image includes: Extracting facial key points from the facial image to obtain facial key point information of the facial image, and performing edge detection on the facial image to obtain facial contour line information of the facial image; fusing the facial key point information and the facial contour line information to obtain facial fusion information of the facial image; Performing a first feature extraction on the facial fusion information to obtain facial features of the face.

3. The method according to claim 2, characterized in that The first feature extraction process is implemented by cascading M layers of first coding layers, where M is an integer greater than 0; the first feature extraction is performed on the facial fusion information to obtain the facial features of the face, including: Calling the first encoding layer of the M layers of first encoding layers to encode the facial fusion information, and obtaining facial features of the face output by the first encoding layer of the first layer; Calling the i-th first coding layer of the M-layer first coding layer to encode the facial features output by the (i-1)-th first coding layer to obtain the facial features of the face output by the i-th first coding layer; Traverse the i to obtain the facial features of the face output by each first encoding layer of the M first encoding layers, where i is an integer greater than 0 and not greater than M.

4. The method according to claim 3, characterized in that The denoising process is implemented by a first image denoising model, wherein the first image denoising model includes a first encoder and a first decoder, wherein the first decoder includes a cascade of M first decoding layers, and the first decoding layer corresponds to the first encoding layer in one-to-one correspondence; the denoising process is performed on the noisy latent variable in combination with the facial features to obtain the denoised latent variable, including: Calling the first encoder to encode the noisy latent variable to obtain an encoded latent variable; Calling the Mth first decoding layer of the Mth first decoding layer to decode the encoded latent variable and the facial features output by the Mth first encoding layer to obtain a decoded latent variable output by the Mth first decoding layer; Calling the i-th first decoding layer of the M-layer first decoding layers to decode the decoding latent variables output by the (i+1)-th first decoding layer and the facial features output by the i-th first encoding layer to obtain the decoding latent variables output by the i-th first decoding layer; Traversing the i, obtaining the decoded latent variable output by the first decoding layer of the M first decoding layers, and using the decoded latent variable output by the first decoding layer of the first layer as the denoised latent variable obtained by denoising the denoised latent variable.

5. The method according to claim 4, characterized in that The facial features are facial structural features; after extracting the facial features of the face in the facial image, the method further includes: Extracting facial element features of the face in the facial image; The step of encoding the noise added latent variable to obtain the encoded latent variable comprises: The noisy latent variable is encoded in combination with the facial element features to obtain an encoded latent variable.

6. The method according to claim 4, characterized in that The decoding of the encoded latent variables and the facial features output by the first encoding layer of the Mth layer to obtain the decoded latent variables output by the first decoding layer of the Mth layer includes: Based on the first weight value of the encoded latent variable and the second weight value of the facial feature output by the first encoding layer of the Mth layer, weighted sum is performed on the encoded latent variable and the facial feature output by the first encoding layer of the Mth layer to obtain a second splicing feature; Decoding the second concatenated features to obtain a decoding latent variable output by the first decoding layer of the Mth layer; The decoding latent variable output by the (i+1)th first decoding layer and the facial features output by the i-th first encoding layer are decoded to obtain the decoding latent variable output by the i-th first decoding layer, including: Based on the third weight value of the decoded latent variable output by the (i+1)th first decoding layer and the fourth weight value of the facial feature output by the i-th first encoding layer, weighted sum is performed on the decoded latent variable output by the (i+1)th first decoding layer and the facial feature output by the i-th first encoding layer to obtain a third splicing feature; The third concatenated feature is decoded to obtain a decoding latent variable output by the first decoding layer of the i-th layer.

7. The method according to claim 2, characterized in that The step of fusing the facial key point information and the facial contour line information to obtain facial fusion information of the facial image includes: Performing convolution processing of multiple different scales on the facial key point information to obtain first convolution features at each scale, and merging the first convolution features at each scale to obtain a first merged feature; Performing convolution processing of multiple scales on the facial contour line information to obtain second convolution features at each scale, and merging the second convolution features at each scale to obtain a second merged feature; The first combined feature and the second combined feature are spliced ​​to obtain a first spliced ​​feature, and the first spliced ​​feature is used as the facial fusion information of the facial image.

8. The method according to claim 1, characterized in that The extracting facial features of the face in the facial image includes: Extracting facial key points from the facial image to obtain facial key point information of the facial image, and performing second feature extraction on the facial key point information to obtain facial features of the face; Alternatively, edge detection is performed on the facial image to obtain facial contour line information of the facial image, and a third feature extraction is performed on the facial contour line information to obtain facial features of the face; Alternatively, facial elements are extracted from the facial image to obtain facial element information of the facial image, and fourth features are extracted from the facial element information to obtain facial features of the face.

9. The method according to claim 1, characterized in that The denoising process is implemented by a second image denoising model, the second image denoising model includes a second encoder and a second decoder, the second encoder includes a cascaded N-layer second encoding layer, and N is an integer greater than 0; the denoising process is performed on the noisy latent variable in combination with the facial features to obtain the denoised latent variable, including: The noisy latent variable and the facial feature are concatenated to obtain a fourth concatenated feature; the first second coding layer of the N second coding layers is called to perform self-attention processing on the fourth concatenated feature to obtain a first self-attention result, and the first self-attention result is encoded to obtain an encoding result output by the first second coding layer; The encoding result output by the (j-1)th second encoding layer and the facial feature are concatenated to obtain a fifth concatenated feature; the jth second encoding layer of the Nth second encoding layer is called to perform self-attention processing on the fifth concatenated feature to obtain a jth self-attention result, and the jth self-attention result is encoded to obtain an encoding result output by the jth second encoding layer; Traversing the j to obtain the encoding result output by the second encoding layer of the Nth layer, where j is an integer greater than 0 and not greater than the N; The encoding result output by the Nth second encoding layer is decoded to obtain the denoising latent variable.

10. The method according to claim 1, characterized in that The facial image belongs to the object image of the target object; after the image reconstruction is performed on the denoising latent variables to obtain the face enhanced image of the facial image, the method further includes: Acquire an object enhanced image of the object image, and determine a target area in the object enhanced image corresponding to the facial image; The target area is replaced with the face enhanced image to obtain a first enhanced image.

11. The method according to claim 1, characterized in that The face enhancement image has a plurality of image edges; after replacing the target area with the face enhancement image to obtain a first enhanced image, the method further includes: Generate a rectangular target gradient map corresponding to each of the image sides respectively, the length of the target gradient map is the same as the length of the image side, the brightness of the target gradient map decreases from the center line to both sides, and the length of the center line is the length of the target gradient map; Adding each of the target gradient images to the target image to obtain an image mask, wherein the pixel value of each pixel point of the target image is 0, and the midline of each of the target gradient images in the image mask coincides with the corresponding image edge in the face enhanced image; Based on the image mask, the first enhanced image is restored to obtain a second enhanced image.

12. The method according to claim 1, characterized in that The step of obtaining the latent variable of the facial image to be enhanced comprises: encoding the facial image to be enhanced to obtain the latent variable; The performing image reconstruction on the denoising latent variables to obtain the face-enhanced image of the face image includes: decoding the denoising latent variables to obtain the face-enhanced image of the face image.

13. The method according to claim 1, characterized in that The denoising process includes T denoising processes, where T is an integer greater than 0; the denoising process is performed on the noisy latent variable in combination with the facial features to obtain the denoised latent variable of the facial image, including: In combination with the facial features, the first denoising process is performed on the noisy latent variable to obtain an intermediate denoising latent variable output by the first denoising process; In combination with the facial features, performing a t-th denoising process on the intermediate denoising latent variable output by the (t-1)-th denoising process to obtain an intermediate denoising latent variable output by the t-th denoising process; The t is traversed to obtain an intermediate denoising latent variable output by the T-th denoising process, and the intermediate denoising latent variable output by the T-th denoising process is used as the denoising latent variable of the facial image.

14. An image enhancement device, characterized in that: The device comprises: An acquisition module, used for acquiring latent variables of a facial image to be enhanced, and adding noise to the latent variables to obtain noisy latent variables of the facial image, wherein the facial image is an image of the face of a target object; An extraction module, used for extracting facial features of the face in the facial image; A denoising module, used to denoise the noisy latent variable in combination with the facial features to obtain a denoised latent variable of the facial image; The reconstruction module is used to perform image reconstruction on the denoising latent variables to obtain a face-enhanced image of the face image.

15. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the image enhancement method according to any one of claims 1 to 13 when executing the computer executable instructions stored in the memory.

16. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image enhancement method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image enhancement method according to any one of claims 1 to 13 is implemented.

Images