Image enhancement method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

The present application provides an image enhancement method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent traffic and assisted driving. The method comprises: acquiring a latent variable of an object image to be enhanced, and adding noise to the latent variable to obtain a noisy latent variable of the object image, the object image being an image of a target object; extracting object structure features of the target object in the object image; on the basis of the object structure features, denoising the noisy latent variable to obtain a denoised latent variable of the object image; and on the basis of the denoised latent variable, performing image reconstruction to obtain a first object enhanced image of the object image.

Description

Image enhancement method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 2023110586437 and application date of August 21, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of artificial intelligence technology, and in particular to an image enhancement method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0004] Artificial Intelligence (AI) is a comprehensive field in computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0005] Image enhancement is also a key application area for artificial intelligence. Related technologies often enhance object images (such as sketches of people) by gradually adding noise to the image until it becomes a completely random noise image. Starting from the random noise image, the noise is then gradually removed to obtain the final object-enhanced image. However, due to the difficulty in controlling the denoising intensity during the denoising process, the final object-enhanced image tends to lose a large number of original features, resulting in poor enhancement effects.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image enhancement method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the image enhancement effect of an object image.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] The present application provides an image enhancement method, which is applied to an electronic device and includes:

[0010] Acquire a latent variable of an object image to be enhanced, and add noise to the latent variable to obtain a noisy latent variable of the object image, where the object image is an image of a target object;

[0011] extracting object structural features of the target object in the object image;

[0012] Denoising the noisy latent variable in combination with the object structure feature to obtain a denoised latent variable of the object image;

[0013] Performing image reconstruction on the denoising latent variables to obtain a first object enhanced image of the object image.

[0014] The present application also provides an image enhancement device, including:

[0015] an acquisition module configured to acquire a latent variable of an object image to be enhanced, and add noise to the latent variable to obtain a noisy latent variable of the object image, wherein the object image is an image of a target object;

[0016] an extraction module configured to extract object structural features of the target object in the object image;

[0017] a denoising module configured to perform denoising processing on the noisy latent variable in combination with the object structure feature to obtain a denoised latent variable of the object image;

[0018] A reconstruction module is configured to perform image reconstruction on the denoising latent variable to obtain a first object enhanced image of the object image.

[0019] An embodiment of the present application further provides an electronic device, including:

[0020] a memory configured to store computer-executable instructions;

[0021] The processor is configured to implement the image enhancement method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0022] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the image enhancement method provided in the embodiment of the present application is implemented.

[0023] An embodiment of the present application further provides a computer program product, comprising computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the image enhancement method provided in the embodiment of the present application is implemented.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] Applying the above-mentioned embodiment of the present application, the latent variables of the object image to be enhanced are first obtained, and noise is added to the latent variables to obtain the noisy latent variables of the object image; then, the object structural features of the target object in the object image are extracted; thereby, the noisy latent variables are denoised in combination with the object structural features to obtain the denoised latent variables of the object image; finally, the denoised latent variables are image reconstructed to obtain a first object-enhanced image of the object image. Here, during the image enhancement process of the object image, the denoising process of the noisy latent variables is supplemented with the control of the object structural features of the object image, that is, the denoising process of the noisy latent variables is combined with the object structural features, thereby reducing the loss of the object structural features of the object image during the denoising process, so that the object-enhanced image obtained in the end can retain more of the object structural features of the object image, thereby improving the image enhancement effect of the object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram of the architecture of an image enhancement system provided in an embodiment of the present application;

[0027] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0028] FIG3 is a flow chart of an image enhancement method according to an embodiment of the present application;

[0029] FIG4 is a flow chart of an image enhancement method according to an embodiment of the present application;

[0030] FIG5 is a processing diagram of a structural fusion model provided in an embodiment of the present application;

[0031] FIG6 is a schematic diagram of the structure of an image enhancement model provided in an embodiment of the present application;

[0032] FIG7 is a schematic diagram of the structure of the image generation diffusion model provided in an embodiment of the present application;

[0033] FIG8 is a flow chart of an image enhancement method according to an embodiment of the present application;

[0034] FIG9 is a schematic diagram of controlling denoising processing by object structural features according to an embodiment of the present application;

[0035] FIG10 is a schematic diagram of an image super-resolution processing model provided in an embodiment of the present application;

[0036] FIG11 is a schematic diagram of an image mask of an image block provided in an embodiment of the present application;

[0037] FIG12 is a schematic diagram showing the display of an object enhanced image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) Client: An application running in a terminal that provides various services, such as a client that supports image enhancement.

[0044] 2) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0045] 3) U-Net model: It is an algorithm that uses a fully convolutional network for semantic segmentation.

[0046] 4) Attention: A method that mimics human attention and aims to quickly filter out high-value information from a large amount of information. It is primarily used to address the difficulty of obtaining a reasonable vector representation for long input sequences in time series models. This approach involves retaining the intermediate results of the time series model, learning them with a new model, and associating them with the output to achieve information filtering.

[0047] 5) Latent variable: Also known as a latent variable or potential variable, in contrast to an observed variable, a latent variable refers to a random variable that is not directly observable. Latent variables can be inferred from observed data using mathematical models. A mathematical model that explains observed variables using latent variables is called a latent variable model. In machine learning, a latent variable, while not directly observable, explains the behavior or characteristics of observable data to some extent. Latent variable models assume that observed data are generated by latent variables, which themselves are unobservable. By analyzing the observed data, the value of the latent variable can be inferred.

[0048] 6) Receptive Field: In machine learning, particularly deep learning, the receptive field refers to the portion of the input data that a neuron or group of neurons in a neural network can influence. It is a crucial concept because it determines which features in the input data the feature detectors (i.e., neurons) in the neural network can detect. In convolutional neural networks, the receptive field is typically related to the filter size and stride length. As the filter slides over the input image (i.e., performing the convolution operation), each step covers a new area of ​​the image, which is the filter's receptive field. As the network becomes deeper, the filter size tends to increase, and therefore the receptive field also increases. The importance of the receptive field lies in determining the size and location of features that the model can capture. The receptive field is a key concept in deep learning, directly influencing which features a model can learn from the input data and how these features affect the model's performance on specific tasks.

[0049] 7) Encoding: In AI, encoding refers to converting input data into a form more suitable for processing, typically a numerical representation, so that it can be analyzed and learned by machine learning algorithms. The encoding process can be a direct conversion of raw data into numbers or by abstracting and simplifying the data to create a more compact and useful representation.

[0050] 8) Decoding: In artificial intelligence, decoding generally refers to converting data in a certain format into another format that can be understood or manipulated. The decoding process can involve various types of data, including text, images, audio, etc. In machine vision, decoding may involve converting image data into recognizable objects, scenes, or emotional states. For example, an image is encoded by a convolutional neural network, and then the encoded image features are converted back to the original image data through a decoder (such as the decoder in a generative adversarial network). In short, decoding is a key concept in artificial intelligence that involves converting complex digital representations into a data format that is easier to understand and manipulate. The purpose of decoding is to extract useful information from the input data and convert it into a task that the machine can understand and execute.

[0051] Based on the above description of the nouns and terms involved in the embodiments of the present application, the embodiments of the present application are described in detail below. The embodiments of the present application provide an image enhancement method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the image enhancement effect of an object image.

[0052] It should be noted that the collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0053] The following describes an image enhancement system provided by an embodiment of the present application. Referring to Figure 1 , FIG1 is a schematic diagram of the architecture of the image enhancement system provided by an embodiment of the present application. To support an exemplary application, image enhancement system 100 includes a server 200, a network 300, and a terminal 400. Terminal 400 is connected to server 200 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of the two, using either wireless or wired links for data transmission.

[0054] Here, the terminal 400 (for example, running a client that supports image enhancement) responds to an image enhancement instruction for an object image to be enhanced, and sends an object image and an image enhancement request for the object image to the server 200, where the object image is an image of a target object; the server 200 receives the object image and the image enhancement request; in response to the image enhancement request, obtains latent variables of the object image, and adds noise to the latent variables to obtain noisy latent variables of the object image; extracts object structural features of the target object in the object image; denoises the noisy latent variables in combination with the object structural features to obtain denoised latent variables of the object image; reconstructs the denoised latent variables to obtain a first object enhanced image of the object image; returns the first object enhanced image to the terminal 400; the terminal 400 receives and displays the first object enhanced image.

[0055] In some embodiments, the image enhancement method provided in the embodiments of the present application is implemented by an electronic device. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server in collaboration. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, video, animation, games, metaverse, image generation, user-generated content (UGC), etc.

[0056] In some embodiments, the electronic device for implementing the image enhancement method provided in the embodiments of the present application may be various types of terminals or servers. Among them, the server (such as server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal (such as terminal 400) may be a laptop, a tablet computer, a desktop computer, a smart phone, an intelligent voice interaction device (such as a smart speaker), a smart home appliance (such as a smart TV), a smart watch, a car terminal, a wearable device, a virtual reality (VR) device, etc., but is not limited to this. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application do not limit this.

[0057] In some embodiments, the image enhancement method provided in the embodiments of the present application can be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing resources and storage resources. As an example, the server (such as server 200) can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0058] In some embodiments, the image enhancement method provided in the embodiments of the present application can be implemented with the aid of blockchain technology. Blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. As an example, multiple servers can be composed into a blockchain, and the servers are nodes on the blockchain. There can be an information connection between each node in the blockchain, and information can be transmitted between nodes through the information connection. Among them, the data related to the image enhancement method provided in the embodiments of the present application (such as image denoising model, image enhancement model, and object enhanced image (such as first object enhanced image)) can be stored on the blockchain.

[0059] In some embodiments, the terminal or server can implement the image enhancement method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0060] The following describes an electronic device for implementing an image enhancement method provided in an embodiment of the present application. Referring to FIG. 2 , FIG. 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 provided in an embodiment of the present application can be a terminal or a server. As shown in FIG. 2 , the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It can be understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are labeled as bus system 540 in FIG.

[0061] In some embodiments, the image enhancement device provided in the embodiments of the present application can be implemented in software. Figure 2 shows an image enhancement device 555 stored in the memory 550, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 5551, an extraction module 5552, a denoising module 5553 and a reconstruction module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0062] The image enhancement method provided in the embodiment of the present application is described. As previously mentioned, the image enhancement method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. Therefore, the execution entities of each step will not be repeatedly described below. Referring to Figure 3, Figure 3 is a flow chart of the image enhancement method provided in the embodiment of the present application. The image enhancement method provided in the embodiment of the present application includes:

[0063] Step 101: Obtain a latent variable of the object image to be enhanced, and add noise to the latent variable to obtain the noisy latent variable of the object image.

[0064] The object image is an image of a target object.

[0065] In step 101, when performing image enhancement on an object image, the latent variables of the object image can first be obtained. For example, the object image can be encoded to obtain the latent variables of the object image; here, the encoding process can be implemented by an encoder, and the encoding process is a downsampling process, that is, by downsampling the object image, the object image is encoded to obtain the latent variables of the object image. After the latent variables of the object image are obtained, noise is added to the latent variables to obtain the noisy latent variables of the object image. For example, the noise can be sampled from a target distribution; the target distribution includes multiple random data that conform to a specific data distribution type (such as normal distribution, standard normal distribution, uniform distribution), and the random data that conform to the specific data distribution type can be generated based on a random data generation algorithm. For example, the noise can be Gaussian noise, Poisson noise, salt and pepper noise, Gaussian white noise, etc. Taking Gaussian noise as an example, a random data generation algorithm can be used to generate random data that conforms to the Gaussian distribution, and then a target random number that conforms to the Gaussian distribution is sampled from the random data that conforms to the Gaussian distribution, and the target random number that conforms to the Gaussian distribution is used as the noise.

[0066] It should be noted that the object image is an image of a target object. The target object can be a virtual object, such as a virtual person, virtual animal, virtual anime character, virtual game character, etc., or a real object, such as a real person or object. The object image is an image that requires image enhancement. The image enhancement process is the process of refining the image content. For example, the object image can be an object sketch (such as a virtual character sketch for art design). By performing image enhancement on the object sketch, the object sketch can be refined.

[0067] Step 102: Extracting object structural features of the target object in the object image.

[0068] Step 103: De-noising the noisy latent variables in combination with the object structure features to obtain the de-noised latent variables of the object image.

[0069] In step 102, the object structure features of the target object are extracted from the object image. In step 103, the noisy latent variables can be denoised in combination with the object structure features to obtain the denoised latent variables of the object image, thereby controlling the denoising process through the object structure features of the object image.

[0070] In some embodiments, the object structure features of the target object in the object image can be extracted by performing the following steps: obtaining at least one of the following object information of the target object in the object image: a depth map of the object image, object posture information of the target object, and object line drawing information of the target object; performing feature extraction on the object information to obtain object structure features.

[0071] Here, the object structural features may include at least one of the following features: object posture features extracted from the object posture information of the target object (used to characterize the posture movement of the target object, etc.), object line draft features extracted from the object line draft information of the target object (used to characterize the structural details, contours, etc. of the target object), and depth map features extracted from the depth map of the object image, where the depth map records the distance from the pixel point in the object image to the camera and can characterize the surface structural features of the target object in the object image. It should be noted that the object posture information can be obtained by performing object posture detection on the object image, such as by using a pre-trained posture detection model; the object line draft information can be obtained by performing object line draft extraction on the object image, such as by using a pre-trained line draft extraction model; and the depth map can be obtained by performing depth map estimation on the object image, such as by using a pre-trained depth estimation model.

[0072] In some embodiments, the process of extracting the feature of the object structure feature can be implemented by cascading M layers of first coding layers, where M is an integer greater than 0. Based on this, taking the extraction of object posture features from object posture information as an example, the process of extracting the feature of the object structure feature includes: calling the first layer of the first coding layer of the M layers of first coding layers to encode the fused object posture information to obtain the object structure feature output by the first layer of the first coding layer; calling the i-th layer of the first coding layer of the M layers of first coding layers to encode the object structure feature output by the (i-1)-th layer of the first coding layer to obtain the object structure feature output by the i-th layer of the first coding layer; traversing i to obtain the object structure feature output by each layer of the first coding layer of the M layers of first coding layers, where i is an integer greater than 0 and not greater than M. It should be noted that i here can start from 1 and traverse to M, and i increases by 1 each time during the traversal process.

[0073] Based on this, the above-mentioned denoising process can be implemented by an image denoising model, which includes a first encoder and a first decoder, and the first decoder includes a cascade of M layers of first decoding layers, and the first decoding layer corresponds to the first encoding layer one-to-one; the denoising process includes: calling the first encoder to encode the noisy latent variable to obtain the encoded latent variable; calling the Mth first decoding layer of the M-layer first decoding layer to decode the encoded latent variable and the object structure features output by the M-layer first encoding layer to obtain the decoded latent variable output by the M-layer first decoding layer; calling the i-th first decoding layer of the M-layer first decoding layer to decode the decoded latent variable output by the (i+1)-th first decoding layer and the object structure features output by the i-th first encoding layer to obtain the decoded latent variable output by the i-th first decoding layer; traversing i to obtain the decoded latent variable output by the 1st first decoding layer of the M-layer first decoding layer, and using the decoded latent variable output by the 1st first decoding layer as the denoised latent variable obtained by denoising the noisy latent variable. It should be noted that i here can start from M and traverse to 1. During the traversal process, i decreases by 1 each time.

[0074] It should be noted that the process of extracting object line features from object line information and the process of extracting depth map features from the depth map are similar to the above-mentioned process of extracting object posture features from object posture information, and will not be repeated here. Furthermore, the object structure features (including at least one of the object posture features, object line features, and depth map features) can be combined to perform denoising on the noisy latent variables to obtain the denoised latent variables of the object image. In this way, the control of the object structure features of the object image (including the object structure features extracted from at least one of the object information of the object posture information, object line information, and depth map) is added to the denoising of the noisy latent variables, that is, the noisy latent variables are denoised in combination with the object structure features, thereby reducing the loss of the object structure features of the object image during the denoising process, so that the object enhanced image finally obtained can maintain more of the object structure features of the object image and improve the image enhancement effect of the object image.

[0075] In some embodiments, object structural features are obtained by fusing a depth map, object pose information, and object line drawing information. Referring to FIG4 , step 102 shown in FIG3 can be implemented through steps 1021 to 1024 shown in FIG4 : step 1021, generating a depth map of the object image; step 1022, performing object pose detection on the object image to obtain object pose information of the target object, and performing object line drawing extraction on the object image to obtain object line drawing information of the target object; step 1023, fusing the depth map, object pose information, and object line drawing information to obtain fused object structural information; and step 1024, performing feature extraction on the fused object structural information to obtain object structural features.

[0076] Here, the object structure feature is extracted from the fused object structure information, which is obtained by fusing the object pose information of the target object, the object line drawing information of the target object, and the depth map of the object image. In step 1023, a structure fusion model can be called for implementation. The structure fusion model can be a Unet network structure (composed of an encoder and a decoder, as shown in Figure 5), wherein the object pose information, the object line drawing information, and the depth map each have their own encoder. They can first extract feature maps through their respective encoders, and then fuse their respective feature maps through element-wise summation in a skip connection to obtain the fused object structure information. Specifically, the object posture information is encoded by the encoder of the object posture information to obtain the object posture coding features; the object line draft information is encoded by the encoder of the object line draft information to obtain the object line draft coding features; the depth map is encoded by the encoder of the depth map to obtain the depth map coding features; it should be noted that each encoder includes multiple layers of encoding processing, and each layer of encoding processing outputs corresponding features; accordingly, the decoder also includes multiple layers of decoding processing, and the levels of decoding processing and encoding processing are in a one-to-one correspondence. The input of each layer of decoding processing includes the output of the encoding processing of the corresponding layer (that is, the object posture coding features, object line draft coding features and depth map coding features output by the encoding processing of the corresponding layer. When performing decoding processing, the object posture coding features, object line draft coding features and depth map coding features can be spliced, and the spliced ​​features can be input as part of the decoding processing of this layer), and the output of the last layer of decoding processing is the fused object structure information.

[0077] In this way, by fusing the depth map, object posture information and object line drawing information to extract the object structure features, the feature expression ability of the object structure features for the target object can be improved, making the object structure features more accurate. Therefore, when the noisy latent variables are denoised in combination with the object structure features, the loss of the object structure features of the object image can be reduced during the denoising process, so that the final object enhanced image can retain the object structure features of the object image more and more accurately, thereby improving the image enhancement effect of the object image.

[0078] In some embodiments, the feature extraction process of step 1024 shown in FIG4 can be implemented by cascading M layers of first coding layers, where M is an integer greater than 0. Based on this, the implementation process of step 1024 shown in FIG4 includes: calling the first layer of the M layers of first coding layers to encode the fused object structure information to obtain the object structure features output by the first layer of the first coding layer; calling the i-th layer of the M layers of first coding layers to encode the object structure features output by the (i-1)-th layer of the first coding layer to obtain the object structure features output by the i-th layer of the first coding layer; traversing i to obtain the object structure features output by each layer of the first coding layer of the M layers of first coding layers, where i is an integer greater than 0 and not greater than M. It should be noted that i here can start from 1 and traverse to M, and i is incremented by 1 each time during the traversal process.

[0079] Here, the first coding layer performs a downsampling function, and the encoding process is also a downsampling process. The first coding layer can include a convolution layer and a self-attention layer. For example, when the first coding layer encodes the fused object structure information, it can first perform self-attention processing on the fused object structure information to obtain a self-attention result, and then perform convolution processing on the self-attention result to obtain the object structure features output by the first coding layer. For another example, when the i-th first coding layer encodes, it can first perform self-attention processing on the object structure features output by the (i-1)-th first coding layer to obtain a self-attention result, and then perform convolution processing on the self-attention result to obtain the object structure features output by the i-th first coding layer. In actual implementation, the cascaded M layers of first coding layers constitute a feature extraction model, for example, the feature extraction model can be a ControlNet model. In this way, the cascaded M layers of first coding layers perform feature extraction on the fused object structure information, extracting the object structure features output by each of the M layers of first coding layers, i.e., M object structure features.

[0080] In some embodiments, the feature size of the object structure features output by each first coding layer may be different. For example, the feature size of the object structure features outputted from the 1st first coding layer to the Mth first coding layer may become larger. Thus, different feature sizes indicate that different first coding layers focus on different feature extraction ranges (receptive fields) when extracting object structure features. Thus, the M object structure features that can be extracted can more accurately and comprehensively express the features of the target object. This allows the object structure features to be combined with the object structure features to perform denoising on the noisy latent variables, thereby reducing the loss of the object structure features of the object image during the denoising process. This allows the resulting object-enhanced image to more comprehensively and accurately retain the object structure features of the object image, thereby improving the image enhancement effect of the object image.

[0081] In some embodiments, the denoising process is implemented by an image denoising model, which includes a first encoder and a first decoder, wherein the first decoder includes a cascade of M first decoding layers, with the first decoding layer corresponding to the first encoding layer. Based on this, the denoising process includes: calling the first encoder to encode the noisy latent variable to obtain the encoded latent variable; calling the Mth first decoding layer of the M first decoding layers to decode the encoded latent variable and the object structure features output by the Mth first encoding layer to obtain the decoded latent variable output by the Mth first decoding layer; calling the i-th first decoding layer of the M first decoding layers to decode the decoded latent variable output by the (i+1)th first decoding layer and the object structure features output by the i-th first encoding layer to obtain the decoded latent variable output by the i-th first decoding layer; traversing i to obtain the decoded latent variable output by the first first decoding layer of the M first decoding layers, and using the decoded latent variable output by the first first decoding layer of the M first decoding layers as the denoising latent variable. It should be noted that i here can start from M and traverse to 1, and i is decremented by 1 each time during the traversal process.

[0082] Here, the image denoising model used for denoising comprises a first encoder and a first decoder consisting of M layers of first decoding layers. Each first decoding layer in the M layers of first decoding layers is used to denoise the object structural features output by each first decoding layer in the M layers of first encoding layers. Specifically, the first encoder is first called to encode the noisy latent variables to obtain encoded latent variables; the Mth first decoding layer of the M layers of first decoding layers is called to decode the encoded latent variables and the object structural features output by the Mth first encoding layer to obtain decoded latent variables output by the Mth first decoding layer; the i-th first decoding layer of the M layers of first decoding layers is called to decode the decoded latent variables output by the (i+1)th first decoding layer and the object structural features output by the i-th first encoding layer to obtain decoded latent variables output by the i-th first decoding layer; the decoded latent variables output by the first decoding layer of the M layers of first decoding layers are traversed through i to obtain the decoded latent variables output by the first first decoding layer of the M layers of first decoding layers, and the decoded latent variables output by the first first decoding layer of the M layers of first decoding layers are used as the denoising latent variables. In this way, the object structure features output by the first encoding layer of each layer are combined, and the noisy latent variables are denoised through the first decoding layers of M layers. This not only preserves the object structure features of the object image more comprehensively and accurately, but also improves the denoising effect of the noisy latent variables, thereby improving the image enhancement effect.

[0083] In some embodiments, the size of the input data for each of the M first decoding layers can be different. For example, the size of the input data for the i-th first decoding layer can be equal to the feature size of the object structure features output by the i-th first coding layer. This ensures that the first decoding layer can quickly and accurately process the object structure features output by the first coding layer, thereby improving the processing efficiency and accuracy of the first decoding layer.

[0084] In some embodiments, the first decoding layer includes a convolution layer and a self-attention layer. For example, the convolution layer of the first decoding layer of the Mth layer is called to perform convolution processing on the encoded latent variables and the object structure features output by the first encoding layer of the Mth layer to obtain the convolution features of the Mth layer, and then the self-attention layer of the first decoding layer of the Mth layer is called to perform self-attention processing on the convolution features of the Mth layer to obtain the decoding latent variables output by the first decoding layer of the Mth layer; the convolution layer of the first decoding layer of the i-th layer is called to perform convolution processing on the decoding latent variables output by the first decoding layer of the (i+1)th layer and the object structure features output by the first encoding layer of the i-th layer to obtain the convolution features of the i-th layer, and then the self-attention layer of the first decoding layer of the i-th layer is called to perform self-attention processing on the convolution features of the i-th layer to obtain the decoding latent variables output by the first decoding layer of the i-th layer.

[0085] In some embodiments, the encoding latent variable and the object structure feature output by the first encoding layer of the Mth layer can be decoded in the following manner to obtain the decoding latent variable output by the first decoding layer of the Mth layer: based on the first weight value of the encoding latent variable and the second weight value of the object structure feature output by the first encoding layer of the Mth layer, the encoding latent variable and the object structure feature output by the first encoding layer of the Mth layer are weightedly summed to obtain the first splicing feature; the first splicing feature is decoded to obtain the decoding latent variable output by the first decoding layer of the Mth layer; accordingly, the (i+1)th layer can be decoded in the following manner The decoding latent variable output by the first decoding layer and the object structure feature output by the first encoding layer of the i-th layer are decoded to obtain the decoding latent variable output by the first decoding layer of the i-th layer: based on the third weight value of the decoding latent variable output by the first decoding layer of the (i+1)-th layer and the fourth weight value of the object structure feature output by the first encoding layer of the i-th layer, the decoding latent variable output by the first decoding layer of the (i+1)-th layer and the object structure feature output by the first encoding layer of the i-th layer are weightedly summed to obtain the second splicing feature; the second splicing feature is decoded to obtain the decoding latent variable output by the first decoding layer of the i-th layer.

[0086] It should be noted that the first, second, third, and fourth weight values ​​can be pre-set and, in actual implementation, can be adjusted according to actual conditions. Thus, in the denoising process of the noisy latent variables, the control of the object structure features is added, and the influence of the object structure features on the denoising effect can be controlled by the weight values ​​set for the object structure features, thereby increasing the flexibility of the influence of the object structure features on the denoising process, and users can set them according to their needs.

[0087] As an example, referring to FIG6 , the image enhancement model includes a feature extraction model (such as a controlnet model), an image denoising model, a noise addition model, a structural fusion model, an image encoder, and an image decoder; wherein the image denoising model is actually a Unet network model, including a first encoder of Unet and a first decoder of Unet, the feature extraction model includes a cascaded M-layer first encoding layer, the first decoder includes a cascaded M-layer first decoding layer, and the first decoding layer and the first encoding layer correspond one to one.

[0088] Thus, based on the image enhancement model shown in FIG6 , 1) the image encoder is called to encode the object image to be enhanced to obtain a latent variable. 2) the noise addition model is called to add noise to the latent variable to obtain a noisy latent variable. 3) the structural fusion model is called to fuse the object posture information, the object line drawing information and the depth map to obtain the fused object structure information. 4) the feature extraction model (including the cascaded M-layer first coding layer) is called to extract the object structure features: the first-layer first coding layer of the M-layer first coding layer is called to encode the fused object structure information to obtain the object structure features output by the first-layer first coding layer; the i-th first coding layer of the M-layer first coding layer is called to encode the object structure features output by the (i-1)-th first coding layer to obtain the object structure features output by the i-th first coding layer; traversing i, the object structure features output by each first coding layer of the M-layer first coding layer are obtained. 5) the first encoder is called to encode the noisy latent variable to obtain the encoded latent variable. 6) Calling the Mth first decoding layer of the Mth first decoding layer to decode the encoded latent variables and the object structure features output by the Mth first decoding layer to obtain the decoded latent variables output by the Mth first decoding layer; calling the i-th first decoding layer of the Mth first decoding layer to decode the decoded latent variables output by the (i+1)th first decoding layer and the object structure features output by the i-th first encoding layer to obtain the decoded latent variables output by the i-th first decoding layer; traversing i to obtain the decoded latent variables output by the 1st first decoding layer of the Mth first decoding layer, and using the decoded latent variables output by the 1st first decoding layer as the denoised latent variables obtained by denoising the noisy latent variables. 7) Calling the image decoder to decode the denoised latent variables to obtain a first object enhanced image of the object image.

[0089] By applying the above embodiment, in the image enhancement process of the object image, the denoising process of the noisy latent variables is increased by controlling the object structure features of the object image (i.e., the object structure features extracted from the fused object structure information of the object posture information, the object line drawing information, and the depth map). That is, the noisy latent variables are denoised in combination with the object structure features, thereby reducing the loss of the object structure features of the object image, so that the final object enhanced image can maintain more object structure features of the object image, thereby improving the image enhancement effect of the object image.

[0090] In practical applications, the image enhancement model may be based on an image generation diffusion model, such as a stable diffusion model, a deepfloy_if model, etc.; the image denoising model may be a Denoising Unet model based on a Unet network.

[0091] Continuing to refer to Figure 6, the first encoder includes a cascade of P third encoding layers, where P is an integer greater than 0; each third encoding layer includes a convolution layer and a self-attention layer; based on this, the process of calling the first encoder to encode the noisy latent variable includes: calling the first third encoding layer of the P third encoding layers to encode the noisy latent variable, and obtaining the encoding result output by the first third encoding layer; calling the pth third encoding layer of the P third encoding layers to encode the encoding result output by the (p-1)th third encoding layer, and obtaining the encoding result output by the pth third encoding layer; traversing p to obtain the encoding result output by the Pth third encoding layer, where p is an integer greater than 0 and not greater than P; and using the encoding result output by the Pth third encoding layer as the encoding latent variable.

[0092] Here, the first encoder can also be composed of at least one third encoding layer, each third encoding layer includes a convolution layer and a self-attention layer. Specifically, the convolution layer of the first third encoding layer is called to perform convolution processing on the noisy latent variable to obtain convolution features, and then the self-attention layer of the first third encoding layer is called to perform self-attention processing on the convolution features to obtain the encoding result output by the first third encoding layer; the convolution layer of the pth third encoding layer is called to perform convolution processing on the encoding result output by the (p-1)th third encoding layer to obtain convolution features, and then the self-attention layer of the first third encoding layer is called to perform self-attention processing on the convolution features to obtain the encoding result output by the pth third encoding layer. In this way, by encoding the noisy latent variable at least once through at least one third encoding layer, more detailed features can be extracted, thereby improving the encoding effect of the first encoder and providing more feature details for subsequent processing.

[0093] In some embodiments, the denoising process includes T denoising processes, where T is an integer greater than 0; in combination with the object structure features, the denoising process can be performed on the noisy latent variable by performing the following steps to obtain the denoised latent variable of the object image: in combination with the object structure features, the noisy latent variable is subjected to a first denoising process to obtain an intermediate denoised latent variable output by the first denoising process; in combination with the object structure features, the intermediate denoised latent variable output by the (t-1)th denoising process is subjected to a tth denoising process to obtain an intermediate denoised latent variable output by the tth denoising process; traversing t to obtain the intermediate denoised latent variable output by the Tth denoising process, and using the intermediate denoised latent variable output by the Tth denoising process as the denoised latent variable of the object image. It should be noted that t here can start from 1 and traverse to T, and t increases by 1 each time during the traversal process.

[0094] It should be noted that the denoising process in the embodiment of the present application is a forward diffusion process, that is, noise is gradually added to the latent variable until the noisy latent variable is obtained; and the denoising process is a backward diffusion process, that is, the noisy latent variable is subjected to T denoising processes to gradually remove the noise and obtain the denoised latent variable. The T denoising processes can be understood as denoising processes for T time steps, with one denoising process completed for each time step. Each denoising process can be implemented using the denoising logic provided in the above embodiment. In this way, the denoising effect of the noisy latent variable can be improved, so that the obtained denoised latent variable retains the characteristics of the object image as much as possible, thereby improving the image enhancement effect of the object image.

[0095] Step 104: Perform image reconstruction on the denoised latent variables to obtain a first object enhanced image of the object image.

[0096] In step 104, the image reconstruction process is to decode the denoising latent variables to obtain a first object enhanced image of the object image. The decoding process can be implemented by a decoder, and the decoding process is an upsampling process.

[0097] In some embodiments, the image enhancement process for the object image can be performed multiple times, that is, after performing image reconstruction on the denoised latent variables to obtain a first object-enhanced image of the object image, the first object-enhanced image can further be enhanced. This can be achieved by performing the following steps: obtaining the target latent variables of the first object-enhanced image and adding noise to the target latent variables to obtain the target denoised latent variables of the first object-enhanced image; extracting the target object structural features of the target object in the first object-enhanced image; combining the target object structural features, denoising the target denoised latent variables to obtain the target denoised latent variables of the first object-enhanced image; and reconstructing the target denoised latent variables to obtain a second object-enhanced image of the object image. In the embodiment of the present application, multiple image enhancements of the object image are performed, and each image enhancement process can be implemented using the processing process of steps 101 to 104. Except for the first image enhancement (the input of the first image enhancement is the object image), the input of each image enhancement is the output of the previous image enhancement. In this way, through multiple cascaded image enhancements, the image enhancement effect of the object image can be further improved.

[0098] In some embodiments, after performing image reconstruction on the denoised latent variables to obtain a first object enhanced image of the object image, image super-resolution processing can also be performed on the first object enhanced image to obtain an object super-resolution image; the object super-resolution image is divided into multiple image blocks; image enhancement is performed on each image block to obtain an enhanced image block of each image block; and the enhanced image blocks are spliced ​​to obtain a third object enhanced image of the object image.

[0099] Image super-resolution (ISR) is a computer vision technique that aims to reconstruct a higher-resolution, high-definition image from a low-resolution image. This technique has wide applications in image restoration, video processing, medical imaging, satellite image analysis, and augmented reality. The basic idea behind image super-resolution is to estimate pixel values ​​at a higher resolution by analyzing features and patterns in a low-resolution image. For example, image super-resolution can be achieved by using the R-ESRGAN (Real Enhanced Super-Resolution Generative Adversarial Networks) model to achieve double super-resolution of an image. By performing image super-resolution on the first object-enhanced image, a super-resolved image of the object is obtained. However, since image super-resolution does not increase detail but simply enlarges the image size to match the original image, some originally blurry details may become even blurrier. Therefore, image enhancement can also be performed on local areas of the image to improve quality and increase detail. Specifically, the super-resolved object image is divided into multiple image blocks (each image block is a tile), and then image enhancement is performed on each image block to obtain an enhanced image block for each image block. These enhanced image blocks are then spliced ​​together to obtain a third object-enhanced image of the object image. This not only increases the image resolution of the first object-enhanced image through image super-resolution processing, but also performs local image enhancement on each image block while increasing the image resolution, further improving the image enhancement effect of the object image.

[0100] In some embodiments, image enhancement can be performed on each image block to obtain an enhanced image block for each image block: for each image block, the following processing is performed: noise is added to the block latent variables of the image block to obtain the noisy block latent variables of the image block; feature extraction is performed on the image block to obtain image block features of the image block; image block denoising is performed on the noisy block latent variables based on the image block features to obtain denoised block latent variables of the image block; and image reconstruction is performed on the denoised block latent variables to obtain an enhanced image block of the image block. It should be noted that each processing step for the image block can be the same as each processing step for the object image described above.

[0101] For example, the feature extraction process of an image block is implemented by cascading N layers of second coding layers, where N is an integer greater than 0. Based on this, the image block feature can be extracted by executing the following steps to obtain the image block feature of the image block: calling the first second coding layer of the N second coding layers to encode the image block to obtain the image block feature output by the first second coding layer; calling the jth second coding layer of the N second coding layers to encode the image block feature output by the (j-1)th second coding layer to obtain the image block feature output by the jth second coding layer; traversing j to obtain the object structure feature output by each first coding layer of the N first coding layers, where j is an integer greater than 0 and not greater than N. It should be noted that i here can start from 1 and traverse to M, and i increases by 1 each time during the traversal process.

[0102] For example, the image block denoising process is implemented by an image denoising model, which includes a second encoder and a second decoder. The second decoder includes a cascade of N layers of second decoding layers, and the second decoding layer and the second encoding layer have a one-to-one correspondence. Based on this, the image block denoising process includes: calling the second encoder to encode the noise block latent variable to obtain the coding block latent variable; performing weighted summation based on the weight value on the coding block latent variable and the image block feature output by the N-th layer second encoding layer to obtain a third splicing feature, and calling the N-th layer second decoding layer of the N-th layer second decoding layer to decode the third splicing feature to obtain the N-th layer second decoding layer. The decoding block latent variable output by the second decoding layer is obtained; the decoding block latent variable output by the (j+1)th layer second decoding layer and the image block feature output by the jth layer second encoding layer are weighted summed based on the weight value to obtain the fourth splicing feature, and the jth layer second decoding layer of the Nth layer second decoding layer is called to decode the fourth splicing feature to obtain the decoding block latent variable output by the jth layer second decoding layer; traverse j to obtain the decoding block latent variable output by the first layer second decoding layer of the Nth layer second decoding layer, and use the decoding block latent variable output by the first layer second decoding layer as the denoised block latent variable obtained by performing image block denoising on the noisy block latent variable. It should be noted that i here can start from M and traverse to 1. During the traversal process, i is decremented by 1 each time.

[0103] In some embodiments, before obtaining the latent variables of the object image to be enhanced, a target image of the target object can be obtained; the target image is resized to multiple different sizes to obtain resized images at each size; and the resized images at each size are used as the object image. Here, the object image to be enhanced is obtained by resizing the target image to multiple different sizes. The resizing processes may include both size reduction and size enlargement. The multiple resized images may include the target image at its original size. This allows for achieving corresponding image enhancement effects for target images of different sizes. Furthermore, different image enhancement effects are achieved for images of different sizes within the same image enhancement process, thereby increasing the selectivity of the resulting enhanced object image.

[0104] In an exemplary scenario, the embodiment of the present application can be applied to a game design scenario, specifically a design scenario for a virtual object in a game scenario. For example, an object image of a virtual object designed by a game designer (e.g., an object sketch of a virtual object) is obtained, and then the latent variables of the object image are extracted, and noise is added to the latent variables to obtain the noisy latent variables of the object image, where the object image is an image of the target object; the object structure features of the target object in the object image are extracted; the noisy latent variables are denoised in combination with the object structure features to obtain the denoised latent variables of the object image; the denoised latent variables are image reconstructed to obtain a first object enhanced image of the object image. In this way, in the image enhancement process of the object image, the denoising process of the noisy latent variables increases the control of the object structure features of the object image, that is, the noisy latent variables are denoised in combination with the object structure features, thereby reducing the loss of the object structure features of the object image during the denoising process, so that the object enhanced image finally obtained can maintain more of the object structure features of the object image, improve the image enhancement effect of the object image and the game design effect; and can achieve automatic image enhancement effect, thereby improving the efficiency of game design.

[0105] In an exemplary scenario, the embodiment of the present application can be applied to an animation design scenario, specifically a design scenario for an animated object in an animation. For example, an object image of an animated object designed by an animation designer (e.g., an object sketch of an animated object) is obtained, and then the latent variables of the object image are extracted, and noise is added to the latent variables to obtain the noisy latent variables of the object image, where the object image is an image of the target object; the object structure features of the target object in the object image are extracted; the noisy latent variables are denoised in combination with the object structure features to obtain the denoised latent variables of the object image; the denoised latent variables are image reconstructed to obtain a first object enhanced image of the object image. In this way, in the image enhancement process of the object image, the denoising process of the noisy latent variables is increased to control the object structure features of the object image, that is, the noisy latent variables are denoised in combination with the object structure features, thereby reducing the loss of the object structure features of the object image during the denoising process, so that the object enhanced image finally obtained can maintain more of the object structure features of the object image, improve the image enhancement effect of the object image and the animation design effect; and can achieve automatic image enhancement effect, thereby improving the efficiency of animation design.

[0106] In an exemplary scenario, the embodiment of the present application can be applied to a UCG scenario. For example, an object image of a target object to be produced by a user (such as a personal image of the user, an object sketch of an object designed by the user) is obtained, and then the latent variables of the object image are extracted, and noise is added to the latent variables to obtain the noisy latent variables of the object image, where the object image is an image of the target object; the object structure features of the target object in the object image are extracted; the noisy latent variables are denoised in combination with the object structure features to obtain the denoised latent variables of the object image; the denoised latent variables are image reconstructed to obtain a first object enhanced image of the object image. In this way, in the image enhancement process of the object image, the denoising process of the noisy latent variables increases the control of the object structure features of the object image, that is, the noisy latent variables are denoised in combination with the object structure features, thereby reducing the loss of the object structure features of the object image during the denoising process, so that the object enhanced image finally obtained can maintain more of the object structure features of the object image, improve the image enhancement effect and content generation effect of the object image; and can achieve automatic image enhancement effect, improve content generation efficiency, and increase user stickiness of the UCG platform.

[0107] Applying the above-mentioned embodiment of the present application, the latent variables of the object image to be enhanced are first obtained, and noise is added to the latent variables to obtain the noisy latent variables of the object image; then, the object structural features of the target object in the object image are extracted; thereby, the noisy latent variables are denoised in combination with the object structural features to obtain the denoised latent variables of the object image; finally, the denoised latent variables are image reconstructed to obtain a first object-enhanced image of the object image. Here, during the image enhancement process of the object image, the denoising process of the noisy latent variables is supplemented with the control of the object structural features of the object image, that is, the denoising process of the noisy latent variables is combined with the object structural features, thereby reducing the loss of the object structural features of the object image during the denoising process, so that the object-enhanced image obtained in the end can retain more of the object structural features of the object image, thereby improving the image enhancement effect of the object image.

[0108] The following takes image enhancement (i.e., image refinement) of an object sketch (i.e., an object image) as an example to illustrate an exemplary application of an embodiment of the present application in an actual application scenario.

[0109] In art design, the design of an object (such as a person or an animal) is usually divided into three stages, including concept setting, sketch design, and sketch refinement. Among them, (1) Concept setting stage: based on the basic requirements and characteristics of the object image discussed with the client or team members (such as the gender, age, personality, etc. of the object), the designer conducts preliminary design and conception to obtain a preliminary concept setting. (2) Sketch design stage: Designers usually use tools such as sketches or hand-drawing to further design and conceive the concept setting to obtain the corresponding object sketch. (3) Sketch refinement stage: Designers usually further polish and refine the object sketch to obtain a more detailed and realistic object image. Designers will make more detailed designs and concepts for each part of the object, including facial expressions, clothing details, muscle lines, etc. Since each step in the above object image design process requires manual participation of the designer, the production efficiency of the original painting design is very low. Therefore, the production efficiency of the original painting design can be improved by automating the process, such as automating the sketch refinement stage.

[0110] In related art, object sketch refinement often involves gradually adding noise to the object image until it becomes a completely random noise image. Starting from this random noise image, the noise is then gradually removed to produce the final object-enhanced image. However, due to the difficulty in controlling the denoising intensity during the denoising process, the final object-enhanced image tends to lose a large number of original features, resulting in poor enhancement effects.

[0111] Based on this, an embodiment of the present application provides an image enhancement method to at least solve the above-mentioned problems. An embodiment of the present application provides an image enhancement model, which is based on the Image Generative Diffusion Model. In order to enhance the feature retention of the object sketch, the embodiment of the present application 1) adds object structure features including object posture information, object line draft information and depth map information on the basis of the image generative diffusion model. The three types of information have their own focus, and the fusion of each other is conducive to enhancing structural control and improving structural consistency in refinement. 2) Use super-resolution technology to improve image resolution, and then use tile control to increase image details in each image area in units of tiles, and splice the tiles of the entire image to obtain a refined image. 3) Multi-scale refinement strategy. It is proposed to perform multi-scale refinement of object sketches to cover use cases with different degrees of refinement.

[0112] Next, we will explain the image generation diffusion model. The image generation diffusion model is a generative model based on a diffusion process, used to generate high-quality images with rich texture and detail. The image generation process in the image generation diffusion model is a diffusion process in which noise gradually fades away. In this diffusion process, the model starts with an original image containing random noise and gradually removes the noise from the initial image to obtain the generated image. The generation process of the image generation diffusion model is divided into a forward diffusion process and a reverse diffusion process. In the forward diffusion process, the model gradually adds noise to the original image until it becomes a completely random noise image. In the reverse diffusion process, the model starts with a completely random noise image and gradually removes the noise to obtain the generated image. During the reverse diffusion process, the model needs to learn how to remove noise at each time step. Therefore, the model typically uses a neural network structure, such as a convolutional neural network (CNN) or a variational autoencoder (VAE).

[0113] As an example, refer to Figure 7, which is a structural diagram of the image generation diffusion model provided in an embodiment of the present application. Here, the image generation diffusion model is implemented based on a stable diffusion network, including an encoder (Encoder), a decoder (Decoder), a denoising module (Diffusion Process), and an image denoising model (built based on a Unet network, such as Denoising Unet). Among them, the encoder is used to convert the input image x into a latent variable (latent representation) z. The latent variable can capture the important features of the input image and help the model remove noise during the reverse diffusion process. The denoising module is used to perform noise processing on the latent variable (such as adding Gaussian white noise) to obtain the noisy latent variable z T The image denoising model is used to add noise to the latent variable z T Perform denoising to obtain the denoising latent variable z0. The denoising process is divided into multiple time steps. The image denoising model can predict the noise item that needs to be removed at each time step and remove the corresponding noise at each time step to obtain z in turn. T-1 、z T-2 , ... z1, z0. The decoder is used to reconstruct the image based on the denoising latent variable z0 to obtain the generated image During model training, it learns to generate images by minimizing the reconstruction error, which is calculated as the difference between the generated image and the original image. By optimizing the reconstruction error, the model can learn to gradually remove noise during the back-diffusion process, ultimately generating high-quality images.

[0114] In practical applications, the image refinement of the object sketch needs to balance the degree of refinement completion, the degree of structural restoration of the refined image, and the image quality of the refined image. See Figure 8, which is a flow chart of the image enhancement method provided in an embodiment of the present application. Here, the object sketch is input, and after multiple rounds of processing by the image enhancement model based on the control of the object structure features, the object enhanced image 1 is obtained; then the object enhanced image 1 is super-resolution processed, and the object enhanced image 1 is enlarged to twice the original size to obtain an object super-resolution image; the object super-resolution image is divided into multiple tiles (i.e., image blocks) row by row and column by column, and tile control is used to perform image restoration (i.e., image enhancement) on each tile, to improve the local details of each tile, and finally all tiles are stitched together to obtain the object enhanced image 2. The following are explained separately.

[0115] Phase (1) is based on the processing of the image enhancement model controlled by the object structure feature. In the processing process of phase (1), as shown in Figure 8, multiple image enhancement models are cascaded to achieve this. The cascade of multiple image enhancement models can alleviate the problem of the image enhancement model being highly dependent on the denoising strength.

[0116] Here, the structure of the image enhancement model is shown in Figure 6. The image enhancement model is based on the image generation diffusion model shown in Figure 7, and a structural control branch is added. That is, the image enhancement model provided in the embodiment of the present application includes: a feature extraction model (such as a controlnet model), an image denoising model (such as the Denoising Unet built based on the Unet network in Figure 7), a noise addition model, a structural fusion model, an image encoder and an image decoder; wherein, the image denoising model is actually a Unet network model, including the first encoder of Unet and the first decoder of Unet, the feature extraction model includes a cascade of M layers of first encoding layers, and the first decoder includes a cascade of M layers of first decoding layers, and the first decoding layer corresponds to the first encoding layer one by one. wherein, the first encoder includes multiple encoding layers, each encoding layer is composed of a convolutional network and a self-attention network; the first decoder includes multiple decoding layers, each decoding layer is also composed of a convolutional network and a self-attention network; the feature extraction model includes multiple feature extraction layers, each feature extraction layer is also composed of a convolutional network and a self-attention network.

[0117] For an object sketch, the pose detection model extracts the object pose information, the line drawing extraction model extracts the object line drawing information, and the depth estimation model extracts the depth map of the object sketch. The object pose information represents the pose and motion of the target object, the line drawing information represents the structural details of the target object in the image, and is two-dimensional structural information. The depth map records the distance between pixels in the image and the camera, representing the surface structural information of the target object. The fusion of these three types of information enhances structural control and improves structural consistency during refinement. The structural fusion model fuses the object pose information, line drawing information, and depth map to generate fused object structural information. During the fusion process, the object pose information, line drawing information, and depth map each have their own encoders. Since the structural fusion model is a Unet network model, each encoder extracts features. Then, the feature maps of the three structural information are fused through element-wise summation in a skip connection to generate fused object structural information. The feature extraction model extracts features from the fused object structural information to generate object structural features.

[0118] As shown in Figure 9, the structure on the left is the structure of the image denoising model Denoising Unet, including the first encoder (including multiple encoding layers Encoder Block) and the first decoder (including multiple decoding layers Decoder Block and middle layer Middle Block); the structure on the right is the structure of the feature extraction model (i.e., the controlnet model), including multiple encoding layers Encoder Block, middle layer Middle Block, and multiple zero convolution layers. Each encoding layer of the feature extraction model outputs object structure features, and the respective object structure features are input into the corresponding layer of the first decoder of Unet (including the decoding layer and the middle layer) through the zero convolution layer, so that the first decoder of Unet can combine the object structure features and the first encoder of Unet from the noise latent variable x. t The multi-level image features extracted (obtained by denoising the latent variables of the object sketch x) generate the denoised latent variables x0, and then generate the object enhanced image based on the denoised latent variables x0 In this way, the generated object enhanced image can be influenced by the object structure features of the target object.

[0119] The Encoder Block, Decoder Block, and Middle Block layers can be based on the Squeeze-and-Excitation (SD) module. The Squeeze-and-Excitation module is used to enhance the network's receptive field and expressive power for features. The SD module improves the quality of feature representation through two steps: squeezing and excitation. Squeeze compresses the feature map through global average pooling to produce a single feature vector that incorporates the global context of the input feature map. Excitation learns the importance of features through two fully connected layers, then applies the learned weights to the original feature map to enhance important features and suppress unimportant ones. The SD module can be inserted anywhere in a convolutional neural network to increase feature interactivity and contextual information at different levels. This approach can improve model performance, particularly in visual tasks such as image recognition and object detection.

[0120] In addition, in stage (1), there are several important hyperparameters: the number of models in the cascaded image enhancement model, the denoising strength of the single image enhancement model, and the structural control strength of the object structure feature. Among them, the structural control strength refers to the weight ratio of the object structure feature fused to the first decoder. The greater the structural control strength, the higher the weight of the object structure feature fusion. For example, the values ​​of the above hyperparameters can be shown in Table 1 below, and can be adjusted as needed in actual applications.

[0121] Table 1

[0122] In practical applications, during the training of the above-mentioned cascaded image enhancement model, the structure fusion model and feature extraction model (such as the controlnet model) involved need to be trained, and the other parameters of the model can use pre-trained weights and do not need to participate in training. Specifically, a training data set can be collected, and the posture map, line drawing map and depth map of each image sample can be calculated during the preprocessing process. In addition, image degradation processing (such as Gaussian blur) is used to generate the corresponding "original image" data. For example, the training parameters can be as follows: batch size equals 64, learning rate equals 1e-4, number of training steps 200000, and optimizer Adam. The loss function can be the image reconstruction loss function L LDM :

[0123] Where, ∈ represents Gaussian noise, θ represents the model parameters of the Unet-based image denoising model; z t represents the noise added latent variable; t represents the sampling time point.

[0124] Stage (2) Image super-resolution + image restoration based on tile control.

[0125] a) Image super-resolution. In stage (2), the object enhanced image 1 obtained in stage (1) is first subjected to image super-resolution processing to improve the image resolution and obtain an object super-resolution image. In practical applications, this can be achieved through an image super-resolution processing model, such as the R-ESRGAN (Real Enhanced Super-Resolution Generative Adversarial Networks) model. The R-ESRGAN model is an image super-resolution method based on a generative adversarial network. The R-ESRGAN model consists of two parts: a generator and a discriminator. Among them, 1) the generator is responsible for upsampling the low-resolution image to a high-resolution image. As shown in (1) in Figure 10, it is a structural diagram of the generator. The generator includes multiple convolutional layers (conv), upsampling layers (Upsampling), and multiple residual scaling layers (Residual-in-Residual Dense Block, RRDB). 2) The discriminator is responsible for distinguishing between the generated high-resolution image and the real high-resolution image. As shown in (2) in Figure 10, it is a schematic diagram of the structure of the discriminator, which includes multiple convolutional layers (conv) and multiple spectral normalization layers (spectral norm).

[0126] Since image super-resolution processing does not increase details but only enlarges the image size according to the original image, some originally blurred details may become even more blurred. Therefore, image enhancement can also be performed on local areas of the image to improve quality and increase details.

[0127] b) Tile-controlled image restoration. After image super-resolution processing, the original (H, W) image size increases to (2H, 2W). Since images used in general design scenarios are often larger than (1024, 1024), the video memory required for image enhancement processing of (2H, 2W) can easily exceed the 32GB of video memory on a single V100 graphics card. Therefore, the super-resolution image of the object can be divided into multiple tiles for refinement to save video memory.

[0128] In practical applications, multi-tile serial image generation can be achieved through image restoration. Specifically, each network input is the current refined image (as shown in (1) in Figure 11) and the mask image of the tile, as shown in (2) in Figure 11. The white area in the mask represents the corresponding area of ​​the input image, which is the tile to be refined. Image restoration is the same as the image enhancement process in stage 1. After the tile is encoded into latent by the autoencoder, a certain number of steps of Gaussian noise are added to generate a noisy latent map, and then denoised by the back diffusion process to generate an enhanced tile. In image restoration, in order to ensure that the black area in the mask does not change, the noisy latent predicted by the model in each step of the denoising process is replaced with the noisy latent of the same level in the denoising process. After the current tile is restored, the position in the entire image is automatically updated and participates in the restoration of the next tile. In the image restoration process of each tile, tile control is used to generate new details for the local image and ensure the consistency of the overall image structure. The network structure of the tile control branch is consistent with the control branch of the object structure feature in stage (1), but its input image is replaced by the object super-resolution image.

[0129] In stage (2), there are several important hyperparameters, including: the denoising strength of the model, the tile control strength (weight value), and the length and width of a single tile. Among them, the tile control strength refers to the weight ratio of the tile control feature fusion to the first decoder. The greater the tile control strength, the higher the weight when the tile control feature is fused, and the more details will be added to the image. The length and width of a single tile controls the size of the image repair area each time. The smaller the area, the more details will be added after the same strength of tile control. For example, the values ​​of the above hyperparameters can be shown in Table 2 below, and can be adjusted as needed in actual applications.

[0130] Table 2

[0131] Phase (3) Multi-size refinement strategy. As the image size increases, the redundancy of information in the data increases, and the denoising task of large images at the same intensity becomes simpler, that is, the degree of change of the output image compared with the input image at the same denoising intensity is different for images of different sizes. The larger the size, the smaller the image change at the same denoising intensity. Since the degree of image refinement expected by users in different images is different, a multi-size refinement strategy can be adopted. In specific implementation, the object sketch can be scaled to different sizes (such as 0.5 times, 1 times, 2 times), and then the object sketches of different sizes are input into the above image refinement process to obtain object enhanced images of different sizes, so that object enhanced images of multiple sizes can be returned to the user at the same time for selection. In actual implementation, it is found that the small images of the embodiment of the present application can often generate more associations and details, which is especially suitable for the case where the object sketch is very low in completion; and the large image after the refinement of the embodiment of the present application will be more faithful to the input image.

[0132] It should be noted that the embodiments of this application can be applied to various scenarios (such as game concept art design, animation or comic design, advertising design, etc.), including the refinement of object sketches, and different model parameters can be designed according to the scenario requirements. It should be noted that the faces appearing in the figures provided in the embodiments of this application are synthetic and not real faces.

[0133] By applying the above-mentioned embodiment of the present application, it is possible to automatically polish and refine the object sketch, and achieve good image enhancement effects on large areas in the object sketch (such as hair, clothing, skin, etc.). In actual implementation, it can help art designers quickly improve an object sketch with only 30%-40% completion to 80% or higher completion, greatly improving the designer's production efficiency. Referring to FIG12, which is a schematic diagram of a comparison of a set of object refinement results, it can be seen that: (1) Comparing the results of stage (1) of the embodiment of the present application with the results of single image enhancement, the results of stage (1) of the embodiment of the present application are more consistent with the original image (i.e., the object sketch), including hair, clothing, skin, etc. (2) Comparing the results of only performing tile control with the results of stage (2) of the embodiment of the present application, the refinement completion of the results of only performing tile control is relatively low. (3) The result of stage (1) can obtain an object image with realistic texture, but the image quality is not high; the image resolution of the result of image super-resolution processing in stage (2) is improved, but in fact, when zoomed in, the degree of refinement of local areas such as the face, forehead, and eyes is not enough, and local details such as the ends of the hair and the right wrist are blurred; the result of tile control in stage (2) directly improves the completion and quality of local areas.

[0134] The following continues to describe an exemplary structure of the image enhancement device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 2, the software modules stored in the image enhancement device 555 in the memory 550 may include: an acquisition module 5551, configured to acquire latent variables of the object image to be enhanced, and add noise to the latent variables to obtain noisy latent variables of the object image, where the object image is an image of the target object; an extraction module 5552, configured to extract object structural features of the target object in the object image; a denoising module 5553, configured to denoise the noisy latent variables in combination with the object structural features to obtain denoised latent variables of the object image; and a reconstruction module 5554, configured to perform image reconstruction on the denoised latent variables to obtain a first object enhanced image of the object image.

[0135] In some embodiments, the extraction module 5552 is further configured to generate a depth map of the object image; perform object posture detection on the object image to obtain object posture information of the target object, and perform object line draft extraction on the object image to obtain object line draft information of the target object; fuse the depth map, the object posture information and the object line draft information to obtain fused object structure information; perform feature extraction on the fused object structure information to obtain the object structure feature.

[0136] In some embodiments, the feature extraction process is implemented by cascading M layers of first coding layers, where M is an integer greater than 0; the extraction module 5552 is further configured to call the 1st layer of the M layers of first coding layers to encode the fused object structure information to obtain the object structure features output by the 1st layer of the first coding layer; call the i-th layer of the M layers of first coding layers to encode the object structure features output by the (i-1)th layer of the first coding layer to obtain the object structure features output by the i-th layer of the first coding layer; traverse the i to obtain the object structure features output by each layer of the first coding layer of the M layers of first coding layers, where i is an integer greater than 0 and not greater than the M.

[0137] In some embodiments, the denoising process is implemented by an image denoising model, which includes a first encoder and a first decoder, wherein the first decoder includes a cascade of M layers of first decoding layers, and the first decoding layer corresponds to the first encoding layer one-to-one; the denoising module 5553 is further configured to call the first encoder to encode the noisy latent variable to obtain an encoded latent variable; call the Mth first decoding layer of the M-layer first decoding layer to decode the encoded latent variable and the object structure features output by the M-layer first encoding layer to obtain a decoded latent variable output by the M-layer first decoding layer; call the i-th first decoding layer of the M-layer first decoding layer to decode the decoded latent variable output by the (i+1)-th first decoding layer and the object structure features output by the i-th first encoding layer to obtain a decoded latent variable output by the i-th first decoding layer; traverse the i to obtain the decoded latent variable output by the 1st first decoding layer of the M-layer first decoding layer, and use the decoded latent variable output by the 1st first decoding layer as the denoising latent variable.

[0138] In some embodiments, the denoising module 5553 is further configured to perform weighted summation of the encoded latent variable and the object structure feature output by the M-th first encoding layer based on the first weight value of the encoded latent variable and the second weight value of the object structure feature output by the M-th first encoding layer to obtain a first splicing feature; decode the first splicing feature to obtain a decoded latent variable output by the M-th first decoding layer; the denoising module 5553 is further configured to perform weighted summation of the decoded latent variable output by the (i+1)-th first decoding layer and the object structure feature output by the i-th first encoding layer based on the third weight value of the decoded latent variable output by the (i+1)-th first decoding layer and the fourth weight value of the object structure feature output by the i-th first encoding layer to obtain a second splicing feature; decode the second splicing feature to obtain a decoded latent variable output by the i-th first decoding layer.

[0139] In some embodiments, the first encoder includes a cascade of P layers of third coding layers, where P is an integer greater than 0; the denoising module 5553 is further configured to call the first layer of the third coding layer of the P layers to encode the noisy latent variable to obtain the encoding result output by the first layer of the third coding layer; call the pth layer of the third coding layer of the P layers to encode the encoding result output by the (p-1)th layer of the third coding layer to obtain the encoding result output by the pth layer of the third coding layer; traverse p to obtain the encoding result output by the Pth layer of the third coding layer, where p is an integer greater than 0 and not greater than P; and use the encoding result output by the Pth layer of the third coding layer as the encoding latent variable.

[0140] In some embodiments, the reconstruction module 5554 is further configured to, after performing image reconstruction on the denoised latent variable to obtain a first object enhanced image of the object image, obtain a target latent variable of the first object enhanced image, and add noise to the target latent variable to obtain a target denoised latent variable of the first object enhanced image; extract the target object structural features of the target object in the first object enhanced image; perform denoising on the target denoised latent variable in combination with the target object structural features to obtain a target denoised latent variable of the first object enhanced image; perform image reconstruction on the target denoised latent variable to obtain a second object enhanced image of the object image.

[0141] In some embodiments, the reconstruction module 5554 is further configured to, after performing image reconstruction on the denoised latent variables to obtain a first object enhanced image of the object image, perform image super-resolution processing on the first object enhanced image to obtain an object super-resolution image; divide the object super-resolution image into multiple image blocks; perform image enhancement on each of the image blocks respectively to obtain an enhanced image block of each image block; and splice the enhanced image blocks to obtain a third object enhanced image of the object image.

[0142] In some embodiments, the reconstruction module 5554 is further configured to perform the following processing on each of the image blocks: adding noise to the block latent variables of the image block to obtain the noisy block latent variables of the image block; performing feature extraction on the image block to obtain the image block features of the image block; performing image block denoising on the noisy block latent variables in combination with the image block features to obtain the denoised block latent variables of the image block; performing image reconstruction on the denoised block latent variables to obtain the enhanced image block of the image block.

[0143] In some embodiments, the feature extraction process is implemented by cascading N layers of second coding layers, where N is an integer greater than 0; the reconstruction module 5554 is further configured to call the first second coding layer of the N layers of second coding layers to encode the image block to obtain the image block features output by the first second coding layer; call the jth second coding layer of the N layers of second coding layers to encode the image block features output by the (j-1)th second coding layer to obtain the image block features output by the jth second coding layer; traverse the j to obtain the object structure features output by each first coding layer of the N layers of first coding layers, where j is an integer greater than 0 and not greater than the N.

[0144] In some embodiments, the image block denoising process is implemented by an image denoising model, the image denoising model includes a second encoder and a second decoder, the second decoder includes a cascade of N layers of second decoding layers, and the second decoding layer and the second encoding layer have a one-to-one correspondence; the denoising module 5553 is further configured to call the second encoder to encode the noise block latent variable to obtain the coding block latent variable; the coding block latent variable and the image block feature output by the N-th layer of the second encoding layer are weighted summed based on the weight value to obtain a third splicing feature, and the N-th layer of the second decoding layer is called to encode the third The splicing features are decoded to obtain the decoding block latent variables output by the N-th second decoding layer; the decoding block latent variables output by the (j+1)-th second decoding layer and the image block features output by the j-th second encoding layer are weightedly summed based on the weight value to obtain a fourth splicing feature, and the j-th second decoding layer of the N-th second decoding layers is called to decode the fourth splicing feature to obtain the decoding block latent variables output by the j-th second decoding layer; traverse the j to obtain the decoding block latent variables output by the 1st second decoding layer of the N-th second decoding layers, and use the decoding block latent variables output by the 1st second decoding layer as the denoising block latent variables.

[0145] In some embodiments, the acquisition module 5551 is further configured to acquire a target image of the target object before acquiring the latent variables of the object image to be enhanced; perform resizing processing on the target image at multiple different sizes to obtain adjusted images at each size; and use the adjusted images at each size as the object image.

[0146] In some embodiments, the acquisition module 5551 is further configured to encode the object image to be enhanced to obtain the latent variable; the reconstruction module 5554 is further configured to decode the denoised latent variable to obtain the first object enhanced image of the object image.

[0147] In some embodiments, the denoising process includes T denoising processes, where T is an integer greater than 0; the denoising module 5553 is further configured to perform a first denoising process on the noisy latent variable in combination with the object structure features to obtain an intermediate denoised latent variable output by the first denoising process; perform a t-th denoising process on the intermediate denoised latent variable output by the (t-1)th denoising process in combination with the object structure features to obtain an intermediate denoised latent variable output by the t-th denoising process; traverse t to obtain the intermediate denoised latent variable output by the T-th denoising process, and use the intermediate denoised latent variable output by the T-th denoising process as the denoising latent variable of the object image.

[0148] In some embodiments, the extraction module 5552 is further configured to obtain at least one of the following object information of the target object in the object image: a depth map of the object image, object posture information of the target object, and object line drawing information of the target object; and perform feature extraction on the object information to obtain the object structural features.

[0149] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments, and is not further elaborated here. Any unfinished technical details of the image enhancement device provided in the embodiments of this application can be understood based on the description of the technical details in the method embodiments described above.

[0150] The present application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or the computer program from the computer-readable storage medium and executes the computer-executable instructions or the computer program, causing the electronic device to perform the image enhancement method provided in the present application.

[0151] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image enhancement method provided in the embodiment of the present application.

[0152] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0153] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0154] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0155] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0156] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image enhancement method, applied to an electronic device, the method comprising: Acquire a hidden variable of an object image to be enhanced, and add noise to the hidden variable to obtain a noisy hidden variable of the object image, where the object image is an image of a target object; extracting object structural features of the target object in the object image; In combination with the object structure features, the noisy latent variable is denoised to obtain a denoised latent variable of the object image; Performing image reconstruction on the denoising latent variables to obtain a first object enhanced image of the object image.

2. The method of claim 1, wherein: The extracting the object structure feature of the target object in the object image includes: generating a depth map of the object image; Performing object posture detection on the object image to obtain object posture information of the target object, and performing object line draft extraction on the object image to obtain object line draft information of the target object; Fusing the depth map, the object posture information, and the object line draft information to obtain fused object structure information; Feature extraction is performed on the fused object structure information to obtain the object structure feature.

3. The method of claim 2, wherein: The feature extraction process is implemented by cascading M layers of first coding layers, where M is an integer greater than 0; extracting features from the fused object structure information to obtain the object structure features includes: Calling the first coding layer of the M layers of first coding layers to encode the fused object structure information to obtain the object structure features output by the first coding layer of the first layer; Calling the i-th first coding layer of the M-th first coding layer to encode the object structure feature output by the (i-1)-th first coding layer to obtain the object structure feature output by the i-th first coding layer; Traverse the i to obtain the object structure features output by each first coding layer of the M first coding layers, where i is an integer greater than 0 and not greater than M.

4. The method of claim 3, wherein: The denoising process is implemented by an image denoising model, the image denoising model includes a first encoder and a first decoder, the first decoder includes a cascade of M first decoding layers, and the first decoding layer corresponds to the first encoding layer in one-to-one correspondence; The step of performing denoising on the noisy latent variable in combination with the object structure feature to obtain the denoised latent variable comprises: Calling the first encoder to encode the noisy latent variable to obtain an encoded latent variable; Calling the Mth first decoding layer of the Mth first decoding layer to decode the encoded latent variable and the object structure feature output by the Mth first encoding layer to obtain the decoded latent variable output by the Mth first decoding layer; Calling the i-th first decoding layer of the M-th first decoding layer to decode the decoding latent variable output by the (i+1)-th first decoding layer and the object structure feature output by the i-th first encoding layer to obtain the decoding latent variable output by the i-th first decoding layer; Traverse the i to obtain the decoding latent variable output by the first decoding layer of the M first decoding layers, and use the decoding latent variable output by the first decoding layer of the first layer as the denoising latent variable.

5. The method of claim 4, wherein: The decoding of the encoded latent variables and the object structure features output by the first encoding layer of the Mth layer to obtain the decoded latent variables output by the first decoding layer of the Mth layer includes: Based on the first weight value of the encoding latent variable and the second weight value of the object structure feature output by the first encoding layer of the Mth layer, weighted summation is performed on the encoding latent variable and the object structure feature output by the first encoding layer of the Mth layer to obtain a first splicing feature; Decoding the first concatenated features to obtain a decoding latent variable output by the first decoding layer of the Mth layer; The decoding latent variable output by the (i+1)th first decoding layer and the object structure feature output by the i-th first encoding layer are decoded to obtain the decoding latent variable output by the i-th first decoding layer, including: Based on the third weight value of the decoded latent variable output by the (i+1)th first decoding layer and the fourth weight value of the object structure feature output by the i-th first encoding layer, weighted sum is performed on the decoded latent variable output by the (i+1)th first decoding layer and the object structure feature output by the i-th first encoding layer to obtain a second splicing feature; The second concatenated feature is decoded to obtain a decoding latent variable output by the first decoding layer of the i-th layer.

6. The method of claim 5, wherein: The first encoder includes a cascaded P-layer third encoding layer, where P is an integer greater than 0; calling the first encoder to encode the noisy latent variable to obtain the encoded latent variable includes: Calling the first third coding layer of the P third coding layer to encode the noisy latent variable to obtain the encoding result output by the first third coding layer; Calling the p-th third coding layer of the P-th third coding layer to encode the coding result output by the (p-1)-th third coding layer to obtain the coding result output by the p-th third coding layer; Traverse p to obtain the encoding result output by the third encoding layer of the Pth layer, where p is an integer greater than 0 and not greater than P; The encoding result output by the third encoding layer of the Pth layer is used as the encoding latent variable.

7. The method according to any one of claims 1 to 6, wherein: After performing image reconstruction on the denoising latent variables to obtain a first object enhanced image of the object image, the method further includes: Acquire a target latent variable of the first object enhanced image, and add noise to the target latent variable to obtain a target noisy latent variable of the first object enhanced image; extracting a target object structural feature of the target object in the first object enhanced image; In combination with the structural features of the target object, denoising the target noisy latent variable to obtain a target denoised latent variable of the first object enhanced image; Performing image reconstruction on the target denoising latent variable to obtain a second object enhanced image of the object image.

8. The method according to any one of claims 1 to 7, wherein: After performing image reconstruction on the denoising latent variables to obtain a first object enhanced image of the object image, the method further includes: Performing image super-resolution processing on the first object enhanced image to obtain an object super-resolution image; Dividing the object super-resolution image into a plurality of image blocks; Performing image enhancement on each of the image blocks respectively to obtain an enhanced image block of each of the image blocks; The enhanced image blocks are spliced ​​together to obtain a third object enhanced image of the object image.

9. The method of claim 8, wherein: The performing image enhancement on each of the image blocks to obtain an enhanced image block of each of the image blocks comprises: For each of the image blocks, the following processing is performed respectively: Adding noise to the block latent variable of the image block to obtain the noisy block latent variable of the image block; Performing feature extraction on the image block to obtain image block features of the image block; In combination with the image block features, performing image block denoising processing on the noisy block latent variable to obtain a denoised block latent variable of the image block; Performing image reconstruction on the denoising block latent variables to obtain an enhanced image block of the image block.

10. The method of claim 9, wherein: The feature extraction process is implemented by cascading N layers of second coding layers, where N is an integer greater than 0; extracting features from the image block to obtain image block features of the image block includes: Calling the first second coding layer of the N second coding layers to encode the image block, and obtaining image block features output by the first second coding layer; Calling the jth second coding layer of the N second coding layers to encode the image block features output by the (j-1)th second coding layer to obtain the image block features output by the jth second coding layer; Traverse the j to obtain the object structure features output by each first coding layer of the N first coding layers, where j is an integer greater than 0 and not greater than N.

11. The method of claim 10, wherein: The image block denoising process is implemented by an image denoising model, wherein the image denoising model includes a second encoder and a second decoder, wherein the second decoder includes N cascaded second decoding layers, and the second decoding layers correspond to the second encoding layers in a one-to-one manner; The step of performing image block denoising on the noise-added block latent variable in combination with the image block feature to obtain the denoised block latent variable of the image block comprises: Calling the second encoder to encode the noise-added block latent variable to obtain an encoded block latent variable; The coding block latent variable and the image block feature output by the second coding layer of the Nth layer are weighted summed based on the weight value to obtain a third splicing feature, and Calling the Nth second decoding layer of the N second decoding layers to decode the third splicing feature to obtain a decoding block latent variable output by the Nth second decoding layer; The decoded block latent variables output by the second decoding layer of the (j+1)th layer and the image block features output by the second encoding layer of the jth layer are weighted summed based on the weight value to obtain the fourth splicing feature, and Calling the jth second decoding layer of the N second decoding layers to decode the fourth splicing feature to obtain a decoding block latent variable output by the jth second decoding layer; Traverse the j to obtain the decoding block latent variables output by the first second decoding layer of the N second decoding layers, and use the decoding block latent variables output by the first second decoding layer as the denoising block latent variables.

12. The method according to any one of claims 1 to 11, wherein: Before obtaining the latent variables of the object image to be enhanced, the method further includes: Acquire a target image of the target object; Performing a resizing process on the target image at multiple different sizes to obtain an adjusted image at each size; The adjusted images at the respective sizes are respectively used as the target images.

13. The method according to any one of claims 1 to 12, wherein: The step of obtaining the latent variable of the object image to be enhanced comprises: encoding the object image to be enhanced to obtain the latent variable; The performing image reconstruction on the denoising latent variables to obtain a first object enhanced image of the object image includes: decoding the denoising latent variables to obtain the first object enhanced image of the object image.

14. The method according to any one of claims 1 to 13, wherein: The denoising process includes T denoising processes, where T is an integer greater than 0; the denoising process is performed on the noisy latent variable in combination with the object structure feature to obtain the denoised latent variable of the object image, including: In combination with the object structure feature, the first denoising process is performed on the noisy latent variable to obtain an intermediate denoising latent variable output by the first denoising process; In combination with the object structure feature, performing a t-th denoising process on the intermediate denoising latent variable output by the (t-1)-th denoising process to obtain the intermediate denoising latent variable output by the t-th denoising process; The t is traversed to obtain an intermediate denoising latent variable output by the T-th denoising process, and the intermediate denoising latent variable output by the T-th denoising process is used as the denoising latent variable of the object image.

15. The method according to any one of claims 1 to 14, wherein: The extracting the object structure feature of the target object in the object image includes: Acquire at least one of the following object information of the target object in the object image: a depth map of the object image, object posture information of the target object, and object line draft information of the target object; Feature extraction is performed on the object information to obtain the object structure feature.

16. An image enhancement device, comprising: an acquisition module, configured to acquire a latent variable of an object image to be enhanced, and add noise to the latent variable to obtain a noisy latent variable of the object image, wherein the object image is an image of a target object; an extraction module, configured to extract object structural features of the target object in the object image; A denoising module, configured to perform denoising processing on the noisy latent variable in combination with the object structure feature to obtain a denoised latent variable of the object image; A reconstruction module is configured to perform image reconstruction on the denoising latent variable to obtain a first object enhanced image of the object image.

17. An electronic device, comprising: a memory configured to store computer executable instructions; The processor is configured to implement the image enhancement method described in any one of claims 1 to 15 when executing the computer executable instructions stored in the memory.

18. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a processor, the image enhancement method according to any one of claims 1 to 15 is implemented.

19. A computer program product, comprising computer executable instructions or a computer program, wherein when the computer executable instructions or the computer program is executed by a processor, the image enhancement method according to any one of claims 1 to 15 is implemented.

Images