Face image processing method and device, electronic equipment, storage medium and program product

Abstract

The application provides a face image processing method and device, electronic equipment, storage medium and program product. The method comprises: performing multi-level identity feature extraction on a first face image of a target object to obtain identity features of the target object in multiple sizes; obtaining face optimization requirement information set for the first face image; based on a coding model matched with the face optimization requirement information, performing multi-level coding on the face optimization requirement information and the first face image to obtain coding feature maps in multiple sizes; based on a decoding model matched with the face optimization requirement information, performing multi-level decoding on the coding feature maps in multiple sizes and the identity features in multiple sizes to obtain a noise map; and performing denoising processing on the first face image based on the noise map to obtain a second face image. Through the application, the consistency of the optimized face image and the original image in the identity features can be ensured, thereby improving the accuracy and stability of image processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a facial image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the widespread use of cameras and other mobile devices with camera lenses, taking photos has become increasingly convenient, and the number of photos taken is also increasing. After taking a photo, people usually apply beauty filters before sharing it with others on social media platforms.

[0003] In related technologies, features are typically extracted and modeled separately for facial skin, eyes, mouth, nose, and other regions according to their functions. Independent beautification strategies are designed for different functions in different regions, and the final result is obtained by fusing the different regions. Due to the fragmented nature of processing different regions separately, this method results in poor image fusion after beautification, and it is easy to lose facial identity information. Summary of the Invention

[0004] This application provides a facial image processing method, apparatus, electronic device, storage medium, and program product that can optimize facial images while ensuring consistency in identity features between the optimized facial image and the original image, thereby improving the accuracy and stability of image processing.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a facial image processing method, the method comprising:

[0007] Multiple levels of identity feature extraction are performed on the first facial image of the target object to obtain identity features of the target object in multiple dimensions, wherein one level corresponds to one dimension;

[0008] Obtain facial optimization requirements information for the first facial image;

[0009] Based on the encoding model that matches the facial optimization requirement information, the facial optimization requirement information and the first facial image are encoded at multiple levels to obtain encoded feature maps of multiple sizes.

[0010] Based on a decoding model that matches the facial optimization requirements, the encoded feature maps of multiple sizes and the identity features of multiple sizes are decoded at multiple levels to obtain a noise map. For each level, the input to the decoding of the level is the output of the decoding of the previous level, the identity features with the same size as the decoded feature map of the level output, and the encoded feature map.

[0011] The first face image is denoised based on the noise map to obtain the second face image.

[0012] This application provides a facial image processing apparatus, including:

[0013] The feature extraction module is used to extract identity features at multiple levels from the first facial image of the target object, thereby obtaining identity features of the target object at multiple dimensions, wherein one level corresponds to one dimension;

[0014] The acquisition module is used to acquire facial optimization requirement information set for the first facial image;

[0015] The encoding module is used to encode the face optimization requirement information and the first face image at multiple levels based on an encoding model that matches the face optimization requirement information, so as to obtain the encoded feature maps of the multiple sizes.

[0016] The decoding module is used to decode the encoded feature maps of multiple sizes and the identity features of multiple sizes at multiple levels based on a decoding model that matches the facial optimization requirements information, to obtain a noise map. For each level, the input to the decoding of the level is the output of the decoding of the previous level, the identity features with the same size as the decoded feature map of the output of the level, and the encoded feature map.

[0017] The denoising module is used to denoise the first face image based on the noise map to obtain the second face image.

[0018] This application provides an electronic device, the electronic device comprising:

[0019] Memory is used to store executable instructions or computer programs.

[0020] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the facial image processing method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the facial image processing method provided in this application when executed by a processor.

[0022] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the facial image processing method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] Based on an encoding model that matches the facial optimization requirements, the facial optimization requirements are combined with the first facial image of the target object to generate encoded feature maps of multiple sizes. A decoding model matching the facial optimization requirements is then used to decode these features, resulting in a noise map. This ensures that personalized optimization needs are integrated into the model processing, allowing the optimization results to accurately meet specific requirements and improve the accuracy of image processing. Multi-level identity feature extraction is performed on the first facial image to obtain the identity features of the target object at different sizes. Using a decoding model, encoded feature maps of multiple sizes are fused with the identity features, and noise maps are generated layer by layer. The decoding input for each level includes not only the decoding output of the previous level but also the identity features of the same size as the output of that level, ensuring that identity features are not lost during multi-level decoding and effectively guaranteeing the stability of identity features during optimization. Finally, the first facial image is denoised based on the generated noise map to obtain a second facial image. This achieves consistency in identity features between the optimized facial image and the original image while optimizing the facial image, thereby improving the accuracy and stability of image processing. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the facial image processing system provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0027] Figure 3 This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 1 ;

[0028] Figure 4 This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 2 ;

[0029] Figure 5 This is a schematic diagram of the architecture of the first identity feature extraction model provided in the embodiments of this application;

[0030] Figure 6 This is a schematic diagram of the diffusion model provided in the embodiments of this application;

[0031] Figure 7 This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 3 ;

[0032] Figure 8 This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 4 ;

[0033] Figure 9This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 5 ;

[0034] Figure 10 This is a schematic diagram of the diffusion model and identity feature extraction model provided in the embodiments of this application;

[0035] Figure 11 This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 6 ;

[0036] Figure 12 This is a flowchart illustrating the beautification process based on a diffusion model, as provided in an embodiment of this application.

[0037] Figure 13 This is a schematic diagram of the diffusion model principle provided in the embodiments of this application;

[0038] Figure 14 This is a schematic diagram of the facial feature analysis and processing provided in the embodiments of this application;

[0039] Figure 15 This is a schematic diagram of the beautification result provided in the embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0045] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0046] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0047] 1) Identification features: These are key information that can be used to distinguish the identities of different target objects. They can include facial contours, facial features (such as eyes, nose, mouth, etc.), skin texture, and overall proportions. For non-human face target objects, identification features can also include unique facial landmarks or structures (such as the facial features of cartoon characters, the facial contours of animals, etc.).

[0048] 2) Facial Optimization Requirement Information: This refers to the specific description or type definition of facial image optimization requested by the user or system. Facial optimization requirement information can be a specific optimization type (such as a "natural makeup" style beautification) or a description of the requirement (such as "make the face look smoother without sacrificing realism"). Facial optimization requirement information guides the diffusion model in processing the image, ensuring that the final optimization result meets the expected effect.

[0049] 3) Diffusion Model: A deep learning model based on a probabilistic generative framework that mimics the diffusion process in physics. It gradually adds noise to disrupt the data distribution (forward diffusion), and then learns a reverse denoising process to recover the original data (reverse generation). Specifically, the forward diffusion process transforms the input data (such as an image) into Gaussian noise through multi-step progressive noise superposition; the reverse generation process trains a neural network (such as U-Net) to predict the noise at each time step and iteratively denoises, ultimately generating a high-quality image from random noise.

[0050] 4) Graph-to-Graph Task Based on Diffusion Model: This is a conditional generation task based on the diffusion model, aiming to transform an input image into an output image aligned with target conditions (such as text, another image, or semantic labels). Specifically, in the reverse denoising process, the generation direction is guided by conditional encoding (such as Contrastive Language-Image Pre-training (CLIP) text embeddings or image features), so that the denoising step depends simultaneously on the initial noise, the initial input image, and the input conditional information.

[0051] 5) Encoding Model: This is a crucial component of the diffusion model, typically implemented using the U-Net architecture. The encoding model's role is to transform input data (such as the first facial image of the target object and facial optimization requirements) into multi-level feature representations. Specifically, the encoding model extracts features from different levels of the image (such as local details and global structure) and combines these features with facial optimization requirements to generate encoded feature maps suitable for subsequent processing.

[0052] 6) Decoding Model: This is another important component of the diffusion model, typically implemented based on the U-Net architecture. The decoding model is the inverse process of the encoding model, responsible for progressively transforming the encoded feature maps into the final optimized result. In this embodiment, the decoding model combines features from multiple levels (including identity features and encoded feature maps) and progressively generates a face image that meets the optimization requirements.

[0053] 7) Conditional Control Network (ControlNet): This is a conditional control framework for diffusion models that precisely guides the image generation process by introducing external conditions (such as edge maps, depth maps, semantic segmentation maps, etc.). Its core mechanism is to freeze the parameters of the original diffusion model while adding a trainable copy network (containing an encoder and intermediate layers) for each control condition. Conditional information is gradually injected through "zero-convolutional" layers (convolutions with initial weights of zero), ensuring that the generated result retains the high-quality characteristics of the diffusion model while strictly aligning with the geometric, semantic, or stylistic constraints of the input conditions.

[0054] 8) Identity Preservation: Based on a diffusion model, identity preservation refers to using the generative capabilities of a diffusion model in image generation or editing tasks (such as face synthesis, style transfer, attribute modification, and beautification) to ensure that the output result is highly consistent with the identity features (such as face ID and key object attributes) of the input conditions (such as a face image) through specific technical means. This method aims to maintain the essential identity information of the target object during image generation (e.g., preserving facial features, biometric attributes, or the core structure of an object), while flexibly adjusting other irrelevant attributes (such as expression, pose, skin smoothness, and lighting).

[0055] 9) Human-computer interaction interface: An interface used to provide human-computer interaction functions. Examples include graphical user interfaces (GUIs), augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing viewing of stereoscopic images without special glasses), multimodal interfaces (interfaces combining multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interfaces (BMIs).

[0056] In related technologies, facial beautification of photos typically requires users to manually set beautification parameters one by one (such as skin smoothing intensity, eye magnification ratio, facial contour adjustment, face slimming, etc.), which is cumbersome and inconvenient for users. When implementing beautification processing, the face is usually divided into independent sub-regions such as skin, eyes, and lips, and local feature extraction and modeling are performed separately for sub-functions such as whitening, wrinkle removal, blemish removal, eye magnification, and aegyo sal generation to achieve regional facial beautification. This method customizes independent algorithm modules for different regions, such as designing independent beautification strategies for different functions in different regions (e.g., using a separate strategy for skin texture optimization and facial feature structure adjustment), and finally, the overall beautification is completed by stitching together the results from multiple modules in different regions. However, this fragmented processing mechanism has the following two problems: First, the optimization logic of each area lacks synergy, resulting in a decrease in the overall coordination of the face. In other words, the fragmentation of processing different areas separately leads to a lack of overall beauty effect. Second, isolated processing may destroy the global correlation of facial features (such as facial proportions and texture continuity), and the final beauty effect is often stiff and fragmented, making it difficult to maintain the unity of facial features and the stability of identity (ID).

[0057] To address the aforementioned problems, this application provides a facial image processing method, apparatus, electronic device, storage medium, and program product, which can ensure the consistency of the optimized facial image with the original image in terms of identity features, thereby improving the accuracy and stability of image processing. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. The following will describe exemplary applications when the device is implemented as a terminal or server.

[0058] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of a facial image processing system provided in this application embodiment. To support a facial image processing application, the facial image processing system 100 includes at least a terminal 400, a network 300, and a server 200. The terminal 400 is connected to the server 200 through the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. In some embodiments, the server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this application embodiment.

[0059] In some embodiments, the embodiments of this application can be implemented by the terminal 400 alone. For example, a user can input a first facial image of a target object through the client of the terminal 400. After receiving the first facial image, the terminal 400 performs multi-level identity feature extraction on the first facial image of the target object to obtain identity features of the target object in multiple sizes, and obtains facial optimization requirement information set for the first facial image. Based on an encoding model that matches the facial optimization requirement information, the facial optimization requirement information and the first facial image are encoded in multiple levels to obtain encoded feature maps of multiple sizes. Based on a decoding model that matches the facial optimization requirement information, the encoded feature maps of multiple sizes and the identity features of multiple sizes are decoded in multiple levels to obtain a noise map. The first facial image is denoised based on the noise map to obtain a second facial image. If the number of iterations for encoding and decoding has not reached the set value, the terminal 400 uses the second face image as the new first face image and returns to execute the encoding model based on matching the face optimization requirement information. This involves encoding the face optimization requirement information and the first face image at multiple levels to obtain encoded feature maps of multiple sizes. If the number of iterations for encoding and decoding has reached the set value, the terminal 400 uses the second face image obtained in the last iteration as the optimization result of the first face image. The terminal 400 displays this second face image on the current interface.

[0060] In some embodiments, the embodiments of this application can be implemented collaboratively by a server and a terminal. For example, a user can input a first facial image of a target object through the client of terminal 400. After receiving the first facial image of the target object, terminal 400 encapsulates the first facial image of the target object into an image processing request and sends the image processing request to server 200 through network 300. In response to the received image processing request, server 200 extracts identity features from the first facial image of the target object at multiple levels to obtain identity features of the target object at multiple sizes, and obtains facial optimization requirement information set for the first facial image; based on an encoding model matching the facial optimization requirement information, it encodes the facial optimization requirement information and the first facial image at multiple levels to obtain encoded feature maps at multiple sizes; based on a decoding model matching the facial optimization requirement information, it decodes the encoded feature maps and identity features at multiple levels at multiple levels to obtain a noise map; and based on the noise map, it performs denoising processing on the first facial image to obtain a second facial image. If the number of iterations for encoding and decoding has not reached the set value, server 200 uses the second face image as the new first face image and returns to execute the encoding model based on matching the face optimization requirement information. This involves encoding the face optimization requirement information and the first face image at multiple levels to obtain encoded feature maps of multiple sizes. If the number of iterations for encoding and decoding has reached the set value, server 200 uses the second face image obtained in the last iteration as the optimization result of the first face image. Server 200 sends this second face image to terminal 400 so that terminal 400 displays the optimization result of the first face image on the current interface.

[0061] The facial image processing method provided in this application can be applied to any scenario that requires processing images containing faces, improving the accuracy and stability of image processing. Specific application scenarios include:

[0062] 1) Mobile Device Beautification: When users use photo or video call applications on mobile devices, they desire real-time optimization of their facial images while maintaining a natural and realistic look. The user opens a beauty camera app and selects the "Natural Makeup" style as their facial optimization request. The device extracts the user's identity features from their initial facial image. Through the encoding process in a diffusion model, an coded feature map is generated based on the "Natural Makeup" text information and the initial facial image. Through the decoding process in the diffusion model, an optimized facial image is generated in real-time based on the coded feature map and the user's identity features.

[0063] 2) Online Virtual Makeup Try-On Service: Users access the online makeup try-on platform, upload their photos, and select a makeup style (such as "elegant red lips" or "fresh nude makeup"). The platform then returns an optimized makeup try-on image. Alternatively, users can upload a photo of their face and input their desired face optimization (e.g., "I want an elegant and glamorous style"). The server parses this information to determine the desired optimization type as "elegant red lips." Through multi-level feature extraction of the facial photo and the encoding and decoding process of a diffusion model matching "elegant red lips," a makeup try-on image with a red lip effect is generated.

[0064] 3) Medical Aesthetics Consultation System: Users wish to simulate certain facial adjustment effects (such as face slimming, eyebrow lifting, etc.) through the medical aesthetics consultation system to facilitate communication between doctors and users. Users upload facial photos and input facial optimization needs (such as "slight face slimming" or "eyebrow lifting 5mm"). The server extracts the user's facial features and encodes the facial optimization needs and facial photos using an encoding model that matches the facial optimization needs information to obtain an encoded feature map. Then, using a decoding model that matches the facial optimization needs information, the encoded feature map and facial features are decoded to obtain a noise map. The noise map is used to denoise the facial photo to obtain an image with the simulated optimization effect, which is then displayed to the user and doctor.

[0065] 4) Anime Character Design Tool: Anime designers want to quickly generate character facial images that match a specific style (such as "cute girl" or "rugged man"). Designers input a basic image of the character's face and information on facial optimization requirements (such as "cute girl"). The terminal's design tool extracts the character's facial features and generates an optimized facial image. The optimized facial image can be used as reference material for character design.

[0066] 5) Social media platforms offer image optimization features. Users frequently upload photos or videos to social networks. These photos may contain noise (such as graininess or blurriness) in the facial images due to shooting environment, equipment limitations, or other reasons. To improve user experience, social media platforms can provide an AI-based facial image optimization function to help users automatically repair and enhance facial image quality or beautify faces. Users upload a photo containing their face (the first facial image) through a social media application. After receiving the first facial image, the social media platform's server extracts the user's facial identity features. Using an encoding model that matches the user's input facial optimization requirements, it encodes the facial optimization requirements and the photo to obtain an encoded feature map. Then, using a decoding model that matches the facial optimization requirements, it decodes the encoded feature map and identity features to obtain a noise map. This noise map is then used to denoise the photo, resulting in an optimized image. The optimized image is then published on the social media platform, where other users can view, like, and comment on it.

[0067] In some embodiments, the electronic device implementing the facial image processing method provided in this application may be Figure 1 Terminal 400 in the middle. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0068] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0069] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0070] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0071] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0072] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0073] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0074] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0075] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0076] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0077] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A face image processing apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: a feature extraction module 4551, an acquisition module 4552, an encoding module 4553, a decoding module 4554, and a noise reduction module 4555. These modules are logically linked and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0078] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the facial image processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0079] In some embodiments, the terminal or server can implement the facial image processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as live streaming APPs or instant messaging APPs; or they can be mini-programs that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run, such as beauty mini-programs embedded in instant video messaging APPs. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0080] The following describes the facial image processing method provided in the embodiments of this application. As mentioned above, the electronic device implementing the facial image processing method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0081] The facial image processing method provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0082] See Figure 3 , Figure 3 This is a flowchart illustrating the facial image processing method provided in the embodiments of this application. Figure 1 , will combine Figure 3 The steps shown are explained as follows: Figure 3 As shown, taking the terminal as the execution subject of the facial image processing method as an example, the method includes the following steps 101 to 105:

[0083] In step 101, multiple levels of identity features are extracted from the first facial image of the target object to obtain identity features of the target object in multiple dimensions.

[0084] Each level corresponds to one size.

[0085] Here, the first face image of the target object is an image containing the face of the target object, which can be a real person, animal, anime character, etc. The first face image can be a real human face image, an anime character face image, an animal face image, etc. The first face image can be an image directly input by the user to the terminal, such as a face image taken by the user through the front-facing camera. Alternatively, the first face image can be an image obtained from the original image input by the user through face detection and cropping. In practical applications, the original image input by the user may not be directly captured of the face, but rather an image containing more background information or other objects. Therefore, before step 101, face detection and cropping need to be performed on the original image to extract the face region of the target object as the first face image. Face detection uses algorithms to identify whether a face exists in the original image and locates the position and extent of the face. The detection result is a face bounding box, representing the position of the face in the original image. After completing face detection, the face region is cropped from the original image based on the detected face bounding box to obtain the first face image.

[0086] Identity features are key information extracted from the first facial image that identifies the target object. These features include both local details (such as skin texture, pores, and facial feature boundaries) and global structure (such as facial contours, symmetry, and facial proportions). Multiple levels of identity feature extraction can be performed on the first facial image, with each level outputting an identity feature of a specific size. Different sized identity features reflect information about the target object's face at different resolutions. For example, larger identity features (lower resolution) contain global structural information about the target object's face, while smaller identity features (higher resolution) contain local structural information about the target object's face.

[0087] For example, a user wants to optimize their selfie using a beauty app and uploads the selfie as the first facial image to the beauty app. The first facial image undergoes four levels of feature extraction, resulting in four feature sizes: 64×64, 32×32, 16×16, and 8×8.

[0088] In some embodiments, the identity features of the target object include the target object's facial edge features and facial depth features. See also Figure 4 , Figure 4 The step 101 illustrates the extraction of multiple levels of identity features from the first facial image of the target object, yielding identity features of the target object at multiple dimensions. This can be achieved through steps 1011 to 1014:

[0089] In step 1011, contour detection is performed on the first face image to obtain the facial contour map of the target object.

[0090] Here, facial edge features are features that reflect the contour information of the target object's face and the contour information of its facial features. For example, facial edge features reflect information such as the contour of the eyes, the edge of the nose, and the boundary of the mouth. Facial depth features are global structural information of the target object's face, including the proportional relationships, symmetry, and high-level semantic information of the face.

[0091] A contour detection algorithm can be used to extract the facial contour map from a first face image. The facial contour map is represented in binary or grayscale form, highlighting the contours of the face and facial features. This application does not limit the contour detection algorithm; for example, the Canny edge detection algorithm or an edge extraction module in a deep learning model can be used. For instance, if the Canny edge detection algorithm is used to extract the facial contour map from the first face image, the first face image is first converted to a grayscale image, and a Gaussian blur filter is applied to blur the grayscale image to obtain a filtered image. Then, edge detection is performed on the filtered image to generate a binary or grayscale facial edge map. Alternatively, if a deep learning model (such as a U-Net network or a lightweight facial landmark detection model) is used to directly extract the facial edge map, the first face image is first input into a pre-trained semantic segmentation model or landmark detection model to obtain a probability map or mask map representing the regional distribution of the face and facial features. A clear facial contour map is then generated based on the probability map or mask map.

[0092] In step 1012, edge features are extracted from the facial contour map at multiple levels to obtain facial edge features of the target object at multiple sizes.

[0093] Here, multiple convolutional operations in a convolutional neural network can be used to extract edge features from the facial contour map at multiple levels, resulting in facial edge features of the target object at multiple sizes. Each convolutional operation outputs a facial edge feature of a specific size. The size of the facial edge features output at each level gradually decreases as the number of levels increases.

[0094] For example, a user uploads a selfie as the first facial image. Canny edge detection is used to process the first facial image to obtain the facial contour map of the target object. The facial edge features in the facial contour map extracted based on the convolutional neural network include facial edge features of multiple sizes: 64×64, 32×32, 16×16, and 8×8.

[0095] In some embodiments, edge feature extraction at multiple levels is performed sequentially, and the output of the edge feature extraction at the previous level is the input of the edge feature extraction at the next level. The size of the facial edge features output at the next level is smaller than the size of the facial edge features output at the previous level.

[0096] Here, a progressive, multi-level edge feature extraction approach is adopted, with the output of each level serving as the input for the next level. Simultaneously, the size of the facial edge features decreases layer by layer. First, the first-level edge feature extraction applies convolution operations and a non-linear activation function (e.g., ReLU) to the input facial contour map to obtain an initial feature map. Pooling operations (e.g., max pooling) are then used to reduce the spatial resolution of the feature map, resulting in the first-level output facial edge features. The second-level edge feature extraction continues by applying convolution operations and a non-linear activation function to the first-level output facial edge features to obtain another feature map. Pooling operations are again used to reduce the spatial resolution of the feature map, resulting in the second-level facial edge features. These steps are repeated for each level until edge feature extraction for multiple levels is completed, resulting in facial edge features of multiple sizes.

[0097] For example, if the facial contour image is 128×128 pixels, the first-level edge feature extraction processes the facial contour image to obtain facial edge features of size 64×64. The second-level edge feature extraction processes the 64×64 facial edge features to obtain facial edge features of size 32×32. The third-level edge feature extraction processes the 32×32 facial edge features to obtain facial edge features of size 16×16. The fourth-level edge feature extraction processes the 16×16 facial edge features to obtain facial edge features of size 8×8.

[0098] This application's embodiments extract multi-level facial edge features in a progressive manner, capturing the contour information of the face and facial features from local to global perspectives. This recursive design not only reduces computational complexity but also ensures the effective extraction of features at different scales. These facial edge features of different sizes play a crucial role in subsequent optimization processing, ensuring that the optimized image retains the realism and detail of the target object.

[0099] In some embodiments, multi-level edge feature extraction is achieved through a first identity feature extraction model, which includes a zero convolutional layer and an N-level first identity feature extraction network, where N is a positive integer greater than 1. Step 1012 involves multi-level edge feature extraction of the facial contour map to obtain facial edge features of multiple sizes for the target object. This can be achieved as follows: First, based on the zero convolutional layer in the first identity feature extraction model, the facial contour map is convolved to obtain intermediate edge features; then, based on the first-level first identity feature extraction network, the intermediate edge features are downsampled to obtain first-level facial edge features; finally, based on the (n+1)-level first identity feature extraction network, the n-level facial edge features are downsampled according to a preset step size to obtain (n+1)-level facial edge features. The size of the (n+1)-level facial edge features is proportionally reduced to that of the n-level facial edge features based on the preset step size, where n is a positive integer that increases sequentially, 1 ≤ n ≤ N-1.

[0100] Here, the model structure of the first identity feature extraction model is not limited in this embodiment; for example, it can be a convolutional neural network model, a deep learning model, etc. In this embodiment, the Conditional Control Net model can be used as the first identity feature extraction model. Figure 5 This is a schematic diagram of the architecture of the first identity feature extraction model provided in this application embodiment. The conditional control model includes a zero convolutional layer 501 and N first identity feature extraction networks 502. Each first identity feature extraction network 502 includes a zero convolutional layer and an encoder.

[0101] Zero-convolutional layers do not change the size of facial edge features, only the number of channels. Therefore, after the facial contour image is processed by zero-convolutional layer 501, the intermediate edge features are obtained with the same size as the facial contour image. N first identity feature extraction networks 502 adopt a progressive approach, performing multi-level downsampling processing on the intermediate edge features. The output of each level serves as the input of the next level, while the size of the facial edge features decreases layer by layer. The downsampling processing is a convolution operation with a preset stride greater than 1. The preset stride is the downsampling ratio of each level. For example, when the preset stride is 2, the size of the facial edge features output by each level is 1 / 2 of the facial edge features output by the previous level. Specifically, the encoder in the first identity feature extraction network 502 convolves the output of the previous level with a preset stride to obtain edge features with reduced size. Then, the edge features are convolved by the zero-convolutional layer in the first identity feature extraction network 502 to obtain the facial edge features of that level.

[0102] For example, N=4, the preset stride is 2, and the size of the facial contour map is 128×128. The zero-convolutional layer 501 in the first identity feature extraction model performs an initial convolution operation on the input facial contour map to generate intermediate edge features. The size of the intermediate edge features remains 128×128. The encoder in the first-level first identity feature extraction network 502 convolves the intermediate edge features with a stride of 2, resulting in a 64×64 edge feature in the first level. The zero-convolutional layer in the first-level first identity feature extraction network 502 convolves the 64×64 edge features with a stride of 2, resulting in a 64×64 facial edge feature. The encoder in the second-level first identity feature extraction network 502 convolves the 64×64 facial edge features with a stride of 2, resulting in a 32×32 edge feature in the second level. The zero-convolutional layer in the first identity feature extraction network 502 of the second layer convolves the 32×32 edge features with a stride of 2 to obtain 32×32 facial edge features. The remaining layers process them sequentially to obtain facial edge features of size 16×16 output from the third layer and facial edge features of size 8×8 output from the fourth layer.

[0103] In this embodiment, the first identity feature extraction network at each level progressively reduces the spatial resolution of facial edge features through an encoder and zero-convolutional layers. The resulting facial edge features of multiple sizes comprehensively capture the local details and global structural information of the face and facial features. Initially, the zero-convolutional layers do not change the size of the input facial contour map, only adjusting the number of channels to suit subsequent processing needs. This design ensures the integrity of the input information and avoids early information loss. The downsampling operations at each level (such as convolutions with a stride of 2) significantly reduce the size of the facial edge features, thereby reducing computational complexity. By extracting facial edge features of different sizes layer by layer, the identity features of the target object can be better preserved during subsequent optimization.

[0104] In step 1013, depth detection is performed on the first face image to obtain the facial depth map of the target object.

[0105] Here, depth detection algorithms (such as monocular depth estimation models) can be used to perform depth detection on the first face image to obtain a face depth map. Specifically, the monocular depth estimation model can be built based on a convolutional neural network and trained with a large amount of labeled data, enabling it to predict the corresponding depth information from a single two-dimensional image. In practice, the preprocessed first face image is first input into the monocular depth estimation model. The model calculates the depth value of each pixel based on clues such as texture, edges, and lighting in the first face image, thereby generating a face depth map reflecting the depth distribution of the face. The face depth is represented in the form of a grayscale image or a heatmap, reflecting the depth distribution of the face. Different grayscale values ​​or colors represent different depth levels: lighter colors (such as white or red) represent areas closer to the camera, while darker colors (such as black or blue) represent areas farther away from the camera. The face depth map not only contains the overall structural information of the target object's face but also accurately reflects the relative positional relationships between facial features, such as the height of the bridge of the nose, the degree of eye hollowing, and the degree of chin protrusion.

[0106] It should be noted that the execution order of steps 1011 and 1013 is not limited in this embodiment of the application, and steps 1011 and 1013 can be executed simultaneously.

[0107] In step 1014, depth features are extracted from the face depth map at multiple levels to obtain face depth features of the target object at multiple sizes.

[0108] Here, multiple convolutional operations in a convolutional neural network can be used to extract depth features from the facial depth map at multiple levels, resulting in facial depth features of the target object at multiple sizes. Each convolutional operation outputs a facial depth feature of a specific size. The size of the facial depth features output at each level gradually decreases as the number of levels increases.

[0109] For example, a user uploads a selfie as the first facial image, and a monocular depth estimation model is used to extract the facial depth map of the target object from the first facial image. The facial depth features extracted based on the convolutional neural network include facial depth features of multiple sizes: 64×64, 32×32, 16×16, and 8×8.

[0110] In some embodiments, the depth feature extraction of multiple layers is performed sequentially, and the output of the depth feature extraction of the previous layer is the input of the depth feature extraction of the next layer. The size of the facial depth feature output by the next layer is smaller than the size of the facial depth feature output by the previous layer.

[0111] Here, a progressive, multi-level approach is adopted for deep feature extraction. The output of each level serves as the input for the next level, while the size of the facial depth features decreases with each level. First, the first-level deep feature extraction applies convolution operations and a non-linear activation function (e.g., ReLU) to the input facial depth map to obtain an initial feature map. Pooling operations (e.g., max pooling) are then used to reduce the spatial resolution of the feature map, resulting in the first-level output facial depth features. The second-level deep feature extraction applies convolution operations and a non-linear activation function to the first-level output facial depth features to obtain another feature map. Pooling operations are again used to reduce the spatial resolution of the feature map, resulting in the second-level facial depth features. These steps are repeated for each level until multiple levels of deep feature extraction are completed, resulting in facial depth features of multiple sizes.

[0112] For example, if the face depth map is 128×128 pixels, the first-level depth feature extraction processes the face depth map to obtain a face depth feature of size 64×64. The second-level depth feature extraction processes the face depth feature of size 64×64 to obtain a face depth feature of size 32×32. The third-level depth feature extraction processes the face depth feature of size 32×32 to obtain a face depth feature of size 16×16. The fourth-level depth feature extraction processes the face depth feature of size 16×16 to obtain a face depth feature of size 8×8.

[0113] In this embodiment, the depth feature extraction at each level involves performing convolution and pooling operations on the facial depth features output from the previous level. This progressively reduces the spatial resolution of the facial depth features, thereby capturing facial depth features at different scales and ultimately comprehensively capturing the global structure and local details of the face and facial features. Convolution operations are used to extract local depth features, while pooling operations (such as max pooling) significantly reduce computation by lowering the spatial resolution of the feature map, while preserving crucial depth information. By extracting facial depth features of different sizes layer by layer, the identity features of the target object can be better preserved during subsequent optimization.

[0114] In some embodiments, multi-level deep feature extraction is achieved through a second identity feature extraction model, which includes a zero-convolutional layer and an N-level second identity feature extraction network, where N is a positive integer greater than 1. Step 1014 involves multi-level deep feature extraction of the facial depth map to obtain facial depth features of multiple sizes for the target object. This can be achieved as follows: First, based on the zero-convolutional layer in the second identity feature extraction model, the facial depth map is convolved to obtain intermediate depth features; then, based on the first-level second identity feature extraction network, the intermediate depth features are downsampled to obtain the first-level facial depth features; finally, based on the (n+1)-level second identity feature extraction network, the n-level facial depth features are downsampled according to a preset step size to obtain the (n+1)-level facial depth features. The size of the (n+1)-level facial depth features is proportionally reduced to that of the n-level facial depth features based on the preset step size, where n is a positive integer that increases sequentially, 1 ≤ n ≤ N-1.

[0115] Here, the model structure of the second identity feature extraction model is not limited in this embodiment; for example, it can be a convolutional neural network model, a deep learning model, etc. In this embodiment, the Conditional Control Network (ControlNet) model can be used as the second identity feature extraction model. Exemplarily, the model structure of the second identity feature extraction model is similar to... Figure 5 The structure of the first identity feature extraction model is similar and will not be described further.

[0116] After the facial depth map is processed by the first zero-convolutional layer, an intermediate depth feature with the same size as the facial depth map is obtained. N secondary identity feature extraction networks employ a progressive approach, performing multi-level downsampling on the intermediate depth feature. The output of each level serves as the input to the next level, while the size of the facial depth feature decreases layer by layer. The downsampling process is essentially a convolution operation with a preset stride greater than 1. The preset stride is the downsampling ratio of each level; for example, with a preset stride of 2, the size of the facial depth feature output at each level is half that of the previous level. Specifically, the encoder in the secondary identity feature extraction network convolves the output of the previous level with a preset stride to obtain a reduced-size depth feature. This reduced-size feature is then convolved by the zero-convolutional layer in the secondary identity feature extraction network to obtain the facial depth feature at that level.

[0117] It should be noted that the number of layers in the first identity feature extraction network in the first identity feature extraction model is the same as the number of layers in the second identity feature extraction network in the second identity feature extraction model. The preset step size used by the first identity feature extraction network when performing downsampling is the same as the preset step size used by the second identity feature extraction network when performing downsampling. This application embodiment does not limit the value of the preset step size and can set it according to actual needs.

[0118] For example, N=4, the preset stride is 2, and the size of the face depth map is 128×128. The zero-convolutional layer in the second identity feature extraction model performs an initial convolution operation on the input face depth map to generate intermediate depth features. The size of the intermediate depth features remains 128×128. The encoder in the first-level second identity feature extraction network convolves the intermediate depth features with a stride of 2, resulting in a 64×64 depth feature in the first level. The zero-convolutional layer in the first-level second identity feature extraction network convolves the 64×64 depth feature with a stride of 2, resulting in a 64×64 face depth feature. The encoder in the second-level second identity feature extraction network convolves the 64×64 face depth feature with a stride of 2, resulting in a 32×32 depth feature in the second level. The zero-convolutional layer in the second-level second identity feature extraction network convolves the 32×32 depth feature with a stride of 2, resulting in a 32×32 face depth feature. The remaining layers are processed sequentially to obtain the facial depth features of the third layer with a size of 16×16 and the facial depth features of the fourth layer with a size of 8×8.

[0119] In this embodiment, the second identity feature extraction network at each level gradually reduces the spatial resolution of the depth features by performing convolution and downsampling operations on the depth features output from the previous level, thereby capturing facial depth information at different scales. The resulting facial depth features of multiple sizes can comprehensively capture the global structure and local details of the face and facial features. In the initial stage, the zero-convolutional layer does not change the size of the input facial depth map, only adjusting the number of channels to adapt to subsequent processing needs. This design ensures the integrity of the input information and avoids early information loss. The downsampling operation at each level (such as convolution with a stride of 2) significantly reduces the size of the depth features, thereby reducing computational complexity. Simultaneously, the progressive approach ensures that the computational load at each step is controllable, improving overall efficiency. By extracting facial depth features of different sizes layer by layer, the identity features of the target object can be better preserved during subsequent optimization.

[0120] In step 102, facial optimization requirements information set for the first facial image is obtained.

[0121] Here, the facial optimization requirement information can be either the facial optimization type or a description of the optimization requirements. The facial optimization type is a predefined optimization category; users can predefine multiple types. For example, in a beauty mode, the facial optimization type could be a beauty style, such as "natural nude makeup," "3D contours," or "Korean cool white." The facial optimization requirement description text uses natural language to describe the optimization needs, such as "make the skin look smoother without sacrificing realism" or "adjust the facial contours to make them more sculpted."

[0122] In some embodiments, the user can select the face optimization type. Preset face optimization type options (such as drop-down menus, button groups, etc.) are provided in beauty apps or image processing tools, allowing users to choose the appropriate type based on their needs. For example, a user might select "Natural Nude Makeup" as the face optimization type for this image processing.

[0123] In some embodiments, a text description of facial optimization needs can be obtained through user input. A text input box is provided in beauty apps or image processing tools, allowing users to describe their optimization needs in natural language. For example, a user could input "make the skin look smoother without losing its naturalness" into the text input box. Other modal input methods, such as voice input, can also be provided, and the resulting multimodal information can be modally converted to obtain the text description of facial optimization needs.

[0124] In some embodiments, facial optimization needs information can be obtained through system recommendations. Based on feature analysis of a first facial image (such as skin tone, facial proportions, age, etc.), the system can automatically recommend suitable facial optimization types or descriptions of facial optimization needs. Users can choose to accept the recommendations or make adjustments. For example, if the system detects that the target subject's skin tone is dark, it recommends the "brighten skin tone" facial optimization type. After user confirmation, the system uses this facial optimization type as the facial optimization needs information.

[0125] In step 103, based on the encoding model that matches the facial optimization requirement information, the facial optimization requirement information and the first facial image are encoded at multiple levels to obtain encoded feature maps of multiple sizes.

[0126] Here, multiple facial image processing models matching specific facial optimization types can be pre-trained, such as a facial image processing model matching "natural nude makeup" or a facial image processing model matching "three-dimensional contours." After obtaining facial optimization requirement information, the facial optimization type is determined based on this information, and a facial image processing model matching this optimization type is selected from the multiple models. After obtaining the first facial image, identity features, and facial optimization type, the first facial image can be optimized based on the facial image processing model matching the optimization type, combined with the identity features and the facial optimization type, to obtain the optimized result of the first facial image.

[0127] The facial image processing model can be a diffusion model. The process of optimizing the first facial image using the diffusion model can be divided into a forward noise addition process and a reverse noise reduction process. Figure 6 This is a schematic diagram of the diffusion model provided in an embodiment of this application. See also... Figure 6 First, the diffusion model adds noise to the first face image for a preset number of rounds (T rounds, where T is a positive integer), resulting in a noisy first face image. Then, the denoising module 603 (e.g., a U-net network) in the diffusion model performs denoising on the noisy first face image for a preset number of rounds, obtaining an optimized result for the first face image. The denoising process of the denoising module 603 is implemented by an encoding model 601 and a decoding model 602. Therefore, after selecting a face image processing model that matches the face optimization type, the noisy first face image and the face optimization type are first encoded at multiple levels based on the encoding model 601 in the face image processing model, resulting in encoded feature maps of multiple sizes.

[0128] In some embodiments, before encoding the face optimization requirement information and the first face image at multiple levels based on an encoding model that matches the face optimization requirement information to obtain encoded feature maps of multiple sizes, the noisy image in the t-th round is denoised to obtain the noisy face image in the t-th round, and the noisy face image in the t-th round is determined as the noisy image in the (t+1)-th round, where the noisy image in the 1-th round is the first face image, t is a positive integer that increases sequentially, 1≤t≤T, and T is a positive integer; the noisy face image in the T-th round is used as the first face image input to the encoding model.

[0129] Here, for the t-th round, the Gaussian noise for the t-th round can be randomly determined from the one-dimensional standard Gaussian noise, and the Gaussian noise can be superimposed on the noisy image of the t-th round to obtain the noisy face image of the t-th round.

[0130] For example, T is a preset round. When T=1, after inputting the face optimization requirements information and the first face image into the diffusion model, the diffusion model only needs to add noise to the first face image once to obtain the noisy face image of the first round, which is the noisy first face image. The noisy face image of the first round is used as the first face image input into the encoding model.

[0131] For example, when T=2, after inputting the face optimization requirement information and the first face image into the diffusion model, the diffusion model adds noise to the first face image to obtain a noisy face image in the first round. This noisy face image in the first round is then determined as the noisy face image in the second round. Noise is added to the noisy face image in the second round to obtain a noisy face image in the second round, and this noisy face image in the second round is used as the first face image input into the encoding model.

[0132] This application embodiment adds noise to the first face image multiple times (from the first round to the Tth round). The Gaussian noise in each round is randomly sampled from a one-dimensional standard Gaussian distribution. This randomness ensures the controllability and diversity of the noise addition process, simulates the uncertainty in real-world scenarios, and the noisy face image after multiple rounds of noise addition can better adapt to the optimization needs in complex environments, improving the model's generalization ability in different scenarios.

[0133] In some embodiments, when the facial optimization requirement information is a facial optimization type, the encoding model matching the facial optimization requirement information is an encoding model pre-associated with the facial optimization type, and the decoding model matching the facial optimization requirement information is a decoding model pre-associated with the facial optimization type.

[0134] Here, when the face optimization requirement information is a face optimization type, a face image processing model pre-associated with that face optimization type can be selected from multiple face image processing models. A face image processing model, as a diffusion model, includes an encoding model and a decoding model. Determining the face image processing model pre-associated with the face optimization type means determining the encoding and decoding models pre-associated with the face optimization type.

[0135] For example, the facial optimization requirement information is the "natural nude makeup" facial optimization type selected by the user. Based on the "natural nude makeup" facial optimization type, the facial image processing model pre-associated with the "natural nude makeup" facial optimization type is invoked. The encoding and decoding models in the facial image processing model are used to optimize the first facial image after adding noise, resulting in an optimized image that conforms to the "natural nude makeup" style.

[0136] In this embodiment, each facial optimization type (such as "natural makeup," "skin smoothing and whitening," "face slimming," etc.) is pre-associated with a specific encoding and decoding model. These models are specifically trained or adjusted according to the characteristics of the optimization type, thus more accurately meeting the corresponding optimization requirements. Each optimization type corresponds to an independent encoding and decoding model; this modular design allows the system to be easily extended to support new optimization types. By using the encoding and decoding models pre-associated with the facial optimization type in the image optimization process, the relevance and quality of the optimization results can be significantly improved.

[0137] In some embodiments, where the facial optimization requirement information is a description of facial optimization requirements, see [link to relevant documentation]. Figure 7 Before step 103, the facial image processing method provided in this application embodiment further includes the following steps 201 to 204:

[0138] In step 201, the text features of the requirement description text and the type features of multiple face optimization types are extracted.

[0139] Here, Natural Language Processing (NLP) techniques can be used to extract textual features from the requirement description text. These textual features are vector representations of the requirement description text. For example, a pre-trained model (such as the BERT encoding model) can be used to convert the requirement description text into vector-represented textual features. Multiple face optimization types are predefined during model training, such as "natural nude makeup" and "3D contours." The same pre-trained model can be used to convert each face optimization type into a vector-represented type feature. The vector lengths of the textual features and the type features are consistent.

[0140] For example, the user inputs a description of their needs as "to make the skin look smoother without sacrificing realism." This description is encoded into a 512-character text feature using the BERT encoding model. Three facial optimization types are predefined: "natural nude makeup," "3D contour," and "Korean cool white." Each of these three facial optimization types is encoded into a 512-character feature using the BERT encoding model.

[0141] In step 202, probability prediction is performed based on text features and type features to obtain the matching probability of each face optimization type corresponding to the face optimization requirement information.

[0142] Here, a probabilistic prediction model can be used to perform probability prediction based on text features and type features to obtain the matching probability of each face optimization type corresponding to the face optimization requirement information. This application embodiment does not limit the model structure of the probabilistic prediction model; for example, the probabilistic prediction model is a classification model, and the model structure can be a logistic regression network or a neural network, etc. The probabilistic prediction model can calculate the similarity between text features and each type feature. In this application embodiment, similarity can be calculated using methods such as cosine similarity and Euclidean distance. After obtaining the similarity between text features and each type feature, the probabilistic prediction model can perform activation processing on the similarity based on an activation function (e.g., the Softmax function) to obtain the matching probability of each face optimization type corresponding to the face optimization requirement information.

[0143] For example, if the user inputs a description of their needs as "to make the skin look smoother without losing its realism", and three facial optimization types are predefined as "natural nude makeup", "3D contouring" and "Korean cool white", the predicted matching probability for the facial optimization needs is 0.7 for "natural nude makeup", 0.1 for "3D contouring" and 0.1 for "Korean cool white".

[0144] The probabilistic prediction model can be trained as follows: Multiple training samples are obtained, each including a requirement description text sample, a face optimization type, and a matching label. The matching label indicates whether the requirement description text sample matches the face optimization type (1 indicates a match, 0 indicates no match). Training samples can be obtained from historical data or manually labeled data. For example, the training samples include the requirement description text sample "Requirement Description Text Sample", the face optimization type "Natural Nude Makeup", and the matching label "1". Features are extracted from the requirement description text sample and the face optimization type in each training sample to obtain text feature samples and type feature samples. Based on the text feature samples and type feature samples, the probabilistic prediction model predicts the matching probability samples between the requirement description text sample and the face optimization type. A loss function (e.g., cross-entropy loss function) is calculated based on the difference between the matching probability samples and the label. The gradient is calculated according to the loss function, and the model parameters in the probabilistic prediction model are updated using an optimization algorithm until the loss function converges or the maximum number of iterations is reached, resulting in the trained probabilistic prediction model.

[0145] In step 203, at least one face optimization type is selected from multiple face optimization types in descending order of matching probability.

[0146] Here, multiple face optimization types are sorted according to their matching probabilities to obtain the sorting results. If only one face optimization type is selected, the face optimization type with the highest matching probability is selected from the sorting results. Alternatively, if multiple face optimization types are selected, the top K (integers greater than 1) face optimization types with the highest matching probabilities are selected from the sorting results according to actual needs.

[0147] For example, the predicted matching probability for the facial optimization requirement information is 0.7 for "natural nude makeup," 0.1 for "3D contour," and 0.1 for "Korean cool white." You can choose only "natural nude makeup" as the final facial optimization type, or, if using a multi-select model (K=2), choose both "natural nude makeup" and "Korean cool white" as the final facial optimization type.

[0148] In step 204, at least one encoding model corresponding to a face optimization type is determined as an encoding model that matches the face optimization requirement information.

[0149] Here, if there is only one face optimization type, the coding model pre-associated with that face optimization type is determined as the coding model that matches the face optimization requirement information. If there are multiple face optimization types, the coding model pre-associated with each face optimization type is determined as the coding model that matches the face optimization requirement information.

[0150] For example, if the "Natural Nude Makeup" face optimization type is selected, the diffusion model pre-associated with the "Natural Nude Makeup" face optimization type is invoked, and the encoding model in the diffusion model is determined to be the encoding model that matches the face optimization requirement information. If both "Natural Nude Makeup" and "Korean Cool White" are selected, both the diffusion model pre-associated with "Natural Nude Makeup" and the diffusion model pre-associated with "Korean Cool White" are invoked, and the encoding models in both diffusion models are determined to be the encoding models that match the face optimization requirement information.

[0151] This application utilizes natural language processing technology to transform user-inputted requirement description text into computable feature vectors, achieving accurate parsing of complex optimization requirements. Predefined facial optimization types are represented using features to facilitate subsequent matching calculations. By calculating the matching probability between the requirement description text and each facial optimization type, the system sorts and selects the optimal facial optimization type based on the matching probability, avoiding the subjectivity of manual intervention and ensuring that the selection best meets the user's needs. Each facial optimization type is pre-associated with a specific encoding model, allowing the system to quickly call the corresponding encoding model based on the selection result, reducing runtime dynamic adjustments.

[0152] In some embodiments, where the facial optimization requirement information is a description of the facial optimization requirement, see [link to relevant documentation]. Figure 8 Before step 103, the facial image processing method provided in this application embodiment further includes the following steps 301 to 302:

[0153] In step 301, the language model is invoked based on the first prompt word to trigger the language model to perform intent recognition on the demand description text and obtain at least one face optimization type that matches the intent of the demand description text.

[0154] The first cue word is used to instruct the language model to identify at least one face optimization type that matches the intent of the text describing the requirements.

[0155] Here, the first prompt word can provide multiple optional facial optimization models. The first prompt word is concatenated with the requirement description text and then input into the language model. For example, if the user's requirement description text is "to make the skin look smoother without sacrificing realism," the first prompt word could be "Based on the following requirement description text, recommend one or more suitable facial optimization types: [requirement description text]. Optional facial optimization types include, but are not limited to: natural nude makeup, sculpted contours, Korean cool white, etc." The content input to the language model would then be "Based on the following requirement description text, recommend one or more suitable facial optimization types: to make the skin look smoother without sacrificing realism. Optional facial optimization types include, but are not limited to: natural nude makeup, sculpted contours, Korean cool white, etc."

[0156] This application does not limit the model structure of the language model; for example, it can be a pre-trained large language model. The language model first parses the first prompt word, clarifying that the current task is to identify the intent of the requirement description text, and generates at least one face optimization type that matches this intent. The language model analyzes the requirement description text layer by layer to obtain the intent information. The language model can extract keywords from the requirement description text as intent information through an attention mechanism. For example, if the requirement description text is "to make the skin look smoother without losing its realism," the keywords "skin" and "smooth" can be extracted as intent information. The language model performs semantic matching between the intent information and multiple face optimization types pre-stored in the database, or multiple face optimization types set in the first prompt word, to obtain at least one matching face optimization type.

[0157] In step 302, at least one encoding model corresponding to a face optimization type is determined as an encoding model that matches the face optimization requirement information.

[0158] Here, the specific process of determining the encoding model corresponding to at least one face optimization type as the encoding model that matches the face optimization requirement information can be referred to the implementation process of step 204 above, and will not be described again.

[0159] This application's embodiments can accurately extract key intent information from requirement description text, ensuring high relevance of subsequent matching results. Through semantic understanding mechanisms, the language model can handle complex natural language descriptions, avoiding erroneous matching due to ambiguity. The language model can handle various forms of requirement description text (such as simple instructions, complex descriptions, etc.), adapting to the expression habits of different users. By updating the language model's database or first prompt words, new facial optimization types and their corresponding intents can be easily supported.

[0160] In some embodiments, the encoding model includes a first encoder and a second encoder with N levels, where N is a positive integer greater than 1. In step 103, based on the encoding model matching the facial optimization requirement information, the facial optimization requirement information and the first facial image are encoded at multiple levels to obtain encoded feature maps of multiple sizes. This can be achieved as follows: The facial optimization requirement information is encoded based on the first encoder to obtain the first encoding vector; First, the first facial image is encoded based on the first-level second encoder to obtain the first-level second encoding vector, and cross-attention processing is performed on the first encoding vector and the first-level second encoding vector to obtain the first-level encoded feature map; Then, based on the (n+1)-level second encoder, the n-level encoded feature map is encoded according to a preset step size to obtain the (n+1)-level second encoding vector, where n is a positive integer that increases sequentially, 1≤n≤N-1; Finally, cross-attention processing is performed on the first encoding vector and the (n+1)-level second encoding vector to obtain the (n+1)-level encoded feature map, where the size of the (n+1)-level encoded feature map is proportionally reduced compared to the n-level encoded feature map based on the preset step size.

[0161] See here. Figure 6 The encoding model 601 includes a first encoder 605 and N layers of second encoders 608 (only the case of N=2 is shown in the figure). The first encoder is a text encoder, such as a CLIP encoder, a BERT encoder, etc. The second encoders are encoders in the diffusion model U-net network. The first encoder 605 encodes the face optimization type to obtain a first encoding vector, and then inputs the first encoding vector into each of the second encoders 608. The second encoder 608 at each layer encodes the encoded feature map output from the previous layer according to a preset stride to obtain the second encoding vector of the current layer. The encoding process can be a convolution operation with a preset stride greater than 1. For example, when the preset stride is 2, the size of the encoded feature map output from each layer is 1 / 2 of the size of the encoded feature map output from the previous layer. The second encoder 608 at each layer performs cross-attention processing on the first encoding vector and the second encoding vector to obtain the encoded feature map of that layer.

[0162] For example, N=4. The second encoder of the first level encodes the first face image (128×128) with a preset step size of 2 to obtain the second encoding vector of the first level. Cross-attention processing is then performed on the first encoding vector and the second encoding vector of the first level to obtain the encoding feature map of the first level (64×64). The second encoder of the second level encodes the encoding feature map of the first level to obtain the second encoding vector of the second level. Cross-attention processing is then performed on the first encoding vector and the second encoding vector of the second level to obtain the encoding feature map of the second level (32×32). The second encoder of the third level encodes the encoding feature map of the second level to obtain the second encoding vector of the third level. Cross-attention processing is then performed on the first encoding vector and the second encoding vector of the third level to obtain the encoding feature map of the third level (16×16). The second encoder of the fourth level encodes the coding feature map of the third level to obtain the second coding vector of the fourth level, and performs cross-attention processing on the first coding vector and the second coding vector of the fourth level to obtain the coding feature map of the fourth level (8×8).

[0163] In this embodiment, the first encoder transforms facial optimization requirement information into a first encoding vector. At each level, a cross-attention mechanism is used to fuse the first encoding vector with the second encoding vector of the current level, ensuring that the facial optimization requirement information permeates the entire encoding process and guides feature extraction at different scales. The second encoder at each level progressively reduces the spatial resolution of the feature map by performing convolution and downsampling operations on the input data (such as the first facial image or the encoded feature map from the previous level). The resulting encoded feature maps of multiple sizes comprehensively capture the global structure and local details of the facial image.

[0164] In some embodiments, where at least one face optimization type is multiple, see [link to relevant documentation]. Figure 9 , Figure 9 The diagram illustrates that in step 103, a coding model based on matching the facial optimization requirement information is used to encode the facial optimization requirement information and the first facial image at multiple levels, resulting in coded feature maps of multiple sizes. This can be achieved through steps 1031 to 1032:

[0165] In step 1031, for each face optimization type, based on the encoding model corresponding to the face optimization type, the face optimization requirement information and the first face image are encoded at multiple levels to obtain face feature maps of multiple sizes for the face optimization type.

[0166] Here, the specific process of encoding the face optimization requirement information and the first face image at multiple levels based on the encoding model corresponding to the face optimization type to obtain face feature maps of multiple sizes for the face optimization type can be referred to in the above embodiment, which describes the process of encoding the face optimization requirement information and the first face image at multiple levels based on the encoding model matching the face optimization requirement information to obtain encoded feature maps of multiple sizes. This will not be described again here. In this embodiment, the encoding model corresponding to each face optimization type outputs face feature maps of multiple sizes.

[0167] In step 1032, for each size, facial feature maps of the same size corresponding to multiple facial optimization types are superimposed to obtain an encoded feature map that matches multiple facial optimization types.

[0168] Here, the method of superimposing multiple facial feature maps in this embodiment is not limited. For example, the encoded feature map is essentially a multi-dimensional array, where the value at each position is called an element. Multiple facial feature maps corresponding to the same size and facial optimization type can be directly added element-wise to obtain an encoded feature map that matches all facial optimization types. Alternatively, multiple facial feature maps corresponding to the same size and facial optimization type can be concatenated along the channel dimension to obtain an encoded feature map that matches all facial optimization types. Alternatively, multiple facial feature maps corresponding to the same size and facial optimization type can be superimposed using max pooling, that is, taking the maximum value of multiple facial feature maps at the same position, with the maximum value at each position constituting the final encoded feature map that matches all facial optimization types.

[0169] For example, assuming N=4, the selected face optimization types are "natural nude makeup" and "Korean cool white". For the 4th level, obtain the 8×8 face feature map a output by the second encoder of the 4th level in the coding model corresponding to "natural nude makeup", and the 8×8 face feature map b output by the second encoder of the 4th level in the coding model corresponding to "Korean cool white". Superimpose face feature map a and face feature map b to obtain a coding feature map that matches both "natural nude makeup" and "Korean cool white".

[0170] This application embodiment, by superimposing facial feature maps corresponding to multiple facial optimization types, generates an encoded feature map that can simultaneously satisfy multiple optimization requirements, improving the diversity and flexibility of the optimization results. It can balance the influence of different optimization types and avoid one type excessively dominating the optimization results. This application embodiment provides various superposition strategies (such as element-wise addition, channel concatenation, max pooling, etc.), which can be flexibly selected according to actual needs. These operations are all based on the basic units (elements) of the feature maps, making the calculation simple and efficient, thus improving encoding efficiency.

[0171] In some embodiments, for each size, facial feature maps of the same size corresponding to multiple face optimization types are superimposed to obtain an encoded feature map that matches all face optimization types. This can be achieved as follows: First, for each face optimization type, attention encoding is performed on the facial feature maps of the same size corresponding to the face optimization type to obtain the attention weights of the same size corresponding to the face optimization type; then, based on the attention weights of the sizes corresponding to the multiple face optimization types, the facial feature maps of the same size corresponding to the multiple face optimization types are superimposed to obtain an encoded feature map that matches all face optimization types.

[0172] Here, for each face optimization type, multiple sizes of facial feature maps output by the corresponding encoding model are obtained. For any given size, attention encoding is performed on the facial feature map of that size to obtain the attention weight for that size corresponding to the face optimization type. The attention weight is a numerical value that measures the importance of each position or channel in the facial feature map, and the value range of the attention weight is [0, 1]. Specifically, the first encoding vector can be used as the query vector Q, and the facial feature map of that size can be used as the key vector K and value vector V. Cross-attention calculation is performed on the query vector and value vector to obtain the attention score for each position. The attention score is then normalized to obtain the attention weight for that size of the face optimization type. Based on the attention weight of each face optimization type at that size, the facial feature maps of multiple face optimization types corresponding to that size are weighted and summed to obtain the encoded feature map of that size that matches multiple face optimization types.

[0173] It should be noted that the above-described scheme of the encoding model in the embodiments of this application can also be applied to the decoding model, and will not be described further.

[0174] In this embodiment, the attention weights are dynamically calculated based on the first encoding vector (optimization requirement information), which can accurately measure the importance of each position or channel in the current optimization objective. Multiple facial feature maps of different optimization types are fused together using a weighted summation method to ensure that the contribution ratio of each optimization type matches its importance. Through the attention mechanism, the most relevant regions (such as eyes and nose) to the optimization objective can be automatically identified and strengthened, while suppressing the influence of irrelevant regions. When the objectives of different optimization types are contradictory (such as slimming the face and enlarging the eyes), the attention mechanism can balance these objectives through weight allocation, preventing a single optimization type from dominating the result.

[0175] In step 104, based on the decoding model that matches the facial optimization requirements, the encoded feature maps of multiple sizes and the identity features of multiple sizes are decoded at multiple levels to obtain a noise map.

[0176] For each level, the input to the decoding is the output of the previous level, identity features of the same size as the decoded feature map of the level output, and the encoded feature map.

[0177] Here, the specific process of determining the decoding model that matches the facial optimization requirement information is similar to the specific process of determining the encoding model that matches the facial optimization requirement information in the above embodiments, and will not be described again. See Figure 6 The denoising module 603 includes a decoding model 602, which in turn includes multiple levels of decoders 607. After the N levels of second encoders 608 output encoded feature maps of multiple sizes, each second encoder 608 can skip connections with the decoders 607 at the same level. That is, each level of decoder 607 acquires an encoded feature map of the same size as the decoded feature map output by the current level, decodes the identity features and encoded feature maps of the same size, and predicts the noise map.

[0178] In some embodiments, the decoding model includes M levels of decoders, where M is a positive integer greater than 1. Step 104, based on the decoding model matching the facial optimization requirements, decodes multiple sizes of encoded feature maps and multiple sizes of identity features at multiple levels to obtain a noise map. This can be achieved as follows: First, based on the first-level decoder, the identity features and encoded feature maps of the corresponding sizes at the first level, as well as the smallest encoded feature map, are fused to obtain the first-level identity enhancement image; then, based on the first-level decoder, the first-level identity enhancement image is upsampled to obtain the first-level decoded feature map; then, based on the (m+1)th level decoder... The device fuses the identity feature and encoded feature map of the (m+1)th level with the decoded feature map of the mth level to obtain the identity enhancement image of the (m+1)th level, where 1≤m≤M-1. Finally, based on the decoder of the (m+1)th level, the identity enhancement image of the (m+1)th level is upsampled according to a preset step size to obtain the decoded feature map of the (m+1)th level, where the size of the decoded feature map of the (m+1)th level increases proportionally to that of the decoded feature map of the mth level based on the preset step size. When the value of m iterates to M-1, the decoded feature map of the Mth level is used as a noise map.

[0179] See here. Figure 6The decoding model 602 includes M levels of decoders 607 (only the case where M=2 is shown in the figure). The values ​​of M and N can be the same. Decoder 607 is the decoder in the diffusion model U-net network. Each level of decoder 607 acquires the encoded feature map output by the encoder 608 of the same level, as well as the identity feature of the same level. It then performs a weighted sum of the decoded feature map output by the previous level decoder 607, the encoded feature map, and the identity feature to obtain the identity enhancement image of the current level. The current level decoder 607 upsamples the identity enhancement image according to a preset stride (e.g., convolution operation) to obtain the decoded feature map of the current level. For example, when the preset stride is 2, the size of the decoded feature map output by each level is twice that of the encoded feature map output by the previous level. The preset stride used in the encoding and decoding models is the same.

[0180] It should be noted that the first-level decoder cannot obtain the decoded feature map output by the previous level decoder as input. Instead, it can directly use the encoded feature map output by the Nth-level encoder as input, which is the encoded feature map with the smallest size. Alternatively, the diffusion model can include an intermediate module between multiple encoders and decoders to perform convolution processing on the encoded feature map output by the Nth-level encoder without changing its size. The resulting encoded feature map is then used as the input to the first-level decoder.

[0181] For example, M=4. The coded feature map (8×8) output by the second encoder at level 4 is skipped to the decoder at level 1. The decoder at level 1 performs a weighted summation of the 8×8 identity features, the 8×8 coded feature map output by the second encoder, and the 8×8 coded feature map output by the intermediate module to obtain the identity enhancement image at level 1. Then, the decoder at level 1 upsamples the identity enhancement image at level 1 by a preset step size of 2 to obtain the decoded feature map at level 1. The coded feature map (16×16) output by the second encoder at level 3 is skipped to the decoder at level 2. The decoder at level 2 performs a weighted summation of the 16×16 identity features, the 16×16 coded feature map output by the second encoder, and the decoded feature map at level 1 to obtain the identity enhancement image at level 2. The decoder at level 2 upsamples the identity enhancement image at level 2 by a preset step size of 2 to obtain the decoded feature map at level 3. The coded feature map (32×32) output by the second encoder of level 2 is skipped to the decoder of level 3. The coded feature map (64×64) output by the second encoder of level 1 is skipped to the decoder of level 4.

[0182] In this embodiment, at each level, an identity-enhanced image is generated by fusing identity features of the corresponding size, encoded feature maps, and decoded feature maps from the previous level. This design ensures that the decoding process always focuses on the identity features of the target object, avoiding distortion caused by over-optimization. The process gradually recovers from the smallest encoded feature map to the largest decoded feature map, fusing identity features and encoded feature maps of the current size at each level. This achieves effective multi-scale feature fusion, preserving the realism of the overall structure and local details of the target object while meeting optimization requirements.

[0183] Figure 10 This is a schematic diagram of the diffusion model and identity feature extraction model provided in the embodiments of this application.

[0184] See Figure 10 N and M are both equal to 4. The encoding model includes second encoders A, B, C, and D. The identity feature extraction model sequentially replicates the second encoders A, B, C, and D, forming four layers of the identity feature extraction network: encoders A, B, C, and D. The encoders in the identity feature extraction model have the same model parameters as the encoders at the same level in the encoding model. Since the actual diffusion model processes images through multiple time steps T, each iteration requires inputting the current time step t into the encoder at that time, and then inputting the resulting encoded vector along with the first encoded vector output by the first encoder into the second encoder.

[0185] In step 105, the first face image is denoised based on the noise map to obtain the second face image.

[0186] Here, the first face image can be obtained by element-wise subtraction of the noise image. Alternatively, the first face image and the noise image can be weighted and fused to obtain the second face image. If the time step T of the diffusion model is set to 1, the second face image is directly used as the optimized result of the first face image. If the time step T of the diffusion model is set to greater than 1, it is necessary to determine whether the current number of iterations for encoding and decoding has reached the set value, which is the preset time step T of the diffusion model, and is the same value as the time step T of the noise addition process in the diffusion model.

[0187] In some embodiments, encoding and decoding are performed iteratively. After denoising the first face image based on the noise map to obtain the second face image, the method further includes: if the number of iterations for encoding and decoding has not reached a set value, then using the second face image as a new first face image, returning an encoding model based on matching the face optimization requirement information, and encoding the face optimization requirement information and the first face image at multiple levels to obtain encoded feature maps of multiple sizes. If the number of iterations for encoding and decoding has reached a set value, then using the second face image obtained in the last iteration as the optimization result of the first face image.

[0188] See here. Figure 6 After obtaining the second face image, if the number of iterations for encoding and decoding has not reached the set value T, the second face image continues to be denoised based on the denoising module 603 until the set value T has been reached at this iteration point. The second face image of the Tth iteration is then used as the optimized result of the first face image. That is, when the number of iterations for encoding and decoding has not reached the set value T, the second face image is used as the new first face image, and steps 102 to 105 are repeated.

[0189] In each iteration of this application, a noise map is regenerated based on the current second face image, and denoising processing is performed. This stepwise optimization method can gradually reduce imperfections or unnecessary information in the image while preserving the identity features of the target object. As the number of iterations increases, the optimization process can more finely adjust the image content, ensuring that the final result meets the optimization requirements while maintaining a natural realism. By limiting the number of iterations to a set value T (e.g., 10), over-optimization (such as loss of identity features, image distortion, etc.) caused by too many iterations can be effectively avoided.

[0190] In some embodiments, see Figure 11 In the facial image processing method provided in this application embodiment, the encoding model and decoding model are trained through the following steps 401 to 405:

[0191] In step 401, face image samples with optimized facial features are selected from the dataset.

[0192] Among them, the facial image samples are optimized based on the facial optimization type matched with facial optimization requirement information.

[0193] Here, after determining a certain face optimization type, a large number of face image samples that conform to that face optimization type can be selected from the dataset.

[0194] For example, the face optimization type is "natural nude makeup". Multiple face images after beautification with the "natural nude makeup" style are selected from the dataset as face image samples.

[0195] In step 402, any time step is selected as the current time step, and Gaussian noise corresponding to the current time step is generated.

[0196] Here, the time step t in the diffusion model represents the degree of noise addition; the smaller the time step, the lower the noise intensity; the larger the time step, the higher the noise intensity. Assuming the total time step is T, then t can be any integer between 1 and T. Gaussian noise refers to noise whose probability density function follows a Gaussian distribution (i.e., a normal distribution). It is a common type of noise in digital images, with noise appearing at every point and at random noise depths.

[0197] For example, if T = 10, time step 3 can be selected as the current time step, and Gaussian noise corresponding to the current time step 3 can be randomly generated.

[0198] In step 403, Gaussian noise corresponding to the current time step is added to the face image sample to obtain the noisy image sample of the current time step.

[0199] Here, Gaussian noise can be directly added to the face image sample to obtain a noisy image sample at the current time step. Alternatively, a noisy image sample at the current time step can be generated based on the current time step t, Gaussian noise, and the face image sample. (Noisy image sample at the current time step) in, It is Gaussian noise. For the difference time table (cumulative forward diffusion coefficient, used to represent the cumulative noise level at each time step during the diffusion process), α t =1-β t , β is an arithmetic sequence containing T values, such as β1 = 1. e-4 ,β T =1 e-2 .

[0200] In step 404, the prediction noise for the current time step is determined based on the face optimization type and the noisy image samples.

[0201] Here, the face optimization type and noisy image samples can be passed as input to the encoding and decoding models so that the decoding model can ultimately predict the prediction noise at the current time step. Determining the prediction noise at the current time step is similar to determining the noise map in step 104, but does not include the processing of identity features, and will not be described further here.

[0202] In step 405, the noise error between the Gaussian noise and the predicted noise is determined, and the parameters of the encoding model and the decoding model are updated based on the noise error.

[0203] Here, the noise error between Gaussian noise and prediction noise can be calculated using a loss function (e.g., mean squared error loss function). The parameters of the encoding and decoding models are updated based on the noise error using a backpropagation algorithm (e.g., gradient descent algorithm) until the total noise error at all time steps reaches the minimum value, thus obtaining the pre-associated encoding and decoding models of the successfully trained face optimization model.

[0204] This application selects facial image samples optimized based on facial optimization requirement information from the dataset, ensuring that the training data is highly relevant to the actual application scenario. Based on the noisy image samples and facial optimization type at the current time step, the predicted noise at the current time step is determined, enabling the model to learn how to progressively remove noise from the image. Through backpropagation, the parameters of the encoding and decoding models are updated according to the noise error, gradually improving the model's denoising ability and optimization effect.

[0205] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0206] Current beautification technologies suffer from the following significant problems in practical applications: First, parameter adjustment heavily relies on manual intervention. Users need to adjust dozens of beautification parameters, such as skin smoothing and facial feature distortion, one by one, resulting in a high operational threshold and difficulty in achieving "one-click" intelligent optimization. Second, modular processing leads to algorithmic fragmentation. The face is broken down into independent functional modules such as skin, eyes, and lips, and a divide-and-conquer strategy is used for local beautification (such as designing separate algorithms for whitening, freckle removal, and eye enlargement), resulting in a lack of cross-regional feature synergy between modules. This fragmented processing leads to three core problems: 1. Lack of global consistency: Local optimization struggles to maintain facial lighting transitions, texture continuity, and facial feature proportions, easily producing a "collage-like" appearance or biometric distortion; 2. Degradation of identity (ID): Isolated adjustments may damage key identity features such as texture and facial topology, leading to facial recognition feature shifts; 3. Insufficient scene adaptation: Traditional technologies have poor robustness to complex lighting and occlusion scenarios, often requiring secondary manual correction, severely restricting user experience and commercial viability.

[0207] Based on this, this application provides a facial image processing method. In beautification scenarios, this method is a diffusion model-based beautification processing method. It achieves end-to-end integration through a phased diffusion framework. By constructing a three-level diffusion generation architecture of "facial feature analysis - identity consistency injection - overall beautification generation," it integrates traditional modular processing solutions (such as independent optimization of skin and facial features) into a unified process, eliminating the fragmentation caused by multi-algorithm stitching. During the reverse denoising process, the diffusion model uses the original facial image as initial input, combining facial contour detection and depth map analysis to achieve dynamic decoupling of beautification attributes and identity features, avoiding the global imbalance caused by independent local optimization in traditional methods.

[0208] The facial image processing method provided in this application can be applied to any program on a terminal that has facial image processing capabilities, such as the beauty function module in a communication app. Users can upload photos on the interface of the beauty function module. After uploading the first facial image, they can click the "One-Click Beauty" function. A list below the function displays preset styles of different beauty effects (corresponding to the facial optimization types in the above embodiments), such as "Natural Nude Makeup," "3D Contours," and "Korean Cool White Skin." Users can also input text commands (corresponding to facial optimization requirements in the above embodiments), such as "Add soft light effect, retain realistic skin texture," which the beauty function module will match to obtain the desired beauty style. After selecting a style, the client interface enters the one-click beauty waiting area. After the waiting time is complete, the beauty function module automatically generates a globally optimized beauty result without requiring manual adjustment of local parameters. The user obtains the photo result after one-click beauty (corresponding to the optimization result of the first facial image in the above embodiments).

[0209] Figure 12 This is a flowchart illustrating the beautification process based on a diffusion model, as provided in this application embodiment. In actual use, users can select a preset beautification style and beautification model, in conjunction with... Figure 12 The diffusion generation framework designed in the middle enables one-click beautification.

[0210] See Figure 12 Step 1201, Diffusion model training.

[0211] This process involves collecting corresponding beauty style data (corresponding to the optimized facial image samples in the above embodiments) based on pre-selected beauty styles. Different beauty styles can be categorized to obtain beauty style data with a common style effect. For example, multiple beautified facial images with different beauty styles can be prepared directly as beauty style data. Alternatively, multiple unbeautified facial images can be prepared and processed using other beauty tools to obtain multiple beautified facial images with different beauty styles as beauty style data. The diffusion model is trained based on the beauty style data to obtain a beauty model capable of generating facial images of that beauty style (corresponding to the facial image processing model in the above embodiments). For example, a "natural nude makeup" diffusion model can be trained using beauty style data of a natural nude makeup style, a "3D contour" diffusion model can be trained using beauty style data of a 3D contour style, and a "Korean cool white" diffusion model can be trained using beauty style data of a Korean cool white style.

[0212] Figure 13 This is a schematic diagram illustrating the diffusion model principle provided in an embodiment of this application. See also... Figure 13The image X0 is input into the diffusion model. The forward process of the diffusion model involves adding Gaussian noise to the image X0 through T iterations, successively obtaining images X1, X2, ..., X... t-1 X t ... X T The larger t is, the closer the resulting image is to pure noise; the forward process is... Figure 13 q(X) t |X t-1 The process. The inverse process of the diffusion model is the denoising inference process, that is, the denoising inference process for the noisy image X. T The process of obtaining image X0 after T iterations of denoising (p(X) in the figure) t-1 |X t )process).

[0213] The training process of the diffusion model is illustrated below using the "natural makeup" style as an example. In each training round, a face image in the "natural makeup" style is selected (corresponding to the face image sample in the above embodiment), a random time step t is selected (corresponding to the current time step in the above embodiment), and one-dimensional Gaussian noise is randomly sampled and added to the face image to obtain a noisy face image (corresponding to the noisy image sample in the above embodiment). in, It is a standard one-dimensional Gaussian distributed noise, α t =1-β t , β is an arithmetic sequence containing T values, such as β1 = 1. e-4 ,β T =1 e-2 The time step t is encoded as a time vector, and "natural makeup" (corresponding to the face optimization type in the above embodiment) is encoded as a text vector. The noisy face image, time vector, and text vector are input into the diffusion model to obtain the predicted noise. The loss function (e.g., mean-square error (MSE)) between the actual added Gaussian noise and the predicted noise is calculated. The model parameters are updated using gradient descent or other optimization algorithms until the loss function converges or the preset training rounds are reached.

[0214] Step 1202, Input text conditions.

[0215] This involves preparing textual conditional inputs based on a certain beautification style. These textual conditions are used to set the beautification style of the beautification model and should be consistent with the textual conditions used during the training of the diffusion model.

[0216] Step 1203: User inputs image.

[0217] This involves obtaining the image to be processed input by the user (corresponding to the first face image in the above embodiment).

[0218] Step 1204, facial feature analysis.

[0219] In this embodiment, facial feature analysis is performed on the user-input image to obtain a facial contour map and a facial depth map as dual facial feature conditions. Specifically, facial contour detection can be performed on the image to obtain a facial contour map, and facial depth detection can be performed on the image to obtain a facial depth map. Figure 14 This is a schematic diagram of facial feature analysis and processing provided in an embodiment of this application. See also... Figure 14 , Figure 14 In the image, (a) is the image to be processed, (b) is the facial contour map, and (c) is the facial depth map.

[0220] For example, edge detection algorithms (such as Canny, HED, or MLSD) can be used to detect facial contours in the image to be processed, resulting in a facial contour map I. edge This process captures structural information of the image (such as face boundaries and facial contours). A depth map detection algorithm is used to perform face depth detection on the image to be processed, resulting in a face depth map I. depth It captures the depth information of the image (such as the three-dimensionality of the face and the distribution of facial features).

[0221] Step 1205, Identity Consistency Injection Model.

[0222] Specifically, the facial feature conditions mentioned above are input into the identity consistency injection model (i.e., Controlnet, which requires no training and corresponds to the identity feature extraction model in the above embodiments) to extract control feature conditions (corresponding to the identity features in the above embodiments), which are used to accurately guide the subsequent image generation process. By injecting the control feature conditions into the subsequent beautification model, the resulting image generated by the beautification model retains the high-quality characteristics of the diffusion model while strictly aligning with the geometric, spatial distribution, and semantic constraints of the identity consistency conditions.

[0223] This application embodiment has two independent identity consistency injection models. The identity consistency injection model replicates the encoder and intermediate layers of the U-net structure in the diffusion model, but the parameters are independent and trainable. The facial contour map I... edge and facial depth map I depth Input the encoders of two identity consistency injection models respectively, and extract multi-scale facial edge features F. edge-cond and multi-scale facial depth features F depth-cond Extracting multi-scale facial edge features F edge-condThis refers to an identity consistency injection model that has multiple identity feature extraction networks (blocks). Each block's structure can include a trainable encoder layer (the encoder in the copy diffusion model U-net network) and a zero convolutional layer. Each block outputs facial edge features at one scale.

[0224] Step 1206, beautification model processing.

[0225] In this process, the image to be processed, control feature conditions, and text conditions are input into the beautification model to obtain a result image of one-click beautification of the image to be processed (corresponding to the optimization result of the first face image in the above embodiment).

[0226] First, noise injection is performed. The beautification model adds Gaussian noise of a preset number of steps (corresponding to the number of rounds T in the above embodiment, for example, set to 10) to the image to be processed, generating a noisy face image X. T (T is the time step); then, conditional binding is performed, and the noisy face image X is... t Together with the text conditions, they serve as the initial input to the U-net network in the beautification model, initiating the graph-to-graph task.

[0227] By injecting identity consistency into the model with an initial zero-weight convolutional layer, the extracted facial edge features F are... edge-cond and facial depth features F depth-cond The original features F are progressively injected into each layer of the decoder of the beautification model; in each residual block of the U-Net network, the original features F are... unet (The features output by the encoder in the beautification model, corresponding to the encoded feature map in the above embodiment) are added to the control feature conditions output by the identity consistency injection model to form the conditional enhancement feature (corresponding to the identity enhancement image in the above embodiment) F. fused =F unet +α×F edge-cond +β×F depth-cond , where α and β are control intensity coefficients, for example, α and β are 0.45.

[0228] Inverse Denoising Generation: Iterative Denoising: From Noise X T Start (T is the total number of steps), and perform the following operations step by step until t=0: U-Net network predicts noise: Input the current noisy image X T(Corresponding to the first face image of the input encoding model in the above embodiment), time step t, text features and control feature conditions, and predicted noise (corresponding to the noise map in the above embodiment). Condition-guided correction: The control feature conditions adjust the noise prediction direction through feature fusion, forcing the structure of the generated image to align with the facial edge features and facial depth features, maintaining the original facial contour and the original user's facial feature distribution; Denoising update: The image is updated according to the predicted noise and the diffusion sampling rule. Figure 15 This is a schematic diagram of the beautification result provided in an embodiment of this application. See also... Figure 15 Image (a) is the image to be processed, image (b) is the result of the beautification process, and image (c) is a comparison of details before and after beautification.

[0229] This application's embodiments construct a phased diffusion generation framework, using beautification styles to replace the attribute settings for each part of the face region. This integrates the traditional beautification process to achieve one-click beautification generation that takes into account all parts of the face, reducing the inconvenience of manually setting parameters item by item. The one-click beautification processing method is internally decomposed into a multi-level processing of "facial feature analysis - identity consistency injection - overall beautification generation". In the reverse denoising process of the diffusion model, facial contour detection and facial depth map are introduced as dual facial feature analysis input conditions. In the denoising process, the original face image is used as the initial input image to import explicit identity consistency information through the image-to-image task, dynamically separating beautification attributes (such as skin smoothing, face slimming, and makeup style) from identity features (such as facial structure and biometric features), ensuring that identity information is not lost when the beautification operation is applied to the target area.

[0230] The following description continues to illustrate the exemplary structure of the facial image processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the face image processing device 455 in the memory 450 may include:

[0231] The feature extraction module 4551 is used to extract identity features at multiple levels from the first face image of the target object, and obtain identity features of the target object at multiple sizes, wherein one level corresponds to one size;

[0232] The acquisition module 4552 is used to acquire facial optimization requirement information set for the first facial image;

[0233] The encoding module 4553 is used to encode the face optimization requirement information and the first face image at multiple levels based on an encoding model that matches the face optimization requirement information, so as to obtain encoded feature maps of multiple sizes.

[0234] The decoding module 4554 is used to decode multiple encoded feature maps and multiple identity features of multiple sizes at multiple levels based on a decoding model that matches the facial optimization requirements information, to obtain a noise map. For each level, the input of the decoding is the output of the previous level, the identity features of the same size as the decoded feature map of the level output, and the encoded feature map.

[0235] The denoising module 4555 is used to denoise the first face image based on the noise map to obtain the second face image.

[0236] In some embodiments, the identity features of the target object include facial edge features and facial depth features of the target object. The feature extraction module 4551 is further configured to perform contour detection on the first face image to obtain a facial contour map of the target object; perform edge feature extraction at multiple levels on the facial contour map to obtain facial edge features of multiple sizes of the target object; perform depth detection on the first face image to obtain a facial depth map of the target object; and perform depth feature extraction at multiple levels on the facial depth map to obtain facial depth features of multiple sizes of the target object.

[0237] In some embodiments, edge feature extraction at multiple levels is achieved through a first identity feature extraction model, which includes a zero convolutional layer and a first identity feature extraction network of N levels, where N is a positive integer greater than 1. The feature extraction module 4551 is further configured to convolve the facial contour map based on the zero convolutional layer in the first identity feature extraction model to obtain intermediate edge features; downsample the intermediate edge features based on the first-level first identity feature extraction network to obtain the first-level facial edge features; and downsample the n-level facial edge features based on the (n+1)-level first identity feature extraction network according to a preset step size to obtain the (n+1)-level facial edge features. The size of the (n+1)-level facial edge features is proportionally reduced to that of the n-level facial edge features based on the preset step size, where n is a positive integer that increases sequentially, and 1 ≤ n ≤ N-1.

[0238] In some embodiments, multi-level deep feature extraction is achieved through a second identity feature extraction model, which includes a zero convolutional layer and an N-level second identity feature extraction network, where N is a positive integer greater than 1. The feature extraction module 4551 is further configured to convolve the face depth map based on the zero convolutional layer in the second identity feature extraction model to obtain intermediate depth features; downsample the intermediate depth features based on the first-level second identity feature extraction network to obtain the first-level face depth features; and downsample the n-level face depth features based on the (n+1)-level second identity feature extraction network according to a preset step size to obtain the (n+1)-level face depth features. The size of the (n+1)-level face depth features is proportionally reduced to that of the n-level face depth features based on the preset step size, where n is a positive integer that increases sequentially, and 1 ≤ n ≤ N-1.

[0239] In some embodiments, when the face optimization requirement information is a face optimization requirement description text, the face image processing device 455 further includes a model selection module for extracting text features of the requirement description text and type features of multiple face optimization types; performing probability prediction based on text features and type features to obtain the matching probability of each face optimization type corresponding to the face optimization requirement information; selecting at least one face optimization type from multiple face optimization types in descending order of matching probability; and determining the encoding model corresponding to at least one face optimization type as the encoding model that matches the face optimization requirement information.

[0240] In some embodiments, when the face optimization requirement information is a face optimization requirement description text, the model selection module is further configured to call a language model based on a first prompt word to trigger the language model to perform intent recognition on the requirement description text and obtain at least one face optimization type that matches the intent of the requirement description text, wherein the first prompt word is used to instruct the language model to identify at least one face optimization type that matches the intent of the requirement description text; and to determine the encoding model corresponding to at least one face optimization type as the encoding model that matches the face optimization requirement information.

[0241] In some embodiments, when there are multiple face optimization types, the encoding module 4553 is further configured to encode the face optimization requirement information and the first face image at multiple levels based on the encoding model corresponding to the face optimization type for each face optimization type, thereby obtaining face feature maps of multiple sizes for the face optimization type; and for each size, superimposing the face feature maps of the same size corresponding to multiple face optimization types to obtain an encoded feature map that matches all multiple face optimization types.

[0242] In some embodiments, the encoding module 4553 is further configured to perform attention encoding on the facial feature map of the same size corresponding to each facial optimization type for each facial optimization type, to obtain the attention weight of the same size corresponding to the facial optimization type; and to superimpose the facial feature maps of the same size corresponding to multiple facial optimization types based on the attention weights of the sizes corresponding to multiple facial optimization types, to obtain an encoded feature map that matches multiple facial optimization types.

[0243] In some embodiments, encoding and decoding are performed iteratively multiple times. The face image processing device 455 further includes an iteration module, which is used to, if the number of iterations of encoding and decoding has not reached a set value, take the second face image as a new first face image, return an encoding model based on matching the face optimization requirement information, and encode the face optimization requirement information and the first face image at multiple levels to obtain an encoding feature map of multiple sizes; if the number of iterations of encoding and decoding has reached a set value, take the second face image obtained in the last iteration as the optimization result of the first face image.

[0244] In some embodiments, the face image processing device 455 further includes a noise-adding module for adding noise to the noisy image in the t-th round to obtain a noisy face image in the t-th round, and determining the noisy face image in the t-th round as the noisy image in the (t+1)-th round, wherein the noisy image in the 1st round is the first face image, t is a positive integer that increases sequentially, 1≤t≤T, and T is a positive integer; and the noisy face image in the T-th round is used as the first face image input to the encoding model.

[0245] In some embodiments, the encoding model includes a first encoder and a second encoder with N levels, where N is a positive integer greater than 1. The encoding module 4553 is further configured to encode facial optimization requirement information based on the first encoder to obtain a first encoding vector; encode the first facial image based on the second encoder at the first level to obtain a second encoding vector at the first level, and perform cross-attention processing on the first encoding vector and the second encoding vector at the first level to obtain a first-level encoding feature map; encode the encoding feature map at the nth level based on the second encoder at the (n+1)th level according to a preset step size to obtain a second encoding vector at the (n+1)th level, where n is a positive integer that increases sequentially, 1≤n≤N-1; and perform cross-attention processing on the first encoding vector and the second encoding vector at the (n+1)th level to obtain an encoding feature map at the (n+1)th level, wherein the size of the encoding feature map at the (n+1)th level is proportionally reduced compared to the encoding feature map at the nth level based on the preset step size.

[0246] In some embodiments, the decoding model includes M levels of decoders, where M is a positive integer greater than 1. The decoding module 4554 is further configured to, based on the first-level decoder, fuse the identity features and encoded feature maps of the corresponding size at the first level, as well as the encoded feature map of the smallest size, to obtain the identity enhancement image at the first level; based on the first-level decoder, upsample the identity enhancement image at the first level to obtain the decoded feature map at the first level; and based on the (m+1)th level decoder, fuse the identity features of the corresponding size at the (m+1)th level... The identity enhancement image at level m is obtained by fusing the coded feature map and the decoded feature map at level m+1, where 1≤m≤M-1. Based on the decoder at level m+1, the identity enhancement image at level m+1 is upsampled by a preset step size to obtain the decoded feature map at level m+1, where the size of the decoded feature map at level m+1 increases proportionally to that at level m based on the preset step size. When the value of m iterates to M-1, the decoded feature map at level M is used as a noise map.

[0247] In some embodiments, the face image processing apparatus 455 further includes a training module for training an encoding model and a decoding model by: selecting face-optimized face image samples from a dataset, wherein the face image samples are optimized based on the face optimization type matched with face optimization requirement information; selecting any time step as the current time step and generating Gaussian noise corresponding to the current time step; adding the Gaussian noise corresponding to the current time step to the face image samples to obtain noisy image samples at the current time step; determining the prediction noise at the current time step based on the face optimization type and the noisy image samples; determining the noise error between the Gaussian noise and the prediction noise, and updating the parameters of the encoding model and the decoding model based on the noise error.

[0248] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the facial image processing method described in this application.

[0249] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the facial image processing method provided in this application. For example, ... Figure 3 The method for processing facial images is shown.

[0250] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0251] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0252] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0253] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0254] In summary, this application's embodiments achieve dynamic decoupling of beautification attributes and identity features through a phased diffusion framework and dual constraints of facial depth maps and contour detection. While preserving facial structure and biometric features, it supports one-click global optimization. The latent space of the diffusion model collaboratively controls skin color, face shape, and lighting, eliminating the patchwork effect of traditional modular processing and ensuring compatibility with complex scenes such as low light and multiple poses. In terms of application value, users can obtain natural beautification results without individually adjusting parameters, improving operational efficiency. This application's embodiments can be widely applied in social media, e-commerce, and other fields, solving the dual problems of distorted effects and low efficiency in traditional methods.

[0255] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A facial image processing method, characterized in that, The method includes: Multiple levels of identity feature extraction are performed on the first facial image of the target object to obtain identity features of the target object in multiple dimensions, wherein one level corresponds to one dimension; Obtain facial optimization requirements information for the first facial image; Based on the encoding model that matches the facial optimization requirement information, the facial optimization requirement information and the first facial image are encoded at multiple levels to obtain encoded feature maps of multiple sizes. Based on a decoding model that matches the facial optimization requirements, the encoded feature maps of multiple sizes and the identity features of multiple sizes are decoded at multiple levels to obtain a noise map. For each level, the input to the decoding of the level is the output of the decoding of the previous level, the identity features with the same size as the decoded feature map of the level output, and the encoded feature map. The first face image is denoised based on the noise map to obtain the second face image.

2. The method according to claim 1, characterized in that, The identity features of the target object include the facial edge features and facial depth features of the target object. The process involves extracting identity features at multiple levels from the first facial image of the target object, resulting in identity features of the target object at multiple dimensions, including: Contour detection is performed on the first face image to obtain the facial contour map of the target object; Multiple levels of edge feature extraction are performed on the facial contour map to obtain facial edge features of the target object at multiple dimensions; Depth detection is performed on the first face image to obtain the face depth map of the target object; Multiple levels of depth feature extraction are performed on the facial depth map to obtain facial depth features of the target object at multiple dimensions.

3. The method according to claim 2, characterized in that, The multiple levels of edge feature extraction are performed sequentially, and the output of the edge feature extraction of the previous level is the input of the edge feature extraction of the next level. The size of the facial edge feature output by the next level is smaller than the size of the facial edge feature output by the previous level.

4. The method according to claim 3, characterized in that, The multi-level edge feature extraction is achieved through a first identity feature extraction model, which includes a zero convolutional layer and an N-level first identity feature extraction network, where N is a positive integer greater than 1. The step of extracting edge features at multiple levels from the facial contour map to obtain facial edge features of the target object at multiple dimensions includes: Based on the zero convolutional layer in the first identity feature extraction model, the facial contour map is convolved to obtain the middle edge features; Based on the first identity feature extraction network of the first level, the intermediate edge features are downsampled to obtain the face edge features of the first level; Based on the first identity feature extraction network of the (n+1)th level, the facial edge features of the nth level are downsampled according to a preset step size to obtain the facial edge features of the (n+1)th level. The size of the facial edge features of the (n+1)th level is proportionally reduced to that of the facial edge features of the nth level based on the preset step size, where n is a positive integer that increases sequentially, 1≤n≤N-1.

5. The method according to claim 2, characterized in that, The multiple levels of depth feature extraction are performed sequentially, and the output of the previous level of depth feature extraction is the input of the next level of depth feature extraction. The size of the facial depth feature output by the next level is smaller than the size of the facial depth feature output by the previous level.

6. The method according to claim 5, characterized in that, The multi-level deep feature extraction is achieved through a second identity feature extraction model, which includes zero convolutional layers and an N-level second identity feature extraction network, where N is a positive integer greater than 1. The process of extracting depth features at multiple levels from the facial depth map to obtain facial depth features of the target object at multiple dimensions includes: Based on the zero convolutional layer in the second identity feature extraction model, the face depth map is convolved to obtain intermediate depth features; Based on the second identity feature extraction network of the first level, the intermediate depth features are downsampled to obtain the face depth features of the first level; Based on the second identity feature extraction network of the (n+1)th level, the facial depth features of the nth level are downsampled according to a preset step size to obtain the facial depth features of the (n+1)th level. The size of the facial depth features of the (n+1)th level is proportionally reduced to that of the facial depth features of the nth level based on the preset step size, where n is a positive integer that increases sequentially, 1≤n≤N-1.

7. The method according to claim 1, characterized in that, When the facial optimization requirement information is a facial optimization type, the encoding model matching the facial optimization requirement information is the encoding model pre-associated with the facial optimization type, and the decoding model matching the facial optimization requirement information is the decoding model pre-associated with the facial optimization type.

8. The method according to claim 1, characterized in that, When the facial optimization requirement information is a description of facial optimization requirements, the method further includes the following steps before encoding the facial optimization requirement information and the first facial image at the multiple levels: Extract the text features of the requirement description text, as well as the type features of multiple face optimization types; Based on the text features and the type features, probability prediction is performed to obtain the matching probability of each face optimization type corresponding to the face optimization requirement information; At least one face optimization type is selected from the plurality of face optimization types in descending order of matching probability; The encoding model corresponding to the at least one facial optimization type is determined as the encoding model that matches the facial optimization requirement information.

9. The method according to claim 1, characterized in that, When the facial optimization requirement information is a description of facial optimization requirements, the method further includes the following steps before encoding the facial optimization requirement information and the first facial image at the multiple levels: The language model is invoked based on the first prompt word to trigger the language model to perform intent recognition on the requirement description text and obtain at least one face optimization type that matches the intent of the requirement description text. The first prompt word is used to instruct the language model to identify at least one face optimization type that matches the intent of the requirement description text. The encoding model corresponding to the at least one facial optimization type is determined as the encoding model that matches the facial optimization requirement information.

10. The method according to claim 8 or 9, characterized in that, When there are multiple face optimization types, the encoding model based on matching the face optimization requirement information encodes the face optimization requirement information and the first face image at multiple levels to obtain encoded feature maps of multiple sizes, including: For each of the aforementioned face optimization types, based on the encoding model corresponding to the face optimization type, the face optimization requirement information and the first face image are encoded at multiple levels to obtain face feature maps of multiple sizes for the face optimization type. For each of the dimensions, facial feature maps corresponding to the same dimension for multiple facial optimization types are superimposed to obtain an encoded feature map that matches all of the multiple facial optimization types.

11. The method according to claim 10, characterized in that, For each of the dimensions, the facial feature maps corresponding to the same size from multiple facial optimization types are superimposed to obtain an encoded feature map that matches all of the multiple facial optimization types, including: For each face optimization type, attention encoding is performed on the face feature map of the same size corresponding to the face optimization type to obtain the attention weights of the same size corresponding to the face optimization type; Based on the attention weights corresponding to the dimensions of the multiple face optimization types, the face feature maps of the multiple face optimization types corresponding to the same dimension are superimposed to obtain an encoded feature map that matches all of the multiple face optimization types.

12. The method according to any one of claims 1 to 11, characterized in that, The encoding and decoding are performed iteratively multiple times. After denoising the first face image based on the noise map to obtain the second face image, the method further includes: If the number of iterations for encoding and decoding has not reached the set value, then the second face image is used as the new first face image, and the encoding model based on the face optimization requirement information is returned. The face optimization requirement information and the first face image are encoded at the multiple levels to obtain the encoded feature maps of the multiple sizes. If the number of iterations for encoding and decoding reaches the set value, then the second face image obtained in the last iteration is used as the optimized result of the first face image.

13. The method according to any one of claims 1 to 11, characterized in that, Before encoding the face optimization requirement information and the first face image at multiple levels using the encoding model based on matching the face optimization requirement information to obtain the encoded feature maps of the multiple sizes, the method further includes: The noisy image in round t is noise-added to obtain the noisy face image in round t, and the noisy face image in round t is determined as the noisy image in round t+1, where the noisy image in round 1 is the first face image, t is a positive integer that increases sequentially, 1≤t≤T, and T is a positive integer; The noisy face image from round T is used as the first face image input to the encoding model.

14. The method according to any one of claims 1 to 11, characterized in that, The encoding model includes a first encoder and a second encoder with N levels, where N is a positive integer greater than 1. The encoding model based on matching facial optimization requirement information encodes the facial optimization requirement information and the first facial image at multiple levels to obtain encoded feature maps of multiple sizes, including: The first encoder encodes the facial optimization requirement information to obtain a first encoding vector; The first face image is encoded based on the second encoder of the first level to obtain the second encoding vector of the first level. Cross attention processing is then performed on the first encoding vector and the second encoding vector of the first level to obtain the encoding feature map of the first level. Based on the second encoder of the (n+1)th level, the encoded feature map of the nth level is encoded according to a preset step size to obtain the second encoded vector of the (n+1)th level, where n is a positive integer that increases sequentially, 1≤n≤N-1; Cross-attention processing is performed on the first encoding vector and the second encoding vector at the (n+1)th level to obtain the encoding feature map at the (n+1)th level. The size of the encoding feature map at the (n+1)th level is proportionally reduced compared to the encoding feature map at the nth level, based on the preset step size.

15. The method according to any one of claims 1 to 11, characterized in that, The decoding model includes M levels of decoders, where M is a positive integer greater than 1. The decoding model based on matching facial optimization requirements performs multi-level decoding on the encoded feature maps of multiple sizes and the identity features of multiple sizes to obtain a noise map, including: Based on the first-level decoder, the identity features and encoded feature maps of the corresponding size of the first level, as well as the encoded feature map of the smallest size, are fused to obtain the first-level identity enhancement image; Based on the first-level decoder, the first-level identity enhancement image is upsampled to obtain the first-level decoded feature map; Based on the decoder at level m+1, the identity features and encoded feature maps of the corresponding size at level m+1, as well as the decoded feature map at level m, are fused to obtain the identity enhancement image at level m+1, where 1≤m≤M-1; Based on the decoder at the (m+1)th level, the identity enhancement image at the (m+1)th level is upsampled by a preset step size to obtain the decoded feature map at the (m+1)th level. The size of the decoded feature map at the (m+1)th level is proportionally increased to that at the (m)th level based on the preset step size. When the value of m iterates to M-1, the decoded feature map of the Mth level is used as the noise map.

16. The method according to any one of claims 1 to 11, characterized in that, The method further includes: The encoding model and the decoding model are trained in the following manner: Select optimized facial image samples from the dataset, wherein the facial image samples are optimized based on the facial optimization type matched with the facial optimization requirement information; Select any time step as the current time step and generate Gaussian noise corresponding to the current time step; Add Gaussian noise corresponding to the current time step to the face image sample to obtain a noisy image sample at the current time step; Based on the face optimization type and the noisy image sample, determine the predicted noise at the current time step; The noise error between the Gaussian noise and the predicted noise is determined, and the parameters of the encoding model and the decoding model are updated based on the noise error.

17. A facial image processing device, characterized in that, The device includes: The feature extraction module is used to extract identity features at multiple levels from the first facial image of the target object, thereby obtaining identity features of the target object at multiple dimensions, wherein one level corresponds to one dimension; The acquisition module is used to acquire facial optimization requirement information set for the first facial image; The encoding module is used to encode the face optimization requirement information and the first face image at multiple levels based on an encoding model that matches the face optimization requirement information, so as to obtain the encoded feature maps of the multiple sizes. The decoding module is used to decode the encoded feature maps of multiple sizes and the identity features of multiple sizes at multiple levels based on a decoding model that matches the facial optimization requirements information, to obtain a noise map. For each level, the input to the decoding of the level is the output of the decoding of the previous level, the identity features with the same size as the decoded feature map of the output of the level, and the encoded feature map. The denoising module is used to denoise the first face image based on the noise map to obtain the second face image.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the facial image processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the facial image processing method according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the face image processing method according to any one of claims 1 to 16 is implemented.

Images