Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and belongs to the technical field of computers. According to the method, the face region map and the face depth information of the reference object are acquired, so that the illumination atmosphere conforming to the template image and the image for keeping the structure and details of the user face are generated. By determining the text description and face features of the input image including the face of the user, the generated image can retain the key face features of the user. Compared with an existing image processing scheme, the similarity between the image generated by the scheme and the face of the user is higher, the color expression and the light fusion effect are better, the fusion of the five sense organs and other structures and details of the face is natural and smooth, and the aesthetic degree is higher.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] With the development of Internet technology, face swapping of existing photos is a common demand. In this way, users can quickly obtain high-quality photos and image resources based on their own faces. At present, most face-swapping technologies are based on generative models. However, there are many difficult-to-solve problems with generative models, such as low facial similarity with the user whose face is swapped, insufficient fusion of color and light between the replaced face and the template image, stiff fusion of facial features, and low aesthetics. Therefore, a new image processing method is urgently needed to achieve a high-quality face-swapping solution. Summary of the invention

[0003] The present disclosure provides an image processing method, device, equipment and storage medium. Compared with existing image processing solutions, the image generated by this method has a higher similarity with the user's face, better color expression and light fusion effect, and the structure and details of the facial features are naturally and smoothly integrated, with a higher aesthetics.

[0004] According to one aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0005] Acquire a face region map and face depth information of a first image, where the first image is a template image including a reference object;

[0006] Determining a text description and a facial feature based on a second image, wherein the second image is an input image including a user's face, the text description is used to describe the content of the second image, and the facial feature is used to represent a feature of the user's face;

[0007] Based on the face area map, the face depth information, the text description and the face features, a third image is generated, where the third image is an image obtained by replacing the face of the reference object in the template image with the user's face.

[0008] According to another aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0009] an acquisition unit configured to acquire a face region map and face depth information of a first image, wherein the first image is a template image including a reference object;

[0010] a determining unit configured to determine a text description and a facial feature based on a second image, wherein the second image is an input image including a user's face, the text description is used to describe the content of the second image, and the facial feature is used to represent a feature of the user's face;

[0011] A generating unit is configured to generate a third image based on the face area map, the face depth information, the text description and the face features, wherein the third image is an image obtained by replacing the face of the reference object in the template image with the user's face.

[0012] In some embodiments, the acquisition unit is configured to determine the face range of the reference object in the first image and crop it to obtain the face area map; determine the corresponding face depth map based on the face area map; and extract the face depth information from the face depth map.

[0013] In some embodiments, the acquisition unit is configured to acquire the face area map and face depth information of the first image from a cache database according to an image identifier of the first image.

[0014] In some embodiments, the determination unit is configured to perform image segmentation on the second image to obtain multiple image blocks; perform linear projection on the multiple image blocks respectively to obtain an embedded representation of each image block, the embedded representation includes a position code of the corresponding image block, and the position code is used to represent the spatial information of the corresponding image block; based on a self-attention mechanism, encode the embedded representation to obtain the text description, and the text description includes multiple image labels.

[0015] In some embodiments, the generating unit is configured to generate an intermediate face image based on the face region map, the face depth information, the text description and the face features, wherein the intermediate face image includes the features of the user's face, and the face three-dimensional structure of the intermediate face image is consistent with the face region map. Figure 1 synthesizing the intermediate face image and the first image to obtain the third image.

[0016] In some embodiments, the generation unit is configured to input the face area map, the face depth information, the text description and the face features to generate an intermediate face map as reference information into a diffusion model, and the diffusion model outputs the intermediate face map. The diffusion model is used to generate an image starting from random noise and gradually denoising based on the reference information.

[0017] In some embodiments, the generating unit is configured to perform key point detection on the intermediate face image to obtain the positional relationship between the user's face in the intermediate face image and the face in the first image; replace the face area in the first image with a mask; insert the intermediate face image at the mask position and fuse them to obtain the third image.

[0018] In some embodiments, the apparatus further comprises:

[0019] The adjustment unit is configured to obtain color distribution information of the face area map; and adjust the color and style of the intermediate face map based on the color distribution information.

[0020] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0021] one or more processors;

[0022] A memory for storing program codes executable by the processor;

[0023] The processor is configured to execute the program code to implement the above-mentioned image processing method.

[0024] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned image processing method.

[0025] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0026] The disclosed embodiment provides an image processing solution, which helps to generate an image that matches the lighting atmosphere of the template image and maintains the structure and details of the user's face by acquiring the face area map and face depth information of the reference object. By determining the text description and face features of the input image including the user's face, the generated image can retain the user's key facial features. Compared with existing image processing solutions, the image generated by the above solution has a higher similarity to the user's face, and has better color expression and light fusion effects. The structure and details of the facial features are naturally and smoothly integrated, and the aesthetics are high.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0029] Figure 1 It is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment.

[0030] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0031] Figure 3 is a flowchart of another image processing method according to an exemplary embodiment.

[0032] Figure 4 It is a flowchart of an image processing method provided according to an exemplary embodiment.

[0033] Figure 5 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0034] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the images involved in this disclosure are all obtained with full authorization.

[0038] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0039] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and the application is used to play information stream data, such as multimedia data or live broadcast data. The application is associated with the server 102, and the server 102 provides background services to the terminal 101.

[0040] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0041] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0042] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0043] In step S201, a face region map and face depth information of a first image are obtained.

[0044] In the disclosed embodiment, the first image is a template image including a reference object, that is, the first image is a template for reference when performing image processing. The reference object can be called a model object or a target object for face replacement. Among them, the face area map is an image of the range where the face is cropped from the first image. Face depth information Face depth information refers to the position relationship and distance information of the face in three-dimensional space.

[0045] In step S202, text description and facial features are determined based on the second image.

[0046] In the embodiment of the present disclosure, the second image is an input image including a user's face, that is, the second image is an image to be processed when performing image processing, and the second image includes the user's face. The text description is used to describe the content of the second image. The face feature is used to represent the features of the user's face.

[0047] In step S203, a third image is generated based on the face area map, the face depth information, the text description and the face features.

[0048] In the disclosed embodiment, reference information is provided based on the face region map, face depth information, text description and face features, and a diffusion model is used to gradually remove noise from the noise image to generate a face image, and then the obtained face image is synthesized with the template image to generate a third image. The third image is an image after the face of the reference object in the template image is replaced with the user's face.

[0049] The disclosed embodiment provides an image processing solution, which helps to generate an image that matches the lighting atmosphere of the template image and maintains the structure and details of the user's face by acquiring the face area map and face depth information of the reference object. By determining the text description and face features of the input image including the user's face, the generated image can retain the user's key facial features. Compared with existing image processing solutions, the image generated by the above solution has a higher similarity to the user's face, and has better color expression and light fusion effects. The structure and details of the facial features are naturally and smoothly integrated, and the aesthetics are high.

[0050] In some embodiments, obtaining a face region map and face depth information of a first image includes:

[0051] Determine the face range of the reference object in the first image and crop it to obtain a face area map;

[0052] Determine a corresponding face depth map based on the face region map;

[0053] Extract face depth information from face depth map.

[0054] By determining and cropping the face range of the reference object in the first image to obtain a face area map, and then determining a face depth map and extracting face depth information, it helps to more accurately analyze facial features and provide more accurate data support for subsequent processing processes.

[0055] In some embodiments, obtaining a face region map and face depth information of a first image includes:

[0056] According to the image identifier of the first image, a face region map and face depth information of the first image are obtained from a cache database.

[0057] By directly acquiring the face area map and face depth information from the cache database according to the image identifier of the first image, the efficiency of data acquisition can be improved and the time cost of repeated calculation and processing can be reduced.

[0058] In some embodiments, based on the second image, the step of determining the text description includes:

[0059] Performing image segmentation on the second image to obtain a plurality of image blocks;

[0060] Performing linear projection on multiple image blocks respectively to obtain an embedded representation of each image block, where the embedded representation includes a position code of the corresponding image block, and the position code is used to represent spatial information of the corresponding image block;

[0061] Based on the self-attention mechanism, the embedded representation is encoded to obtain a text description, which includes multiple image tags.

[0062] By segmenting and linearly projecting the second image to obtain an embedded representation containing position coding, and then encoding it based on the self-attention mechanism to obtain a text description including multiple image labels, it is possible to make full use of the spatial information of the image for efficient encoding, accurately generate a text description of the image content, and facilitate subsequent processing.

[0063] In some embodiments, generating a third image based on the face area map, the face depth information, the text description, and the face features includes:

[0064] An intermediate face image is generated based on the face region map, face depth information, text description and face features. The intermediate face image includes the features of the user's face, the face three-dimensional structure and face region of the intermediate face image. Figure 1 To;

[0065] The intermediate face image is synthesized with the first image to obtain a third image.

[0066] By using the face area map, face depth information, text description and face features, a user's face features and a three-dimensional face structure and face area are generated. Figure 1 The intermediate face image is obtained by synthesizing the intermediate face image and the first image to obtain the third image, thereby realizing personalized image synthesis.

[0067] In some embodiments, generating an intermediate face map based on the face region map, face depth information, text description, and face features includes:

[0068] The face area map, face depth information, text description and face feature generated intermediate face map are input into the diffusion model as reference information, and the diffusion model outputs the intermediate face map. The diffusion model is used to generate an image starting from random noise and gradually denoising based on the reference information.

[0069] By inputting the face area map, face depth information, text description and face features as reference information into the diffusion model to generate an intermediate face map, and using the diffusion model to gradually denoise the random noise to generate an image, it is possible to fully combine multiple information to generate a high-quality intermediate face map with specific features, providing a more accurate foundation for subsequent image synthesis.

[0070] In some embodiments, synthesizing the intermediate face image and the first image to obtain a third image includes:

[0071] Perform key point detection on the intermediate face image to obtain the positional relationship between the user's face in the intermediate face image and the face in the first image;

[0072] Replace the face region in the first image with the mask;

[0073] The intermediate face image is inserted at the mask position and fused to obtain the third image.

[0074] By performing key point detection on the intermediate face image to determine its positional relationship with the face in the first image, the face area of ​​the first image is replaced with a mask, and then the intermediate face image is inserted and fused to obtain a third image, accurate image fusion can be achieved to generate a natural image.

[0075] In some embodiments, the method further comprises:

[0076] Obtain color distribution information of the face area map;

[0077] Based on the color distribution information, adjust the color and style of the intermediate face image.

[0078] By obtaining the color distribution information of the face area map and adjusting the color and style of the intermediate face map accordingly, the intermediate face map can be better integrated into the original image, thereby enhancing the naturalness and coordination of image synthesis.

[0079] Above Figure 2 Shown is a flow chart of an image processing method of the present disclosure. The image processing solution provided by the present disclosure is further elaborated below. Figure 3 is a flowchart of another image processing method according to an exemplary embodiment. Figure 3 The method is performed by an electronic device and comprises the following steps:

[0080] In step S301 , a text description is determined based on the second image.

[0081] In the disclosed embodiment, the second image is an input image including a user's face. The user's face is the face of the current user. The input image means that the second image is input by the current user. In other words, the user's face is the face that is retained after the image processing is completed and the face-changing processing is performed. The text description is used to describe the content of the second image. Optionally, the text description can describe the facial features, expressions, skin color, etc. of the user in the image, and can also describe the background, lighting conditions, etc. of the image.

[0082] For example, in the second image, the user's face is smiling and the background is a beautiful sea of ​​flowers, so the text description can be "The user is smiling, surrounded by beautiful flowers, and looks very happy." If the user's face is serious and the light is dim, the text description may be "The user has a serious expression and is in a dark environment, showing a deep feeling."

[0083] In some embodiments, the electronic device determines a text description through an attention mechanism, and the text description is a plurality of image tags. Accordingly, the electronic device determines the step of the text description based on the second image, including: first, the electronic device performs image segmentation on the second image to obtain a plurality of image blocks. Then, the electronic device performs linear projection on the plurality of image blocks respectively to obtain an embedded representation of each image block, and the embedded representation includes a position code of the corresponding image block, and the position code is used to represent the spatial information of the corresponding image block. Finally, the electronic device encodes the embedded representation based on the self-attention mechanism to obtain a text description, and the text description includes a plurality of image tags. Optionally, the size of the plurality of image patches is a fixed size. Wherein, for each image patch, the image patch is converted into a corresponding embedding vector by linear projection. The electronic device adds a position code to each embedding vector to retain spatial information. Wherein, the position code can represent the position of the image patch in the entire image, such as which part of the image the image patch is in, and the relative position relationship with other image patches. By setting the position code, the electronic device can better utilize the spatial layout information of the image patch when processing the image later, so as to more accurately understand and process the image. Optionally, the electronic device encodes the above-mentioned embedded representation based on the Transformer encoder. Accordingly, the electronic device inputs the embedded representation of each image block into a series of Transformer encoder layers. Each encoder layer uses a multi-head self-attention mechanism and a feed-forward neural network to process the input embedded representation. Optionally, the electronic device outputs a text description by using a multi-label classification method. Optionally, the text description includes age description, skin description, accessories description, and environment description.

[0084] Optionally, the electronic device extracts a text description of the image input by the user through a ViT (Vision Transformer) visual image tagging model. The ViT model includes an image segmentation module, a linear embedding module, a position encoding module, a Transformer encoder, and a multi-label classification module. The image segmentation module is used to divide the input image into blocks of fixed size. The linear embedding module is used to convert the image blocks into embedding vectors through linear projection. The position encoding module is used to add position encoding to each embedding vector to retain spatial information. The Transformer encoder uses a multi-head self-attention mechanism (Multi-Head Self-Attention) and a feedforward neural network to encode the embedding vector. The multi-label classification module is used to predict multiple related description tags. The advantage of the ViT model is that it can capture the global relationship in the image, so that a more comprehensive and contextual image description, that is, a text description, can be generated. The above text description provides important semantic information for the subsequent face replacement process, which helps to generate a more matching and natural replacement result. Compared with the traditional convolutional neural network (CNN) method, the image processing method based on the ViT model performs better in processing long-distance dependencies and global information, especially in complex scenes and images with rich details. Optionally, the ViT model can be replaced with other multimodal large language models that support image recognition. It should be noted that changing the model may lead to richer concepts and more controllable output prompts, but the inference cost will also increase.

[0085] In step S302, a face region map of the first image is obtained.

[0086] In the disclosed embodiment, the first image is a template image including a reference object. The first image is used as a template for reference during image processing, and the purpose of image processing is to replace the face of the reference object in the first image with the face of the user. The electronic device can obtain a face area map by determining the face range of the reference object in the first image and cropping it.

[0087] Optionally, the electronic device recognizes and segments the face portion in the first image by using an image segmentation model. By using the image segmentation model, a very accurate segmentation result can be generated even in a complex background, and the image segmentation model has the characteristic of high precision. Moreover, the image segmentation model is applicable to various scenes and can be segmented according to different types of prompts. In addition, the image segmentation model also has a strong generalization ability and can segment object categories that have not been seen during the training process.

[0088] In step S303, facial depth information of the first image is obtained.

[0089] In the embodiments of the present disclosure, face depth information refers to the position information of the face in three-dimensional space. Face depth information can reflect the distance between various parts of the face and the image acquisition device (such as a camera). The methods for obtaining face depth information include structured light technology, time of flight method, etc., which are not limited in the embodiments of the present disclosure.

[0090] In some embodiments, the electronic device extracts the face depth information through the face depth map. Accordingly, the electronic device determines the corresponding face depth map based on the face area map, and then extracts the face depth information from the face depth map. Optionally, the face depth information can also be replaced by face line information.

[0091] In some embodiments, the electronic device uses an estimation preprocessor to obtain a face cropping image, that is, a face depth map of a face region map, and the face depth map includes contour information such as depth or lines, such as depth information or line information. The electronic device inputs the face depth map into the Depth ControlNet model for processing to obtain face depth information. Among them, the working principle of ControlNet is: first, the face depth map is input into ControlNet as a condition. Then ControlNet uses the extracted information to modulate the feature map of the original diffusion model. Then, through the cross-attention mechanism, the depth information is integrated into the image generation process. It should be noted that the above method can adjust the influence of the depth information to balance the relationship between maintaining the original structure and allowing creative generation. By using the ControlNet model to extract face depth information, the face depth information helps to maintain the three-dimensional or two-dimensional line structure of the face during the generation process, and can also help the model generate results consistent with the lighting conditions of the original image, and can also better process the subtle features of the face, such as bumps and contours, according to the depth information.

[0092] Optionally, important facial depth information can be obtained through the above steps, but because the structure and details of the user's face are different from the structure and details of the face in the template image, the facial depth map of the template image can be compressed to a small size, weakening the structure and detail information, retaining some lighting consistency or line consistency information, which helps to generate a lighting atmosphere that is more consistent with the template image and a replacement result that better maintains the structure and details of the user's face.

[0093] Optionally, the electronic device can also obtain the face area map and face depth information of the first image from the cache database according to the image identifier of the first image. The cache database is used to store relevant information of the template image, including but not limited to the face area map and face depth information of the template image. Since the template image is fixed, by pre-processing and caching certain intermediate results, it is possible to avoid repeated calculations at each request, thereby improving the efficiency and response speed of the system and reducing the consumption of computing resources. In addition, it can also ensure that multiple requests for the same template image obtain consistent intermediate results with high consistency. The cache strategy can also be dynamically adjusted according to the system load and resource conditions, with high flexibility.

[0094] It should be noted that when setting up the cache database, the following implementation considerations should be taken: 1. The expected number of template images and system resources should be used to reasonably set the cache size; 2. Consider compressing the cached image data to save storage space; 3. When the system starts, commonly used template images can be pre-processed and the cache can be filled; 4. Implement a monitoring system to track cache hit rates and performance indicators, and continuously optimize the cache strategy. By implementing the above cache strategy, the efficiency and scalability of the entire face replacement system can be significantly improved, thereby providing a faster user experience and more efficient use of computing resources, thereby laying the foundation for large-scale deployment and use.

[0095] In step S304, facial features are determined based on the second image.

[0096] In the embodiment of the present disclosure, facial features are used to represent features of a user's face.

[0097] In some embodiments, the electronic device can extract the user's facial information through the feature extraction model provided by the face ID retention project, and convert it into a format that can be understood by the diffusion model, that is, facial features. Among them, the step of extracting facial features includes: first, the user's facial image, that is, the second image, is input into the feature extraction model. Then, the key features of the face are extracted and encoded by the feature extraction model to obtain facial features. Optionally, the above facial features can be converted into a format that can be understood by the ControlNet model. The ControlNet model uses the facial features to generate a control signal. The control signal is used to guide the diffusion model to retain the user's facial features during the generation process. Extracting facial features through the feature extraction model can effectively capture and retain the user's unique facial features, and also allows the degree of retention of identity features to be controlled to varying degrees. Accordingly, through precise feature extraction and control, more realistic and personalized results can be generated.

[0098] In step S305, an intermediate face map is generated based on the face region map, face depth information, text description and face features.

[0099] In the embodiment of the present disclosure, the intermediate face image includes the features of the user's face, the three-dimensional structure of the face and the face area of ​​the intermediate face image. Figure 1 In other words, the intermediate face image is an image after the user's face is transformed, and the structure of the intermediate face image is consistent with the face of the reference object, but the content is the user's face.

[0100] In some embodiments, the electronic device generates an intermediate face image through a diffusion model. Accordingly, the electronic device generates an intermediate face image using a face region map, face depth information, text description, and face features as reference information and inputs the intermediate face image into the diffusion model, which outputs the intermediate face image. The diffusion model is used to generate an image starting from random noise and gradually denoising based on the reference information. Optionally, the diffusion model is a conditional diffusion model, which can guide the image generation process according to a variety of input conditions.

[0101] In some embodiments, the workflow of the conditional diffusion model includes the following five steps:

[0102] Step 1: Input integration. In this step, the text description is used as information to provide semantic guidance. The face region map is used as information to provide structural guidance for the initial image. The face depth information is used as information to provide the three-dimensional structure of the face. The face features are used as information to provide identity features.

[0103] Step 2: Conditional diffusion process. This step is initialized first, starting from random noise and gradually denoising to generate an image. Then, in each denoising process, the conditional diffusion model will refer to all the input conditions in step 1. Different conditional information is fused by using methods such as cross attention mechanism or feature splicing to achieve multi-condition fusion.

[0104] Step 3: Iterative refinement. When generating an image, the conditional diffusion model gradually increases from low resolution to high resolution to continuously refine the details of the generated image. Optionally, the image generation process is implemented through multiple iterations. In each iteration process, the conditional diffusion model is adjusted according to the current generation results and conditions.

[0105] Step 4: Identity feature transfer. When generating images, the conditional diffusion model retains the key facial features of the user's face and naturally integrates the facial features of the user's face into the structure of the template image.

[0106] Step 5: Detail adjustment. The conditional diffusion model can generate an intermediate face image with a suitable three-dimensional structure based on the face depth information. Then adjust the skin color, skin texture and other details according to the text description.

[0107] The final output of the conditional diffusion model is an image that integrates the user's facial features, conforms to the structure of the template image, maintains three-dimensional consistency, and is rich in details, that is, the intermediate face image.

[0108] The face region map is generated by the conditional diffusion model mentioned above. Due to the integration of multiple conditional information, precise control of the generation process is achieved. For example, facial features allow creative changes while retaining the user's identity features, facial depth information can maintain the natural three-dimensional structure of the face, and text descriptions allow fine-grained adjustments to the generated results. Moreover, each input condition has an impact on the final generated result. This method achieves high-quality, personalized facial feature migration.

[0109] Optionally, the diffusion model may be driven by a diffusion model such as SDXL or a DiT image generation model similar to FLUX, and the embodiments of the present disclosure are not limited thereto.

[0110] In step S306, the color and style of the intermediate face image are adjusted based on the color distribution information of the face region image.

[0111] In the disclosed embodiment, after the intermediate face image is generated, the color and style of the face image may be adjusted so that the adjusted intermediate face image is consistent with the template image in color and style.

[0112] In some embodiments, the electronic device adjusts the color and style through color migration technology. First, the electronic device performs color statistical analysis on the face area map obtained in the above step S302, calculates the color mean and variance of the first image, and obtains the color distribution information of the face area map. Optionally, the color distribution information can be directly obtained from the cache database, which is not limited by the embodiments of the present disclosure. Then, the electronic device performs style migration on the intermediate face image based on the above color distribution information to obtain an adjusted intermediate face image. Optionally, the color migration technology can use AdaIN (Adaptive Instance Normalization) or other similar methods, which is not limited by the embodiments of the present disclosure. Among them, AdaIN is a simple and effective style migration method, the principle of which is to adjust the mean and variance of the content image to match the style image.

[0113] By adjusting the color and style of the intermediate face image, the facial details and texture information are retained while color matching is performed, ensuring that the final generated image is not only similar to the user in terms of facial features, but also harmonious with the original template image in terms of overall color atmosphere, achieving a more natural color transition. Optionally, the matching strength can also be adjusted according to different image characteristics, improving the naturalness and realism of face replacement.

[0114] Optionally, the adjustment of the intermediate face image includes but is not limited to aligning the color performance of the intermediate face image and the template image, adjusting the overall color performance (saturation, brightness, etc.) of the stitched image, adding a post-beautification filter to the intermediate face image, etc.

[0115] In step S307, the intermediate face image and the first image are synthesized to obtain a third image.

[0116] In the disclosed embodiment, this step synthesizes the intermediate face image adjusted in the previous step with the original template image, that is, the first image, to obtain a third image, that is, a complete face-changing image.

[0117] In some embodiments, the electronic device synthesizes the intermediate face image with the first image to obtain the step of obtaining the third image, including: first, the electronic device performs key point detection on the intermediate face image to obtain the positional relationship between the user's face in the intermediate face image and the face in the first image, so as to achieve an accurate match between the face in the generated intermediate face image and the facial position in the template image. Then, the electronic device replaces the facial area in the first image with a mask. The mask can accurately represent the area to be replaced, including smooth transition edges. Finally, the electronic device inserts the intermediate face image at the mask position and fuses it to obtain the third image. Optionally, the electronic device can use advanced image fusion techniques such as Poisson image editing or gradient domain fusion to ensure that the continuity of local structure and texture is maintained during the fusion process. Optionally, the electronic device can apply adaptive feathering technology to the fusion edge to create a soft transition effect and avoid obvious splicing marks.

[0118] By performing image synthesis in the above manner, the generated intermediate face image can be combined with the original template image, so that the replaced face and the original image are naturally integrated, and the continuity of illumination, texture and structure is maintained, thereby obtaining a natural and realistic final image. Optionally, the fusion strategy can be dynamically adjusted according to the image characteristics to fuse different images.

[0119] It should be noted that in order to make the image processing solution provided by the embodiment of the present disclosure easier to understand, see Figure 4 As shown, Figure 4 FIG. 1 is a flow chart of an image processing method provided according to an exemplary embodiment. Figure 4As shown, input the user input image (the second image above). Step 1: Extract the text description of the user input image. Step 2: Get the face range of the template image, which is the face area image cut from the template image. Step 3: Extract the face depth information in the template image. Step 4: Extract the facial features in the user input image. Step 5: Based on the output of the above 4 steps, generate the image. Step 6: Post-process the generated intermediate face image, that is, adjust the color and style. Step 7: Combine the intermediate face image and the template image to obtain the final composite image.

[0120] After testing, the image processing solution provided by the present invention is superior to face-changing methods using the Gan model or other unknown processes in terms of overall generation cost (computing cost, labor cost, production cycle), the similarity between the generated image and the original image, and the aesthetics of the generated image.

[0121] The disclosed embodiment provides an image processing solution, which helps to generate an image that matches the lighting atmosphere of the template image and maintains the structure and details of the user's face by acquiring the face area map and face depth information of the reference object. By determining the text description and face features of the input image including the user's face, the generated image can retain the user's key facial features. Compared with existing image processing solutions, the image generated by the above solution has a higher similarity to the user's face, and has better color expression and light fusion effects. The structure and details of the facial features are naturally and smoothly integrated, and the aesthetics are high.

[0122] Figure 5 FIG. 1 is a block diagram of an image processing apparatus according to an exemplary embodiment. Figure 5 As shown, the device includes: an acquisition unit 501, a determination unit 502 and a generation unit 503.

[0123] An acquisition unit 501 is configured to acquire a face region map and face depth information of a first image, where the first image is a template image including a reference object;

[0124] The determining unit 502 is configured to determine a text description and a facial feature based on a second image, wherein the second image is an input image including a user's face, the text description is used to describe the content of the second image, and the facial feature is used to represent a feature of the user's face;

[0125] The generating unit 503 is configured to generate a third image based on the face area map, face depth information, text description and face features, where the third image is an image obtained by replacing the face of the reference object in the template image with the user's face.

[0126] In some embodiments, the acquisition unit 501 is configured to determine the face range of the reference object in the first image and crop it to obtain a face area map; determine the corresponding face depth map based on the face area map; and extract face depth information from the face depth map.

[0127] In some embodiments, the acquisition unit 501 is configured to acquire the face area map and face depth information of the first image from a cache database according to the image identifier of the first image.

[0128] In some embodiments, the determination unit 502 is configured to perform image segmentation on the second image to obtain multiple image blocks; perform linear projection on the multiple image blocks respectively to obtain an embedded representation of each image block, the embedded representation includes a position encoding of the corresponding image block, and the position encoding is used to represent the spatial information of the corresponding image block; based on the self-attention mechanism, the embedded representation is encoded to obtain a text description, and the text description includes multiple image labels.

[0129] In some embodiments, the generating unit 503 is configured to generate an intermediate face image based on the face region map, the face depth information, the text description and the face features, wherein the intermediate face image includes the features of the user's face, the face three-dimensional structure and the face region of the intermediate face image. Figure 1 The intermediate face image is synthesized with the first image to obtain a third image.

[0130] In some embodiments, the generation unit 503 is configured to input the face area map, face depth information, text description and face features to generate an intermediate face map as reference information into a diffusion model, and the diffusion model outputs the intermediate face map. The diffusion model is used to generate an image starting from random noise and gradually denoising based on the reference information.

[0131] In some embodiments, the generation unit 503 is configured to perform key point detection on the intermediate face image to obtain the positional relationship between the user's face in the intermediate face image and the face in the first image; replace the face area in the first image with a mask; insert the intermediate face image at the mask position and fuse them to obtain a third image.

[0132] In some embodiments, the apparatus further comprises:

[0133] The adjustment unit is configured to obtain color distribution information of the face region map; and adjust the color and style of the intermediate face map based on the color distribution information.

[0134] The disclosed embodiment provides an image processing device, which helps to generate an image that matches the lighting atmosphere of a template image and maintains the structure and details of a user's face by acquiring a face area map and face depth information of a reference object. By determining the text description and face features of an input image including the user's face, the generated image can retain the user's key facial features. Compared with existing image processing schemes, the image generated by the above scheme has a higher similarity to the user's face, and has better color expression and light fusion effects. The structure and details of the facial features are naturally and smoothly integrated, and the aesthetics are high.

[0135] It should be noted that the image processing device provided in the above embodiment is only illustrated by the division of the above functional units. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0136] Regarding the image processing device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0137] In the embodiments of the present disclosure, the electronic device may be a terminal or a server. When the electronic device is a terminal, the terminal is used as an execution subject to implement the technical solution provided in the embodiments of the present disclosure; when the electronic device is a server, the server is used as an execution subject to implement the technical solution provided in the embodiments of the present disclosure; or, the technical solution provided in the present disclosure is implemented through interaction between the terminal and the server. The embodiments of the present disclosure are not limited to this.

[0138] Figure 6 6 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 600 includes: a processor 601 and a memory 602 .

[0139] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0140] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one program code, which is used to be executed by the processor 601 to implement the image processing method provided by the method embodiment of the present disclosure.

[0141] In some embodiments, the electronic device 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607 and a power supply 608.

[0142] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0143] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0144] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, and the front panel of the electronic device 600 is set; in other embodiments, the display screen 605 can be at least two, which are respectively set on different surfaces of the electronic device 600 or are folded; in some other embodiments, the display screen 605 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0145] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0146] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0147] The power supply 608 is used to power various components in the electronic device 600. The power supply 608 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 608 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0148] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device 600, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0149] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the instructions can be executed by a processor 601 of an electronic device 600 to complete the above-mentioned image processing method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0150] A computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method is implemented.

[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0152] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a face region map and face depth information of a first image, where the first image is a template image including a reference object; Determining a text description and a facial feature based on a second image, wherein the second image is an input image including a user's face, the text description is used to describe the content of the second image, and the facial feature is used to represent a feature of the user's face; Based on the face area map, the face depth information, the text description and the face features, a third image is generated, where the third image is an image obtained by replacing the face of the reference object in the template image with the user's face.

2. The image processing method according to claim 1, characterized in that: The step of obtaining the face region map and face depth information of the first image includes: Determining the face range of the reference object in the first image and cropping the face range to obtain the face region map; Determine a corresponding face depth map based on the face area map; The face depth information is extracted from the face depth map.

3. The image processing method according to claim 1, characterized in that: The step of obtaining the face region map and face depth information of the first image includes: According to the image identifier of the first image, a face region map and face depth information of the first image are obtained from a cache database.

4. The image processing method according to claim 1, characterized in that: Based on the second image, the step of determining a text description comprises: Performing image segmentation on the second image to obtain a plurality of image blocks; Performing linear projection on the multiple image blocks respectively to obtain an embedded representation of each image block, wherein the embedded representation includes a position code of the corresponding image block, and the position code is used to represent spatial information of the corresponding image block; Based on a self-attention mechanism, the embedded representation is encoded to obtain the text description, where the text description includes multiple image tags.

5. The image processing method according to claim 1, characterized in that: The step of generating a third image based on the face area map, the face depth information, the text description and the face features includes: Generate an intermediate face image based on the face region image, the face depth information, the text description and the face features, the intermediate face image including the features of the user's face, and the face three-dimensional structure of the intermediate face image is consistent with the face region image; The intermediate face image and the first image are synthesized to obtain the third image.

6. The image processing method according to claim 5, characterized in that: The generating an intermediate face image based on the face area image, the face depth information, the text description and the face features comprises: The face area map, the face depth information, the text description and the face features generate an intermediate face map as reference information and input it into a diffusion model, which outputs the intermediate face map. The diffusion model is used to generate an image starting from random noise and gradually denoising based on the reference information.

7. The image processing method according to claim 5, characterized in that: The synthesizing the intermediate face image and the first image to obtain the third image includes: Perform key point detection on the intermediate face image to obtain a positional relationship between the user's face in the intermediate face image and the face in the first image; Replacing the face region in the first image with the mask; The intermediate face image is inserted into the mask position and fused to obtain the third image.

8. The image processing method according to any one of claims 5 to 7, characterized in that: The method further comprises: Obtaining color distribution information of the face region map; Based on the color distribution information, the color and style of the intermediate face image are adjusted.

9. An image processing device, characterized in that: The device comprises: an acquisition unit configured to acquire a face region map and face depth information of a first image, wherein the first image is a template image including a reference object; a determining unit configured to determine a text description and a facial feature based on a second image, wherein the second image is an input image including a user's face, the text description is used to describe the content of the second image, and the facial feature is used to represent a feature of the user's face; A generating unit is configured to generate a third image based on the face area map, the face depth information, the text description and the face features, wherein the third image is an image obtained by replacing the face of the reference object in the template image with the user's face.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the image processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

Images