Face fusion method and device based on parallel denoising, equipment and medium

Abstract

The present application relates to the fields of finance, medical health and artificial intelligence technology, and provides a face fusion method, device, equipment and medium based on parallel denoising, which can detect face key point coordinates in a target image based on a face key point detector, extract identity features of a source face image based on an identity encoder, calculate texture features of the source face image based on a texture statistics extractor, so as to provide a data basis for subsequent fusion; inject the identity features into a denoising process based on the face key point coordinates using an identity maintaining path, generate skin texture and illumination details using a texture synthesis path, decompose a single path diffusion model into two parallel and function-specific processing paths, so as to avoid coupling conflicts when simultaneously optimizing identity and texture; and fuse output features of the identity maintaining path and the texture synthesis path using a dynamic gating module, so that the fusion strategy can adapt to the needs of different denoising stages, thereby generating a high-quality face fusion image.

Description

Technical Field

[0001] This invention relates to the fields of finance, healthcare and artificial intelligence, and in particular to a face fusion method, apparatus, device and medium based on parallel denoising. Background Technology

[0002] Face fusion technology is an important research direction in the field of computer vision. Its goal is to seamlessly transfer the identity features of a source face to the pose, expression, and lighting conditions of a target face. Face fusion tasks are involved in many fields such as finance and healthcare.

[0003] For example, in the financial sector, face fusion can replace low-quality real-time faces with images that match the facial features of the user's stored high-definition identity and have natural textures, thus helping the identity verification system to more accurately identify the user's identity; in the healthcare sector, face fusion can replace the patient's face in real surgical videos and medical records with a virtual image, thereby avoiding the leakage of patient privacy.

[0004] Existing technologies, such as Face2Face (face-to-face) based on feature point alignment and traditional face fusion algorithms based on image blending, can achieve basic replacement, but they struggle to handle high-resolution details and complex lighting conditions, resulting in obvious artifacts and identity distortion in the synthesized results. With the rise of deep learning, such as DeepFakes, although it can achieve preliminary end-to-end replacement through autoencoder structures, it is limited by the stability issues of adversarial training, often producing blurry or inconsistent results. Summary of the Invention

[0005] In view of the above, it is necessary to provide a face fusion method, apparatus, device and medium based on parallel denoising, which aims to solve the problem of being unable to generate high-quality face fusion images.

[0006] A face fusion method based on parallel denoising, the face fusion method based on parallel denoising includes:

[0007] In response to a face fusion instruction that fuses a source face image into a target image, the system detects the coordinates of facial key points in the target image based on a facial key point detector, extracts the identity features of the source face image based on an identity encoder, and calculates the texture features of the source face image based on a texture statistics extractor.

[0008] Noise is added to the target image to obtain a noisy image, and the noisy image is then input into an improved diffusion model for denoising.

[0009] In the denoising process, the identity preservation path in the diffusion model is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates. The texture synthesis path in the diffusion model is used to generate skin texture and lighting details in the denoising process based on the texture features. The dynamic gating module in the diffusion model is used to fuse the output features of the identity preservation path and the texture synthesis path.

[0010] Once denoising is complete, acquire the output data of the diffusion model;

[0011] The output data is input into the decoder to obtain a face fusion image.

[0012] A face fusion device based on parallel denoising, the face fusion device based on parallel denoising includes:

[0013] The feature extraction unit is used to respond to a face fusion instruction that fuses a source face image into a target image, detect the coordinates of facial key points in the target image based on a facial key point detector, extract the identity features of the source face image based on an identity encoder, and calculate the texture features of the source face image based on a texture statistics extractor.

[0014] An input unit is used to add noise to the target image to obtain a noisy image, and input the noisy image into an improved diffusion model for denoising;

[0015] The denoising unit is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates using the identity preservation path in the diffusion model, and to generate skin texture and lighting details in the denoising process based on the texture features using the texture synthesis path in the diffusion model, and to fuse the output features of the identity preservation path and the texture synthesis path using the dynamic gating module in the diffusion model.

[0016] An acquisition unit is used to acquire the output data of the diffusion model when denoising is completed;

[0017] The input unit is also used to input the output data into the decoder to obtain a face fusion image.

[0018] A computer device, the computer device comprising:

[0019] Memory, storing at least one instruction; and

[0020] The processor executes the instructions stored in the memory to implement the face fusion method based on parallel denoising.

[0021] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the face fusion method based on parallel denoising.

[0022] As can be seen from the above technical solutions, this invention can detect the coordinates of facial key points in the target image based on the facial key point detector, extract the identity features of the source face image based on the identity encoder, and calculate the texture features of the source face image based on the texture statistical extractor, so as to provide the data foundation for subsequent fusion. In the denoising process, the identity preservation path in the diffusion model is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates, and the texture synthesis path in the diffusion model is used to generate skin texture and lighting details in the denoising process based on the texture features. The single-path diffusion model is decomposed into two parallel and functionally specialized processing paths to avoid coupling conflicts between optimizing identity and texture at the same time. The dynamic gating module in the diffusion model is used to fuse the output features of the identity preservation path and the texture synthesis path, so that the fusion strategy can adapt to the needs of different denoising stages, thereby generating a high-quality face fusion image. Attached Figure Description

[0023] Figure 1 This is a flowchart of a preferred embodiment of the face fusion method based on parallel denoising of the present invention.

[0024] Figure 2 This is a functional block diagram of a preferred embodiment of the face fusion device based on parallel denoising of the present invention.

[0025] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the parallel denoising-based face fusion method of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the face fusion method based on parallel denoising of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0028] The parallel denoising-based face fusion method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0029] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0030] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0031] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0035] S10, in response to the face fusion instruction to fuse the source face image into the target image, detect the coordinates of the face key points in the target image based on the face key point detector, extract the identity features of the source face image based on the identity encoder, and calculate the texture features of the source face image based on the texture statistics extractor.

[0036] In this embodiment, the source face image refers to the face image that needs to be replaced in the target image.

[0037] In this embodiment, the target image refers to an image in which the face needs to be replaced with the source face image.

[0038] The source face image and the target image may differ in different fields.

[0039] For example, in marketing activities in the financial sector, the source facial image can be a spokesperson's facial image, and the target image can be a marketing image taken by someone else; in the healthcare sector, the source facial image can be a virtual facial image, and the target image can be a patient image.

[0040] In this embodiment, the facial landmark detector can be used to locate specific key positions on a face. For example, the facial landmark detector can detect the coordinates of 68 facial landmarks in the target image.

[0041] In this embodiment, the identity encoder may include ArcFace.

[0042] In this embodiment, the texture statistics extractor may include a VGG-19 (Visual Geometry Group 19-layer network) network.

[0043] The above embodiments enable the construction of input data for subsequent networks to provide a data foundation.

[0044] S11, add noise to the target image to obtain a noisy image, and input the noisy image into the improved diffusion model for denoising.

[0045] In this embodiment, the U-Net (U-Net Convolutional Neural Network) backbone network of the diffusion model replaces the original convolutional blocks with blocks from the Swin Transformer (Swin Transformer: HierarchicalVision Transformer using Shifted Windows).

[0046] The U-Net backbone network includes four downsampling and upsampling stages, each stage embedding a window-based multi-head self-attention module, cross-scale feature skip connections, and dynamic deep convolutional residual blocks.

[0047] With the improved architecture described above, the hierarchical attention of the Swin-Transformer can be used to maintain high-resolution details (such as facial pores and wrinkles) while ensuring the consistency of the global structure (such as the overall outline of the face and the layout of facial features).

[0048] S12, In the denoising process, the identity preservation path in the diffusion model is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates. The texture synthesis path in the diffusion model is used to generate skin texture and lighting details in the denoising process according to the texture features. The dynamic gating module in the diffusion model is used to fuse the output features of the identity preservation path and the texture synthesis path.

[0049] In this embodiment, the step of injecting the identity features into the denoising process of the noisy image based on the facial key point coordinates using the identity preservation path in the diffusion model includes:

[0050] The noise latent representation of the noise image is obtained in the key point attention layer in the identity preservation path. The attention mapping between the noise latent representation and the facial key point coordinates is calculated. The noise latent representation is enhanced based on the attention mapping to obtain the first intermediate feature map.

[0051] In the identity embedding fusion layer of the identity preservation path, the identity features and the first intermediate feature map are fused based on a two-layer fully connected network to obtain a second intermediate feature map;

[0052] In the identity preservation path, the structure-preserving residual block uses four cascaded residual units to adaptively modulate the second intermediate feature map; wherein each residual unit employs dynamic convolution.

[0053] In the above embodiments, the key point attention layer enables the network to focus on identity-sensitive regions such as eyes, nose, and mouth, and enhances the feature consistency of these regions during the denoising process; the identity embedding fusion layer enables the smooth injection of identity features; and the structure-preserving residual block enables adaptive feature modulation.

[0054] Specifically, calculating the attention mapping between the noise latent representation and the facial keypoint coordinates includes:

[0055] The attention map is calculated using the following formula:

[0056]

[0057] Where A represents the attention mapping; W q W k Represents the learnable projection matrix; z t The noise latent representation is represented by L; the facial key point coordinates are represented by d. k This represents the dimension of the key vector.

[0058] Specifically, the adaptive feature modulation of the second intermediate feature map using four cascaded residual units includes:

[0059] For each residual unit, the dynamic convolution is calculated using the following formula:

[0060]

[0061] Among them, Conv dyn (x) represents the dynamic convolution of each residual unit; x represents the input data of each residual unit, the input data corresponding to the first residual unit is the second intermediate feature map, and the input data of other residual units is the dynamic convolution output by the cascaded previous residual unit; π i (x) represents the mixed weights predicted based on x; i represents the index variable, with values ​​of 1, 2, and 3, corresponding to 3 different regular convolution operations Conv. i (x).

[0062] The above embodiments can enhance the consistency of features in identity-sensitive areas, making the replaced face more closely match the target identity in key identity feature areas (such as facial features, position, etc.).

[0063] In this embodiment, the step of generating skin texture and lighting details during the denoising process based on the texture features using the texture synthesis path in the diffusion model includes:

[0064] In the AdaIN (Adaptive Instance Normalization) modulation module of the texture synthesis path, the latent noise representation is normalized according to the texture features using the following formula to obtain a standard noise representation: Among them, AdaIN(z t σ(y) represents the standard noise representation, y represents the feature map of the source face image, σ(y) represents the channel-level standard deviation of y in the texture feature, and μ(z) represents the standard noise representation. t ) represents the z in the texture feature t The channel-level mean, σ(z) t ) represents the z in the texture feature t The channel-level standard deviation, μ(y) represents the channel-level mean of y in the texture feature;

[0065] The high-frequency enhancement block in the texture synthesis path captures the skin texture using parallel dilated convolution branches and regular convolution branches according to the following formula:

[0066] ;in, Represents the skin texture, Conv 1×1 DConv2 represents a 1×1 convolution operation, DConv4 represents a dilated convolution operation with a dilation rate of 2, DConv8 represents a dilated convolution operation with a dilation rate of 8, and Conv represents a regular convolution operation.

[0067] In the multilayer perceptron layer of the texture synthesis path, the lighting details are added to the skin texture using the following formula: in, M represents the output feature of the texture synthesis path. light and b light This represents the lighting details predicted based on the multilayer perceptron layer.

[0068] In the above embodiments, the AdaIN modulation module can standardize the latent representation of noise based on texture statistics, effectively capture fine texture features such as pores and wrinkles based on the multi-scale fusion mechanism of the high-frequency enhancement block, and further generate lighting details based on the multilayer perceptron layer, thereby helping to generate more realistic and natural textures, with lighting matching the target face environment, and improving the realism of textures after face replacement.

[0069] In this embodiment, fusing the output features of the identity preservation path and the texture synthesis path using the dynamic gating module in the diffusion model includes:

[0070] The pixel-level fusion weights in the spatial dimension are predicted by a convolutional long short-term memory network based on the output features of the identity preservation path and the texture synthesis path.

[0071] At different U-Net layers, independent gating networks are applied based on a hierarchical fusion strategy to differentially enhance the pixel-level fusion weights; specifically, the weight of the texture synthesis path is increased for high-resolution shallow features, and the weight of the identity preservation path is increased for low-resolution deep features.

[0072] Based on the gated network, the temporal embedding vector of the diffusion time step is obtained, and the pixel-level fusion weight is adjusted in stages according to the temporal embedding vector using the gated function.

[0073] The weights obtained after differentiation enhancement and phased adjustments will be determined as the target weights.

[0074] The output features of the identity preservation path and the texture synthesis path are fused according to the target weight; wherein the target weight includes a first weight corresponding to the identity preservation path and a second weight corresponding to the texture synthesis path.

[0075] For example, the weight of high-resolution shallow features can be configured to 0.7, and the weight of low-resolution deep features can be configured to 0.6.

[0076] The fused features can be represented as follows: in, G represents the fused features. t1 G represents the first weight. t2 This represents the second weight. This represents the output feature of the identity preservation path.

[0077] The convolutional long short-term memory network can capture long-range dependencies while maintaining resolution in the spatial dimension, solving the pixel-level feature fusion problem. It generates independent fusion weights for each pixel, allowing the model to adaptively balance identity and texture features for different regions of the face (such as eyes and cheeks).

[0078] The hierarchical fusion strategy applies independent gated networks at different U-Net levels to form a hierarchical fusion strategy, which solves the feature fusion problem at the network level. Based on the pixel-level fusion weights, it utilizes the shallow (high resolution, texture sensitive) and deep (low resolution, identity sensitive) characteristics of U-Net to make biased adjustments to the pixel-level weights at different levels (e.g., shallow layers emphasize texture, and deep layers emphasize identity).

[0079] The temporal embedding vector addresses the feature fusion problem in the denoising stage. Based on pixel-level fusion weights and differentiation enhancement, it further introduces an embedding vector for the diffusion time step, allowing the fusion weights to dynamically change with the denoising process (such as initial denoising and later denoising), thereby adapting to the needs of identity preservation and texture synthesis at different stages.

[0080] Through the above embodiments, the features of the two paths can be dynamically and intelligently fused, so that the fusion result can balance the needs of identity and texture at different scales and different denoising stages, thereby improving the overall denoising quality.

[0081] S13, when denoising is completed, obtain the output data of the diffusion model.

[0082] In this embodiment, the output data is a vector representation that needs to be further decoded to reconstruct an image.

[0083] S14, the output data is input to the decoder to obtain a face fusion image.

[0084] In this embodiment, the face fusion image is the image obtained by fusing the source face image into the target image.

[0085] In this embodiment, the model training can adopt a progressive training strategy, such as phased training from 256×256 to 512×512, thereby utilizing more data and making the generated results more generalizable in terms of resolution.

[0086] In a 512×512 resolution face fusion task, this embodiment improves identity similarity by approximately 23% (based on the ArcFace metric) and texture realism by 18% (based on the LPIPS (Learned Perceptual Image Patch Similarity) metric) compared to existing methods, while maintaining real-time inference speed (approximately 0.5 seconds / frame on an NVIDIA V100 GPU (NVIDIA Volta V100 Graphics Processing Unit).

[0087] This embodiment can be used for face fusion in various fields.

[0088] For example, in the financial sector, security monitoring in financial institution branches may require enhancing or replacing faces in surveillance videos to simulate potentially risky facial features for drills or anomaly detection. This embodiment utilizes the identity preservation path to enhance the identity features of the target risky face, the texture synthesis path to restore realistic skin texture and lighting details, and the dynamic gating module to fuse a realistic simulated face for training a security recognition model. This improves the model's ability to recognize abnormal faces in complex scenes, thereby ensuring the security of financial venues.

[0089] For example, in healthcare settings, medical data sharing and research require the desensitization of medical images containing patient faces (such as surgical videos and physical examination images) while retaining necessary facial features for case association. Using this embodiment, the identity preservation path can adjust facial identity features to prevent them from corresponding to a real patient, while still preserving facial anatomical features to aid medical analysis. The texture synthesis path can reshape skin texture, and the dynamic gating module can fuse images that are both desensitized and retain the facial information needed for medical analysis, thus protecting patient privacy without affecting the reasonable use of medical data.

[0090] As can be seen from the above technical solutions, this invention can detect the coordinates of facial key points in the target image based on the facial key point detector, extract the identity features of the source face image based on the identity encoder, and calculate the texture features of the source face image based on the texture statistical extractor, so as to provide the data foundation for subsequent fusion. In the denoising process, the identity preservation path in the diffusion model is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates, and the texture synthesis path in the diffusion model is used to generate skin texture and lighting details in the denoising process based on the texture features. The single-path diffusion model is decomposed into two parallel and functionally specialized processing paths to avoid coupling conflicts between optimizing identity and texture at the same time. The dynamic gating module in the diffusion model is used to fuse the output features of the identity preservation path and the texture synthesis path, so that the fusion strategy can adapt to the needs of different denoising stages, thereby generating a high-quality face fusion image.

[0091] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the face fusion device based on parallel denoising of the present invention. The face fusion device 11 based on parallel denoising includes a feature extraction unit 110, an input unit 111, a denoising unit 112, and an acquisition unit 113. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0092] The feature extraction unit 110 is used to respond to a face fusion instruction that fuses a source face image into a target image, detect the coordinates of facial key points in the target image based on a facial key point detector, extract the identity features of the source face image based on an identity encoder, and calculate the texture features of the source face image based on a texture statistics extractor.

[0093] In this embodiment, the source face image refers to the face image that needs to be replaced in the target image.

[0094] In this embodiment, the target image refers to an image in which the face needs to be replaced with the source face image.

[0095] The source face image and the target image may differ in different fields.

[0096] For example, in marketing activities in the financial sector, the source facial image can be a spokesperson's facial image, and the target image can be a marketing image taken by someone else; in the healthcare sector, the source facial image can be a virtual facial image, and the target image can be a patient image.

[0097] In this embodiment, the facial landmark detector can be used to locate specific key positions on a face. For example, the facial landmark detector can detect the coordinates of 68 facial landmarks in the target image.

[0098] In this embodiment, the identity encoder may include ArcFace.

[0099] In this embodiment, the texture statistics extractor may include a VGG-19 (Visual Geometry Group 19-layer network) network.

[0100] The above embodiments enable the construction of input data for subsequent networks to provide a data foundation.

[0101] The input unit 111 is used to add noise to the target image to obtain a noisy image, and input the noisy image into an improved diffusion model for denoising.

[0102] In this embodiment, the U-Net (U-Net Convolutional Neural Network) backbone network of the diffusion model replaces the original convolutional blocks with blocks from the Swin Transformer (Swin Transformer: HierarchicalVision Transformer using Shifted Windows).

[0103] The U-Net backbone network includes four downsampling and upsampling stages, each stage embedding a window-based multi-head self-attention module, cross-scale feature skip connections, and dynamic deep convolutional residual blocks.

[0104] With the improved architecture described above, the hierarchical attention of the Swin-Transformer can be used to maintain high-resolution details (such as facial pores and wrinkles) while ensuring the consistency of the global structure (such as the overall outline of the face and the layout of facial features).

[0105] The denoising unit 112 is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates using the identity preservation path in the diffusion model, generate skin texture and lighting details in the denoising process based on the texture features using the texture synthesis path in the diffusion model, and fuse the output features of the identity preservation path and the texture synthesis path using the dynamic gating module in the diffusion model.

[0106] In this embodiment, the denoising unit 112 utilizes the identity preservation path in the diffusion model to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates, including:

[0107] The noise latent representation of the noise image is obtained in the key point attention layer in the identity preservation path. The attention mapping between the noise latent representation and the facial key point coordinates is calculated. The noise latent representation is enhanced based on the attention mapping to obtain the first intermediate feature map.

[0108] In the identity embedding fusion layer of the identity preservation path, the identity features and the first intermediate feature map are fused based on a two-layer fully connected network to obtain a second intermediate feature map;

[0109] In the identity preservation path, the structure-preserving residual block uses four cascaded residual units to adaptively modulate the second intermediate feature map; wherein each residual unit employs dynamic convolution.

[0110] In the above embodiments, the key point attention layer enables the network to focus on identity-sensitive regions such as eyes, nose, and mouth, and enhances the feature consistency of these regions during the denoising process; the identity embedding fusion layer enables the smooth injection of identity features; and the structure-preserving residual block enables adaptive feature modulation.

[0111] Specifically, calculating the attention mapping between the noise latent representation and the facial keypoint coordinates includes:

[0112] The attention map is calculated using the following formula:

[0113]

[0114] Where A represents the attention mapping; W q W k Represents the learnable projection matrix; z t The noise latent representation is represented by L; the facial key point coordinates are represented by d. k This represents the dimension of the key vector.

[0115] Specifically, the adaptive feature modulation of the second intermediate feature map using four cascaded residual units includes:

[0116] For each residual unit, the dynamic convolution is calculated using the following formula:

[0117]

[0118] Among them, Conv dyn (x) represents the dynamic convolution of each residual unit; x represents the input data of each residual unit, the input data corresponding to the first residual unit is the second intermediate feature map, and the input data of other residual units is the dynamic convolution output by the cascaded previous residual unit; π i (x) represents the mixed weights predicted based on x; i represents the index variable, with values ​​of 1, 2, and 3, corresponding to 3 different regular convolution operations Conv. i (x).

[0119] The above embodiments can enhance the consistency of features in identity-sensitive areas, making the replaced face more closely match the target identity in key identity feature areas (such as facial features, position, etc.).

[0120] In this embodiment, the denoising unit 112 utilizes the texture synthesis path in the diffusion model to generate skin texture and lighting details during the denoising process based on the texture features, including:

[0121] In the AdaIN (Adaptive Instance Normalization) modulation module of the texture synthesis path, the latent noise representation is normalized according to the texture features using the following formula to obtain a standard noise representation: Among them, AdaIN(z t σ(y) represents the standard noise representation, y represents the feature map of the source face image, σ(y) represents the channel-level standard deviation of y in the texture feature, and μ(z) represents the standard noise representation. t ) represents the z in the texture feature t The channel-level mean, σ(z) t ) represents the z in the texture feature t The channel-level standard deviation, μ(y) represents the channel-level mean of y in the texture feature;

[0122] The high-frequency enhancement block in the texture synthesis path captures the skin texture using parallel dilated convolution branches and regular convolution branches according to the following formula:

[0123] ;in, Represents the skin texture, Conv 1×1 DConv2 represents a 1×1 convolution operation, DConv4 represents a dilated convolution operation with a dilation rate of 2, DConv8 represents a dilated convolution operation with a dilation rate of 8, and Conv represents a regular convolution operation.

[0124] In the multilayer perceptron layer of the texture synthesis path, the lighting details are added to the skin texture using the following formula: in, M represents the output feature of the texture synthesis path. light and b light This represents the lighting details predicted based on the multilayer perceptron layer.

[0125] In the above embodiments, the AdaIN modulation module can standardize the latent representation of noise based on texture statistics, effectively capture fine texture features such as pores and wrinkles based on the multi-scale fusion mechanism of the high-frequency enhancement block, and further generate lighting details based on the multilayer perceptron layer, thereby helping to generate more realistic and natural textures, with lighting matching the target face environment, and improving the realism of textures after face replacement.

[0126] In this embodiment, the denoising unit 112 utilizes the dynamic gating module in the diffusion model to fuse the output features of the identity preservation path and the texture synthesis path, including:

[0127] The pixel-level fusion weights in the spatial dimension are predicted by a convolutional long short-term memory network based on the output features of the identity preservation path and the texture synthesis path.

[0128] At different U-Net layers, independent gating networks are applied based on a hierarchical fusion strategy to differentially enhance the pixel-level fusion weights; specifically, the weight of the texture synthesis path is increased for high-resolution shallow features, and the weight of the identity preservation path is increased for low-resolution deep features.

[0129] Based on the gated network, the temporal embedding vector of the diffusion time step is obtained, and the pixel-level fusion weight is adjusted in stages according to the temporal embedding vector using the gated function.

[0130] The weights obtained after differentiation enhancement and phased adjustments will be determined as the target weights.

[0131] The output features of the identity preservation path and the texture synthesis path are fused according to the target weight; wherein the target weight includes a first weight corresponding to the identity preservation path and a second weight corresponding to the texture synthesis path.

[0132] For example, the weight of high-resolution shallow features can be configured to 0.7, and the weight of low-resolution deep features can be configured to 0.6.

[0133] The fused features can be represented as follows: in, G represents the fused features. t1 G represents the first weight. t2 This represents the second weight. This represents the output feature of the identity preservation path.

[0134] The convolutional long short-term memory network can capture long-range dependencies while maintaining resolution in the spatial dimension, solving the pixel-level feature fusion problem. It generates independent fusion weights for each pixel, allowing the model to adaptively balance identity and texture features for different regions of the face (such as eyes and cheeks).

[0135] The hierarchical fusion strategy applies independent gated networks at different U-Net levels to form a hierarchical fusion strategy, which solves the feature fusion problem at the network level. Based on the pixel-level fusion weights, it utilizes the shallow (high resolution, texture sensitive) and deep (low resolution, identity sensitive) characteristics of U-Net to make biased adjustments to the pixel-level weights at different levels (e.g., shallow layers emphasize texture, and deep layers emphasize identity).

[0136] The temporal embedding vector addresses the feature fusion problem in the denoising stage. Based on pixel-level fusion weights and differentiation enhancement, it further introduces an embedding vector for the diffusion time step, allowing the fusion weights to dynamically change with the denoising process (such as initial denoising and later denoising), thereby adapting to the needs of identity preservation and texture synthesis at different stages.

[0137] Through the above embodiments, the features of the two paths can be dynamically and intelligently fused, so that the fusion result can balance the needs of identity and texture at different scales and different denoising stages, thereby improving the overall denoising quality.

[0138] The acquisition unit 113 is used to acquire the output data of the diffusion model when denoising is completed.

[0139] In this embodiment, the output data is a vector representation that needs to be further decoded to reconstruct an image.

[0140] The input unit 111 is also used to input the output data to the decoder to obtain a face fusion image.

[0141] In this embodiment, the face fusion image is the image obtained by fusing the source face image into the target image.

[0142] In this embodiment, the model training can adopt a progressive training strategy, such as phased training from 256×256 to 512×512, thereby utilizing more data and making the generated results more generalizable in terms of resolution.

[0143] In a 512×512 resolution face fusion task, this embodiment improves identity similarity by approximately 23% (based on the ArcFace metric) and texture realism by 18% (based on the LPIPS (Learned Perceptual Image Patch Similarity) metric) compared to existing methods, while maintaining real-time inference speed (approximately 0.5 seconds / frame on an NVIDIA V100 GPU (NVIDIA Volta V100 Graphics Processing Unit).

[0144] This embodiment can be used for face fusion in various fields.

[0145] For example, in the financial sector, security monitoring in financial institution branches may require enhancing or replacing faces in surveillance videos to simulate potentially risky facial features for drills or anomaly detection. This embodiment utilizes the identity preservation path to enhance the identity features of the target risky face, the texture synthesis path to restore realistic skin texture and lighting details, and the dynamic gating module to fuse a realistic simulated face for training a security recognition model. This improves the model's ability to recognize abnormal faces in complex scenes, thereby ensuring the security of financial venues.

[0146] For example, in healthcare settings, medical data sharing and research require the desensitization of medical images containing patient faces (such as surgical videos and physical examination images) while retaining necessary facial features for case association. Using this embodiment, the identity preservation path can adjust facial identity features to prevent them from corresponding to a real patient, while still preserving facial anatomical features to aid medical analysis. The texture synthesis path can reshape skin texture, and the dynamic gating module can fuse images that are both desensitized and retain the facial information needed for medical analysis, thus protecting patient privacy without affecting the reasonable use of medical data.

[0147] As can be seen from the above technical solutions, this invention can detect the coordinates of facial key points in the target image based on the facial key point detector, extract the identity features of the source face image based on the identity encoder, and calculate the texture features of the source face image based on the texture statistical extractor, so as to provide the data foundation for subsequent fusion. In the denoising process, the identity preservation path in the diffusion model is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates, and the texture synthesis path in the diffusion model is used to generate skin texture and lighting details in the denoising process based on the texture features. The single-path diffusion model is decomposed into two parallel and functionally specialized processing paths to avoid coupling conflicts between optimizing identity and texture at the same time. The dynamic gating module in the diffusion model is used to fuse the output features of the identity preservation path and the texture synthesis path, so that the fusion strategy can adapt to the needs of different denoising stages, thereby generating a high-quality face fusion image.

[0148] like Figure 3 The diagram shown is a structural schematic of a computer device that implements a preferred embodiment of the parallel denoising-based face fusion method of the present invention.

[0149] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a face fusion program based on parallel denoising.

[0150] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0151] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0152] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as code for a face fusion program based on parallel denoising, but also to temporarily store data that has been output or will be output.

[0153] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a face fusion program based on parallel denoising) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0154] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the parallel denoising-based face fusion method described above, for example... Figure 1 The steps are shown.

[0155] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a feature extraction unit 110, an input unit 111, a noise reduction unit 112, and an acquisition unit 113.

[0156] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the parallel denoising-based face fusion method described in the various embodiments of the present invention.

[0157] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0158] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0159] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0160] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0161] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0162] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0163] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0164] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0165] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0166] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0167] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a face fusion method based on parallel denoising, and the processor 13 can execute the multiple instructions to achieve the following:

[0168] In response to a face fusion instruction that fuses a source face image into a target image, the system detects the coordinates of facial key points in the target image based on a facial key point detector, extracts the identity features of the source face image based on an identity encoder, and calculates the texture features of the source face image based on a texture statistics extractor.

[0169] Noise is added to the target image to obtain a noisy image, and the noisy image is then input into an improved diffusion model for denoising.

[0170] In the denoising process, the identity preservation path in the diffusion model is used to inject the identity features into the denoising process of the noisy image based on the facial key point coordinates. The texture synthesis path in the diffusion model is used to generate skin texture and lighting details in the denoising process based on the texture features. The dynamic gating module in the diffusion model is used to fuse the output features of the identity preservation path and the texture synthesis path.

[0171] Once denoising is complete, acquire the output data of the diffusion model;

[0172] The output data is input into the decoder to obtain a face fusion image.

[0173] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0174] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0175] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0176] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0177] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0179] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0180] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0181] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A face fusion method based on parallel denoising, characterized in that, The face fusion method based on parallel denoising comprises: In response to a face fusion instruction for fusing a source face image into a target image, detecting face key point coordinates in the target image based on a face key point detector, extracting identity features of the source face image based on an identity encoder, and calculating texture features of the source face image based on a texture statistics extractor; Adding noise to the target image to obtain a noise image, and inputting the noise image into an improved diffusion model for denoising; In the denoising process, injecting the identity features into the denoising process of the noise image based on the face key point coordinates by using an identity preservation path in the diffusion model, and generating skin texture and lighting details in the denoising process according to the texture features by using a texture synthesis path in the diffusion model, and fusing output features of the identity preservation path and the texture synthesis path by using a dynamic gating module in the diffusion model; When the denoising is completed, obtaining output data of the diffusion model; Inputting the output data into a decoder to obtain a face fusion image; The method comprises the following steps:

2. The face fusion method based on parallel denoising according to claim 1, wherein, The U-Net backbone network of the diffusion model uses a Block block of Swin Transformer to replace an original convolution block; The U-Net backbone network comprises four down-sampling and up-sampling stages, each stage embedding a window-based multi-head self-attention module, a cross-scale feature skip connection, and a dynamic deep convolution residual block.

3. The face fusion method based on parallel denoising according to claim 1, wherein, The method comprises the following steps: The attention map is calculated by using the following formula: ； wherein A denotes the attention map; denotes a learnable projection matrix; denotes the noise latent representation; denotes the face keypoint coordinates; denotes the key vector dimension.

4. The face fusion method based on parallel denoising according to claim 1, wherein, The method comprises the following steps: For each residual unit, the dynamic convolution is calculated by using the following formula: ； wherein, denotes a dynamic convolution of each residual unit; denotes input data of each residual unit, the input data of the first residual unit is the second intermediate feature map, and the input data of other residual units is the dynamic convolution output by the last residual unit in cascade; denotes that the prediction is based on a predicted mixing weight; denotes an index variable, the value of is 1, 2, or 3, corresponding to three different normal convolution operations .

5. The face fusion method based on parallel denoising according to claim 3, wherein, The method comprises the following steps: In the AdaIN modulation module of the texture synthesis path, the latent noise representation is standardized according to the texture features using the following formula to obtain a standard noise representation: ;in, This represents the standard noise representation. This represents the feature map of the source face image. Indicating the texture features Channel-level standard deviation, Indicating the texture features Channel-level mean, Indicating the texture features Channel-level standard deviation, Indicating the texture features Channel-level mean; In the high-frequency enhancement block in the texture synthesis path, a parallel hollow convolution branch and a conventional convolution branch are utilized to capture the skin texture according to the following formula: ; wherein, represents the skin texture, represents a 1x1 convolution operation, represents a hollow convolution operation with a dilation rate of 2, represents a hollow convolution operation with a dilation rate of 4, represents a hollow convolution operation with a dilation rate of 8, represents a conventional convolution operation; In a multi-layer perceptron layer in the texture synthesis path, the following equation is used to add the lighting details to the skin texture: ; wherein, represents the output feature of the texture synthesis path, and represents the lighting details predicted from the multi-layer perceptron layer.

6. The face fusion method based on parallel denoising according to claim 1, wherein, The method comprises the following steps: The method comprises the following steps: predicting pixel-level fusion weights in a spatial dimension according to output features of the identity preserving path and the texture synthesis path through a convolutional long short-term memory network; applying independent gating networks based on a hierarchical fusion strategy to different U-Net levels to differentially enhance the pixel-level fusion weights; wherein the weight of the texture synthesis path is increased for high-resolution shallow features, and the weight of the identity preserving path is increased for low-resolution deep features; obtaining a time embedding vector of a diffusion time step based on the gating network, and using a gating function to stage adjust the pixel-level fusion weights according to the time embedding vector; determining the weights obtained after differential enhancement and stage adjustment as target weights; fusing output features of the identity preserving path and the texture synthesis path according to the target weights; wherein the target weights include first weights corresponding to the identity preserving path and second weights corresponding to the texture synthesis path.

7. A face fusion device based on parallel denoising, characterized in that, The face fusion device based on parallel denoising includes: A feature extraction unit is configured to, in response to a face fusion instruction of fusing a source face image into a target image, detect face key point coordinates in the target image based on a face key point detector, extract identity features of the source face image based on an identity encoder, and calculate texture features of the source face image based on a texture statistics extractor; An input unit is configured to add noise to the target image to obtain a noisy image, and input the noisy image to an improved diffusion model for denoising; A denoising unit is configured to, in a denoising process, inject the identity features into the denoising process of the noisy image based on the face key point coordinates using an identity preserving path in the diffusion model, and generate skin texture and lighting details in the denoising process according to the texture features using a texture synthesis path in the diffusion model, and fuse output features of the identity preserving path and the texture synthesis path using a dynamic gating module in the diffusion model; An acquisition unit is configured to, when the denoising is completed, acquire output data of the diffusion model; The input unit is further configured to input the output data to a decoder to obtain a face fusion image. The identity feature is injected into a denoising process of the noisy image based on the face key point coordinates by using an identity preserving path in the diffusion model, including: obtaining a noise latent representation of the noisy image at a key point attention layer in the identity preserving path, calculating an attention mapping between the noise latent representation and the face key point coordinates, and performing enhancement processing on the noise latent representation based on the attention mapping to obtain a first intermediate feature map; performing fusion processing on the identity feature and the first intermediate feature map based on a two-layer fully connected network at an identity embedding fusion layer in the identity preserving path to obtain a second intermediate feature map; and performing adaptive feature modulation on the second intermediate feature map by using four cascaded residual units at a structure preserving residual block in the identity preserving path; wherein each residual unit adopts dynamic convolution.

8. A computer device, comprising: The computer device comprises: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the face fusion method based on parallel denoising according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the face fusion method based on parallel denoising according to any one of claims 1 to 6.

Images