Image facial feature extraction and fusion method, system and memory

Abstract

The invention discloses an image facial feature extraction and fusion method and system and a memory, and the method comprises the steps: carrying out the feature extraction of a source role facial image, and generating a basic feature map and an identity feature vector; performing decoupling processing on the basic feature map by taking the identity feature vector as a control condition, and separating an expression feature channel and an illumination feature channel; inputting the decoupled expression feature channel and illumination feature channel into a scene perception gating network, and generating an adaptive fusion feature map in combination with the target scene image; executing Poisson fusion and generative adversarial refinement on the adaptive fusion feature map, and outputting an optimized fusion image; through a decoupling mechanism guided by an identity feature vector, texture distortion and illumination sensitivity are avoided, a scene perception gating network dynamically adjusts a feature fusion weight, cross-scene adaptability is remarkably improved, poisson-generative adversarial two-stage fusion introduces antagonism refinement on the basis of gradient domain optimization, boundary splicing traces are eliminated, and the fusion precision is improved. And high-fidelity and cross-scene facial feature fusion is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, and memory for extracting and fusing facial features from an image. Background Technology

[0002] Facial feature extraction and fusion technology, as a core branch of computer vision, has been widely applied in film and television production, virtual live streaming, short video special effects, and digital human development. Especially under the wave of film and television industrialization, this technology has become a key support for realizing the digital reproduction of characters and cross-scene shooting, significantly reducing the cost of live-action shooting and overcoming physical shooting limitations. With the explosion of emerging business models such as AI short dramas and real-time interactive advertising, the market demand for high-fidelity, high-efficiency facial fusion technology has surged, requiring it to stably generate film-quality images under complex lighting and multi-pose scenes.

[0003] Current technologies primarily rely on feature point matching mechanisms (such as SIFT / SURF algorithms) to achieve facial alignment and fusion by locating the coordinates of key facial points. However, this technology is extremely sensitive to changes in lighting and has fundamental limitations: highlight areas are prone to overexposure and loss of feature points, while shadow areas reduce feature contrast, leading to drift in facial feature positioning; lighting differences cause color block breaks in the fused skin texture (such as a harsh transition between the highlight area of ​​the bridge of the nose and the shadow area of ​​the cheek); in scenarios such as outdoor advertising shooting that require adaptation to real-time lighting changes, traditional methods require repeated manual corrections, which is inefficient.

[0004] Therefore, it is necessary to improve the existing technology for extracting facial features from images to solve the technical problem that the light sensitivity cannot meet the requirements for continuous shooting across different scenes. Summary of the Invention

[0005] The purpose of this invention is to provide an image facial feature extraction and fusion method, system, and memory to solve the above-mentioned technical problems.

[0006] To achieve this objective, the present invention adopts the following technical solution: A method for extracting and fusing facial features from an image, comprising the following steps: S1, extract features from the source character's facial image, and generate basic feature maps and identity feature vectors through a convolutional neural network model; S2, using the identity feature vector as a control condition, decouple the basic feature map to separate the expression feature channel and the illumination feature channel; S3, input the decoupled facial expression feature channel and illumination feature channel into the scene-aware gating network, and generate an adaptive fusion feature map by combining the target scene image; S4, perform Poisson fusion and generative adversarial refinement on the adaptive fusion feature map, and output the optimized fusion image.

[0007] Optionally, step S1 includes the following steps: S11, perform multi-scale segmentation on the source character's facial image to generate multi-scale hierarchical features that include global structural information and local detail information; S12, input the multi-scale hierarchical features into the convolutional neural network, extract detailed feature maps of skin texture and facial contours through the bottom convolutional layer, and extract semantic feature maps of facial muscle groups and facial relationships through the high-level convolutional layer. S13, Perform a global average pooling operation on the semantic feature map to generate an identity feature vector; S14, the detail feature map and the semantic feature map are concatenated along the channel dimension to generate a basic feature map.

[0008] Optionally, step S14 includes the following steps: S141, Channel attention weighting is performed on the detail feature map, a channel weight vector is generated through a fully connected layer, and a Hadamard product operation is performed with the detail feature map to output an optimized detail feature map; S142, Perform bilinear upsampling on the semantic feature map to improve its spatial resolution and optimize its detail features. Figure 1 To; S143 concatenates the optimized detail feature map and the upsampled semantic feature map along the channel dimension to generate the basic feature map.

[0009] Optionally, step S2 includes the following steps: S21, the identity feature vector is expanded in spatial dimension, and an identity control field with the same spatial size as the base feature map is generated by a convolutional layer. The identity control field and the base feature map are channel-concatenated to form a conditional constraint feature map. S22, the conditional constraint feature map is input into the expression decoupling generator. The expression decoupling generator adopts an encoder-decoder structure, wherein the encoder compresses the feature resolution by downsampling 3 times, the decoder recovers the resolution by upsampling transposed convolution, and embeds the identity invariance loss function in the skip connection to output the initial expression feature channel. S23, the condition constraint feature map is input into the lighting decoupling generator. In the lighting decoupling generator, a 9-dimensional spherical harmonic lighting coefficient is first generated, and then converted into a lighting parameter map through a differentiable rendering layer. The output includes lighting feature channels including directional light intensity, ambient light color temperature and shadow distribution.

[0010] Optionally, after step S23, the method further includes: S24, Perform temporal dynamic filtering on the initial facial expression feature channel: Use a convolutional neural network model to analyze the changes in facial expression features over 5 consecutive frames, generate a dynamic attention mask, and output a purified facial expression feature channel by suppressing identity-related static texture residues. S25: Construct the covariance matrix of facial expression and illumination features, and eliminate the correlation between channels through an orthogonalization algorithm. When the cosine similarity detection value is greater than 0.1, iterative orthogonalization is triggered until linearly independent facial expression feature channels and illumination feature channels are output.

[0011] Optionally, step S3 includes the following steps: S31 performs fast feature extraction on the target scene image and generates scene illumination distribution map, perspective parameter matrix and semantic segmentation mask through lightweight convolutional neural network; S32, perform spherical harmonic coefficient matching between the scene illumination distribution map and the decoupled illumination feature channel, and calculate the illumination migration weight matrix; S33, Perform an affine transformation on the facial expression feature channel based on the perspective parameter matrix to align the source facial expression pose with the perspective relationship of the target scene; S34 inputs the pose-aligned facial feature channels, the illumination transfer weight matrix, and the semantic segmentation mask into a gating fusion network, and generates an adaptive fusion feature map through a three-channel attention mechanism.

[0012] Optionally, step S4 includes the following steps: S41, extract the face region of the target scene image using a mask and generate a constraint mask for the fusion boundary; S42, Solve the Poisson equation based on the constraint mask, embed the gradient domain of the adaptive fusion feature map into the target scene image, and generate a primary fusion image; S43, the primary fused image is input into a generative adversarial network, the key point alignment accuracy is verified by a facial perspective consistency discriminator, and a refined fused image is output; the key points include the corners of the eyes, the wings of the nose, and the corners of the mouth. S44 performs bilateral filtering optimization on the skin transition area of ​​the refined fused image to eliminate texture stitching marks and output an optimized fused image.

[0013] The present invention also provides an image facial feature extraction and fusion system for implementing the image facial feature extraction and fusion method described above, the image facial feature extraction and fusion system comprising: The feature extraction module is configured to extract the basic feature map and identity feature vector of the source character's facial image through a convolutional neural network, including multi-scale pyramid segmentation, hierarchical convolutional feature extraction and channel attention fusion unit; The decoupling module is configured to separate the basic feature map into facial expression feature channels and lighting feature channels using the identity feature vector as the control condition, and integrates the facial expression adversarial decoupling unit, the physical lighting modeling unit, and the feature orthogonalization verification unit. The scene fusion module is configured to generate an adaptive fusion feature map by combining the target scene image, and has a built-in lightweight scene parser, spherical harmonic illumination transferer and three-channel gated fusion network. The optimization module is configured to perform Poisson fusion and generative adversarial refinement, including flexible boundary constraint elements, perspective consistency discriminators, and adaptive bilateral filters.

[0014] The present invention also provides a memory, including a program stored in the memory, the program being invoked to implement the image facial feature extraction and fusion method as described above.

[0015] Compared with existing technologies, this invention has the following advantages: First, it extracts the basic feature map and identity feature vector of the source character's facial image through a convolutional neural network; then, using the identity feature vector as a control condition, it decouples the basic feature map to separate the expression feature channel and the illumination feature channel; the decoupled expression and illumination features are input into a scene-aware gating network, and combined with the target scene image to generate an adaptive fusion feature map; finally, it optimizes the fusion feature map through Poisson fusion and generative adversarial refinement to output a high-quality fusion image; this method avoids texture distortion and illumination sensitivity through the decoupling mechanism guided by the identity feature vector, and the scene-aware gating network dynamically adjusts the feature fusion weights, significantly improving cross-scene adaptability; the Poisson-generative adversarial dual-level fusion introduces adversarial refinement on the basis of gradient domain optimization, eliminating boundary stitching traces and achieving high-fidelity, cross-scene facial feature fusion. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0018] Figure 1This is one of the flowcharts illustrating the image facial feature extraction and fusion method in this embodiment. Figure 2 This is the second flowchart illustrating the image facial feature extraction and fusion method in this embodiment. Detailed Implementation

[0019] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] In the description of this invention, it should be understood that the terms "upper," "lower," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. It should be noted that when a component is considered to be "connected" to another component, it can be directly connected to the other component or there may be a component positioned centrally in the connection.

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0022] Example 1: Combination Figures 1 to 2 As shown, this embodiment of the invention provides a method for extracting and fusing facial features from an image, including the following steps: S1 extracts features from the source character's facial image and generates a basic feature map and identity feature vector through a convolutional neural network model.

[0023] Convolutional neural networks (CNNs) are used to extract features from the facial images of the source character, aiming to fully preserve the structural information and texture details of the face.

[0024] S2 uses the identity feature vector as a control condition to decouple the basic feature map, separating the expression feature channel and the illumination feature channel; the decoupling mechanism helps to improve the flexibility and stability of the model in cross-scene expression transfer and illumination adaptation.

[0025] S3 inputs the decoupled facial expression feature channel and illumination feature channel into the scene-aware gating network, and generates an adaptive fusion feature map by combining the target scene image; compared with single-dimensional fusion, it better balances visual consistency and semantic continuity, thereby improving the naturalness and immersion of the fused image.

[0026] S4 performs Poisson fusion and generative adversarial refinement on the adaptive fusion feature map, outputting an optimized fused image that improves boundary transitions and detail realism. This multi-level optimization process significantly enhances the realism and detail integrity of face-swapped images compared to traditional fusion algorithms, effectively reducing visual artifacts and stitching artifacts.

[0027] The working principle of this invention is as follows: First, a convolutional neural network is used to extract the basic feature map and identity feature vector of the source character's facial image. Then, using the identity feature vector as a control condition, the basic feature map is decoupled to separate the expression feature channel and the illumination feature channel. The decoupled expression and illumination features are input into a scene-aware gating network and combined with the target scene image to generate an adaptive fusion feature map. Finally, the fusion feature map is optimized through Poisson fusion and generative adversarial refinement to output a high-quality fusion image. This method avoids texture distortion and illumination sensitivity through the decoupling mechanism guided by the identity feature vector. The scene-aware gating network dynamically adjusts the feature fusion weights, significantly improving cross-scene adaptability. The Poisson-generative adversarial dual-level fusion introduces adversarial refinement on the basis of gradient domain optimization to eliminate boundary stitching traces and achieve high-fidelity, cross-scene facial feature fusion.

[0028] In this embodiment, step S1 specifically includes the following steps: S11 performs multi-scale segmentation on the source character's facial image to generate multi-scale hierarchical features that include global structural information and local detail information.

[0029] First, the facial image of the source character undergoes multi-scale segmentation to capture structural information and detailed textures of the face at different spatial scales. Specifically, the original image is downsampled at multiple levels of spatial resolution (e.g., original image, 1 / 2 scale, 1 / 4 scale), preserving skin details at high resolution and contour structures at low resolution, respectively. This pyramid-like multi-scale representation helps the subsequent network to simultaneously consider both local minute features and global overall structure when processing images, improving its ability to perceive complex facial features.

[0030] S12 inputs multi-scale hierarchical features into the convolutional neural network, extracts detailed feature maps of skin texture and facial contours through the bottom convolutional layer, and extracts semantic feature maps of facial muscle groups and facial relationships through the high-level convolutional layer.

[0031] After multi-scale image features are input into a convolutional neural network, the network extracts information at different levels based on its depth. The lower convolutional layers (the first few layers closest to the input) primarily extract low-level visual features, such as skin texture, eyelid edges, and nose contours. These detailed feature maps accurately reflect the physical characteristics of a person's appearance. Higher convolutional layers (deeper layers) capture semantic relationships such as the interaction of facial muscles and the relative positions of facial features, extracting more abstract information about facial expressions and identity structure. This layered processing approach is a typical mechanism of convolutional neural networks, characterized by progressively expanding the receptive field and refining feature levels layer by layer.

[0032] S13, perform global average pooling on the semantic feature map to generate an identity feature vector. The acquired semantic feature map is then processed by global average pooling, which calculates the average value of all pixels in each channel, thus compressing the entire feature map into a fixed-length vector. This identity feature vector represents the overall representation of the input face image in the semantic space and can serve as the basis for identity information in subsequent generation and constraint processes. Compared to other methods such as max pooling, global average pooling has higher stability in maintaining the semantic continuity of the image and does not introduce excessively strong local response bias, which is beneficial for consistent modeling of identity features.

[0033] S14: Concatenate the detail feature map and the semantic feature map along the channel dimension to generate the basic feature map.

[0034] The system concatenates the detailed feature maps extracted from the lower-level convolutions with the semantic feature maps obtained from the higher-level convolutions along the channel dimension. Channel concatenation refers to "superimposing" the channels of two feature maps onto a new feature map while maintaining the same spatial resolution, thus forming a "base feature map" containing rich information. This fusion step allows subsequent modules to access both microscopic details such as skin texture and semantic information such as facial expressions and structures, laying a complete feature foundation for the decoupling and fusion stages.

[0035] In this embodiment, step S14 further includes the following steps: S141 performs channel attention weighting on the detail feature map, generates a channel weight vector through a fully connected layer, and performs a Hadamard product operation with the detail feature map to output an optimized detail feature map.

[0036] In this step, the system introduces a channel attention mechanism to weight and adjust the detail feature map, aiming to enhance the perception of key details. Specifically, the detail feature map is first input into a fully connected layer, and the global response of each channel is statistically analyzed to generate a channel-dimensional weight vector, with each value corresponding to the importance score of a feature channel. Then, the system performs element-wise multiplication (i.e., Hadamard product operation) of this channel weight vector with the original detail feature map, thereby suppressing low-response channels and highlighting high-response channels.

[0037] S142, perform bilinear upsampling on the semantic feature map to improve its spatial resolution and optimize detail features. Figure 1 To.

[0038] Since the semantic feature maps extracted in the previous stage have undergone downsampling, their spatial resolution is usually lower than that of the detail feature maps. To achieve spatially consistent fusion operations, this step uses a bilinear upsampling algorithm to enlarge the semantic feature maps to the same spatial size as the optimized detail feature maps. Bilinear upsampling is a common image interpolation method that improves image resolution without introducing artifacts by interpolating and filling the gaps by weighted averaging of adjacent pixel values. It also enhances edge continuity while maintaining the original structural features, making it suitable for semantic structure expansion processing of face regions.

[0039] S143 concatenates the optimized detail feature map and the upsampled semantic feature map along the channel dimension to generate the basic feature map.

[0040] Once the detail feature map and the semantic feature map are consistent in the spatial dimension, the system concatenates them in the channel dimension, that is, it connects the corresponding channel values ​​at each image location in sequence to form a new multi-channel feature map. This fusion method merges the original facial texture details and structural semantic information into a unified representation, which helps the subsequent feature decoupling module to perform targeted feature separation when generating expression and lighting information.

[0041] In this embodiment, step S2 specifically includes the following steps: S21, the spatial dimension of the identity feature vector is expanded, and an identity control field with the same spatial size as the basic feature map is generated through a convolutional layer. The identity control field and the basic feature map are then channel-concatenated to form a conditional constraint feature map.

[0042] The spatial dimension of the generated identity feature vector is expanded to transform the one-dimensional vector into a two-dimensional feature map with spatial structure, facilitating its fusion with the base feature map. Specifically, this is achieved by repeatedly padding or linearly transforming the vector values ​​to map them to an identity control field with the same spatial dimensions as the base feature map. This field is then passed through one or more convolutional layers to further adjust its channel distribution and semantic features. This mapping process is similar to the "conditional injection" mechanism in conditional generative models. Finally, the identity control field is concatenated with the original base feature map along the channel dimension (i.e., "channel cascading"), forming a composite feature map with identity constraint attributes—the conditional constraint feature map.

[0043] S22, Input the conditional constraint feature map into the expression decoupling generator. The expression decoupling generator adopts an encoder-decoder structure, in which the encoder compresses the feature resolution by downsampling 3 times, the decoder restores the resolution by upsampling through transposed convolution, and embeds the identity invariance loss function in the skip connection to output the initial expression feature channel. The conditional feature map is input into the expression decoupling generator, which employs a typical encoder-decoder structure. The encoder progressively compresses the spatial size of the feature map through three consecutive downsampling operations (such as strided convolution or pooling), enhancing its ability to abstractly understand facial expressions. The decoder, on the other hand, upsamples the feature map through transposed convolution to restore its spatial dimension. In each skip connection, an "identity invariance loss function" (such as using cosine similarity to measure the difference between the generated map and the original identity vector) is embedded to ensure that the generated expression features do not alter the person's identity. After this process, the output is the initial expression feature channel, i.e., a structured data map representing the state of facial muscle groups.

[0044] S23, input the condition constraint feature map into the lighting decoupling generator. In the lighting decoupling generator, first generate 9-dimensional spherical harmonic lighting coefficients, and then convert them into lighting parameter maps through a differentiable rendering layer. Output lighting feature channels including directional light intensity, ambient light color temperature and shadow distribution.

[0045] Similar to the expression decoupling process, this step processes the lighting information in the facial image. The system inputs the conditional constraint feature map into the lighting decoupling generator. This module first predicts a set of 9-dimensional spherical harmonic lighting coefficients based on a neural network. This is a common modeling method for natural lighting distribution, which can efficiently represent combinations of various light sources such as directional light and ambient light. Next, the system uses a differentiable rendering layer to convert these spherical harmonic coefficients into a lighting parameter map with spatial structure. This map encodes key attributes such as lighting direction intensity, ambient light color temperature, and shadow region distribution.

[0046] S24, Perform temporal dynamic filtering on the initial facial expression feature channel: Use a convolutional neural network model to analyze the changes in facial expression features over 5 consecutive frames, generate a dynamic attention mask, and output a purified facial expression feature channel by suppressing identity-related static texture residues.

[0047] Considering that static identity texture information can easily be mixed in when extracting facial expression features from a single frame image, a temporal dynamic filtering mechanism is introduced to enhance the time-varying characteristics of facial expression features and suppress identity residue. Specifically, facial expression feature maps from five consecutive frames are input into a convolutional neural network to analyze the dynamic change patterns between frames and generate a dynamic attention mask to highlight pixel regions with temporally changing characteristics. Simultaneously, the mask suppresses pixel regions that remain unchanged throughout the five frames, thereby eliminating static features related to identity texture.

[0048] S25: Construct the covariance matrix of facial expression and illumination features, and eliminate the correlation between channels through an orthogonalization algorithm. When the cosine similarity detection value is greater than 0.1, iterative orthogonalization is triggered until linearly independent facial expression feature channels and illumination feature channels are output.

[0049] It should be noted that the system first performs statistical analysis on the data of each channel in the facial expression feature channel and the illumination feature channel, constructing a covariance matrix to measure the joint variability among the channels. Based on this, a lightweight orthogonalization mechanism is introduced, calculating the cosine similarity of each feature channel pair as a similarity metric. If the cosine similarity value between any two channels exceeds 0.1 (i.e., there is a significant correlation), the system triggers an iterative orthogonalization process.

[0050] Iterative orthogonalization refers to continuously reducing the cross-correlation of feature channels by performing linear projection or basis vector reconstruction on the feature channels in each round until all expression channels and illumination channels are approximately orthogonal in the vector space (i.e., linearly independent). The final output linearly independent expression feature channels and illumination feature channels will have higher feature separation, providing a clearer and decoupled input source for the downstream fusion module.

[0051] In this embodiment, step S3 specifically includes the following steps: S31 performs fast feature extraction on the target scene image and generates a scene illumination distribution map, perspective parameter matrix, and semantic segmentation mask through a lightweight convolutional neural network.

[0052] The system rapidly processes target scene images to extract structural and lighting priors for subsequent fusion. To improve processing efficiency, a lightweight convolutional neural network analyzes the scene images and outputs three key features: first, a scene lighting distribution map describing global lighting direction and intensity; second, a perspective parameter matrix representing geometric information such as camera viewpoint and image projection relationships; and third, a semantic segmentation mask used to distinguish semantic structures such as human and background regions. This process not only provides necessary references for subsequent feature alignment and fusion but also balances operational efficiency with structural integrity.

[0053] S32, perform spherical harmonic coefficient matching between the scene illumination distribution map and the decoupled illumination feature channels, and calculate the illumination migration weight matrix.

[0054] The decoupled lighting feature channels are adaptively adjusted to match the lighting style of the target scene. Specifically, the scene lighting distribution map is transformed into a set of spherical harmonic coefficients (used to approximate the real lighting field), and these coefficients are matched with the spherical harmonic coefficients corresponding to the lighting features obtained from the decoupled source image. The matching process can employ minimum difference matching or regression optimization, generating a lighting transfer weight matrix based on the differences between the two coefficient sets. This matrix reflects how the current lighting should be spatially adjusted to transition from the source image style to the target scene style, providing lighting-level consistency for the final fusion.

[0055] S33 performs an affine transformation on the facial expression feature channels based on the perspective parameter matrix to align the source facial expression pose with the perspective relationship of the target scene.

[0056] Since the source facial image and the target scene may differ significantly in terms of camera angle, head pose, etc., this step performs an affine transformation on the decoupled facial feature channel using the previously extracted perspective parameter matrix. An affine transformation is a spatial mapping method that preserves linear structural relationships (such as lines and angles) and can be used for translation, rotation, and scaling adjustments of facial images. During execution, the system calculates the transformation coefficients based on the perspective matrix and resamples the facial feature map pixel by pixel to ensure that its geometric shape remains consistent with the facial structure in the target scene.

[0057] S34 inputs the pose-aligned facial feature channels, the illumination transfer weight matrix, and the semantic segmentation mask into a gating fusion network, and generates an adaptive fusion feature map through a three-channel attention mechanism.

[0058] In this step, the system inputs three key elements into the fusion module: aligned facial expression feature channels, an illumination transfer weight matrix, and a semantic segmentation mask. The fusion module employs a gated convolutional network structure with an integrated three-channel attention mechanism, weighted for facial expression, illumination, and semantic structure. Within each fusion unit, the network dynamically allocates computational resources and channel weights based on the importance of each channel to enhance key features and suppress conflicting information. The final output adaptive fusion feature map not only preserves the expressiveness of the source image but also achieves deep fusion with the target scene at the illumination and structural semantic levels.

[0059] In this embodiment, step S4 specifically includes the following steps: S41, extract the face region of the target scene image using a mask and generate a constraint mask for the fusion boundary.

[0060] First, the face region is located and a mask is extracted from the target scene image. This primarily serves to define the spatial range and boundary positions for subsequent fusion. Specifically, the foreground contours, including the eyes, nose, and mouth, are extracted from the person region information in the aforementioned semantic segmentation mask. Then, the edges of these regions are dilated or eroded to construct a fusion boundary constraint mask. This mask plays a role in "guiding the fusion range" and "constraining edge transitions" in subsequent gradient embedding operations, ensuring that the fusion content is only performed within the defined area, while preventing edge transitions from producing discontinuities or artifacts.

[0061] S42, based on the constraint mask, solves the Poisson equation and embeds the gradient domain of the adaptive fusion feature map into the target scene image to generate a primary fusion image.

[0062] By solving the Poisson equation, the system embeds the gradient information contained in the adaptive fusion feature map generated in the previous step into the corresponding position of the target scene image. This process is not a simple pixel overlay, but rather reconstructs pixel values ​​by calculating the gradient field inside the fusion region and combining it with boundary conditions (provided by the constraint mask of S41), thereby achieving a primary fusion image with natural transitions and continuous boundaries. Poisson fusion is a classic image processing method, often used for illumination consistency restoration and seamless edge matching, and is particularly suitable for handling skin tone gradients and shadow extensions in face images.

[0063] S43 inputs the initial fused image into the generative adversarial network, verifies the key point alignment accuracy through the facial perspective consistency discriminator, and outputs a refined fused image; the key points include the corners of the eyes, the wings of the nose, and the corners of the mouth.

[0064] To further improve image quality and structural realism, the system refines and optimizes the initial fused image input generative adversarial network (GAN). The generator is responsible for reconstructing natural textures, while the discriminator introduces a specific structural verification module—a facial perspective consistency discriminator. This module focuses on evaluating whether key facial features (such as the corners of the eyes, nose, and mouth) in the fused region are consistent with the perspective relationship of the face in the target scene. The generator is optimized by back-complementing the judgment results of key point alignment accuracy, ensuring that the final output image has high consistency and naturalness in both local structure and overall pose, avoiding any sense of incongruity such as misalignment or disproportion.

[0065] S44 performs bilateral filtering optimization on the skin transition area of ​​the refined fused image to eliminate texture stitching marks and output an optimized fused image.

[0066] While GANs can improve the structural consistency of fused images, slight texture stitching artifacts may still exist at skin texture transitions, manifesting as blurring, color blocks, or discontinuous textures. Therefore, this step employs a bilateral filtering algorithm to further optimize the skin transition regions of the refined image. Bilateral filtering is an image smoothing technique that simultaneously considers pixel spatial location and color similarity. It preserves edge details while suppressing unnecessary texture noise, enhancing the naturalness and consistency of skin tone gradients in facial images. The final optimized fused image output is visually more realistic, smooth, and seamless.

[0067] Example 2: The present invention also provides an image facial feature extraction and fusion system for implementing the image facial feature extraction and fusion method as described in Embodiment 1. The image facial feature extraction and fusion system includes: The feature extraction module is configured to extract the basic feature map and identity feature vector of the source character's facial image through a convolutional neural network, including multi-scale pyramid segmentation, hierarchical convolutional feature extraction, and channel attention fusion unit.

[0068] The decoupling module is configured to separate the basic feature map into facial expression feature channels and lighting feature channels using identity feature vectors as control conditions, and integrates facial expression adversarial decoupling unit, physical lighting modeling unit and feature orthogonalization verification unit.

[0069] The scene fusion module is configured to generate an adaptive fusion feature map by combining the target scene image, and has a built-in lightweight scene parser, spherical harmonic illumination transferer and three-channel gated fusion network.

[0070] The optimization module is configured to perform Poisson fusion and generative adversarial refinement, including flexible boundary constraint elements, perspective consistency discriminators, and adaptive bilateral filters.

[0071] Example 3: The present invention also provides a memory, including a program stored in the memory, the program being invoked to implement the image facial feature extraction and fusion method as described in Embodiment 1.

[0072] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting and fusing facial features from an image, characterized in that, Includes the following steps: S1, extract features from the source character's facial image, and generate basic feature maps and identity feature vectors through a convolutional neural network model; S2, using the identity feature vector as a control condition, decouple the basic feature map to separate the expression feature channel and the illumination feature channel; S3, input the decoupled facial expression feature channel and illumination feature channel into the scene-aware gating network, and generate an adaptive fusion feature map by combining the target scene image; S4, perform Poisson fusion and generative adversarial refinement on the adaptive fusion feature map, and output the optimized fusion image.

2. The image facial feature extraction and fusion method according to claim 1, characterized in that, Step S1 includes the following steps: S11, perform multi-scale segmentation on the source character's facial image to generate multi-scale hierarchical features that include global structural information and local detail information; S12, input the multi-scale hierarchical features into the convolutional neural network, extract detailed feature maps of skin texture and facial contours through the bottom convolutional layer, and extract semantic feature maps of facial muscle groups and facial relationships through the high-level convolutional layer. S13, Perform a global average pooling operation on the semantic feature map to generate an identity feature vector; S14, the detail feature map and the semantic feature map are concatenated along the channel dimension to generate a basic feature map.

3. The image facial feature extraction and fusion method according to claim 2, characterized in that, Step S14 includes the following steps: S141, Channel attention weighting is performed on the detail feature map, a channel weight vector is generated through a fully connected layer, and a Hadamard product operation is performed with the detail feature map to output an optimized detail feature map; S142, Perform bilinear upsampling on the semantic feature map to make its spatial resolution consistent with the optimized detail feature map; S143 concatenates the optimized detail feature map and the upsampled semantic feature map along the channel dimension to generate the basic feature map.

4. The image facial feature extraction and fusion method according to claim 1, characterized in that, Step S2 includes the following steps: S21, the identity feature vector is expanded in spatial dimension, and an identity control field with the same spatial size as the base feature map is generated by a convolutional layer. The identity control field and the base feature map are channel-concatenated to form a conditional constraint feature map. S22, the conditional constraint feature map is input into the expression decoupling generator. The expression decoupling generator adopts an encoder-decoder structure, wherein the encoder compresses the feature resolution by downsampling 3 times, the decoder recovers the resolution by upsampling transposed convolution, and embeds the identity invariance loss function in the skip connection to output the initial expression feature channel. S23, the condition constraint feature map is input into the lighting decoupling generator. In the lighting decoupling generator, a 9-dimensional spherical harmonic lighting coefficient is first generated, and then converted into a lighting parameter map through a differentiable rendering layer. The output includes lighting feature channels including directional light intensity, ambient light color temperature and shadow distribution.

5. The image facial feature extraction and fusion method according to claim 4, characterized in that, Following step S23, the following is also included: S24, Perform temporal dynamic filtering on the initial facial expression feature channel: Use a convolutional neural network model to analyze the changes in facial expression features over 5 consecutive frames, generate a dynamic attention mask, and output a purified facial expression feature channel by suppressing identity-related static texture residues. S25: Construct the covariance matrix of facial expression and illumination features, and eliminate the correlation between channels through an orthogonalization algorithm. When the cosine similarity detection value is greater than 0.1, iterative orthogonalization is triggered until linearly independent facial expression feature channels and illumination feature channels are output.

6. The image facial feature extraction and fusion method according to claim 5, characterized in that, Step S3 includes the following steps: S31 performs fast feature extraction on the target scene image and generates scene illumination distribution map, perspective parameter matrix and semantic segmentation mask through lightweight convolutional neural network; S32, perform spherical harmonic coefficient matching between the scene illumination distribution map and the decoupled illumination feature channel, and calculate the illumination migration weight matrix; S33, Perform an affine transformation on the facial expression feature channel based on the perspective parameter matrix to align the source facial expression pose with the perspective relationship of the target scene; S34 inputs the pose-aligned facial feature channels, the illumination transfer weight matrix, and the semantic segmentation mask into a gating fusion network, and generates an adaptive fusion feature map through a three-channel attention mechanism.

7. The image facial feature extraction and fusion method according to claim 1, characterized in that, Step S4 includes the following steps: S41, extract the face region of the target scene image using a mask and generate a constraint mask for the fusion boundary; S42, Solve the Poisson equation based on the constraint mask, embed the gradient domain of the adaptive fusion feature map into the target scene image, and generate a primary fusion image; S43, the primary fused image is input into a generative adversarial network, the key point alignment accuracy is verified by a facial perspective consistency discriminator, and a refined fused image is output; the key points include the corners of the eyes, the wings of the nose, and the corners of the mouth. S44 performs bilateral filtering optimization on the skin transition area of ​​the refined fused image to eliminate texture stitching marks and output an optimized fused image.

8. A system for extracting and fusing facial features from an image, characterized in that, For implementing the image facial feature extraction and fusion method as described in any one of claims 1 to 7, the image facial feature extraction and fusion system comprises: The feature extraction module is configured to extract the basic feature map and identity feature vector of the source character's facial image through a convolutional neural network, including multi-scale pyramid segmentation, hierarchical convolutional feature extraction and channel attention fusion unit; The decoupling module is configured to separate the basic feature map into facial expression feature channels and lighting feature channels using the identity feature vector as the control condition, and integrates the facial expression adversarial decoupling unit, the physical lighting modeling unit, and the feature orthogonalization verification unit. The scene fusion module is configured to generate an adaptive fusion feature map by combining the target scene image, and has a built-in lightweight scene parser, spherical harmonic illumination transferer and three-channel gated fusion network. The optimization module is configured to perform Poisson fusion and generative adversarial refinement, including flexible boundary constraint elements, perspective consistency discriminators, and adaptive bilateral filters.

9. A memory, characterized in that, Includes a program stored in the memory, the program being invoked to implement the image facial feature extraction and fusion method as described in any one of claims 1 to 7.

Images