Portrait video generation method and apparatus, device, and medium

Abstract

The present application relates to the technical field of image processing, and discloses a portrait video generation method, device, equipment and medium, comprising: performing identity feature coding on a pre-acquired user source identity image to obtain a pure identity code; performing hierarchical decoupling coding on a pre-acquired reference driving video to obtain a global posture code and a hierarchical expression code; performing local fine modeling on the hierarchical expression code to obtain a local expression code; performing multi-source feature adaptive reorganization on the pure identity code, the global posture code and the local expression code to obtain fusion generation features; and generating a target portrait video based on a controllable image generation network and the fusion generation features. The present application can be applied to the retail customer exclusive digital human service video generation scene of financial technology and the practicing physician exclusive health popular science video generation scene of medical technology, and improves the naturalness of the generated portrait video.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and image processing technology, and can be applied to the fields of financial technology and medical technology. In particular, it relates to a method, apparatus, device and medium for generating portrait videos. Background Technology

[0002] Portrait video generation is used to fuse the inherent identity features of a target individual with the head posture and facial expression dynamics information in a reference-driven video. This generates a continuous portrait video that fully preserves the target individual's inherent identity features while accurately reproducing the reference dynamics. This meets the production needs of customized digital human videos in various scenarios, avoiding the problems of high production costs, long production cycles, and insufficient customization flexibility inherent in traditional live-action shooting and editing methods. For example, in the financial sector, this technology can rapidly generate customized videos featuring the portraits of the target customer or physician, significantly improving the exclusivity, credibility, and compliance of service delivery while significantly reducing the production threshold and operating costs of professional and compliant content.

[0003] Currently, in the process of portrait video generation, the relevant facial feature extraction and video generation methods typically employ a holistic encoding and end-to-end direct generation model. This relies heavily on a single encoding network for the overall extraction of facial information. This not only leads to problems such as identity drift, entanglement of pose and expression features, and insufficient precision in local expression control in the generated portrait videos, but also results in poor naturalness of the generated content, weak temporal consistency across consecutive frames, and insufficient flexibility for customized adjustments. Therefore, improving the naturalness of the generated portrait videos has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides an artificial intelligence-based portrait video generation method, apparatus, computer equipment, and medium to improve the naturalness of the generated portrait videos.

[0005] Firstly, a portrait video generation method is provided, including: Obtain a pre-trained portrait video generation model, wherein the portrait video generation model includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network; Based on the identity encoding network, the pre-acquired user source identity image is encoded with identity features to obtain a clean identity code; Based on the driving coding network, the pre-acquired reference driving video is subjected to hierarchical decoupling coding to obtain global pose coding and hierarchical expression coding; Based on the region-aware attention network, the hierarchical expression coding is locally refined and modeled to obtain local expression coding. Based on the adaptive feature fusion network, the pure identity code, the global pose code, and the local expression code are adaptively recombined from multiple sources to obtain fused generated features. Based on the controllable image generation network and the fused generation features, a portrait video is generated to obtain the target portrait video.

[0006] Secondly, a portrait video generation device is provided, comprising: The pre-trained model retrieval module is used to obtain a pre-trained portrait video generation model, wherein the portrait video generation model includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network. The identity feature encoding module is used to encode the identity features of the pre-acquired user source identity image based on the identity encoding network to obtain a clean identity code; The driving video decoupling module is used to perform layered decoupling encoding on the pre-acquired reference driving video based on the driving encoding network to obtain global pose encoding and layered expression encoding. The decoupled facial expression refinement module is used to perform local fine-grained modeling of the hierarchical facial expression code based on the region-aware attention network to obtain local facial expression codes. An adaptive feature fusion module is used to adaptively recombine multi-source features from the clean identity code, the global pose code, and the local expression code based on the adaptive feature fusion network to obtain fused generated features. The portrait video generation module is used to generate a portrait video based on the controllable image generation network and the fused generation features to obtain a target portrait video.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the portrait video generation method described above.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the portrait video generation method described above.

[0009] In the aforementioned portrait video generation method, apparatus, computer equipment, and storage medium, a clean identity code is obtained by encoding the identity features of the pre-acquired user source identity image through an identity encoding network. This stably extracts and preserves the inherent identity information of the person whose portrait is to be generated, providing a reliable content foundation for video generation. Subsequently, a layered decoupling encoding is performed on the pre-acquired reference driving video through a driving encoding network to obtain global pose encoding and layered expression encoding. This achieves the separation and extraction of head pose information and facial expression information, providing a clearly categorized feature foundation for precise control of dynamic information. Finally, the layered expression encoding is locally refined through the region-aware attention network included in the driving encoding network. The model obtains local expression codes, enabling refined processing of facial expression features and effectively improving the control precision and independence of expression information. Then, through an adaptive feature fusion network, multi-source features are adaptively recombined from the pure identity code, global pose code, and local expression code to obtain fused generated features. This achieves accurate adaptation and orderly fusion of multiple types of features, ensuring the complete transmission and collaborative adaptation of various types of information. Finally, a portrait video is generated through a controllable image generation network and fused generated features to obtain the target portrait video. The output is a portrait video that retains the inherent identity information of the target person while completely reproducing the dynamic information corresponding to the reference driving video, greatly improving the fidelity and scene adaptability of the generated content and possessing outstanding practical value. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a portrait video generation method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a portrait video generation method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S10; Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S20; Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S30; Figure 6 This is a schematic diagram of a portrait video generation device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The portrait video generation method based on hierarchical decoupling provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can receive pre-acquired user source identity images and reference driving videos from the client, and perform feature decoupling extraction and fusion generation processing based on the user source identity images and reference driving videos using a pre-trained portrait video generation model. The portrait video generation model includes an identity encoding network, a driving encoding network, an adaptive feature fusion network, and a controllable image generation network. The driving encoding network includes a region-aware attention network. First, the server encodes the user source identity images using the identity encoding network to obtain clean identity codes. Then, based on the driving encoding network, it performs hierarchical decoupling encoding on the reference driving video to obtain global pose codes and hierarchical expression codes. Based on the region-aware attention network, it performs local fine-tuning modeling on the hierarchical expression codes to obtain local expression codes. Finally, based on the adaptive feature fusion network, it adaptively recombines multi-source features from the clean identity codes, global pose codes, and local expression codes to obtain fused generation features. Finally, based on the controllable image generation network and the fused generation features, it generates the portrait video, obtains the target portrait video, and sends the target portrait video back to the corresponding client. In this invention, targeting business entities such as digital human content production, customized services for financial institution clients, and medical and health science popularization, a dual-branch decoupled encoding of identity features and dynamic features is performed based on pre-acquired user source identity images and reference-driven videos. This yields mutually independent pure identity codes, global pose codes, and hierarchical expression codes. Then, a region-aware attention network is used to achieve regional-level refined modeling of facial expressions, transforming the hierarchical expression codes into independently controllable local expression codes. Finally, through multi-source feature adaptive recombination and generation processing, a high-fidelity target portrait video is obtained. This ensures that the generated portrait video fully preserves the identity features of the target person while accurately reproducing the pose and expression dynamics in the reference-driven video. This solves the problems of identity drift, feature entanglement, insufficient expression control precision, and low batch production efficiency in traditional portrait video generation technologies, improving the fidelity, controllability, and production efficiency of portrait video generation. It also optimizes the business implementation capabilities and user experience of customized digital human content production. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, and smart terminal devices equipped with facial image acquisition capabilities. The server-side can be implemented using a standalone server, a server cluster consisting of multiple servers, a cloud computing platform, or an enterprise-grade compliant private cloud server. The invention will now be described in detail through specific embodiments.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a portrait video generation method provided in an embodiment of the present invention includes the following steps: S10: Obtain a pre-trained portrait video generation model, which includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network. S20: Based on the identity coding network, the pre-acquired user source identity image is encoded with identity features to obtain a clean identity code; S30: Based on the driving coding network, the pre-acquired reference driving video is decoupled and encoded in layers to obtain global pose coding and layered expression coding; S40: Based on a region-aware attention network, local fine-grained modeling of hierarchical facial expression coding is performed to obtain local facial expression coding; S50: Based on an adaptive feature fusion network, multi-source features of pure identity encoding, global pose encoding and local expression encoding are adaptively recombined to obtain fused generated features; S60: Based on a controllable image generation network and fused generation features, a portrait video is generated to obtain the target portrait video.

[0015] In the embodiments of this application, steps S10 to S60 involve encoding the pre-acquired user source identity image using an identity encoding network to obtain a clean identity code. This stably extracts and preserves the inherent identity information of the person to be generated in the portrait, providing a reliable content foundation for video generation. Subsequently, a driving encoding network is used to perform hierarchical decoupling encoding on the pre-acquired reference driving video to obtain a global pose code and a hierarchical expression code. This achieves the separation and extraction of head pose information and facial expression information, providing a clearly categorized feature foundation for precise control of dynamic information. Finally, a region-aware attention network included in the driving encoding network is used to perform local fine-grained modeling of the hierarchical expression code to obtain a local expression code. The system employs facial expression encoding to achieve refined processing of facial expression features, effectively improving the control precision and independence of expression information. Then, an adaptive feature fusion network is used to adaptively recombine multi-source features from the pure identity encoding, global pose encoding, and local expression encoding to obtain fused generated features. This achieves precise adaptation and orderly fusion of multiple feature types, ensuring the complete transmission and collaborative adaptation of various information types. Finally, a controllable image generation network and fused generated features are used to generate portrait videos, resulting in target portrait videos. The output retains the inherent identity information of the target person while fully reproducing the dynamic information corresponding to the reference-driven video, significantly improving the fidelity and scene adaptability of the generated content, demonstrating outstanding practical value.

[0016] The portrait video generation method based on layered decoupling provided by this invention can be applied to intelligent video generation systems in the field of digital human content production. This intelligent video generation system is compatible with static facial images and dynamically driven video materials collected from various terminals, enabling the batch generation of high-fidelity, highly personalized portrait videos. For example, in the field of retail financial services in commercial banks, banking institutions often need to generate one-on-one personalized service notification videos for existing individual customers for standardized service scenarios such as updating wealth management rights, account security reminders, and compliance clause notifications. This often requires the use of an intelligent video generation system to fully preserve the customer's portrait characteristics while adapting to the bank's standardized service scripts, actions, and expressions, ensuring the compliance, friendliness, and personalization of the service content, while significantly reducing video production cycles and content production costs.

[0017] For example, the retail banking department of a national joint-stock commercial bank plans to generate personalized annual wealth management benefit update notification videos for its 120,000 existing customers holding platinum-level debit cards. After obtaining the customer's signed authorization ID card photo and a standard script-driven video of the annual benefit notification recorded by a certified top-tier customer service representative from the bank's headquarters, the bank immediately input both materials into its compliant private cloud-based intelligent video generation system. The intelligent video generation system then uses a built-in identity encoding network to encode the customer's authorized facial features, extracting a pure identity code that retains only the customer's inherent identity information. Simultaneously, it uses a built-in driver encoding network to perform layered decoupling encoding on the standard script-driven video, separating the global posture code representing the customer service representative's standardized head movements, as well as the service... The system first performs hierarchical facial expression encoding based on overall facial information. Then, through a region-aware attention network built into the driving encoding network, it performs local fine-grained modeling of the hierarchical facial expression encoding to obtain independent local facial expression codes for facial regions such as eyebrows, eyes, and mouth. Finally, based on the extracted multi-class features, the intelligent video generation system uses an adaptive feature fusion network to adaptively recombine the customer's pure identity code with the global posture code and local expression code of standard services to obtain fused generated features. Then, through a controllable image generation network, it generates a unique portrait video frame by frame, which fully reproduces the customer's portrait, standard customer service actions, and standard service expressions.

[0018] In step S10 of some embodiments, the portrait video generation model refers to an end-to-end deep learning network model that has been pre-trained and optimized with large-scale face data and has the ability to decouple and extract face features, fuse multi-source features, and generate portrait video images. For example, in a financial scenario, the model may be a digital human image generation model deployed in a bank's intelligent customer service system, and in a medical scenario, the model may be a virtual doctor image generation model used in online diagnosis and treatment scenarios.

[0019] An identity encoding network is a deep learning encoding sub-network in a portrait video generation model that is specifically responsible for extracting the inherent identity features of a person in a facial image. It is one of the core components of a dual-branch encoding architecture. For example, in a financial scenario, this network can be a dedicated encoding module for extracting the facial identity features of a customer, while in a medical scenario, it can be a dedicated encoding module for extracting the facial identity features of a doctor.

[0020] The driving encoding network refers to a deep learning encoding sub-network in a portrait video generation model that is specifically responsible for extracting head pose features and facial expression features from the driving video. It is another core component of the dual-branch encoding architecture and integrates a region-aware attention network. For example, in a financial scenario, this network can be an encoding module used to extract action and expression features from driving videos of standard intelligent customer service scripts. In a medical scenario, this network can be an encoding module used to extract action and expression features of doctors in medical science popularization videos.

[0021] Region-aware attention networks are deep learning functional modules embedded within driver coding networks, specifically responsible for refining local regions of hierarchical facial expression coding. They are used to independently extract and optimize facial expression features from different facial regions. For example, in financial scenarios, this network can be a refined processing module for accurately adapting standardized smiling expressions required for financial service scenarios. In medical scenarios, this network can be a refined processing module for adapting gentle and reassuring expressions required for doctor-patient communication scenarios.

[0022] Adaptive feature fusion network refers to a deep learning functional module in a portrait video generation model that is specifically responsible for dimensional adaptation, weight allocation, and fusion and recombination of identity encoding, posture encoding, and facial expression encoding. It is used to generate fused features that can be directly used for video image generation. For example, in financial scenarios, this network can be a fusion module for fusing customer identity features with standard customer service action and facial expression features. In medical scenarios, this network can be a fusion module for fusing doctor identity features with medical science popularization action and facial expression features.

[0023] A controllable image generation network refers to a deep learning generation sub-network in a portrait video generation model that is specifically responsible for generating and temporally stitching portrait video frames based on fused generation features. It is the final output unit of the model. For example, in a financial scenario, this network can be a generation module for generating personalized digital human customer service video frames, and in a medical scenario, it can be a generation module for generating personalized virtual diagnosis and popular science video frames for doctors.

[0024] This application embodiment can obtain a portrait video generation model by calling a portrait video generation model that has completed full-process supervised training and parameter solidification in advance. It should be noted that the portrait video generation model integrates multiple sub-networks with clear functional boundaries and working together. Among them, the identity encoding network and the driving encoding network form a dual-branch parallel encoding architecture, which are responsible for the independent extraction of identity features and dynamic features, respectively. The region-aware attention network is embedded in the driving encoding network and is specifically responsible for the fine processing of facial expression features. The adaptive feature fusion network is responsible for connecting the encoding end and the generation end, and completing the adaptation and recombination of multi-source features. The controllable image generation network is responsible for the generation and output of the final video image. The model can be stored on a local terminal device or a cloud server according to the computing power requirements and deployment requirements of the actual application scenario. When called, the parameters can be directly loaded and initialized.

[0025] It should also be noted that, in the embodiments of this application, a pre-built model architecture can be trained using sample data to obtain a trained portrait video generation model.

[0026] In detail, such as Figure 3 As shown, the training method for the portrait video generation model in step S10 includes the following steps: S11: Obtain the sample face image set and the sample-driven video set. The sample face image set includes identity annotation information and pose annotation information, and the sample-driven video set includes expression annotation information and real video frame annotation information. S12: Based on the preset initial portrait video generation model, perform feature processing on the sample face image set and the sample-driven video set to obtain the sample clean identity code, the sample global pose code, the sample local expression code, the sample fusion generation feature, and the sample generated portrait video frame. S13: Based on the sample pure identity encoding, sample global pose encoding, sample local expression encoding, sample fusion generated features, sample generated portrait video frames, identity annotation information, pose annotation information, expression annotation information and real video frame annotation information, the loss is calculated to obtain the total loss value of the model; S14: Update the parameters of the preset initial portrait video generation model based on the total loss value of the model to obtain the portrait video generation model.

[0027] In step S11 of some embodiments, the sample face image set refers to a collection of a large number of static face images with annotation information used for training the portrait video generation model. It is the basic data carrier for the model to learn facial identity features and pose features. For example, in a financial scenario, the sample face image set can be a collection of static face images of different customers with different head poses collected by a banking institution. In a medical scenario, the sample face image set can be a collection of static face images of different practicing physicians with different head poses collected by a medical institution.

[0028] A sample-driven video set refers to a collection of a large number of labeled facial dynamic videos used for training portrait video generation models. It is the basic data carrier for the model to learn the dynamic features of facial expressions and the logic of video frame generation. For example, in the financial scenario, the sample-driven video set can be a collection of facial dynamic videos corresponding to the standard customer service process of a bank. In the medical scenario, the sample-driven video set can be a collection of facial dynamic videos corresponding to the diagnosis and popular science explanation of a licensed physician.

[0029] Identity labeling information refers to the labeled data attached to sample face images, which is used to uniquely identify the inherent identity attributes of the person corresponding to the image. It is the benchmark information that constrains the accuracy of identity feature extraction during model training. For example, in the financial scenario, identity labeling information can be the identity attribute label corresponding to the unique identity number of a bank customer. In the medical scenario, identity labeling information can be the identity attribute label corresponding to the practicing physician's license number.

[0030] Pose annotation information refers to the annotation data attached to sample face images to identify the rigid motion states of the corresponding person's head, such as rotation, translation, and scale changes. It is the benchmark information that constrains the accuracy of global pose feature extraction during model training. For example, in the financial scenario, pose annotation information can be the annotation of head motion parameters corresponding to the standard service posture of customer service. In the medical scenario, pose annotation information can be the annotation of motion parameters corresponding to the natural head posture of doctors when giving popular science lectures.

[0031] Facial expression annotation information refers to the labeled data attached to the sample-driven video, used to identify the facial expression movement state of different regions of the person's face in the video frame. It is the benchmark information that constrains the accuracy of facial expression feature extraction during model training. For example, in the financial scenario, facial expression annotation information can be the facial expression parameter annotation corresponding to the standard bank service smile. In the medical scenario, facial expression annotation information can be the facial expression parameter annotation corresponding to the gentle and reassuring expression during doctor-patient communication.

[0032] Real video frame annotation information refers to pixel-level annotation data attached to sample-driven videos to identify the original real images of the videos. It serves as the benchmark information for constraining the accuracy of video frame generation during model training. For example, in financial scenarios, real video frame annotation information can be the original high-definition image annotation of a bank's standard customer service video, while in medical scenarios, it can be the original high-definition image annotation of a doctor's popular science explanation video.

[0033] In this embodiment, static facial images and dynamic facial videos covering different ages, genders, and lighting environments can be collected according to the application scenario requirements of the model. The static facial images need to cover multiple head poses and basic expressions of the same person, and the dynamic facial videos need to cover continuous changes in head pose and facial expression. Subsequently, the collected static facial images are labeled, adding unique corresponding identity label information and head pose label information to each image. The collected dynamic facial videos are labeled frame by frame, adding corresponding facial expression label information and original real video frame label information to each video frame. Finally, a sample facial image set and sample-driven video set that meet the training requirements are constructed. For example, in a financial scenario, static facial images of different bank customers and dynamic videos of standard customer service can be collected to complete the sample set construction. In a medical scenario, static facial images of different practicing physicians and dynamic videos of medical science popularization can be collected to complete the sample set construction.

[0034] In step S12 of some embodiments, the preset initial portrait video generation model refers to a deep learning network model that has completed the network architecture construction but has not been trained and optimized, and has basic capabilities for facial feature extraction and fusion and video generation. It is the initial carrier for model training. For example, in the financial scenario, the preset initial portrait video generation model can be the initial deep learning network built for the bank's digital human customer service scenario. In the medical scenario, the preset initial portrait video generation model can be the initial deep learning network built for the medical virtual science popularization scenario.

[0035] Pure identity encoding of samples refers to the feature vector output by the preset initial portrait video generation model after extracting features from the sample face image set during the training process, which contains only the inherent identity attributes of the person.

[0036] Sample-generated portrait video frames refer to single-frame portrait images generated by a preset initial portrait video generation model based on sample fusion during the training process.

[0037] In this embodiment, the constructed sample face image set is first input into the model's identity encoding network. The identity encoding network extracts and eliminates interference from the inherent identity features of the face to obtain the pure identity code of the sample. Simultaneously, the constructed sample-driven video set is input into the model's driving encoding network. The driving encoding network performs hierarchical decoupling extraction of head pose features and facial expression features to obtain the global pose code and hierarchical expression code of the sample. Then, the region-aware attention network embedded in the driving encoding network performs local fine-tuning modeling of the hierarchical expression code to obtain the local expression code of the sample. Subsequently, the pure identity code, global pose code, and local expression code of the sample are input into the model's adaptive feature fusion network to complete the adaptive recombination of multi-source features to obtain the sample fusion generated features. Finally, the sample fusion generated features are input into the model's controllable image generation network to generate a single frame portrait image to obtain the sample generated portrait video frame.

[0038] In step S13 of some embodiments, the total model loss value refers to a value calculated by a preset loss function, which is used to quantify the degree of deviation between the model training output and the corresponding labeled information. It is the core basis for optimizing model parameters. For example, in the financial scenario, the total model loss value can be a value that quantifies the deviation between the digital human customer service generation effect and the standard requirements. In the medical scenario, the total model loss value can be a value that quantifies the deviation between the virtual doctor popular science video generation effect and the real picture.

[0039] In this embodiment, the pure identity code of the sample is first compared with the corresponding identity annotation information to calculate the identity preservation loss value. The global pose code of the sample is compared with the corresponding pose annotation information to calculate the pose constraint loss value. The local expression code of the sample is compared with the corresponding expression annotation information to calculate the expression decoupling loss value. The portrait video frame generated by the sample is compared with the corresponding real video frame annotation information to calculate the image reconstruction loss value. At the same time, the feature consistency loss value is calculated based on the sample fusion generation features. Then, a corresponding preset weight coefficient is assigned to each sub-loss value, and the weighted summation of each sub-loss value is calculated to finally obtain the total model loss value used for model optimization. For example, in the financial scenario, the weight coefficient of the identity preservation loss value can be increased to ensure that the generated digital human customer service is highly consistent with the customer's identity. In the medical scenario, the weight coefficient of the expression decoupling loss value can be increased to ensure that the generated virtual doctor's expression meets the professional requirements of the diagnosis and treatment scenario.

[0040] In step S14 of some embodiments, the calculated total model loss value is first backpropagated, and the gradient values ​​corresponding to the weight parameters of each network layer are calculated layer by layer along the network architecture of the model. Then, based on the preset optimization algorithm and learning rate parameters, all weight parameters of the model are iteratively adjusted according to the calculated gradient values ​​to reduce the total model loss value and narrow the deviation between the model output result and the labeled benchmark information. The process of forward feature processing, loss calculation and backpropagation parameter update is repeated until the total model loss value converges to the preset threshold range and the generation effect of the model meets the preset application requirements. Finally, the training is completed and the portrait video generation model is obtained.

[0041] Steps S11 to S14 of this embodiment first obtain a set of sample face images and a set of sample-driven videos with multi-dimensional annotation information, providing accurate benchmark data and constraints for model training, ensuring the traceability and controllability of the training process. Then, the sample set is processed by a preset initial portrait video generation model, fully reproducing the entire process logic of portrait video generation, providing a full-link processing object for effect constraints during training. Next, loss calculation is performed based on the multi-dimensional model output and corresponding annotation information to obtain the total loss value of the model, realizing quantitative constraints on the full-link effect of model identity feature extraction, posture and expression feature decoupling, feature fusion and image generation, which can accurately locate the optimization direction of the model. Finally, the model parameters are updated based on the total loss value of the model, realizing iterative optimization of model performance, ensuring that the trained portrait video generation model has stable identity fidelity, accurate posture and expression control, and high-quality video generation capabilities.

[0042] In step S20 of some embodiments, the user source identity image refers to a single static image containing the facial information of the target person to be generated as a portrait. It is the source of identity information for generating the target portrait video. For example, in a financial scenario, the user source identity image may be a facial photo of the bank customer's ID card or a facial image collected on-site. In a medical scenario, the user source identity image may be a facial photo of the practicing physician's professional certificate or a facial image collected in the work environment.

[0043] Pure identity encoding refers to a feature vector extracted through an identity encoding network that contains only the inherent identity attributes of the target person and is completely unrelated to dynamic information such as facial expressions and postures. It is the content basis for generating portrait videos. For example, in a financial scenario, pure identity encoding can be a feature vector that only contains the inherent features of the customer's facial features and face shape. In a medical scenario, pure identity encoding can be a feature vector that only contains the inherent features of the doctor's facial features and face shape.

[0044] This application embodiment can perform face region detection and key point alignment processing on the pre-acquired user source identity image to remove background interference information in the image and obtain a standard face region image that meets the model input standard. Then, the standard face region image is input into the identity coding network, and the inherent attribute features of the person are extracted layer by layer through a multi-layer convolutional neural network to obtain an inherent identity feature vector containing inherent attributes such as the person's face shape, facial features, skin color, and hairstyle. Finally, interference elimination processing is performed on the inherent identity feature vector to filter out non-fixed attribute information such as facial expressions and postures, resulting in a pure identity code that is only related to the inherent identity of the person and is completely unrelated to dynamic information.

[0045] In detail, such as Figure 4 As shown, in step S20, which is based on the identity encoding network, the pre-acquired user source identity image is encoded with identity features to obtain a clean identity code, including the following steps: S21: Perform face region detection on the user's source identity image to obtain a standard face region image; S22: Based on the identity coding network, inherent attribute features are extracted from standard face region images to obtain inherent identity feature vectors; S23: Perform interference removal processing on the inherent identity feature vector to obtain a pure identity code.

[0046] In step S21 of some embodiments, the standard face region image refers to a face region image that conforms to the input standard of the identity coding network after face region detection, key point alignment and size normalization. It is the direct input carrier for identity feature extraction. For example, in the financial scenario, the standard face region image can be a fixed-size frontal face image obtained by standardizing a bank customer's face photo. In the medical scenario, the standard face region image can be a fixed-size frontal face image obtained by standardizing a practicing physician's face photo.

[0047] This application embodiment can first call a pre-trained face detection model to perform a full-screen scan of the pre-acquired user source identity image, locate the coordinates of the face region and key facial points in the image, then crop out the effective region containing only the face from the source image based on the located face coordinates, and then align the cropped face region based on the key facial points to correct the face's deflection angle and scale differences. Finally, the aligned face region is normalized in size and standardized in pixel value to obtain a standard face region image that meets the input specifications of the identity encoding network. For example, in a financial scenario, face photos with environmental backgrounds collected on-site by bank customers can be detected, cropped, and aligned to obtain a standard face region image that meets the requirements. In a medical scenario, face photos with office environment uploaded by licensed physicians can be detected, cropped, and aligned to obtain a standard face region image that meets the requirements.

[0048] In step S22 of some embodiments, the inherent identity feature vector refers to a multi-dimensional feature vector that is obtained after the inherent attribute feature extraction and contains inherent identity attributes such as facial features, skin color and hairstyle. It is a digital carrier of inherent identity information of a person. For example, in the financial scenario, the inherent identity feature vector may be a multi-dimensional feature vector containing the inherent facial features of the bank customer. In the medical scenario, the inherent identity feature vector may be a multi-dimensional feature vector containing the inherent facial features of the practicing physician.

[0049] In this embodiment, a preprocessed standard face region image can be input into an identity encoding network. Through the multi-layer convolutional neural network structure of the network, feature extraction and dimensionality compression are performed on the input image layer by layer. During the feature extraction process, the network focuses on fixed attribute information such as facial features, skin color, and hairstyle that do not change with facial expressions and postures, and gradually filters out shallow texture details and non-fixed attribute information. Finally, at the encoding output layer of the network, a multi-dimensional feature vector containing all the core inherent identity attributes of the person is output, namely the inherent identity feature vector. For example, in a financial scenario, the identity encoding network can extract feature vectors that are only related to the inherent facial attributes of the customer from the standard face region image of the customer. In a medical scenario, the identity encoding network can extract feature vectors that are only related to the inherent facial attributes of the doctor from the standard face region image of the doctor.

[0050] In step S23 of some embodiments, the extracted inherent identity feature vector is input into the feature optimization layer of the identity encoding network. Through a pre-trained decoupling constraint mechanism, interference information such as facial expressions, postures, lighting, and shooting angles that are not related to fixed identities is identified and filtered. At the same time, feature constraints ensure that the processed feature vector retains only the core information related to the inherent identity of the person, eliminating the interference of non-identity information on the subsequent generation process. Finally, a pure identity code that is completely unrelated to dynamic information such as facial expressions and postures and only represents the inherent identity attributes of the person is output. For example, in a financial scenario, interference information such as smiling expressions and slight head tilts during shooting can be filtered out from the inherent feature vector of the customer to obtain a pure identity code that only represents the customer's identity. In a medical scenario, interference information such as indoor lighting differences and facial expression changes during shooting can be filtered out from the inherent feature vector of the doctor to obtain a pure identity code that only represents the doctor's identity.

[0051] Steps S21 to S23 as illustrated in this embodiment first obtain a standard face region image by performing face region detection on the user's source identity image. This effectively eliminates background interference and non-standardized factors in the source image, providing a unified and standardized input basis for subsequent identity feature extraction and improving the stability and consistency of the feature extraction process. Subsequently, the inherent attribute features of the standard face region image are extracted through an identity coding network, accurately locking the inherent identity attributes of the person that do not change with dynamic information and the external environment. This provides a core digital carrier for the complete preservation and transmission of the person's identity information. Finally, by performing interference elimination processing on the inherent identity feature vector, non-identity interference information attached to the feature vector is effectively filtered out, resulting in a pure identity code that is only related to the person's inherent identity. This fundamentally avoids the identity drift problem that may occur in the subsequent portrait video generation process and greatly improves the identity fidelity of the generated portrait video.

[0052] In step S30 of some embodiments, the reference driving video refers to a continuous video file containing dynamic information of head posture and facial expression to be transferred. It is the source of action and expression control information for generating the target portrait video. For example, in a financial scenario, the reference driving video may be a demonstration video of actions and expressions corresponding to standard customer service scripts of a bank. In a medical scenario, the reference driving video may be a demonstration video of actions and expressions of doctors corresponding to health science popularization.

[0053] Global pose encoding refers to the feature vector extracted by the driving encoding network, which represents the rigid motion information such as the overall rotation, translation and scale changes of the target person's head. It is used to control the overall head posture of the generated portrait. For example, in the financial scenario, global pose encoding can be a feature vector representing the head rotation information of a customer service representative in a standard sitting posture. In the medical scenario, global pose encoding can be a feature vector representing the natural head swaying information of a doctor when giving a popular science lecture.

[0054] Hierarchical facial expression coding refers to a combined feature vector extracted through a driving coding network, which contains global facial expression context information and preliminary facial expression features of each facial region. It is the processing object for local fine-grained modeling. For example, in a financial scenario, hierarchical facial expression coding can be a vector containing overall information of a customer service representative's standard service expression and preliminary features of each facial region. In a medical scenario, hierarchical facial expression coding can be a vector containing overall information of a doctor's expression during a science popularization explanation and preliminary features of each facial region.

[0055] This application embodiment can perform frame-by-frame face detection and key point alignment processing on the pre-acquired reference driving video to remove background interference information in each frame, thereby obtaining a standardized face driving image set that conforms to the model input standard. The standardized face driving image set is then input into the driving coding network, and head motion features are extracted layer by layer through a dedicated pose coding branch to obtain a global pose code that represents the overall rotation, translation and scale changes of the head. At the same time, a dedicated expression coding branch is used to extract global facial expression features and decompose them layer by layer from the standardized face driving images to obtain a layered expression code that includes global expression context information and preliminary expression features of each facial region.

[0056] In detail, such as Figure 5 As shown, in step S30, which is based on the driving coding network, the pre-acquired reference driving video is subjected to hierarchical decoupling coding to obtain global pose coding and hierarchical expression coding, including the following steps: S31: Perform face detection on the reference driving video frame by frame to obtain a standard face driving image set; S32: Based on the driving coding network, head motion features are extracted from the standard face driving image set to obtain global pose coding; S33: Based on the driving coding network, the expression feature decomposition of the standardized face driving image is performed to obtain the hierarchical expression coding.

[0057] In step S31 of some embodiments, the standard face-driven image set refers to a continuous set of face images that conforms to the input standard of the driving coding network, obtained after face detection, alignment and normalization are performed frame by frame on the reference driving video. It is the direct input carrier for extracting head motion features and expression features. For example, in the financial scenario, the standard face-driven image set can be a continuous set of standardized face images obtained after processing a bank customer service demonstration video. In the medical scenario, the standard face-driven image set can be a continuous set of standardized face images obtained after processing a doctor's popular science video.

[0058] In this embodiment, the pre-acquired reference driving video is first parsed to extract consecutive single frames from the video at a fixed frame rate, forming an original video frame set. Then, a pre-trained face detection and key point recognition model is called to perform a full-screen scan of each frame in the original video frame set, locating the coordinates of the face region and facial key points, removing background interference information from the image, and then aligning the face region in each frame based on the facial key points to correct the face's deflection angle, scale differences, and positional offset. Finally, all aligned face images are normalized in size and standardized in pixel value to form a standard face driving image set with uniform inter-frame specifications that meets the input requirements of the driving coding network. For example, in a financial scenario, frame extraction and frame-by-frame face standardization can be performed on a standard customer service demonstration video of a bank to obtain a standard face driving image set that meets the requirements. In a medical scenario, frame extraction and frame-by-frame face standardization can be performed on a doctor's diagnosis and treatment popular science demonstration video to obtain a standard face driving image set that meets the requirements.

[0059] In step S32 of some embodiments, the preprocessed standard face-driven image set is input into the pose encoding branch of the driving encoding network. Through the multi-layer convolutional neural network structure within the branch, the face region in the continuous frames is extracted layer by layer. The focus is on rigid motion information such as the overall rotation, translation, and scale changes of the head, while filtering out interference from flexible non-rigid information such as facial expression changes. At the same time, through inter-frame temporal correlation modeling, the continuity and consistency of head motion features between consecutive frames are ensured. Finally, at the output layer of the pose encoding branch, a multi-dimensional feature vector representing the overall rigid motion state of the head is output, i.e., global pose encoding. For example, in a financial scenario, the driving encoding network can extract the global pose encoding representing the head motion state under the standard service posture of the customer service representative from the standard face-driven image set of the bank customer service demonstration video. In a medical scenario, the driving encoding network can extract the global pose encoding representing the natural head motion state of the doctor during the explanation from the standard face-driven image set of the doctor's popular science demonstration video.

[0060] In step S33 of some embodiments, a single frame of standardized face-driven image from a standard face-driven image set is input into the expression coding branch of the driving coding network. First, the basic feature extraction layer within the branch extracts global facial expression features from the input image to obtain global expression context features containing the overall semantics of the whole face expression. Then, the hierarchical decoupling layer within the branch decomposes the global expression context features to separate preliminary local expression features corresponding to different facial regions such as eyebrows, eyes, nose, mouth, and cheeks. Finally, the global expression context features and the preliminary local expression features of each region are combined and integrated to obtain a hierarchical expression code that simultaneously contains global facial expression semantics and preliminary expression information of each region. For example, in a financial scenario, the driving coding network can perform feature decomposition on the standard service smile in a standardized face-driven image of a customer service representative to obtain a hierarchical expression code for the corresponding service expression. In a medical scenario, the driving coding network can perform feature decomposition on the gentle explanation expression in a standardized face-driven image of a physician to obtain a hierarchical expression code for the corresponding explanation expression.

[0061] Specifically, based on the driving coding network, expression feature decomposition is performed on standardized face-driven images to obtain hierarchical expression coding, including: Based on the driving coding network, global facial expression features are extracted from standardized face driving images to obtain overall facial expression features. Context-associative encoding is performed on the overall facial expression features to obtain global expression context features. Based on the driving coding network, feature decoupling of the overall facial expression features is performed to obtain a set of local expression features; By combining global facial expression context features with local facial expression feature sets, a hierarchical facial expression coding system is obtained.

[0062] In some embodiments, the overall facial expression feature refers to a multi-dimensional feature vector that contains complete semantic information of the entire facial expression region after global facial expression feature extraction. It is the basic processing object for subsequent context encoding and feature decoupling. For example, in a financial scenario, the overall facial expression feature may be a feature vector containing complete semantic information of a bank customer service smile, while in a medical scenario, the overall facial expression feature may be a feature vector containing complete semantic information of a licensed physician's gentle explanation expression.

[0063] Global facial expression context features refer to multi-dimensional feature vectors that contain the spatial linkage relationship of facial expressions and overall semantic context information after context association encoding. They are the global semantic foundation for constructing hierarchical facial expression encoding. For example, in a financial scenario, global facial expression context features can be feature vectors containing the full-face linkage information of standard service expressions of bank customer service representatives. In a medical scenario, global facial expression context features can be feature vectors containing the full-face linkage information of explanation expressions of licensed physicians.

[0064] Local expression feature set refers to the feature combination obtained after feature decoupling, which includes independent expression features of key facial regions such as eyebrows, eyes, nose, mouth, and cheeks. It is the local feature basis for constructing hierarchical expression coding. For example, in the financial scenario, the local expression feature set can be a combination of independent facial features corresponding to the smile of a bank customer service representative. In the medical scenario, the local expression feature set can be a combination of independent facial features corresponding to the facial expressions of a licensed physician explaining something.

[0065] In one embodiment of this application, global facial expression features of a standardized face-driven image can be extracted through the basic feature extraction layer of the expression coding branch in the driving coding network. Specifically, the preprocessed standardized face-driven image is input into the expression coding branch of the driving coding network. Through the multi-layer convolutional neural network structure within the branch, the input image is subjected to layer-by-layer feature extraction and dimensionality compression. During the extraction process, the focus is on the overall semantic information of the expression covering the entire face region, while filtering out interference information unrelated to the expression, such as the person's identity attributes and head posture. Finally, at the output of the basic feature extraction layer, a multi-dimensional feature vector containing the complete semantic information of the expression of the entire face region is obtained, i.e., the overall facial expression features.

[0066] This application embodiment can also complete the contextual association encoding processing of the overall facial expression features through the context encoding module of the expression encoding branch in the driving encoding network. Specifically, the extracted overall facial expression features are input into the context encoding module. Through the feature association modeling structure within the module, spatial semantic association analysis is performed on the features of each facial region in the overall facial expression features. The expression linkage relationship and overall semantic logic between different facial regions are captured, semantic conflicts between local features are eliminated, and an expression semantic association system covering the entire face is constructed. Finally, a multi-dimensional feature vector containing the spatial linkage relationship of the entire face expression and the overall semantic context information is output, namely, the global expression context features. For example, in the financial scenario, the linkage relationship between the eyebrows, eyes and mouth in the standard smile of a customer service representative can be captured through contextual association encoding processing to generate the global expression context features of the corresponding service expression. In the medical scenario, the linkage relationship between the eyebrows, eyes and cheek muscles in the doctor's soothing expression can be captured through contextual association encoding processing to generate the global expression context features of the corresponding explanatory expression.

[0067] In addition, this application embodiment can achieve feature decoupling of the overall facial expression features through a hierarchical decoupling layer of the expression coding branch in the driving coding network. Specifically, the extracted overall facial expression features are input into the hierarchical decoupling layer. Through a pre-trained decoupling constraint mechanism, the overall facial expression features are regionally split and independently encoded according to the division of key facial regions such as eyebrows, eyes, nose, mouth, and cheeks. This eliminates information coupling and mutual interference between features of different facial regions, so that each feature after splitting corresponds to the expression motion information of only a single facial region. Finally, the independent expression features corresponding to all regions are integrated to obtain a set of local expression features containing the independent expression features of each key facial region. For example, in a financial scenario, feature decoupling can be used to split the overall features of a customer service smile into independent features of regions such as eyebrows, eyes, and mouth to form a set of local expression features corresponding to the service expression. In a medical scenario, feature decoupling can be used to split the overall features of a doctor's explanation expression into independent features of regions such as eyebrows, eyes, and mouth to form a set of local expression features corresponding to the explanation expression.

[0068] It is also important to know that the embodiments of this application can complete the feature combination of global expression context features and local expression feature sets through the feature fusion layer of the expression coding branch in the driving coding network. Specifically, firstly, the features of each region in the global expression context features and local expression feature sets are subjected to dimensional alignment and distribution normalization processing to ensure that the dimensional specifications of all features to be combined are consistent with the feature distribution, eliminating dimensional conflicts in the feature splicing process. Then, the global expression context features that have completed the alignment processing and the local expression features of each region are orderly integrated and spliced ​​so that the final generated features simultaneously contain the overall semantic context information of the whole face expression and the independent local expression information of each facial region. Finally, a multi-dimensional combined feature vector with a clear hierarchical structure and complete information is output, namely, hierarchical expression coding. For example, in the financial scenario, the global context features of customer service expressions can be combined with the local features of each region to generate the hierarchical expression coding of the corresponding service expression. In the medical scenario, the global context features of doctor's explanation expressions can be combined with the local features of each region to generate the hierarchical expression coding of the corresponding explanation expression.

[0069] In this embodiment, a global facial expression feature is first extracted from a standardized face-driven image using a driving coding network. This captures the overall semantic information of the facial expression while filtering out irrelevant interference information, providing a unified and complete set of basic features for the hierarchical processing of expression features. Subsequently, context-related coding is performed on the overall facial expression features to obtain global expression context features. This accurately captures the expression linkage between different facial regions, constructing a complete semantic association system for expressions and ensuring the overall coherence and semantic consistency of expression features. Next, the driving coding network is used to decouple the overall facial expression features to obtain a set of local expression features. This achieves independent splitting and decoupling of expression features in different facial regions, eliminating information coupling between regions and providing an independent and clear local feature foundation for the regionalized and precise control of expressions. Finally, hierarchical expression coding is obtained by combining the global expression context features and the set of local expression features. This achieves the orderly integration of global expression semantics and local regional features, constructing an expression coding system with a clear hierarchical structure and complete and controllable information. This fundamentally avoids the problems of disconnection between global and local expression features and information entanglement between regions, significantly improving the controllability and accuracy of expression features.

[0070] Steps S51 to S53 of this embodiment first obtain a standard face-driven image set by performing frame-by-frame face detection on the reference driving video. This effectively removes background interference information from the driving video, completes the standardization and unification of continuous frame images, and provides a standardized input basis for the stable extraction of pose and expression features, ensuring the continuity and consistency of features between frames. Then, the standard face-driven image set is subjected to head motion feature extraction through a driving coding network to obtain global pose coding, accurately locking the overall rigid motion information of the person's head, effectively filtering out interference from non-pose information such as facial expressions, and providing a precise control basis for the overall head pose control of the generated portrait. Finally, the standardized face-driven images are subjected to expression feature decomposition through a driving coding network to obtain hierarchical expression coding, realizing the hierarchical decomposition of global facial expression information and local area expression information, providing a clear feature basis for the fine control of expressions, and improving the control accuracy and operability of driving information.

[0071] In step S40 of some embodiments, local expression coding refers to the expression feature vectors obtained by refining the hierarchical expression coding through a region-aware attention network. These vectors correspond to different key facial regions, are independent of each other, and have no information entanglement. They are used to independently control the expression actions of the facial regions corresponding to the generated portrait. For example, in a financial scenario, local expression coding may be an independent expression feature vector of the mouth region corresponding to a standard smile of a customer service representative. In a medical scenario, local expression coding may be an independent expression feature vector of the eye region corresponding to a doctor's soothing expression.

[0072] This application embodiment can first adaptively allocate attention weights to different key facial regions such as eyebrows, eyes, nose, mouth, and cheeks based on the facial region information contained in the hierarchical expression coding, so that the network automatically focuses on the expression motion information of the corresponding facial regions. Then, based on the allocated attention weights, the expression features of each facial region are independently encoded and optimized to obtain independent expression features for each facial region. Finally, the independent expression features of each region are deredundant and decoupled to eliminate information entanglement between features of different regions, resulting in mutually independent and individually controllable local expression codes.

[0073] In detail, based on a region-aware attention network, local fine-grained modeling of hierarchical facial expression coding is performed to obtain local facial expression coding, including: Facial region attention weights are obtained by allocating facial region attention based on hierarchical expression coding. Facial region expression features are encoded using a region-aware attention network based on facial region attention weights to obtain independent facial region expression features. Redundancy removal is performed on independent facial expression features to obtain local expression codes.

[0074] In some embodiments, facial region attention weight refers to a quantitative parameter obtained after facial region attention allocation, used to characterize the degree of attention the network pays to the facial expression features of different facial regions. It is the core basis for guiding the network to perform regional feature encoding. For example, in the financial scenario, facial region attention weight can be a quantitative parameter of the degree of attention of each facial region corresponding to customer service expressions. In the medical scenario, facial region attention weight can be a quantitative parameter of the degree of attention of each facial region corresponding to doctor diagnosis expressions.

[0075] Facial region-independent expression features refer to feature vectors obtained by region-aware attention networks independently encoding each facial region based on facial region attention weights. These feature vectors correspond only to the facial expression motion information of a single facial region and serve as intermediate processing objects for generating local expression codes. For example, in a financial scenario, facial region-independent expression features could be an independent expression feature vector corresponding only to the mouth region of a customer service representative's standard smile. In a medical scenario, facial region-independent expression features could be an independent expression feature vector corresponding only to the eye region of a doctor's soothing expression.

[0076] This application embodiment can first perform feature localization and semantic recognition on preset key facial regions such as eyebrows, eyes, nose, mouth, and cheeks based on the global facial expression context information and preliminary features of each facial region contained in the hierarchical expression coding, clarifying the role and importance of each region in the current expression. Then, based on the recognition results, adaptively allocate attention weights to each key facial region, assigning higher weight coefficients to regions carrying core facial movement information and lower weight coefficients to regions with less facial movement information, and finally generating facial region attention weights that correspond one-to-one with each key facial region. For example, in a financial scenario, for a bank customer service representative's standard service smile, higher attention weights can be assigned to the mouth and eye regions that carry the core information of the smile, thus obtaining the facial region attention weights for the corresponding service expression. In a medical scenario, for a practicing physician's gentle and reassuring expression, higher attention weights can be assigned to the eye and cheek regions that carry the reassuring information, thus obtaining the facial region attention weights for the corresponding treatment expression.

[0077] Then, in this embodiment, the generated facial region attention weights and the original hierarchical expression codes can be input into the region feature encoding branch of the region-aware attention network. First, based on the attention weights corresponding to each facial region, the corresponding region features in the hierarchical expression codes are weighted, so that the network automatically focuses on the facial motion information of high-weight regions and suppresses the interference of irrelevant information in low-weight regions. Then, the weighted facial region features are independently convolutionally encoded and optimized, so that the feature output by each encoding corresponds only to the facial motion information of a single facial region and has no information overlap with the features of other facial regions. Finally, independent facial region expression features corresponding one-to-one with each key facial region are generated. For example, in the financial scenario, the region-aware attention network can encode the independent expression features of each region, such as the mouth and eyes, corresponding to the customer service smile based on the attention weights of the service expression. In the medical scenario, the region-aware attention network can encode the independent expression features of each region, such as the eyes and cheeks, corresponding to the doctor's soothing expression based on the attention weights of the diagnosis expression.

[0078] Next, the embodiments of this application can perform correlation analysis on the independent facial expression features of each region, identify and filter out redundant information and coupling interference between features of different regions, and then further orthogonalize the features of each region through a preset decoupling constraint mechanism to strengthen the independence and exclusivity of each region's features, ensuring that the features of each region can only control the facial expression movement of the corresponding region and will not interfere with other regions. Finally, the optimized features of each region are integrated in an orderly manner to obtain local facial expression codes that are independent of each other and free from information entanglement. For example, in the financial scenario, the coupling interference between the features of each region of customer service expressions can be eliminated through redundancy removal to obtain local facial expression codes that can independently control each facial region. In the medical scenario, the coupling interference between the features of each region of doctor's diagnosis expressions can be eliminated through redundancy removal to obtain local facial expression codes that can independently control each facial region.

[0079] This application first obtains facial region attention weights by allocating facial region attention based on hierarchical expression coding, achieving adaptive focusing on the expression features of different facial regions. This allows the network to accurately lock onto key regions carrying core expression information, effectively suppressing information interference from irrelevant regions and providing accurate guidance for subsequent region-level feature coding. Subsequently, facial region expression features are encoded using a region-aware attention network based on facial region attention weights, resulting in independent facial region expression features. This enables independent encoding and extraction of expression features for each facial region, ensuring the unique correspondence of each region's features and providing a clear feature carrier for regional independent control of expressions. Finally, local expression coding is obtained by deduplicating the independent facial region expression features, effectively eliminating redundant information and coupling interference between different region features, strengthening the independence and exclusivity of each region's features, and fundamentally avoiding inter-regional linkage interference problems during expression control, thus significantly improving the control accuracy and naturalness of facial expressions.

[0080] In step S50 of some embodiments, the fused generated feature refers to the fused feature vector obtained by recombining multi-source features through an adaptive feature fusion network. It is dimensionally unified, semantically aligned, and can be directly used for video image generation. It is the input content of the controllable image generation network. For example, in the financial scenario, the fused generated feature can be a feature vector that integrates customer identity information and standard customer service action and expression information. In the medical scenario, the fused generated feature can be a feature vector that integrates doctor identity information and popular science explanation action and expression information.

[0081] In this embodiment, the clean identity code is first used as the basic content feature, and the global pose code and local expression code are used together as the conditional control features. These are then input into an adaptive feature fusion network. The basic content features and conditional control features are then aligned and adapted in terms of feature distribution through adaptive instance normalization to obtain standardized multi-source features with uniform dimensions and consistent distribution. Subsequently, based on a preset weight allocation strategy, the standardized pose features and expression features are dynamically weighted and fused to obtain optimized conditional control features. Finally, the optimized conditional control features are injected into the basic content features in an independent and decoupled manner to complete the semantic alignment and reorganization of multi-source features, resulting in fused generated features that can be directly used for video image generation.

[0082] In detail, based on an adaptive feature fusion network, multi-source features are adaptively recombined from the clean identity code, global pose code, and local expression code to obtain fused generated features, including: Based on an adaptive feature fusion network, the distribution of clean identity encoding, global pose encoding, and local expression encoding is aligned to obtain standard identity encoding, standard pose encoding, and standard expression encoding. Based on a preset weight allocation strategy, standard pose coding and standard facial expression coding are weighted and fused to obtain conditional control features; Conditional control features are injected into the standard identity code to obtain fused generated features.

[0083] In some embodiments, a standard identity code refers to a standardized identity feature vector obtained by distributive alignment of a pure identity code, whose dimensional specifications and feature distribution meet the fusion requirements. It is the core foundation for carrying personal identity information in the process of multi-source feature fusion. For example, in a financial scenario, a standard identity code can be a standardized bank customer identity feature vector, and in a medical scenario, a standardized practicing physician identity feature vector.

[0084] Standard pose encoding refers to the standardized pose feature vector obtained after global pose encoding has undergone distribution alignment processing. The dimensional specifications and feature distribution meet the fusion requirements. It is one of the core foundations for constructing dynamic control features. For example, in financial scenarios, standard pose encoding can be a standardized bank customer service pose feature vector. In medical scenarios, standard pose encoding can be a standardized physician's explanation pose feature vector.

[0085] Standard facial expression encoding refers to the standardized facial expression feature vector obtained after the local facial expression encoding has undergone distribution alignment processing. The dimensional specifications and feature distribution meet the fusion requirements. It is one of the core foundations for constructing dynamic control features. For example, in the financial scenario, standard facial expression encoding can be the standardized facial expression feature vector of bank customer service. In the medical scenario, standard facial expression encoding can be the standardized facial expression feature vector of a licensed physician explaining.

[0086] The preset weight allocation strategy refers to the quantitative rules set in advance according to the application scenario requirements, which are used to allocate fusion weight coefficients to standard posture codes and standard facial expression codes. It is the core basis for realizing the weighted fusion of posture and facial expression features. For example, in the financial scenario, the preset weight allocation strategy can be a weight allocation rule that adapts to the bank customer service scenario and ensures the standardization of posture. In the medical scenario, the preset weight allocation strategy can be a weight allocation rule that adapts to the doctor-patient communication scenario and ensures the naturalness of facial expression.

[0087] Conditional control features refer to the combined feature vector obtained by weighted fusion of standard posture encoding and standard expression encoding based on a preset weight allocation strategy. It integrates all dynamic control information of head posture and facial expression and is the core control carrier used to regulate the generation of dynamic portrait effects. For example, in the financial scenario, the conditional control feature can be a dynamic control feature vector that integrates the posture and expression information of a standard bank customer service representative. In the medical scenario, the conditional control feature can be a dynamic control feature vector that integrates the posture and expression information of a licensed physician.

[0088] In this embodiment, the pre-acquired clean identity code, global pose code, and local expression code are input into the standardization preprocessing module of the adaptive feature fusion network. First, the three types of features from different sources are subjected to dimensional adaptation processing. According to the preset fusion dimensional specification, the dimensional size of the three types of features is adjusted so that all features have a unified dimensional specification, eliminating the fusion obstacles caused by dimensional mismatch. Then, an adaptive instance normalization method is used to perform feature distribution normalization processing on the three types of features that have completed dimensional adaptation, unifying the mean and variance of each feature, eliminating the feature distribution differences caused by different encoding branches, and outputting standard identity codes, standard pose codes, and standard expression codes with unified dimensions and consistent distribution. For example, in a financial scenario, this step can be used to unify the dimensions and distribution of the identity code of a bank customer and the pose and expression codes of a standard customer service representative to obtain standardized features that meet the fusion requirements. In a medical scenario, this step can be used to unify the dimensions and distribution of the identity code of a licensed physician and the pose and expression codes of a science popularization lecture to obtain standardized features that meet the fusion requirements.

[0089] Next, a preset weight allocation strategy matching the application scenario is retrieved, and the fusion weight coefficients corresponding to the standard posture code and the standard expression code are determined. The weight coefficients can be preset and adjusted according to the different requirements of posture control precision and expression control precision in the scenario. Then, the standard posture code and the standard expression code are multiplied by their respective weight coefficients to complete the feature weighting process. The two types of weighted features are then summed and fused so that the fused features simultaneously and completely carry all dynamic control information of head posture and facial expression, resulting in conditional control features that integrate all dynamic control information. For example, in the financial scenario, the standard service posture and expression code can be weighted and fused based on the weight allocation strategy adapted to the customer service scenario to obtain the corresponding conditional control features of customer service dynamics. In the medical scenario, the standard explanation posture and expression code can be weighted and fused based on the weight allocation strategy adapted to the popular science explanation scenario to obtain the corresponding conditional control features of doctor explanation dynamics.

[0090] Then, this embodiment of the application can take the standard identity code that has undergone distribution alignment as the core content basis, and the conditional control features obtained by weighted fusion as the dynamic adjustment information. Through a preset feature injection mechanism, the conditional control features are integrated into the feature space of the standard identity code in an independent and decoupled manner. During the injection process, the semantic alignment of the two types of features is completed simultaneously, ensuring that the inherent identity information of the target person is not tampered with by the dynamic control information. At the same time, it ensures that the dynamic control information of posture and expression can be completely and accurately transmitted to the final fusion feature. Finally, a fusion generation feature that carries both the inherent identity information of the person and the complete dynamic control information, is semantically aligned, and can be directly used for portrait video generation is obtained. For example, in the financial scenario, the conditional control features of customer service dynamics can be injected into the standard identity code of bank customers through this step to obtain a fusion generation feature for generating customer-specific digital human customer service videos. In the medical scenario, the conditional control features of popular science explanation dynamics can be injected into the standard identity code of practicing physicians through this step to obtain a fusion generation feature for generating physician-specific popular science videos.

[0091] This application first uses an adaptive feature fusion network to align the distribution of the clean identity code, global pose code, and local expression code, resulting in three types of standardized features. This effectively eliminates the dimensional and distribution differences between features from different sources, clearing obstacles for the smooth fusion of multi-source features and ensuring the stability and consistency of the fusion process. Subsequently, based on a preset weight allocation strategy, the standard pose code and standard expression code are weighted and fused to obtain conditional control features. This achieves the orderly integration of dynamic control information for pose and expression. At the same time, the preset weights can flexibly adapt to the differentiated needs of pose and expression control in different application scenarios, improving the scenario adaptability and controllability of the solution. Finally, by injecting the conditional control features into the standard identity code, the fused generation features are obtained, achieving accurate fusion and semantic alignment of the inherent identity information and dynamic control information of the person. While completely preserving the identity information of the target person, it achieves precise and controllable regulation of the portrait dynamic effect, fundamentally avoiding the problems of identity information loss and inaccurate dynamic information transmission during the fusion process. This significantly improves the identity fidelity and naturalness of the dynamic effect of the generated portrait video, and can stably adapt to various application scenarios with high requirements for portrait generation quality, such as finance and healthcare.

[0092] In step S60 of some embodiments, the target portrait video refers to the continuous portrait video file that is finally output after the portrait video generation model has processed the entire process, retains the inherent identity characteristics of the target person, and reproduces the posture and facial expression dynamic information in the reference driving video. For example, in the financial scenario, the target portrait video can be a personalized intelligent customer service video of the customer's image, and in the medical scenario, the target portrait video can be a personalized health science popularization video of the doctor's image.

[0093] In this embodiment, the network's built-in temporal consistency module first performs inter-frame motion dynamic modeling on the fusion generation features corresponding to consecutive frames to eliminate feature abrupt changes and image jitter between consecutive frames. Then, the network's generator module generates high-fidelity portrait images frame by frame based on the temporally optimized fusion generation features. Finally, the generated consecutive portrait images are spliced ​​and format optimized to output a target portrait video that retains the inherent identity features of the target person, fully reproduces the posture and facial expression dynamic information in the reference driving video, and has a smooth and coherent temporal sequence.

[0094] As can be seen, in the above scheme, a clean identity code can be obtained by encoding the identity features of the pre-acquired user source identity image through an identity encoding network. This stably extracts and retains the inherent identity information of the person to be generated in the portrait, providing a reliable content foundation for video generation. Subsequently, a hierarchical decoupling encoding of the pre-acquired reference driving video is performed through a driving encoding network to obtain global pose encoding and hierarchical expression encoding. This achieves the separation and extraction of head pose information and facial expression information, providing a clearly categorized feature foundation for precise control of dynamic information. Then, through the region-aware attention network included in the driving encoding network, local fine-grained modeling is performed on the hierarchical expression encoding to obtain local expression encoding. The refined processing of facial expression features effectively improves the control precision and independence of expression information. Then, through an adaptive feature fusion network, multi-source features are adaptively recombined from the pure identity code, global pose code, and local expression code to obtain fused generated features. This achieves accurate adaptation and orderly fusion of multiple types of features, ensuring the complete transmission and collaborative adaptation of various types of information. Finally, a portrait video is generated through a controllable image generation network and fused generated features to obtain the target portrait video. The output portrait video retains the inherent identity information of the target person while completely reproducing the dynamic information corresponding to the reference driving video, greatly improving the fidelity and scene adaptability of the generated content, and has outstanding practical value.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0096] In one embodiment, a portrait video generation apparatus is provided, which corresponds one-to-one with the portrait video generation method described in the above embodiments. For example... Figure 6 As shown, the portrait video generation device includes a pre-trained model retrieval module 101, an identity feature encoding module 102, a video decoupling driving module 103, a decoupled expression refinement module 104, an adaptive feature fusion module 105, and a portrait video generation module 106. Detailed descriptions of each functional module are as follows: The pre-trained model retrieval module 101 is used to acquire a pre-trained portrait video generation model, wherein the portrait video generation model includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network. The identity feature encoding module 102 is used to encode the identity features of the pre-acquired user source identity image based on the identity encoding network to obtain a clean identity code; The driving video decoupling module 103 is used to perform layered decoupling encoding on the pre-acquired reference driving video based on the driving encoding network to obtain global pose encoding and layered expression encoding. The decoupled expression refinement module 104 is used to perform local fine-grained modeling of hierarchical expression codes based on a region-aware attention network to obtain local expression codes. The adaptive feature fusion module 105 is used to adaptively recombine multi-source features from the clean identity code, global pose code, and local expression code based on the adaptive feature fusion network to obtain fused generated features. The portrait video generation module 106 is used to generate portrait videos based on a controllable image generation network and fused generation features to obtain the target portrait video.

[0097] This invention provides a portrait video generation device. First, an identity encoding network encodes the pre-acquired user source identity image to obtain a clean identity code, stably extracting and preserving the inherent identity information of the person to be generated in the portrait, providing a reliable content foundation for video generation. Then, a driving encoding network performs hierarchical decoupling encoding on the pre-acquired reference driving video to obtain global pose encoding and hierarchical expression encoding, achieving the separation and extraction of head pose information and facial expression information, providing a clearly categorized feature foundation for precise control of dynamic information. Next, a region-aware attention network included in the driving encoding network performs local fine-grained modeling of the hierarchical expression encoding to obtain local expression encoding. This method uses a code to achieve refined processing of facial expression features, effectively improving the control precision and independence of expression information. Then, through an adaptive feature fusion network, multi-source features are adaptively recombined from the pure identity code, global pose code, and local expression code to obtain fused generated features. This achieves accurate adaptation and orderly fusion of multiple types of features, ensuring the complete transmission and collaborative adaptation of various types of information. Finally, a portrait video is generated through a controllable image generation network and fused generated features to obtain the target portrait video. The output is a portrait video that retains the inherent identity information of the target person while completely reproducing the dynamic information corresponding to the reference driving video. This significantly improves the fidelity and scene adaptability of the generated content and has outstanding practical value.

[0098] Specific limitations regarding the portrait video generation device can be found in the limitations of the portrait video generation method described above, and will not be repeated here. Each module in the aforementioned portrait video generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0099] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a portrait video generation method on the server side.

[0100] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a portrait video generation method on the client side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain a pre-trained portrait video generation model, which includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network. Based on the identity coding network, the pre-acquired user source identity image is encoded with identity features to obtain a clean identity code; Based on the driving coding network, the pre-acquired reference driving video is subjected to hierarchical decoupling coding to obtain global pose coding and hierarchical expression coding; Based on a region-aware attention network, local fine-grained modeling of hierarchical facial expression coding is performed to obtain local facial expression coding; Based on an adaptive feature fusion network, multi-source features of pure identity encoding, global pose encoding and local expression encoding are adaptively recombined to obtain fused generated features. Based on a controllable image generation network and fused generation features, portrait videos are generated to obtain the target portrait video.

[0101] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain a pre-trained portrait video generation model, which includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network. Based on the identity coding network, the pre-acquired user source identity image is encoded with identity features to obtain a clean identity code; Based on the driving coding network, the pre-acquired reference driving video is subjected to hierarchical decoupling coding to obtain global pose coding and hierarchical expression coding; Based on a region-aware attention network, local fine-grained modeling of hierarchical facial expression coding is performed to obtain local facial expression coding; Based on an adaptive feature fusion network, multi-source features of pure identity encoding, global pose encoding and local expression encoding are adaptively recombined to obtain fused generated features. Based on a controllable image generation network and fused generation features, portrait videos are generated to obtain the target portrait video.

[0102] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0105] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating portrait videos, characterized in that, include: Obtain a pre-trained portrait video generation model, wherein the portrait video generation model includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network; Based on the identity encoding network, the pre-acquired user source identity image is encoded with identity features to obtain a clean identity code; Based on the driving coding network, the pre-acquired reference driving video is subjected to hierarchical decoupling coding to obtain global pose coding and hierarchical expression coding; Based on the region-aware attention network, the hierarchical expression coding is locally refined and modeled to obtain local expression coding. Based on the adaptive feature fusion network, the pure identity code, the global pose code, and the local expression code are adaptively recombined from multiple sources to obtain fused generated features. Based on the controllable image generation network and the fused generation features, a portrait video is generated to obtain the target portrait video.

2. The portrait video generation method as described in claim 1, characterized in that, The portrait video generation model is trained according to the following steps: Obtain a sample face image set and a sample-driven video set, wherein the sample face image set includes identity annotation information and pose annotation information, and the sample-driven video set includes expression annotation information and real video frame annotation information; Based on the preset initial portrait video generation model, feature processing is performed on the sample face image set and the sample-driven video set to obtain the sample clean identity code, sample global pose code, sample local expression code, sample fusion generation features, and sample generated portrait video frames. The total loss value of the model is obtained by calculating the loss based on the sample clean identity encoding, the sample global pose encoding, the sample local expression encoding, the sample fusion generated features, the sample generated portrait video frames, the identity annotation information, the pose annotation information, the expression annotation information, and the real video frame annotation information. The parameters of the preset initial portrait video generation model are updated based on the total loss value of the model to obtain the portrait video generation model.

3. The portrait video generation method as described in claim 1, characterized in that, The step of encoding the pre-acquired user source identity image based on the identity encoding network to obtain a clean identity code includes: Perform face region detection on the user source identity image to obtain a standard face region image; Based on the identity coding network, inherent attribute features are extracted from the standard face region image to obtain an inherent identity feature vector; The inherent identity feature vector is subjected to interference removal processing to obtain the pure identity code.

4. The portrait video generation method as described in claim 1, characterized in that, The step of performing hierarchical decoupling encoding on the pre-acquired reference driving video based on the driving coding network to obtain global pose encoding and hierarchical expression encoding includes: Perform frame-by-frame face detection on the reference driving video to obtain a standard face-driven image set; Based on the driving coding network, head motion features are extracted from the standard face-driven image set to obtain the global pose code. Based on the driving coding network, the standardized face driving image is decomposed into facial expression features to obtain the hierarchical facial expression code.

5. The portrait video generation method as described in claim 4, characterized in that, The step of performing facial expression feature decomposition on the standardized face-driven image based on the driving coding network to obtain the hierarchical facial expression coding includes: Based on the driving coding network, global facial expression features are extracted from the standardized face driving image to obtain overall facial expression features; The overall facial expression features are subjected to context-associative encoding to obtain global expression context features. Based on the driving coding network, the overall facial expression features are decoupled to obtain a set of local expression features; The global facial expression context features are combined with the local facial expression feature set to obtain the hierarchical facial expression encoding.

6. The portrait video generation method according to any one of claims 1-5, characterized in that, The process of performing local fine-grained modeling of the hierarchical expression coding based on the region-aware attention network to obtain local expression coding includes: Facial region attention weights are obtained by allocating facial region attention based on the hierarchical expression coding. The facial region expression features are encoded by the region-aware attention network based on the facial region attention weights to obtain independent facial region expression features. Redundancy removal is performed on the independent facial expression features of the facial region to obtain the local expression code.

7. The portrait video generation method according to any one of claims 1-5, characterized in that, The adaptive feature fusion network is used to adaptively recombine multi-source features from the clean identity code, the global pose code, and the local expression code to obtain fused generated features, including: Based on the adaptive feature fusion network, the pure identity code, the global pose code, and the local expression code are distributed and aligned to obtain the standard identity code, standard pose code, and standard expression code. Based on a preset weight allocation strategy, the standard pose code and the standard facial expression code are weighted and fused to obtain conditional control features; The conditional control features are injected into the standard identity code to obtain the fused generated features.

8. A portrait video generation device, characterized in that, include: The pre-trained model retrieval module is used to obtain a pre-trained portrait video generation model, wherein the portrait video generation model includes an identity encoding network and a driving encoding network, an adaptive feature fusion network and a controllable image generation network, and the driving encoding network includes a region-aware attention network. The identity feature encoding module is used to encode the identity features of the pre-acquired user source identity image based on the identity encoding network to obtain a clean identity code; The driving video decoupling module is used to perform layered decoupling encoding on the pre-acquired reference driving video based on the driving encoding network to obtain global pose encoding and layered expression encoding. The decoupled facial expression refinement module is used to perform local fine-grained modeling of the hierarchical facial expression code based on the region-aware attention network to obtain local facial expression codes. An adaptive feature fusion module is used to adaptively recombine multi-source features from the clean identity code, the global pose code, and the local expression code based on the adaptive feature fusion network to obtain fused generated features. The portrait video generation module is used to generate a portrait video based on the controllable image generation network and the fused generation features to obtain a target portrait video.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the portrait video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the portrait video generation method as described in any one of claims 1 to 7.

Images