Deep learning driven portrait media stream generation system
Through the GAN and AdaIN joint training framework and LSTM network, combined with the rigid body dynamics model, the problems of high cost, low efficiency, temporal incoherence and difficulty in multimodal fusion in traditional portrait generation methods are solved, and high-quality, real-time portrait media stream generation is achieved.
Patent Information
- Application Number
- CN202510721964.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-09-12
AI Technical Summary
Traditional portrait generation methods have problems such as high cost, low efficiency, temporal inconsistency, low style transfer efficiency and difficulty in multimodal fusion, making it difficult to meet the needs of real-time applications.
Using a joint training framework of GAN and AdaIN, combined with an LSTM network and a rigid body dynamics model, high-quality image generation and real-time style transfer are achieved through pre-training, joint training, and effect verification stages. A temporal smoothing loss function and a voice synchronization mechanism are introduced to ensure motion continuity and multimodal fusion.
The quality and speed of style transfer are significantly improved, the phenomenon of motion jumps is reduced, the generated motions are consistent with the biomechanical characteristics of the human body, and the robustness of the system and the consistency of the generated results are improved.
Smart Images
Figure CN120640089A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a deep learning-driven portrait media stream generation system. Background Art
[0002] With the rapid development of the metaverse, virtual reality, and digital human technologies, the demand for portrait media stream generation technology is growing in areas such as virtual anchors, digital human interaction, and film and television special effects. Traditional portrait generation methods mainly rely on manual modeling and motion capture equipment, which have the following technical problems:
[0003] High cost and low efficiency: Traditional methods rely on professional motion capture equipment and manual modeling, which requires a lot of manpower and time investment. The generation process is time-consuming and cannot meet the needs of real-time applications.
[0004] Temporal inconsistency: The lack of an effective temporal consistency optimization mechanism causes the generated portraits to jump or experience sudden changes in expression during continuous actions, disrupting the user experience.
[0005] Inefficient style transfer: Traditional methods usually require retraining models or performing a large number of parameter adjustments when performing style transfer and customization, which cannot meet the needs of real-time style switching.
[0006] Difficulty in multimodal fusion: It is difficult to effectively integrate multimodal information such as vision, speech, and motion, resulting in inconsistencies between the facial expressions, movements, and speech content of the generated portraits.
[0007] While some research has attempted to improve portrait generation technology through deep learning methods, such as GAN-based image generation and CNN-based style transfer, these approaches often focus on optimizing a single aspect and lack systematic solutions. In particular, technical bottlenecks remain in areas such as multimodal information fusion, temporal coherence optimization, and real-time style transfer.
[0008] Therefore, there is an urgent need for a portrait media stream generation system that can comprehensively solve the above problems to meet the growing demand for digital content creation. Summary of the Invention
[0009] To address the shortcomings of existing technologies, this paper proposes a deep learning-driven portrait media stream generation system. This system utilizes a GAN and AdaIN joint training framework, combining generative adversarial networks and adaptive instance normalization techniques to achieve the dual goals of high-quality image generation and real-time style transfer. Through three stages: pre-training, joint training, and validation, it achieves both improved style transfer quality and speed. An LSTM (Long Short-Term Memory) network is used to model temporal dependencies, and a temporal smoothing loss function is introduced to significantly reduce motion jumps. The system analyzes 10-frame sequences to ensure smooth and natural motion transitions between frames, avoiding sudden changes. A rigid-body dynamics model is applied to ensure that the generated motion conforms to human biomechanical characteristics, including joint constraints, collision detection, and balance constraints. The system calculates the center of gravity and support polygons to ensure the stability and realism of the generated poses. Dynamically assessing the quality and reliability of each modal feature and automatically adjusting fusion weights significantly improves system robustness. Through four steps: feature quality assessment, dynamic weight generation, adaptive weight adjustment, and multi-level feature fusion, the system achieves adaptability to diverse environmental conditions. Through layered processing of timing optimization, physical correction, and voice synchronization, a closed-loop optimization system is formed. Each optimization stage influences each other through a feedback mechanism to ensure high quality and consistency of the generated results.
[0010] To achieve the above object, the technical solution of the present invention is as follows:
[0011] A deep learning-driven portrait media stream generation system is characterized in that the system includes an input module, a feature extraction layer module, a multimodal fusion module, an optimization module and an output module, wherein the input module is used to preprocess the input video data, audio data, text data, and style parameters and output a standardized video frame sequence, an audio feature original signal, a text word vector sequence and a style feature vector; the feature extraction layer module is used to extract facial features, body features, action features, and voice features and perform feature aggregation; the multimodal fusion module is used to output attention fusion features using a cross-attention mechanism and output initial portrait frames through joint training of GAN and AdaIN; the optimization module is used to use an LSTM network to perform temporal consistency constraints, perform physical constraints through a rigid body dynamics model, and process voice features in combination with a voice-driven mechanism; and the output module is used to output a final portrait media stream based on the optimized frame sequence.
[0012] A further improvement of the present invention is that the input module includes a preprocessing encoding unit and a parameter output unit, the preprocessing encoding unit is used to preprocess the input video data, audio data, text data, and style parameters; the parameter output unit is used to output a standardized video frame sequence, audio feature original signal, text word vector sequence, and style feature vector.
[0013] A further improvement of the present invention is that the feature extraction layer module includes a facial feature extraction unit, a body feature extraction unit, a motion feature extraction unit, a speech feature extraction unit and a feature aggregation unit. The facial feature extraction unit is used to perform face detection, alignment and feature extraction processing on the standardized video frame sequence to obtain a facial feature vector; the body feature extraction unit is used to perform skeleton detection, standard normalization and time series modeling on the standardized video frame sequence to obtain a body feature vector; the motion feature extraction unit is used to perform optical flow calculation and spatiotemporal feature analysis on a continuous 16-frame standardized video frame sequence to output a motion feature vector; the speech feature extraction unit is used to perform MFCC feature extraction, rhythm feature extraction and speech encoding on the original audio feature signal to output a speech feature vector; the feature aggregation unit is used to normalize and feature splice the facial feature vector, body feature vector, motion feature vector and speech feature vector to output an aggregated feature vector.
[0014] A further improvement of the present invention is that the multimodal fusion module includes a cross-attention mechanism unit and a joint training unit. The cross-attention mechanism unit is used to extract style feature vectors and aggregated feature vectors, perform feature mapping, attention calculation, feature fusion, and output attention fusion features; the joint training unit is used to output the initial portrait frame by jointly training the style feature vectors and attention fusion features through GAN and AdaIN.
[0015] A further improvement of the present invention is that the optimization module includes a timing consistency constraint unit, a physical constraint unit and a speech drive unit; the timing consistency constraint unit is used to perform LSTM timing modeling, smoothing loss, and maximum change rate constraints on a sequence of 10 consecutive initial portrait frames, and output a timing-optimized frame sequence; the physical constraint unit is used to perform joint restrictions, collision detection, and balance constraints on the coordinates of the skeleton key points of the timing-optimized frames, and output a physically corrected frame sequence; the speech drive unit is used for speech feature vectors, outputs lip shape parameters according to speech features through a Transformer model, adjusts facial expression parameters, ensures synchronization of lip shape and phonemes, and outputs an optimized frame sequence.
[0016] A further improvement of the present invention is that the output module is used to perform background fusion, adaptive color correction and encoding operations on the optimized frame sequence to output a final portrait media stream.
[0017] The technical effects of the present invention are as follows:
[0018] This invention utilizes a joint GAN and AdaIN training framework, combining generative adversarial networks (GANs) with adaptive instance normalization (ADN) technology to achieve the dual goals of high-quality image generation and real-time style transfer. Through three stages: pre-training, joint training, and validation, it achieves improvements in both style transfer quality and speed. An LSTM (Long Short-Term Memory) network is used to model temporal dependencies and introduces a temporal smoothing loss function, significantly reducing motion jumps. The system analyzes 10-frame sequences to ensure continuous and natural motion transitions between adjacent frames, avoiding sudden changes. A rigid-body dynamics model is applied to ensure that the generated motion conforms to human biomechanical properties, including joint restrictions, collision detection, and balance constraints. The system calculates the center of gravity and support polygons to ensure the stability and realism of the generated poses. The system dynamically evaluates the quality and reliability of each modal feature and automatically adjusts fusion weights, significantly improving system robustness. Through four steps: feature quality assessment, dynamic weight generation, adaptive weight adjustment, and multi-level feature fusion, the system achieves adaptability to diverse environmental conditions. A closed-loop optimization system is formed through layered processing of timing optimization, physical correction, and voice synchronization. Each optimization stage influences each other through a feedback mechanism to ensure high quality and consistency of the generated results. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0020] Figure 1 This is a structural diagram of a deep learning-driven portrait media stream generation system of the present invention.
[0021] Figure 2 This is an overall flow chart of a deep learning-driven portrait media stream generation system of the present invention. DETAILED DESCRIPTION
[0022] Example 1
[0023] This embodiment proposes a deep learning-driven portrait media stream generation system. This system utilizes a GAN and AdaIN joint training framework, combining generative adversarial networks and adaptive instance normalization techniques to achieve the dual goals of high-quality image generation and real-time style transfer. Through three stages: pre-training, joint training, and validation, it achieves both improved style transfer quality and speed. An LSTM (Long Short-Term Memory) network is used to model temporal dependencies and introduces a temporal smoothing loss function to significantly reduce motion jumps. The system analyzes 10-frame sequences to ensure continuous and natural motion transitions between adjacent frames, avoiding sudden changes. A rigid-body dynamics model is applied to ensure that the generated motion conforms to human biomechanical characteristics, including joint constraints, collision detection, and balance constraints. The system calculates the center of gravity and support polygons to ensure the stability and realism of the generated poses. The system dynamically evaluates the quality and reliability of each modal feature and automatically adjusts the fusion weights, significantly improving system robustness. Through four steps: feature quality assessment, dynamic weight generation, adaptive weight adjustment, and multi-level feature fusion, the system achieves adaptability to diverse environmental conditions. A closed-loop optimization system is formed through layered processing of timing optimization, physical correction, and voice synchronization. Each optimization stage influences each other through a feedback mechanism to ensure high quality and consistency of the generated results.
[0024] like Figure 1 As shown, a deep learning-driven portrait media stream generation system is characterized in that the system includes an input module, a feature extraction layer module, a multimodal fusion module, an optimization module and an output module, wherein the input module is used to preprocess the input video data, audio data, text data, and style parameters and output a standardized video frame sequence, an audio feature original signal, a text word vector sequence and a style feature vector; the feature extraction layer module is used to extract facial features, body features, action features, and voice features and perform feature aggregation; the multimodal fusion module is used to output attention fusion features using a cross-attention mechanism and output the initial portrait frame through joint training of GAN and AdaIN; the optimization module is used to use an LSTM network for temporal consistency constraints, perform physical constraints through a rigid body dynamics model, and process voice features in combination with a voice-driven mechanism; the output module is used to output the final portrait media stream according to the optimized frame sequence.
[0025] In this embodiment, the input module includes a preprocessing encoding unit and a parameter output unit. The preprocessing encoding unit is used to preprocess the input video data, audio data, text data, and style parameters; the parameter output unit is used to output a standardized video frame sequence, an original audio feature signal, a text word vector sequence, and a style feature vector.
[0026] In this embodiment, the feature extraction layer module includes a facial feature extraction unit, a body feature extraction unit, a motion feature extraction unit, a speech feature extraction unit and a feature aggregation unit. The facial feature extraction unit is used to perform face detection, alignment and feature extraction processing on the standardized video frame sequence to obtain a facial feature vector; the body feature extraction unit is used to perform skeleton detection, standard normalization and time series modeling on the standardized video frame sequence to obtain a body feature vector; the motion feature extraction unit is used to perform optical flow calculation and spatiotemporal feature analysis on a continuous 16-frame standardized video frame sequence to output a motion feature vector; the speech feature extraction unit is used to perform MFCC feature extraction, rhythm feature extraction and speech encoding on the original audio feature signal to output a speech feature vector; the feature aggregation unit is used to normalize and feature splice the facial feature vector, body feature vector, motion feature vector and speech feature vector to output an aggregated feature vector.
[0027] In this embodiment, the multimodal fusion module includes a cross-attention mechanism unit and a joint training unit. The cross-attention mechanism unit is used to extract style feature vectors and aggregated feature vectors, perform feature mapping, attention calculation, feature fusion, and output attention fusion features; the joint training unit is used to jointly train the style feature vectors and attention fusion features through GAN and AdaIN to output the initial portrait frame.
[0028] In this embodiment, the optimization module includes a timing consistency constraint unit, a physical constraint unit and a speech drive unit; the timing consistency constraint unit is used to perform LSTM timing modeling, smoothing loss, and maximum change rate constraints on a sequence of 10 consecutive initial portrait frames, and output a timing-optimized frame sequence; the physical constraint unit is used to perform joint restrictions, collision detection, and balance constraints on the coordinates of the skeleton key points of the timing-optimized frames, and output a physically corrected frame sequence; the speech drive unit is used for speech feature vectors, outputs lip shape parameters according to speech features through a Transformer model, adjusts facial expression parameters, ensures that the lip shape is synchronized with the phonemes, and outputs an optimized frame sequence.
[0029] In this embodiment, the output module is used to perform background fusion, adaptive color correction and encoding operations on the optimized frame sequence, and output a final portrait media stream.
[0030] It is important to note that the GAN and AdaIN joint training framework combines generative adversarial networks and adaptive instance normalization technology to achieve the dual goals of high-quality image generation and real-time style transfer. Through three stages: pre-training, joint training, and validation, the system achieves both improved style transfer quality and speed. An LSTM (Long Short-Term Memory) network is used to model temporal dependencies, and a temporal smoothing loss function is introduced to significantly reduce motion jumps. The system analyzes 10-frame sequences to ensure smooth and natural motion transitions between frames, avoiding sudden changes. A rigid-body dynamics model is applied to ensure that the generated motion conforms to human biomechanical characteristics, including joint constraints, collision detection, and balance constraints. The system calculates the center of gravity and support polygons to ensure the stability and realism of the generated poses. The system dynamically evaluates the quality and reliability of each modal feature and automatically adjusts the fusion weights, significantly improving system robustness. Through four steps: feature quality assessment, dynamic weight generation, adaptive weight adjustment, and multi-level feature fusion, the system achieves adaptability to diverse environmental conditions. A layered approach of timing optimization, physical correction, and voice synchronization forms a closed-loop optimization system. Each optimization stage influences each other through a feedback mechanism to ensure high quality and consistency of the generated results.
[0031] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0032] It should be understood that determining B based on A does not mean determining B based solely on A. B can also be determined based on A and / or other information.
[0033] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired network or / and a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.
[0034] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.
[0035] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0036] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only one type. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.
[0037] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0038] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0039] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0040] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A deep learning driven portrait media stream generation system, characterized by: The system includes an input module, a feature extraction layer module, a multimodal fusion module, an optimization module, and an output module. The input module is used to preprocess the input video data, audio data, text data, and style parameters and output a standardized video frame sequence, an original audio feature signal, a text word vector sequence, and a style feature vector; the feature extraction layer module is used to extract facial features, body features, action features, and voice features and perform feature aggregation; the multimodal fusion module is used to output attention fusion features using a cross-attention mechanism and output an initial portrait frame through joint training of GAN and AdaIN; the optimization module is used to use an LSTM network to perform temporal consistency constraints, perform physical constraints through a rigid body dynamics model, and process voice features in combination with a voice-driven mechanism; The output module is used to output the final portrait media stream according to the optimized frame sequence.
2. A deep learning driven portrait media stream generation system according to claim 1, characterized in that: The input module includes a preprocessing encoding unit and a parameter output unit. The preprocessing encoding unit is used to preprocess the input video data, audio data, text data, and style parameters; the parameter output unit is used to output a standardized video frame sequence, an original audio feature signal, a text word vector sequence, and a style feature vector.
3. The deep learning driven portrait media stream generation system according to claim 2, characterized in that: The feature extraction layer module includes a facial feature extraction unit, a body feature extraction unit, a motion feature extraction unit, a speech feature extraction unit and a feature aggregation unit. The facial feature extraction unit is used to perform face detection, alignment and feature extraction processing on the standardized video frame sequence to obtain a facial feature vector; the body feature extraction unit is used to perform skeleton detection, standard normalization and time series modeling on the standardized video frame sequence to obtain a body feature vector; the motion feature extraction unit is used to perform optical flow calculation and spatiotemporal feature analysis on a continuous 16-frame standardized video frame sequence to output a motion feature vector; the speech feature extraction unit is used to perform MFCC feature extraction, rhythm feature extraction and speech encoding on the original audio feature signal to output a speech feature vector; the feature aggregation unit is used to normalize and feature splice the facial feature vector, body feature vector, motion feature vector and speech feature vector to output an aggregated feature vector.
4. The deep learning driven portrait media stream generation system according to claim 3, characterized in that: The multimodal fusion module includes a cross-attention mechanism unit and a joint training unit. The cross-attention mechanism unit is used to extract style feature vectors and aggregated feature vectors, perform feature mapping, attention calculation, feature fusion, and output attention fusion features; the joint training unit is used to jointly train the style feature vectors and attention fusion features through GAN and AdaIN to output the initial portrait frame.
5. The deep learning driven portrait media stream generation system according to claim 4, characterized in that: The optimization module includes a timing consistency constraint unit, a physical constraint unit and a speech drive unit; the timing consistency constraint unit is used to perform LSTM timing modeling, smoothing loss, and maximum change rate constraints on a sequence of 10 consecutive initial portrait frames, and output a timing-optimized frame sequence; the physical constraint unit is used to perform joint restrictions, collision detection, and balance constraints on the coordinates of the skeleton key points of the timing-optimized frames, and output a physically corrected frame sequence; the speech drive unit is used to output lip shape parameters based on speech features through a Transformer model for speech feature vectors, adjust facial expression parameters, ensure that the lip shape is synchronized with the phonemes, and output an optimized frame sequence.
6. The deep learning driven portrait media stream generation system according to claim 5, characterized in that: The output module is used to perform background fusion, adaptive color correction and encoding operations on the optimized frame sequence, and output a final portrait media stream.