Digital human live broadcast microphone connection and video generation method based on computer vision

Abstract

The invention discloses a digital human live broadcast microphone connection and video generation method based on computer vision, and belongs to the technical field of artificial intelligence, and the method comprises the following steps: 1, real-time vision capture and action expression driving: employing a computer vision tool to carry out face key point detection, facial feature points of lips, eyes and the like of a live anchor or a microphone-connected guest are accurately captured; and then a voice conversion model is matched, microphone-connected audio features and lip key point movement are bound, precise synchronization of the sound and the lip is achieved, and for limb actions, limb joint data can be collected through a monocular camera in combination with a posture estimation algorithm or a simple action capture device and mapped to a digital human skeleton model. According to the digital human live broadcast microphone connection and video generation method based on computer vision, through multi-dimensional technical innovation and systematic design, the problems of insufficient sense of reality, interaction lagging, poor scene adaptability and the like existing in current digital human application are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular to a method for live streaming of digital humans based on computer vision, and also relates to a video generation method for live streaming of digital humans based on computer vision. Background Technology

[0002] The core of the computer vision-based digital human live streaming and video generation method is to use visual algorithms to capture real-person movements and expressions, synchronize multi-device interactive data, and then combine rendering and generation technologies to achieve a realistic presentation. However, during live streaming, it is impossible to make real-time adjustments or change the live streaming presentation effect according to the changes in the live stream. Summary of the Invention

[0003] The purpose of this invention is to provide a method for live streaming and video generation of digital humans based on computer vision, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a digital human live-streaming interaction method based on computer vision, comprising the following steps:

[0005] Step 1: Real-time Visual Capture and Motion / Expression Driven: Computer vision tools are used to detect facial key points, accurately capturing facial features such as the lips and eyes of the live stream host or guest. This is then combined with a speech-to-lip model to bind the audio features of the live stream to the movement of lip key points, achieving precise synchronization between sound and lips. For body movements, a monocular camera combined with a pose estimation algorithm, or a simple motion capture device, can be used to collect limb joint data, which is then mapped onto a digital human skeletal model, allowing the digital human to mimic interactive actions such as raising a hand or nodding.

[0006] Step 2: Construct a multimodal capture redundancy system: On the basis of monocular vision, add "infrared camera + depth sensor" to assist in capture, and fuse multi-source data through computer vision algorithms - such as using infrared images to supplement facial feature points when the lighting is insufficient, and using depth data to segment the foreground and background when the background is complex, reducing environmental interference; for occluded scenes, introduce a "feature point prediction model" to fill in missing feature points based on the motion trend before occlusion.

[0007] Step 3: Incorporate emotion-driven and motion smoothing mechanisms: Introduce an "audio emotion analysis model" to associate emotion tags with an emoji library, achieving "semantic-emotion-expression" linkage; add "Bezier curve interpolation" processing to the motion-driven mechanism to optimize motion transition frames, and establish a "digital human muscle dynamics model" to make limb movements conform to biomechanical laws;

[0008] Step 4: Optimize speech-to-lip-sync adaptation capabilities: For multilingual and dialectal scenarios, pre-train subdivided speech-to-lip-sync models and introduce a "phoneme-lip-sync mapping dictionary" to improve the matching accuracy of special pronunciations; at the same time, associate lip movements with other facial areas to achieve full-face coordinated movement.

[0009] Step 5: Multi-device Data Collaboration and Latency Control: Achieve data synchronization between digital humans or real people and digital humans on multiple devices. Utilize computer vision algorithms to perform frame synchronization processing on the live stream. Extract the pose and facial expression feature values ​​of the digital human in each frame using OpenCV and align them with the timestamps of the other party's video stream features. Employ lightweight visual feature encoding technology, transmitting only captured key point data instead of complete video frames to reduce data volume. Simultaneously, combine RTMP push streaming and HLS pull streaming protocols with FFmpeg to complete fast encoding and decoding, reducing audio and video latency during live streams and ensuring smooth interaction.

[0010] Step Six: Construct an "Edge + Center" Collaborative Delay Control System: Deploy edge nodes on the host and guest terminals to process visual feature extraction and encoding locally, reducing the amount of data transmitted across regions; replace some RTMP / HLS scenarios with "WebRTC protocol + UDP transmission" to reduce real-time interaction latency, while adding a "data packet retransmission and redundant encoding" mechanism to cope with network fluctuations; establish a latency monitoring module to automatically reduce the transmission priority of non-critical features when the latency exceeds a threshold.

[0011] Step 7: Develop a multi-platform and multi-style adaptation framework: For different terminal devices, preset visual capture parameter templates, improve image contrast on mobile devices to enhance feature point recognition, and optimize frame rate on PC devices to improve motion smoothness; build a "digital human style feature library" to match exclusive skeletal binding rules and motion mapping coefficients for different styles such as realistic and cartoon, to ensure that the action and image are consistent.

[0012] Step 8: Expand the dimensions of interactive command recognition and response logic: Integrate the "voice recognition + image gesture recognition" module to achieve multimodal fusion recognition of bullet comments, voice, and gesture commands; establish a knowledge base for the association of "command-scene-action", such as when the "show product" command is detected, automatically trigger the digital human to "turn around to get props-point to product with gestures-introduce verbally" to improve the depth of interaction;

[0013] Step Nine: Visual Recognition and Response of Interactive Commands: Commands such as liking and requesting to connect during live streams can be recognized with the assistance of computer vision. For example, image recognition algorithms can be used to capture keywords and emoticons in the comments and convert them into trigger signals. When a request to connect is recognized, the digital human is driven to make response actions such as turning its head or reaching out its hand. When the screen is flooded with likes from the audience, the digital human is controlled to display interactive expressions such as smiling and waving, enhancing the immersive experience of the live stream.

[0014] Step 10: Live Streaming Self-Running and Broadcasting in the No-Instruction State: When the digital human does not receive any instructions, it runs the live stream content according to the preset script to ensure continuous broadcasting. The preset script includes action information and verbal information. When the digital human is running, it adjusts and transforms the preset script according to the instructions of the live stream connection to improve the smoothness, excitement and rationality of the live stream connection.

[0015] As a preferred embodiment, the computer vision tool includes MediaPipe, the speech-to-lip model includes dialect version and bilingual version, and the emotion tags include happy, serious, and surprised.

[0016] A video generation method for digital human live streaming based on computer vision includes the following steps:

[0017] Step 1: Multimodal visual feature fusion preprocessing: First, collect multi-dimensional data and extract visually relevant features. In the image dimension, use CNN to extract the first image features such as texture and contour of the basic image of the digital human. In the lip movement dimension, use 3D point cloud technology to obtain the three-dimensional coordinates of the lips and generate the first point cloud features. Then, associate audio data to extract audio features such as rhythm and intonation. After that, based on the basic image features of the digital human, fuse the correlation between audio and lip point cloud features to generate feature data with higher matching degree, laying the foundation for lip-sound synchronization and image realism.

[0018] Step 2: Dynamic Content Generation and Synthesis: For facial expressions and actions, an expression model is trained based on a GAN network to generate basic expressions such as smiling and frowning. Then, combined with text or audio semantics, the corresponding facial expression and action sequences are matched. For body movements, a trained MotionGAN model is used to convert text instructions into action vectors, which are then mapped to the skeletal nodes of the digital human to generate coherent body animations. For scene synthesis, a generation model such as StableDiffusion is used to generate a virtual background based on the video theme. Then, an image fusion algorithm is used to embed the digital human with expressions and actions into the background. At the same time, edge detection technology is used to optimize the connection between the digital human and the background to avoid incongruity.

[0019] Step 3: Construct a dynamically weighted multimodal feature fusion model: Automatically assign weights to image, point cloud, and audio features based on video themes, such as increasing the feature weights of hand movements and product images in live-streaming e-commerce scenarios; introduce a "temporal attention mechanism" to strengthen the correlation between features in adjacent frames and reduce action jumps; establish a feature cache pool to reuse static features such as digital human basic textures and skeletons during batch generation to improve generation efficiency.

[0020] Step 4: Promote lightweight and engineering optimization of generative models: Optimize models such as GAN and MotionGAN using "model pruning + quantization" technology to reduce GPU computing power requirements; develop a "low-precision fast generation" branch for real-time live streaming scenarios to prioritize frame rate; enable "high-precision distributed generation" mode for offline batch generation, utilizing multiple GPUs to process frame sequences in parallel; and integrate a model acceleration framework to improve inference speed.

[0021] Step 5: Design Differentiated Optimization and Rendering Strategies: Customize optimization algorithms for different styles of digital humans—realistic digital humans use "skin texture-preserving super-resolution" technology, while cartoon digital humans use "edge-enhancing filtering"; differentiate between real-time and offline rendering scenarios, controlling the resolution at 1080P and the frame rate at 60fps during real-time rendering, and supporting 4K resolution and HDR effects during offline rendering. A "real-time lighting and shadow adaptation" module is also added to ensure that the lighting and shadows of the digital human and the virtual background are consistent.

[0022] Step Six: Video Frame Optimization and Rendering Output: Visual optimization is performed on the generated frame-by-frame images. Noise is removed using image filtering algorithms in computer vision, and super-resolution technology is used to improve image clarity. Finally, relying on high-performance GPUs and virtual rendering engines, the processed frame sequences and synchronized audio are integrated and rendered to output digital human videos that meet the specified resolution. It also supports both batch generation and real-time generation modes to adapt to different application scenarios.

[0023] Step 7: Establish a fault degradation and redundancy backup system: Design a multi-level fault response mechanism, reduce background complexity in the case of a level 1 fault, automatically switch to a preset action sequence and virtual background in the case of a level 2 fault, and activate the "real-person emergency broadcast" interface in the case of a level 3 fault; configure backups for critical equipment to ensure seamless switching in the event of hardware failure.

[0024] Step 8: Construct a closed loop for data feedback and model iteration: Embed an "abnormal data collection module" in the live streaming and video generation system to automatically record data on issues such as lip misalignment and abnormal movements; collect audience feedback on the naturalness of the digital human through the user feedback interface, and use the labeled data for incremental model training; regularly update the expression library and action library to adapt to the needs of new scenarios.

[0025] As a preferred embodiment, the resolution of the digital human video is above 1080P, and the video themes include speeches, dances, and product promotions.

[0026] As a preferred embodiment, the light and shadow include light direction and shadow, and the key equipment includes a camera and an encoder.

[0027] As a preferred embodiment, the first-level fault includes feature point extraction delay, the second-level fault includes camera failure, and the third-level fault includes core model collapse.

[0028] As a preferred implementation, the new scenarios include metaverse live streaming and virtual press conferences.

[0029] Compared with the prior art, the technical effects and advantages of the present invention are as follows:

[0030] I. Breaking through the bottleneck of realism to achieve multimodal collaborative driving of digital humans

[0031] The technical solution completely changes the problem of digital humans being "similar in form but lacking in spirit" by constructing a full-link system of "visual capture + emotion-driven + motion optimization". In live streaming scenarios, tools such as MediaPipe are used to accurately capture key facial points, combined with a speech-to-lip model to achieve precise lip-sync, and monocular vision or simple motion capture equipment is used to complete body motion mapping, ensuring the consistency between digital human movements and real people in detail. The multimodal capture redundancy system further solves the problem of feature loss in complex environments such as insufficient lighting and occlusion by fusing multi-source data from infrared cameras and depth sensors, as well as feature point prediction models, thus improving capture stability. At the same time, the introduction of audio emotion analysis models and digital human muscle dynamics models realizes the linkage of "semantics-emotion-expression-movement", and with the optimization of motion transitions by Bezier curve interpolation, the digital human can not only move smoothly, but also convey subtle emotions, greatly improving the realism and immersion of the interaction.

[0032] II. Overcoming the challenge of interactive latency to ensure smooth real-time live streaming interaction

[0033] The solution constructs an edge + center collaborative latency control system, forming a full-process optimization from data acquisition to transmission. At the data processing end, edge nodes complete visual feature extraction and encoding locally, avoiding the massive data volume of transmitting complete video frames. The transmission stage employs lightweight feature encoding technology, combined with WebRTC protocol and UDP transmission, along with RTMP / HLS push-pull streaming protocol and FFmpeg fast encoding / decoding, significantly reducing audio and video latency. Frame synchronization processing extracts feature values ​​through OpenCV and performs timestamp alignment, further ensuring the accuracy of multi-terminal data collaboration. Furthermore, the latency monitoring module and dynamic priority adjustment mechanism can optimize transmission strategies in real time based on network conditions, responding to network fluctuations and ensuring timely command response and smooth interaction during live streaming, effectively avoiding interaction interruptions caused by latency.

[0034] III. Enhance scene adaptation capabilities to achieve full scene coverage across multiple terminals and styles.

[0035] The technical solution, through a layered adaptation framework, meets the diverse needs of different devices, styles, and scenarios. Regarding terminal adaptation, it pre-sets exclusive visual capture parameters for the hardware differences between mobile phones and PCs. On mobile devices, it enhances image contrast to improve feature recognition, while on PCs, it optimizes frame rate to ensure smooth animation, achieving cross-device compatibility. In terms of digital human style, it constructs a style feature library to match exclusive skeletal binding rules and motion mapping coefficients for realistic, cartoon, and other styles, coupled with differentiated rendering strategies—realistic digital humans use skin texture preservation technology, while cartoon digital humans apply edge enhancement filtering to ensure consistency in style and movement. In the video generation stage, it dynamically allocates feature weights for different themes such as speeches, dances, and product sales. For example, in product sales scenarios, it strengthens the weight of hand movements and product images, while also supporting new scenario expansions such as metaverse live streaming and virtual press conferences, significantly improving the technology's industry adaptability.

[0036] IV. Construct a redundancy and iteration system to enhance the stability and scalability of technology applications.

[0037] The solution employs end-to-end redundancy design and a closed-loop data feedback system to ensure stable operation and continuous optimization of the technology. Both the live streaming and video generation systems have established robust fault degradation mechanisms: in live streaming scenarios, from feature point extraction delays to core model crashes, a three-level fault response strategy is implemented, coupled with backups of critical equipment to achieve seamless switching; the video generation stage utilizes multi-level fault handling and emergency broadcast interfaces to prevent system interruptions. Regarding technology iteration, the abnormal data collection module automatically records issues such as lip-syncing misalignment and abnormal movements, combining this with audience feedback data to form a labeled dataset for incremental model training. Regularly updated expression libraries, action libraries, and segmented scene models ensure the technology can continuously adapt to new demands, forming a virtuous cycle of "application-feedback-optimization," providing support for the long-term implementation of digital human technology. Attached Figure Description

[0038] Figure 1 This is a flowchart of the computer vision-based digital human live streaming method of the present invention.

[0039] Figure 2 This is a flowchart of the video generation method for live streaming of digital humans based on computer vision according to the present invention. Detailed Implementation

[0040] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0041] Please see Figures 1 to 2A computer vision-based method for live streaming with digital humans includes the following steps:

[0042] Step 1: Real-time Visual Capture and Motion / Expression Driven: Computer vision tools are used to detect facial key points, accurately capturing facial features such as the lips and eyes of the live stream host or guest. This is then combined with a speech-to-lip model to bind the audio features of the live stream to the movement of lip key points, achieving precise synchronization between sound and lips. For body movements, a monocular camera combined with a pose estimation algorithm, or a simple motion capture device, can be used to collect limb joint data, which is then mapped onto a digital human skeletal model, allowing the digital human to mimic interactive actions such as raising a hand or nodding.

[0043] Step 2: Construct a multimodal capture redundancy system: On the basis of monocular vision, add "infrared camera + depth sensor" to assist in capture, and fuse multi-source data through computer vision algorithms - such as using infrared images to supplement facial feature points when the lighting is insufficient, and using depth data to segment the foreground and background when the background is complex, reducing environmental interference; for occluded scenes, introduce a "feature point prediction model" to fill in missing feature points based on the motion trend before occlusion.

[0044] Step 3: Incorporate emotion-driven and motion smoothing mechanisms: Introduce an "audio emotion analysis model" to associate emotion tags with an emoji library, achieving "semantic-emotion-expression" linkage; add "Bezier curve interpolation" processing to the motion-driven mechanism to optimize motion transition frames, and establish a "digital human muscle dynamics model" to make limb movements conform to biomechanical laws;

[0045] Step 4: Optimize speech-to-lip-sync adaptation capabilities: For multilingual and dialectal scenarios, pre-train subdivided speech-to-lip-sync models and introduce a "phoneme-lip-sync mapping dictionary" to improve the matching accuracy of special pronunciations; at the same time, associate lip movements with other facial areas to achieve full-face coordinated movement.

[0046] Step 5: Multi-device Data Collaboration and Latency Control: Achieve data synchronization between digital humans or real people and digital humans on multiple devices. Utilize computer vision algorithms to perform frame synchronization processing on the live stream. Extract the pose and facial expression feature values ​​of the digital human in each frame using OpenCV and align them with the timestamps of the other party's video stream features. Employ lightweight visual feature encoding technology, transmitting only captured key point data instead of complete video frames to reduce data volume. Simultaneously, combine RTMP push streaming and HLS pull streaming protocols with FFmpeg to complete fast encoding and decoding, reducing audio and video latency during live streams and ensuring smooth interaction.

[0047] Step Six: Construct an "Edge + Center" Collaborative Delay Control System: Deploy edge nodes on the host and guest terminals to process visual feature extraction and encoding locally, reducing the amount of data transmitted across regions; replace some RTMP / HLS scenarios with "WebRTC protocol + UDP transmission" to reduce real-time interaction latency, while adding a "data packet retransmission and redundant encoding" mechanism to cope with network fluctuations; establish a latency monitoring module to automatically reduce the transmission priority of non-critical features when the latency exceeds a threshold.

[0048] Step 7: Develop a multi-platform and multi-style adaptation framework: For different terminal devices, preset visual capture parameter templates, improve image contrast on mobile devices to enhance feature point recognition, and optimize frame rate on PC devices to improve motion smoothness; build a "digital human style feature library" to match exclusive skeletal binding rules and motion mapping coefficients for different styles such as realistic and cartoon, to ensure that the action and image are consistent.

[0049] Step 8: Expand the dimensions of interactive command recognition and response logic: Integrate the "voice recognition + image gesture recognition" module to achieve multimodal fusion recognition of bullet comments, voice, and gesture commands; establish a knowledge base for the association of "command-scene-action", such as when the "show product" command is detected, automatically trigger the digital human to "turn around to get props-point to product with gestures-introduce verbally" to improve the depth of interaction;

[0050] Step Nine: Visual Recognition and Response of Interactive Commands: Commands such as liking and requesting to connect during live streams can be recognized with the assistance of computer vision. For example, image recognition algorithms can be used to capture keywords and emoticons in the comments and convert them into trigger signals. When a request to connect is recognized, the digital human is driven to make response actions such as turning its head or reaching out its hand. When the screen is flooded with likes from the audience, the digital human is controlled to display interactive expressions such as smiling and waving, enhancing the immersive experience of the live stream.

[0051] Step 10: Live Streaming Self-Running and Broadcasting in the No-Instruction State: When the digital human does not receive any instructions, it runs the live stream content according to the preset script to ensure continuous broadcasting. The preset script includes action information and verbal information. When the digital human is running, it adjusts and transforms the preset script according to the instructions of the live stream connection to improve the smoothness, excitement and rationality of the live stream connection.

[0052] The computer vision tool includes MediaPipe, the speech-to-lip model includes dialect and bilingual versions, and the emotion tags include happy, serious, and surprised.

[0053] A video generation method for digital human live streaming based on computer vision includes the following steps:

[0054] Step 1: Multimodal visual feature fusion preprocessing: First, collect multi-dimensional data and extract visually relevant features. In the image dimension, use CNN to extract the first image features such as texture and contour of the basic image of the digital human. In the lip movement dimension, use 3D point cloud technology to obtain the three-dimensional coordinates of the lips and generate the first point cloud features. Then, associate audio data to extract audio features such as rhythm and intonation. After that, based on the basic image features of the digital human, fuse the correlation between audio and lip point cloud features to generate feature data with higher matching degree, laying the foundation for lip-sound synchronization and image realism.

[0055] Step 2: Dynamic Content Generation and Synthesis: For facial expressions and actions, an expression model is trained based on a GAN network to generate basic expressions such as smiling and frowning. Then, combined with text or audio semantics, the corresponding facial expression and action sequences are matched. For body movements, a trained MotionGAN model is used to convert text instructions into action vectors, which are then mapped to the skeletal nodes of the digital human to generate coherent body animations. For scene synthesis, a generation model such as StableDiffusion is used to generate a virtual background based on the video theme. Then, an image fusion algorithm is used to embed the digital human with expressions and actions into the background. At the same time, edge detection technology is used to optimize the connection between the digital human and the background to avoid incongruity.

[0056] Step 3: Construct a dynamically weighted multimodal feature fusion model: Automatically assign weights to image, point cloud, and audio features based on video themes, such as increasing the feature weights of hand movements and product images in live-streaming e-commerce scenarios; introduce a "temporal attention mechanism" to strengthen the correlation between features in adjacent frames and reduce action jumps; establish a feature cache pool to reuse static features such as digital human basic textures and skeletons during batch generation to improve generation efficiency.

[0057] Step 4: Promote lightweight and engineering optimization of generative models: Optimize models such as GAN and MotionGAN using "model pruning + quantization" technology to reduce GPU computing power requirements; develop a "low-precision fast generation" branch for real-time live streaming scenarios to prioritize frame rate; enable "high-precision distributed generation" mode for offline batch generation, utilizing multiple GPUs to process frame sequences in parallel; and integrate a model acceleration framework to improve inference speed.

[0058] Step 5: Design Differentiated Optimization and Rendering Strategies: Customize optimization algorithms for different styles of digital humans—realistic digital humans use "skin texture-preserving super-resolution" technology, while cartoon digital humans use "edge-enhancing filtering"; differentiate between real-time and offline rendering scenarios, controlling the resolution at 1080P and the frame rate at 60fps during real-time rendering, and supporting 4K resolution and HDR effects during offline rendering. A "real-time lighting and shadow adaptation" module is also added to ensure that the lighting and shadows of the digital human and the virtual background are consistent.

[0059] Step Six: Video Frame Optimization and Rendering Output: Visual optimization is performed on the generated frame-by-frame images. Noise is removed using image filtering algorithms in computer vision, and super-resolution technology is used to improve image clarity. Finally, relying on high-performance GPUs and virtual rendering engines, the processed frame sequences and synchronized audio are integrated and rendered to output digital human videos that meet the specified resolution. It also supports both batch generation and real-time generation modes to adapt to different application scenarios.

[0060] Step 7: Establish a fault degradation and redundancy backup system: Design a multi-level fault response mechanism, reduce background complexity in the case of a level 1 fault, automatically switch to a preset action sequence and virtual background in the case of a level 2 fault, and activate the "real-person emergency broadcast" interface in the case of a level 3 fault; configure backups for critical equipment to ensure seamless switching in the event of hardware failure.

[0061] Step 8: Construct a closed loop for data feedback and model iteration: Embed an "abnormal data collection module" in the live streaming and video generation system to automatically record data on issues such as lip misalignment and abnormal movements; collect audience feedback on the naturalness of the digital human through the user feedback interface, and use the labeled data for incremental model training; regularly update the expression library and action library to adapt to the needs of new scenarios.

[0062] The resolution of the digital human videos is above 1080P, and the video themes include speeches, dances, and product promotions.

[0063] The light and shadow effects include the direction of the light and the shadows. Key equipment includes cameras and encoders.

[0064] The first-level faults include feature point extraction delays, the second-level faults include camera malfunctions, and the third-level faults include core model crashes.

[0065] The new scenarios include metaverse live streaming and virtual press conferences.

[0066] It should be noted that in this article, relational terms such as one and two are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for live streaming and co-hosting digital humans based on computer vision, characterized by: Includes the following steps: Step 1: Real-time visual capture and motion / expression driving: Computer vision tools are used to detect facial key points and accurately capture facial feature points such as the lips and eyes of the live anchor or guest; then, a speech-to-lip model is used to bind the audio features of the live broadcast with the movement of the key points of the lips to achieve accurate synchronization of sound and lips. For body movements, a monocular camera combined with a posture estimation algorithm or a simple motion capture device can be used to collect limb joint data and map it to the digital human skeleton model, so that the digital human can imitate interactive actions such as raising a hand and nodding. Step 2: Construct a multimodal capture redundancy system: On the basis of monocular vision, add "infrared camera + depth sensor" to assist in capture, and fuse multi-source data through computer vision algorithms - such as using infrared images to supplement facial feature points when the light is insufficient, and using depth data to segment the foreground and background when the background is complex to reduce environmental interference; for occluded scenes, introduce a "feature point prediction model" to fill in missing feature points based on the motion trend before occlusion. Step 3: Incorporate emotion-driven and motion smoothing mechanisms: Introduce an "audio emotion analysis model" to associate emotion tags with an emoji library, achieving "semantic-emotion-expression" linkage; add "Bezier curve interpolation" processing to the motion-driven mechanism to optimize motion transition frames, and establish a "digital human muscle dynamics model" to make limb movements conform to biomechanical laws; Step 4: Optimize speech-to-lip-sync adaptation capabilities: For multilingual and dialectal scenarios, pre-train subdivided speech-to-lip-sync models and introduce a "phoneme-lip-sync mapping dictionary" to improve the matching accuracy of special pronunciations; at the same time, associate lip movements with other facial areas to achieve full-face coordinated movement. Step 5: Multi-device Data Collaboration and Latency Control: Achieve data synchronization between digital humans or real people and digital humans on multiple devices. Utilize computer vision algorithms to perform frame synchronization processing on the live stream. Extract the pose and facial expression feature values ​​of the digital human in each frame using OpenCV and align them with the timestamps of the other party's video stream features. Employ lightweight visual feature encoding technology, transmitting only captured key point data instead of complete video frames to reduce data volume. Simultaneously, combine RTMP push streaming and HLS pull streaming protocols with FFmpeg to complete fast encoding and decoding, reducing audio and video latency during live streams and ensuring smooth interaction. Step Six: Construct an "Edge + Center" Collaborative Delay Control System: Deploy edge nodes on the host and guest terminals to process visual feature extraction and encoding locally, reducing the amount of data transmitted across regions; replace some RTMP / HLS scenarios with "WebRTC protocol + UDP transmission" to reduce real-time interaction latency, while adding a "data packet retransmission and redundant encoding" mechanism to cope with network fluctuations; establish a latency monitoring module to automatically reduce the transmission priority of non-critical features when the latency exceeds a threshold. Step 7: Develop a multi-platform and multi-style adaptation framework: For different terminal devices, preset visual capture parameter templates, improve image contrast on mobile devices to enhance feature point recognition, and optimize frame rate on PC devices to improve motion smoothness; build a "digital human style feature library" to match exclusive skeletal binding rules and motion mapping coefficients for different styles such as realistic and cartoon, to ensure that the motion and image are consistent. Step 8: Expand the dimensions of interactive command recognition and response logic: Integrate the "voice recognition + image gesture recognition" module to achieve multimodal fusion recognition of bullet comments, voice, and gesture commands; establish a knowledge base for "command-scene-action" association. For example, when the "show product" command is detected, the digital human will automatically trigger a coherent action sequence of "turning around to get props - pointing to the product with gestures - verbally introducing the product", thereby improving the depth of interaction. Step Nine: Visual Recognition and Response of Interactive Commands: Commands such as liking and requesting to connect during live streams can be recognized with the assistance of computer vision. For example, image recognition algorithms can be used to capture keywords and emoticons in the comments and convert them into trigger signals. When a request to connect is recognized, the digital human is driven to make response actions such as turning its head or reaching out its hand. When the screen is flooded with likes from the audience, the digital human is controlled to display interactive expressions such as smiling and waving, enhancing the immersive experience of the live stream. Step 10: Live Streaming Self-Running and Broadcasting in the No-Instruction State: When the digital human does not receive any instructions, it runs the live stream content according to the preset script to ensure continuous broadcasting. The preset script includes action information and verbal information. When the digital human is running, it adjusts and transforms the preset script according to the instructions of the live stream connection to improve the smoothness, excitement and rationality of the live stream connection.

2. The computer vision-based digital human live streaming method according to claim 1, characterized in that: The computer vision tool includes MediaPipe, the speech-to-lip model includes dialect and bilingual versions, and the emotion tags include happy, serious, and surprised.

3. A video generation method for digital human live streaming based on computer vision, generated based on the live streaming co-op method for digital human live streaming based on computer vision as described in any one of claims 1-2, characterized in that: Includes the following steps: Step 1: Multimodal visual feature fusion preprocessing: First, collect multi-dimensional data and extract visually relevant features. In the image dimension, use CNN to extract the first image features such as texture and contour of the basic image of the digital human. In the lip movement dimension, use 3D point cloud technology to obtain the three-dimensional coordinates of the lips and generate the first point cloud features. Then, associate audio data to extract audio features such as rhythm and intonation. After that, based on the basic image features of the digital human, fuse the correlation between audio and lip point cloud features to generate feature data with higher matching degree, laying the foundation for lip-sound synchronization and image realism. Step 2: Dynamic Content Generation and Synthesis: For facial expressions and actions, an expression model is trained based on a GAN network to generate basic expressions such as smiling and frowning. Then, combined with text or audio semantics, the corresponding facial expression and action sequences are matched. For body movements, a trained MotionGAN model is used to convert text instructions into action vectors, which are then mapped to the skeletal nodes of the digital human to generate coherent body animations. For scene synthesis, a generation model such as StableDiffusion is used to generate a virtual background based on the video theme. Then, an image fusion algorithm is used to embed the digital human with expressions and actions into the background. At the same time, edge detection technology is used to optimize the connection between the digital human and the background to avoid incongruity. Step 3: Construct a dynamically weighted multimodal feature fusion model: Automatically assign weights to image, point cloud, and audio features based on video themes, such as increasing the feature weights of hand movements and product images in live-streaming e-commerce scenarios; Introduce a "temporal attention mechanism" to strengthen the correlation between features in adjacent frames and reduce action jumps; Establish a feature cache pool to reuse static features such as basic textures and skeletons of digital humans during batch generation, thereby improving generation efficiency. Step 4: Promote lightweight and engineering optimization of generative models: Optimize models such as GAN and MotionGAN using "model pruning + quantization" technology to reduce GPU computing power requirements; develop a "low-precision fast generation" branch for real-time live streaming scenarios to prioritize frame rate; enable "high-precision distributed generation" mode for offline batch generation, utilizing multiple GPUs to process frame sequences in parallel; and integrate a model acceleration framework to improve inference speed. Step 5: Design Differentiated Optimization and Rendering Strategies: Customize optimization algorithms for different styles of digital humans—realistic digital humans use "skin texture preservation super-resolution" technology, while cartoon digital humans use "edge enhancement filtering"; differentiate between real-time and offline rendering scenarios, controlling the resolution at 1080P and the frame rate at 60fps during real-time rendering, and supporting 4K resolution and HDR effects during offline rendering. A "real-time lighting and shadow adaptation" module is also added to ensure that the lighting and shadows of the digital human and the virtual background are consistent. Step Six: Video Frame Optimization and Rendering Output: Visual optimization is performed on the generated frame-by-frame images. Noise is removed using image filtering algorithms in computer vision, and super-resolution technology is used to improve image clarity. Finally, relying on high-performance GPUs and virtual rendering engines, the processed frame sequences and synchronized audio are integrated and rendered to output digital human videos that meet the specified resolution. It also supports both batch generation and real-time generation modes to adapt to different application scenarios. Step 7: Establish a fault degradation and redundancy backup system: Design a multi-level fault response mechanism. In the case of a Level 1 fault, reduce the background complexity; in the case of a Level 2 fault, automatically switch to a preset action sequence and virtual background; and in the case of a Level 3 fault, activate the "real-person emergency broadcast" interface. Configure backups for critical equipment to ensure seamless switching in the event of hardware failure. Step 8: Construct a closed loop for data feedback and model iteration: Embed an "abnormal data collection module" in the live streaming and video generation system to automatically record data on issues such as lip misalignment and abnormal movements; collect audience feedback on the naturalness of the digital human through the user feedback interface, and use the labeled data for incremental model training; regularly update the expression library and action library to adapt to the needs of new scenarios.

4. The video generation method for live streaming of digital humans based on computer vision according to claim 3, characterized in that: The resolution of the digital human videos is above 1080P, and the video themes include speeches, dances, and product promotions.

5. The video generation method for live streaming of digital humans based on computer vision according to claim 3, characterized in that: The light and shadow effects include the direction of the light and the shadows. Key equipment includes cameras and encoders.

6. The video generation method for live streaming of digital humans based on computer vision according to claim 3, characterized in that: The first-level faults include feature point extraction delays, the second-level faults include camera malfunctions, and the third-level faults include core model crashes.

7. The video generation method for live streaming of digital humans based on computer vision according to claim 3, characterized in that: The new scenarios include metaverse live streaming and virtual press conferences.

Images