Audio-driven true population type broadcast video generation method and system

Abstract

The invention discloses an audio-driven real-population broadcast video generation method and system, and relates to the technical field of video generation. The method comprises the steps of real person material collection, video clipping and expansion, background synthesis, video reverse order processing, face region clipping, audio material synthesis and face mouth shape driving. According to the invention, based on a basic real person video obtained by using a shooting acquisition method, a text speech synthesis technology is combined to generate a real person mouth type broadcast video which can be used for mouth type driving in any scene, and the method can be used in the fields of digital people, virtual anchors and speech assistants. According to the technical scheme provided by the invention, through a series of innovation points, the efficiency and the quality of generating the real population type broadcast video are improved, the application range in various scenes is widened, richer and more real interaction experience is brought to users, and meanwhile, powerful technical support is provided for the development of industries such as digital human broadcast and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of video generation, and relates to an audio-driven real-person mouth-type broadcast video generation method and system. Background Art

[0002] Facial lip-syncing technology is an advanced artificial intelligence application that generates virtual character facial expressions and lip-syncing changes in real time in sync with speech by analyzing speech signals. This technology combines technologies from multiple fields such as audio processing, deep learning, facial animation generation, and rendering synthesis, allowing virtual characters to "perform" in a very natural and realistic way.

[0003] In practical applications, face lip-driven technology is widely used to create virtual anchors, virtual teachers in online education platforms, virtual assistants in customer service, and animation and game characters in the entertainment industry. It not only improves production efficiency and reduces costs, but also provides users with a more immersive and interactive experience.

[0004] The invention discloses an audio-driven real-life mouth-shape broadcasting video generation method, which can be applied to the field of digital humans and provides a fast generation method for audio-based driven mouth-shape broadcasting for 2D real-life digital humans.

[0005] 2D digital humans are virtual images presented on a two-dimensional plane. They use computer graphics (CG) technology to simulate virtual characters similar to real people. 2D digital humans can be divided into two types: 2D real people and 2D cartoons. The production process does not require complex three-dimensional modeling, but is generated through pictures and video materials, which makes their production relatively simple and low-cost.

[0006] When making a 2D digital human, the first thing to do is to generate the character image, which usually involves character design and material collection. Then comes the character expression, including voice generation and animation generation. The mouth shape movement of a 2D digital human can be achieved through intelligent synthesis technology, which can train the model through the association mapping of input text to output audio and output visual information, thereby driving the mouth shape animation. For example, the Wav2lip model can be used to achieve voice-driven intelligent synthesis of 2D digital human mouth shape movements. This method does not require additional training and can modify the mouth shape in the video while keeping the other content of the original video unchanged.

[0007] Other actions besides mouth shape, such as facial expressions, blinking, shaking head, etc., are usually achieved by looping pre-recorded videos or animations. These actions can be selected and played by random or scripted strategies to generate smooth animation effects.

[0008] Existing technologies have disadvantages such as high cost, lack of realism and immersive experience, limited application scenarios, and lack of AI technology drive. Summary of the invention

[0009] In order to make up for the shortcomings of the prior art, the present invention provides an audio-driven real-person mouth shape broadcast video generation method and system. Based on the basic real-person video obtained by the shooting and acquisition method, combined with text-to-speech synthesis technology, a real-person mouth shape broadcast video that can be used for lip-driven in any scenario is generated, which can be used in the fields of digital humans, virtual anchors, and voice assistants.

[0010] The present invention is implemented by the following technical solution: The present invention provides an audio-driven real-person mouth-shaped broadcast video generation method, comprising the following steps: (1) Real-life material collection: Hire actors or models and conduct professional video shooting to obtain high-quality real-life original materials and ensure that the video meets the following requirements: 1.1. The duration is 6-12 seconds, and the lips are still; 1.2. The video size ratio is 9:16; 1.3. Video resolution is 1080P or 4K; 1.4. Frame rate is 25fps; 1.5. The format is MP4; 1.6. The real-person performance meets the customer's psychological expectations and the corresponding scene requirements; (2) Video cropping and expansion: Use video editing software to crop and resize the original video to suit the requirements of the target playback platform, and use high-quality scaling algorithms to avoid image distortion; (3) Background synthesis: Image processing techniques, including semantic segmentation, are used to synthesize the cropped person with a new background, ensuring that the new background matches the person’s lighting, color, and style. (4) Video reversal processing: The synthesized virtual background character video is processed in reverse order, including reverse sorting of video frames and adding inter-frame transition effects to achieve seamless transition between scenes; (5) Face region cropping: crop the face region in the video through an automated face detection algorithm or manual editing, and record the coordinates of the cropped region; (6) Audio material synthesis: Convert text content into speech using professional speech synthesis software, taking into account the intonation, rhythm, and emotion of the speech to match the character’s expressions and movements; (7) Face lip-driven: Analyze the speech features in the audio material and use the lip-driven algorithm to map these features to the video of the face area, accurately adjust the lip shape and facial muscle movements, simulate real speaking movements, and re-synthesize the processed face area into the original video.

[0011] Preferably, the video cropping and expansion steps include improving the video resolution and adjusting the picture ratio.

[0012] Preferably, the background synthesis step includes using advanced image processing technology to identify and separate characters in the video; synthesizing the separated character layer with a new background image or video to create a completely new scene.

[0013] Preferably, the step of driving the face and lip shape includes positioning the driven lip shape back to the coordinates of the original video.

[0014] An audio-driven real-person mouth-type announcement video generation system, comprising: Video acquisition unit, used to collect original live-action footage in high-definition quality; Video editing unit for cropping and expanding videos; Background synthesis unit, used to synthesize characters and new backgrounds; A video reversal processing unit, used for processing the reversal of video frames; A face cropping unit, used for cropping the face area; a speech synthesis unit, for preparing audio material; The lip-drive unit is used to drive the lip shape of the face area.

[0015] The beneficial effects of the present invention are: (1) Improving production efficiency and reducing costs: The present invention significantly improves the production efficiency of digital content through automated video cropping, expansion and high-quality scaling algorithms, as well as an integrated real-person population broadcast video generation system, while reducing dependence on professional equipment and personnel and reducing production costs.

[0016] (2) Enhanced realism and immersive experience: By adopting advanced background synthesis technology and video reversal processing technology, the present invention can create a realistic fusion effect of real people and new backgrounds, as well as smooth scene transitions, providing users with a more realistic immersive experience.

[0017] (3) Improving interactivity and personalized services: Through precise facial region cropping and facial lip shape driving technology, the present invention can achieve highly personalized and natural interaction of the broadcasting characters, and can better simulate the facial expressions and lip shapes of real humans, thereby improving the personalization and naturalness of the service.

[0018] (4) Expanded application scenarios: The technical solution of the present invention can be used not only in the fields of entertainment and education, but also in multiple industries such as finance, medical care, culture and tourism, providing customized virtual anchors, digital humans, voice assistants, etc. for different fields, and promoting the digital transformation of related industries.

[0019] (5) AI technology driven: The present invention utilizes AI technology, especially deep learning models, to drive the broadcast video generation process throughout, thereby improving the intelligent interaction capabilities of digital humans, voice assistants, etc.

[0020] The technical solution of the present invention not only improves the efficiency and quality of real-person population broadcast video generation through a series of innovations, but also broadens the scope of application in various scenarios, bringing users a richer and more realistic interactive experience, while also providing strong technical support for the development of industries such as digital human broadcasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with the accompanying drawings; Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0023] The accompanying drawings are a specific embodiment of an audio-driven real-person mouth-type broadcast video generation method of the present invention. The embodiment includes.

[0024] Real-person material collection: Before starting the lip-sync video generation process, you first need to prepare real-person original materials. This includes hiring actors or models and shooting professional videos according to project requirements. During the shooting process, ensure that the video quality meets high-definition standards so that enough details can be captured in subsequent processing. In addition, the actor's expressions, movements, and language should be natural and smooth so that the generated results can be more realistic and natural. When shooting, you should also pay attention to the simplicity of the background so that it is easier to separate the characters from the background in subsequent steps. The reference shooting requirements are as follows: 1. Duration: 6-12s lip still video; 2. Video size: The best video ratio is 9:16; 3. Video resolution: 1080P or 4K; 4. Frame rate: 25fps; 5. Format: MP4; 6. Status: The real-person performance status in the entire submitted video can meet the customer's psychological expectations and corresponding scene requirements.

[0025] According to the requirements of the video shooting, real-life video shooting is carried out, usually a 10-second green screen video with facial expressions is shot for the original video material of the character. This is the basis for the entire video generation, ensuring that the lighting, angle and background of the recording environment meet high standards in order to capture the highest quality real-life movements and expressions.

[0026] Video cropping and expansion: After obtaining the original video, the next task is to crop and resize the video. First, use video editing software to accurately crop the characters according to their position and size in the picture, ensuring that only the necessary character parts are retained. Then, according to the requirements of the target playback platform, the cropped video is expanded in size, which may involve increasing the resolution and adjusting the aspect ratio. In this process, high-quality scaling algorithms can be used to avoid image distortion.

[0027] The original video is cropped to capture the person, and the video is then expanded to the target size based on the person's size and coordinate position. After recording, the original video is carefully edited. This includes removing unnecessary background elements and accurately cropping the person to ensure that only the necessary part of the person is retained. Next, the cropped video is adjusted to the desired target size to accommodate different playback platforms and viewing devices.

[0028] Background synthesis: After the person video is cropped, the next step is to synthesize the person with a new background. This usually involves using advanced image processing techniques, such as semantic segmentation, to identify and separate the person in the video. The separated person layer is then composited with a new background image or video to create a completely new scene. This step requires ensuring that the new background matches the lighting, color, and style of the person to achieve a realistic visual effect.

[0029] Using advanced semantic segmentation technology, we can accurately extract character pixels from the original footage and synthesize them with the predetermined scene images. This process requires a high degree of technical expertise to ensure that the fusion of the character and the new background is both natural and seamless, thus generating a virtual scene character video with the same length as the original video and a specific background.

[0030] Video reverse processing: In order to ensure the smoothness of the video when switching between different scenes, the synthesized virtual scene character video needs to be reversed. This includes reverse sorting of video frames and possible addition of inter-frame transition effects to ensure that the video can seamlessly transition from the current scene to the next scene when playing.

[0031] The synthesized virtual scene character video is expanded and reversed to ensure seamless connection when switching and playing the video.

[0032] Face area cropping: After the video is processed in reverse order, the face part of the synthesized virtual scene character video is cropped. This step can be achieved through an automated face detection algorithm or through manual editing. Regardless of the method used, it is necessary to ensure that the cropped face area remains consistent and accurate in the video. At the same time, the coordinates of the cropped area are recorded, which will be used to locate the driven lip shape back to the original video later.

[0033] Based on the synthesized video, the face is precisely cropped. This step is crucial because it directly affects the lip position, ultimately the character's expressiveness and the audience's perception. Rapid cropping can be achieved through automatic face detection technology, or manual cropping can be used to obtain finer control. The cropped face video will be used for subsequent lip-driven processing.

[0034] Audio material synthesis: Use speech synthesis technology to synthesize the text to be broadcast into an audio file. While performing face cropping, you also need to prepare audio materials. This usually involves converting text content into speech, which can be achieved using professional speech synthesis software. During the synthesis process, you need to consider the tone, rhythm, and emotion of the speech to ensure that the final audio matches the character's expression and movements.

[0035] Face lip-driven: The last step is face lip-driven, which is the most critical step in the entire broadcast video generation process. First, the speech features in the audio material need to be analyzed, and then these features are mapped to the face area of ​​the video using the lip-driven algorithm. This requires precise adjustment of the shape of the lips and the movement of the facial muscles to simulate real speaking movements. After the lip-driven algorithm is completed, the processed face area is relocated and synthesized into the original video, completing the entire process of generating the real mouth broadcast video.

[0036] The lip-driven algorithm dynamically adjusts the lip shape of the face area according to the audio file. This process involves complex facial animation technology to ensure that the lip shape perfectly matches the speech output.

[0037] The key technical points of the present invention include: High-definition real-person material acquisition technology: This invention uses high-definition video acquisition technology to ensure the high quality of real-person materials and provide clear and detailed raw data for the generation of real-person population broadcast videos. This innovation solves the problem of insufficient material quality in the traditional broadcast video generation process and improves the realism and detail expression of the real-person broadcast scene.

[0038] Precise video cropping and expansion algorithm: Through precise video cropping and expansion algorithm, the present invention can optimize the video according to the requirements of the target playback platform, including resolution enhancement and picture ratio adjustment, while avoiding image distortion. This innovation improves the adaptability and playback quality of the generated video.

[0039] Advanced background synthesis technology: This invention uses advanced image processing technologies such as semantic segmentation to achieve realistic synthesis of characters and new backgrounds. This innovation improves the visual effects and scene authenticity of the video through precise separation of characters and backgrounds, as well as matching of new backgrounds.

[0040] Video reverse processing technology: The present invention uses video reverse processing technology to achieve seamless transition between different scenes of the broadcast video. This innovation improves the fluency and viewing experience of the broadcast video by reversely sorting the video frames and adding inter-frame transition effects.

[0041] Automatic face region cropping technology: This invention uses an automatic face detection algorithm to achieve accurate cropping of the face region in the video. This innovation improves the accuracy and efficiency of face region cropping and provides high-quality basic data for subsequent lip-sync driving and facial expression generation.

[0042] Professional audio material synthesis and matching technology: This invention converts text content into speech through speech synthesis algorithms, etc., and considers the intonation, rhythm and emotion of the speech to match the expressions and actions of the characters. This innovation improves the authenticity and expressiveness of the speech, making the character broadcast more vivid and natural.

[0043] Accurate face lip-driven technology: This invention uses a lip-driven algorithm to accurately map the voice features in the audio material to the video of the face area, achieving accurate simulation of lip shape and facial muscle movement. This innovation improves the authenticity of the mouth shape and facial expression when the person speaks, making the person's broadcast more realistic.

[0044] Integrated real-person mouth type broadcast video generation system: The present invention provides an integrated real-person mouth type broadcast video generation system, including multiple modules such as video acquisition, editing, background synthesis, reverse order processing, face cropping, speech synthesis and lip-activated driving. This innovation realizes the automation and efficiency of the broadcast video generation process through systematic design and integration.

[0045] The present invention achieves a breakthrough in the field of real-person mouth-type broadcast video generation, improves the realism, expressiveness and viewing experience of the character broadcast, and has important practical value and market prospects.

Claims

1. A method for generating an audio-driven real-person mouth-type broadcast video, characterized in that: The following steps are involved: (1) Real-life material collection: Hire actors or models and conduct professional video shooting to obtain high-quality real-life original materials and ensure that the video meets the following requirements: The duration is 6-12 seconds, and the lips are still; The video size ratio is 9:16; Video resolution is 1080P or 4K; The frame rate is 25fps; The format is MP4; The real-life performance meets the customer's psychological expectations and the corresponding scene requirements; (2) Video cropping and expansion: Use video editing software to crop and resize the original video to suit the requirements of the target playback platform, and use high-quality scaling algorithms to avoid image distortion; (3) Background synthesis: Image processing techniques, including semantic segmentation, are used to synthesize the cropped person with a new background, ensuring that the new background matches the person’s lighting, color, and style. (4) Video reversal processing: The synthesized virtual background character video is processed in reverse order, including reverse sorting of video frames and adding inter-frame transition effects to achieve seamless transition between scenes; (5) Face region cropping: crop the face region in the video through an automated face detection algorithm or manual editing, and record the coordinates of the cropped region; (6) Audio material synthesis: Convert text content into speech using professional speech synthesis software, taking into account the intonation, rhythm, and emotion of the speech to match the character’s expressions and movements; (7) Face lip-driven: Analyze the speech features in the audio material and use the lip-driven algorithm to map these features to the video of the face area, accurately adjust the lip shape and facial muscle movements, simulate real speaking movements, and re-synthesize the processed face area into the original video.

2. The audio-driven real-person mouth-type broadcast video generation method according to claim 1 is characterized in that: The video cropping and expansion steps include improving the video resolution and adjusting the picture ratio.

3. The audio-driven real-person mouth-type broadcast video generation method according to claim 1 is characterized in that: The background synthesis step includes using advanced image processing technology to identify and separate characters in the video; synthesizing the separated character layer with a new background image or video to create a new scene.

4. The audio-driven real-person mouth-type broadcast video generation method according to claim 1, characterized in that: The step of driving the face and mouth shape includes positioning the driven mouth shape back to the coordinates of the original video.

5. An audio-driven real-person mouth-type broadcast video generation system, characterized in that: include, Video acquisition unit, used to collect original live-action footage in high-definition quality; Video editing unit for cropping and expanding videos; Background synthesis unit, used to synthesize characters and new backgrounds; A video reversal processing unit, used for processing the reversal of video frames; A face cropping unit, used for cropping the face area; a speech synthesis unit, for preparing audio material; The lip-drive unit is used to drive the lip shape of the face area.

Images