Video generation method and device based on layer separation, equipment and storage medium

Abstract

The invention relates to a video generation method based on layer separation, and the method comprises the steps: segmenting a portrait image containing a character image, and decomposing the image into a foreground layer and a background layer; and driving a character model in the foreground image layer by using the acquired voice data, generating facial expression and mouth shape changes synchronous with the voice, and generating a corresponding character speech video. And generating a static background video sequence by processing the background image layer, and further generating a background video. And performing layer merging on the character speech video and the background video to generate a target speech video. According to the method, the foreground and the background in the image are independently processed, so that the synthesis precision and the processing efficiency of video generation are remarkably improved, and greater flexibility is provided for later dynamic adjustment and personalized processing.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video generation method, device, equipment and storage medium based on layer separation. Background Art

[0002] As the pace of digital transformation in the financial sector accelerates, intelligent and automated technology solutions have become a key means to improve business efficiency and customer experience. As an emerging AI application, video generation technology is gradually being applied and promoted in these traditional industries.

[0003] In the insurance claims process, traditional manual evaluation is often time-consuming and easily affected by human factors. By introducing video generation technology, insurance companies can automatically generate dynamic videos corresponding to voice descriptions or text information to intuitively display the situation of accidents or damages. This technology can not only speed up the claims process, but also improve the accuracy and fairness of claims results.

[0004] In the financial field, video generation technology also has broad application prospects. For example, in customer service, voice-driven virtual customer service can generate animated videos that match the content of customer conversations, providing customers with a more vivid and interactive service experience. In addition, in the marketing and promotion of financial products, video generation technology can quickly generate personalized product introduction videos to improve marketing effectiveness and customer acceptance. However, existing video generation methods still face some challenges in practical applications.

[0005] In traditional video generation methods, the foreground and background are usually processed as a whole, resulting in an unnatural fusion effect of the foreground and background in complex scenes. This processing method is prone to inaccurate synthesis problems, especially when it comes to complex images or scene changes, it is difficult to achieve high-precision synthesis effects. Since the foreground and background cannot be processed independently in existing methods, this increases redundant calculations in the processing process, especially in the processing of high-resolution videos, resulting in a waste of computing resources and reducing the overall processing efficiency of the system. Summary of the invention

[0006] The main purpose of the present invention is to provide a video generation method, device, equipment and storage medium based on layer separation, aiming to solve the technical problem that the prior art fails to effectively separate and independently process the foreground and background in the image, resulting in low synthesis accuracy and poor processing efficiency in the video generation process.

[0007] To achieve the above object, the present invention provides a video generation method based on layer separation, comprising:

[0008] Acquire a portrait image containing a person image, and segment the portrait image to obtain a person foreground layer and a background layer;

[0009] Acquire voice data, and generate a character speech video based on the voice data and the character foreground layer, wherein the voice data is used to drive the facial expression and lip shape changes of the character;

[0010] Copying the background layer frame by frame to generate a still background video sequence, and generating a background video based on the background video sequence;

[0011] The character speech video and the background video are layer-merged to generate a target video.

[0012] In one embodiment, segmenting the portrait image to obtain a person foreground layer and a background layer includes:

[0013] Inputting the portrait image into a pre-trained real-time portrait segmentation network, extracting a mask representing a person region from the portrait image, wherein the mask is used to identify pixel points belonging to the foreground of the person in the portrait image, and forming a binary image, wherein the foreground portion is a valid region and the background portion is an invalid region;

[0014] Using the mask, the person region in the portrait image is separated from the overall image to generate an independent person foreground layer, wherein the person foreground layer only retains pixel information of the person;

[0015] Based on the mask, the person area is removed from the portrait image, the background part is retained, and a background layer is generated. The background layer only contains background pixel information after the person is removed.

[0016] In one embodiment, generating a person speech video based on the voice data and the person foreground layer includes:

[0017] The character foreground layer is used as a reference frame, and a three-dimensional face model is generated based on the reference frame using a three-dimensional modeling technology, wherein the three-dimensional face model describes the static geometric structure of the face and the position data of the key points of the skeleton;

[0018] The three-dimensional face model is driven by the voice data to generate facial dynamic expression parameters and skeletal motion parameters frame by frame, and the generated dynamic expression parameters and skeletal motion parameters are mapped to a two-dimensional space to generate a two-dimensional facial animation for each frame;

[0019] The generated two-dimensional facial animation is rendered frame by frame using a renderer, and a super-resolution reconstruction technique is applied to generate the character's speech video.

[0020] In one embodiment, the three-dimensional face model is driven by the voice data to generate facial dynamic expression parameters and skeletal motion parameters frame by frame, and the generated dynamic expression parameters and skeletal motion parameters are mapped to a two-dimensional space to generate a two-dimensional facial animation for each frame, including:

[0021] Extracting key speech features from the speech data, wherein the key speech features include phonemes, intonation, rhythm, and pause information, so as to drive the three-dimensional face model to generate corresponding facial expression changes;

[0022] Using the key speech features to drive the three-dimensional face model, generating facial dynamic expression parameters and bone movement parameters frame by frame, so that the expression changes in the three-dimensional face model correspond to the key speech features in the speech data;

[0023] Mapping the dynamic expression parameters and bone motion parameters of the generated 3D face model into 2D space to generate corresponding 2D facial animation;

[0024] The generated two-dimensional facial animation is time-aligned with the speech data so that each frame of the animation matches the speech data in time, and the lip animation is made consistent with the speech content through frame-by-frame adjustment.

[0025] In one embodiment, generating a background video based on the background video sequence includes:

[0026] Using a pre-trained video restoration model to process the mask area in the background video sequence, repairing and completing the missing background information frame by frame, and generating a preliminary rough background video;

[0027] Calling the BLIP model to perform spatial feature analysis on each frame of the rough background video, and generating a spatial feature promt for guiding the image redrawing process;

[0028] Based on the generated spatial feature prompt, the rough background video is redrawn frame by frame using the Stable Diffusion technology;

[0029] After redrawing frame by frame, a super-resolution reconstruction technique is applied to each frame of the background video to generate the background video.

[0030] In one embodiment, after applying the super-resolution reconstruction technology to each frame of background video, the method further includes:

[0031] Extracting camera parameters from the person's speech video, and applying the camera parameters to the background video to match the visual effect of the person's speech video;

[0032] Based on the focal length value in the camera parameter, adjusting the focal length in the background video so that the focal length in the background video is consistent with that in the person's speech video;

[0033] Based on the aperture value in the camera parameters, the aperture size of the background video is adjusted so that the aperture sizes of the background video and the person's speech video are consistent;

[0034] Based on the focus point in the camera parameters, the focus point of the background video is adjusted so that the focus of the background video and the person's speech video are consistent;

[0035] Identify lighting conditions from the person's speech video, and adjust lighting parameters of the background video based on the lighting conditions to ensure that the background video and the person's speech video are consistent in lighting effects.

[0036] In one embodiment, merging the character speech video and the background video in layers to generate a target video includes:

[0037] Set the character speech video and background video as independent layers, so that the character speech video and background video occupy different video tracks respectively;

[0038] Adjust the timeline of the character's speech video and the background video so that the starting and ending points of the two are consistent;

[0039] Place the character speech video layer above the background video layer so that the character speech video covers the background video;

[0040] Adjust the transparency and blending mode of the character's speech video layer to make the fusion effect of the character's speech video and the background video meet the target effect;

[0041] The target video is obtained by merging the person's speech video with the background video.

[0042] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a video generation device based on layer separation, wherein the video generation device based on layer separation comprises a memory, a processor, and a video generation program based on layer separation stored in the memory and executable on the processor, wherein the video generation program based on layer separation, when executed by the processor, implements the steps of the video generation method based on layer separation as described above.

[0043] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a video generation program based on layer separation is stored, and when the video generation program based on layer separation is executed by a processor, the steps of the video generation method based on layer separation as described above are implemented.

[0044] Beneficial effects: The present invention relates to a method for generating a video based on layer separation, which decomposes a portrait image containing a human figure into a foreground layer and a background layer by segmenting the image. The acquired voice data is used to drive the character model in the foreground layer to generate facial expressions and lip shape changes synchronized with the voice, and a corresponding character speech video is generated. By processing the background layer, a static background video sequence is generated, and a background video is further generated. The character speech video and the background video are layer-merged to generate a target speech video. The present invention significantly improves the synthesis accuracy and processing efficiency of video generation by independently processing the foreground and background in the image, and provides greater flexibility for later dynamic adjustment and personalized processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0046] Figure 1 It is a flow chart of an embodiment of a method for generating a video based on layer separation according to the present invention;

[0047] Figure 2 It is a functional module diagram of a preferred embodiment of the video generation device based on layer separation of the present invention;

[0048] Figure 3 It is a structural schematic diagram of the device hardware operating environment involved in the embodiment of the video generation device based on layer separation of the present invention. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] As the pace of digital transformation in the financial sector accelerates, intelligent and automated technology solutions have become a key means to improve business efficiency and customer experience. As an emerging AI application, video generation technology is gradually being applied and promoted in these traditional industries.

[0051] In the insurance claims process, traditional manual evaluation is often time-consuming and easily affected by human factors. By introducing video generation technology, insurance companies can automatically generate dynamic videos corresponding to voice descriptions or text information to intuitively display the situation of accidents or damages. This technology can not only speed up the claims process, but also improve the accuracy and fairness of claims results.

[0052] In the financial field, video generation technology also has broad application prospects. For example, in customer service, voice-driven virtual customer service can generate animated videos that match the content of customer conversations, providing customers with a more vivid and interactive service experience. In addition, in the marketing and promotion of financial products, video generation technology can quickly generate personalized product introduction videos to improve marketing effectiveness and customer acceptance. However, existing video generation methods still face some challenges in practical applications.

[0053] In traditional video generation methods, the foreground and background are usually processed as a whole, resulting in an unnatural fusion effect of the foreground and background in complex scenes. This processing method is prone to inaccurate synthesis problems, especially when it comes to complex images or scene changes, it is difficult to achieve high-precision synthesis effects. Since the foreground and background cannot be processed independently in existing methods, this increases redundant calculations in the processing process, especially in the processing of high-resolution videos, resulting in a waste of computing resources and reducing the overall processing efficiency of the system.

[0054] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for generating a video based on layer separation provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0055] like Figure 1 As shown, the video generation method based on layer separation proposed by the present invention includes the following steps:

[0056] S10, acquiring a portrait image containing a person image, and segmenting the portrait image to obtain a person foreground layer and a background layer;

[0057] In this embodiment, the input portrait image can be obtained in various ways, such as camera shooting, image library retrieval or interception from a video frame. The segmentation process is usually completed by a pre-trained real-time portrait segmentation network, which is based on deep learning technology and can automatically identify the character area in the image and generate a corresponding mask. The mask is used to distinguish the foreground and background parts belonging to the character in the image, thereby generating independent foreground layers and background layers. For example, in a financial scenario, the system may need to process the customer's video call image. By segmenting the customer's image from the background, the system can independently analyze the customer's expression, posture, etc., and blur the background part or replace it with a standardized background to ensure privacy protection or create a specific service environment.

[0058] In one specific embodiment, during the insurance claim process, the system obtains a portrait image of the claims agent through the camera of a mobile device. The segmentation network separates the agent in the image from the background to generate a foreground layer and a background layer. The foreground layer contains the agent's facial features and is used to generate a voice-driven facial expression video; the background layer displays a simulation of the accident scene or a photo of the claims scene. Based on the agent's voice data, the system generates a video of the agent's speech, and creates a more intuitive and credible claims explanation video by synthesizing the video with the background simulation. The transparency and efficiency of the claims process are improved, and disputes caused by the complexity of the on-site situation are reduced.

[0059] By separating the foreground and background in the image, higher synthesis accuracy and flexibility are achieved in the video generation process. Independent processing of the foreground layer and the background layer not only improves processing efficiency, but also better responds to dynamic changes and personalized needs. It is suitable for a variety of application scenarios in the financial, insurance and other industries, thereby providing users with more accurate and personalized video generation services.

[0060] S20, acquiring voice data, and generating a character speech video based on the voice data and the character foreground layer, wherein the voice data is used to drive the facial expression and lip shape changes of the character;

[0061] In this embodiment, voice data is used to generate facial animations that match the voice data. By analyzing the phonemes, rhythm, and intonation information in the voice, the facial model in the character foreground layer is driven to achieve accurate lip synchronization and expression changes.

[0062] Voice data can be generated through real-time recording, pre-recorded audio, or text-to-speech (TTS) technology. This process can be performed in a variety of scenarios, such as real-time conversations, customer service call recordings, or pre-recorded voice files. By mapping features in the voice data (such as phonemes, rhythm, intonation, etc.) to the facial model in the character foreground layer, corresponding facial expressions and lip shape changes are generated. This process usually uses deep learning models, such as phoneme-to-lip shape mapping models (Phoneme-to-Viseme Mapping), to ensure that facial movements are synchronized with the speech content. Based on the generated facial expressions and lip shape animations, these changes are applied to the character foreground layer, and a complete speech video matching the voice data is generated through the rendering engine.

[0063] In another specific implementation, in a remote financial advisory service, the system obtains the voice data of the financial advisor and combines it with the advisor's foreground layer. In addition to processing basic voice features, the system also analyzes key financial terms and customer concerns in the voice content through natural language processing (NLP) technology. These key terms and questions are marked as "points of interest". When the system generates a video of the advisor's speech, these points of interest are reflected through changes in facial expressions and body language (such as raised eyebrows and tilted head). This implementation method allows the generated video to not only synchronize the advisor's voice content, but also enhance the communication effect through visual cues.

[0064] By combining voice data and the person foreground layer to generate a synchronized speech video, the authenticity and interactivity of the video content can be effectively improved. Voice-driven facial expressions and lip shape changes make the generated video more vivid and suitable for personalized services and automated processing in a variety of scenarios, providing users with a more natural and accurate video experience.

[0065] S30, copying the background layer frame by frame to generate a static background video sequence, and generating a background video based on the background video sequence;

[0066] In this embodiment, a static background video sequence is created by copying the background layer frame by frame, and then a complete background video is generated based on the sequence. This process is intended to ensure that the background remains consistent during video playback, thereby providing a stable visual environment for the foreground characters.

[0067] Get a single frame image from the background layer and copy it frame by frame on the timeline to form a background video sequence. Each frame in the sequence is still, ensuring that the background remains unchanged throughout the video. Use the generated background video sequence for further video synthesis or optimization. In this process, you can add special effects, perform super-resolution reconstruction, or redraw the background frame by frame through Stable Diffusion technology to improve the quality and visual effects of the background video. For example, in remote communication of financial services, the background can be replaced with the bank's iconic building or company logo. By copying it frame by frame, the background remains consistent throughout the video call, thereby enhancing the brand image and customer experience.

[0068] In one specific implementation, in a financial video conferencing system, when the system copies the acquired background layer frame by frame, the background image of each frame will be dynamically adjusted for lighting and color balance to match the changes in the expression of the person in the foreground. For example, when the conference host uses a certain tone to emphasize a key point, the background color may be slightly darker or the brightness may be reduced. This frame-by-frame background copying not only ensures the consistency of the background, but also enhances the visual and expressive effects of the video through subtle lighting changes.

[0069] In another specific implementation, during the generation of financial advertising videos, the frame-by-frame replication of the background layer is not a simple static background, but is combined with real-time updates of market data. For example, when introducing financial products, the background video sequence will display real-time stock market trends or currency exchange rate changes frame by frame. When the system generates these background video sequences, it combines frame-by-frame replication with real-time data rendering, so that the background can dynamically reflect the actual market fluctuations while ensuring consistency. This method improves the real-time and viewing quality of the advertisement.

[0070] By copying the background layer frame by frame to generate a static background video sequence, and then generating a complete background video based on the sequence, the consistency and stability of the background in the video is ensured. This not only improves the synthesis quality of the video, but also provides greater flexibility for dynamic special effects and subsequent processing, and is suitable for video generation needs in a variety of financial, insurance and other scenarios.

[0071] S40, merging the character speech video and the background video in layers to generate a target video.

[0072] In this embodiment, two independently generated video layers, namely the person's speech video and the background video, are merged to form the final target video. The layer merging technology allows the display order, transparency and blending mode of different layers to be flexibly processed and adjusted during the video production process to achieve the desired visual effect.

[0073] The system uses the generated speech video of the person as the foreground layer and the background video as the background layer, and superimposes the two together. This merging process is not just a simple superposition, but through the transparency of the layer, blending mode and other parameters, the fusion effect of the foreground person and the background is precisely controlled, so that the transition between the person and the background is natural and coordinated. After the layer merging is completed, the system renders the entire video to generate the final target video. The video retains all the necessary visual information, including the dynamic expressions of the foreground person and the detailed display of the background. In the promotional video of financial services, the system can synthesize the speech video of the explainer with the background video showing the characteristics of the financial product to generate a promotional video that not only shows the professional image of the explainer, but also shows the advantages of the product through the background.

[0074] In one specific implementation, in the production of a financial training video, the system merges the speaker's speech video with a dynamic background video. The background video shows the real-time data flow of the financial market, while the foreground video shows the speaker's interpretation of the market dynamics. During the merging process, the system adjusts the transparency and position of the foreground layer so that the market data flows around the speaker's image, creating an immersive visual experience. Through this merging, the generated video can not only intuitively show the real-time changes in the financial market, but also allow the audience to connect the speech content with the market data, thereby deepening their understanding.

[0075] In another specific implementation, in a financial product demonstration video, the system obtains a speech video of the explainer and multiple background videos, which respectively display different product features. During the layer merging process, the system uses dynamic layering technology to switch different background videos according to the explanation content, and at the same time moves the explainer in the foreground speech video to different positions in the background video. This method allows the background video to be closely associated with the explanation content, allowing the audience to see the various features of the product in turn in one video, thereby enhancing the demonstration effect of the video.

[0076] By merging the speech video of the person and the background video into layers to generate the target video, the visual effect is not only natural and coordinated, but also able to display complex information and dynamic changes according to different application scenarios, providing strong support for video production in the fields of finance, insurance, etc.

[0077] The present invention relates to a video generation method based on layer separation, which decomposes a portrait image containing a human figure into a foreground layer and a background layer by segmenting the image. The acquired voice data is used to drive a human model in the foreground layer to generate facial expressions and lip shape changes synchronized with the voice, and a corresponding human speech video is generated. By processing the background layer, a static background video sequence is generated, and a background video is further generated. The human speech video and the background video are layer-merged to generate a target speech video. The present invention significantly improves the synthesis accuracy and processing efficiency of video generation by independently processing the foreground and background in the image, and provides greater flexibility for later dynamic adjustment and personalized processing.

[0078] In one embodiment, in the above S10, segmenting the portrait image to obtain a person foreground layer and a background layer includes:

[0079] S101, inputting the portrait image into a pre-trained real-time portrait segmentation network, extracting a mask representing a person region from the portrait image, wherein the mask is used to identify pixel points belonging to the foreground of the person in the portrait image, and forming a binary image, wherein the foreground portion is a valid region and the background portion is an invalid region;

[0080] S102, using the mask to separate the person region in the portrait image from the overall image, generating an independent person foreground layer, wherein the person foreground layer only retains pixel information of the person;

[0081] S103, based on the mask, removing the person area from the portrait image, retaining the background part, and generating a background layer, wherein the background layer only contains background pixel information after the person is removed.

[0082] In this embodiment, the input portrait image is segmented into a foreground layer and a background layer for subsequent independent processing. Through the pre-trained real-time portrait segmentation network, the technology can accurately identify the human area in the image and distinguish it from the background area.

[0083] Using deep learning models (such as Mask R-CNN or U-Net), these models have been pre-trained on a large amount of annotated data and can process input portrait images in real time. The network generates a mask representing the person area by identifying the person image in the image. The person area is extracted from the input portrait image and a binary image is generated, in which the person area is identified as the valid area (foreground) and the other parts are identified as the invalid area (background). This mask divides the image into foreground and background, where the foreground area is usually white and the background area is black. Using the generated mask, the system separates the person area in the portrait image from the overall image and generates an independent foreground layer. This foreground layer only retains the pixel information of the person, and the other background parts become transparent, so that the person part can be operated separately in subsequent processing. At the same time, the system removes the person area from the portrait image based on the mask, retaining only the pixel information of the background part. The generated background layer contains the background pixel information after the person is removed, which can be processed and optimized independently.

[0084] In one specific implementation, in an intelligent video conferencing system, the system obtains a portrait image of a participant and inputs it into a pre-trained real-time portrait segmentation network. The system separates the foreground and background layers of the person through a mask. Unlike conventional methods, this implementation adopts a dynamic mask update technology to recalculate the mask every several frames to cope with the slight movement of the participant or light changes. This dynamic update ensures that the person in the foreground layer always remains clear and accurate, and the stability of the background layer is enhanced, so that the generated video can maintain good visual effects even under complex lighting conditions.

[0085] In another specific implementation, in a remote customer service scenario, the system obtains a customer portrait image and inputs it into a segmentation network, and separates the foreground and background layers through the generated mask. Different from the traditional binary mask, this implementation adopts a multi-level masking technology to further subdivide the foreground into multiple areas, such as face, gestures, etc., and generate a separate foreground layer for each area. These layers are processed independently and then synthesized, so that during the service process, the system can adjust the display effects of the face and hands according to different changes in voice or gestures, and finally generate a highly interactive customer service video.

[0086] This embodiment achieves high-precision foreground and background separation through mask-based image segmentation technology, ensuring that in the subsequent video generation and processing process, the foreground characters and background can be flexibly and independently operated and optimized. This not only improves the synthesis accuracy of the video, but also provides greater processing flexibility and efficiency for a variety of application scenarios.

[0087] In one embodiment, in the above S20, generating a person speech video based on the voice data and the person foreground layer includes:

[0088] S201, using the person foreground layer as a reference frame, and generating a three-dimensional face model based on the reference frame using a three-dimensional modeling technology, wherein the three-dimensional face model describes the static geometric structure of the face and position data of key skeleton points;

[0089] S202, driving the three-dimensional face model through the voice data, generating facial dynamic expression parameters and skeletal motion parameters frame by frame, mapping the generated dynamic expression parameters and skeletal motion parameters to a two-dimensional space, and generating a two-dimensional facial animation for each frame;

[0090] S203, using a renderer to render the generated two-dimensional facial animation frame by frame, and applying super-resolution reconstruction technology to generate the character's speech video.

[0091] In this embodiment, the three-dimensional face model is driven by voice data, and the final two-dimensional facial animation is generated to create a person speaking video synchronized with the voice.

[0092] The system first uses the foreground layer of the person extracted from the portrait image as a reference frame. Based on this reference frame, a 3D face model is generated using 3D modeling technology. This model accurately describes the static geometric structure of the face and the position data of the key points of the skeleton, providing a basis for subsequent facial animation generation.

[0093] The 3D face model is driven by features such as phonemes and intonation in the speech data to generate facial dynamic expression parameters and bone movement parameters frame by frame. These parameters include subtle changes in mouth shape, eye expression, and facial muscles, and reflect the emotions and tone in the speech.

[0094] The generated dynamic expression parameters and bone motion parameters are mapped to a two-dimensional space to form a two-dimensional facial animation for each frame. This process usually uses projection technology in computer graphics to project the dynamic changes of the three-dimensional model onto a two-dimensional plane to generate facial animation synchronized with speech.

[0095] The generated 2D facial animation is rendered frame by frame using a renderer, and the rendered animation is more realistic and delicate. In order to improve the final quality of the video, the system applies super-resolution reconstruction technology to increase the resolution of the generated video, making the video clearer and more realistic.

[0096] In one specific implementation, at a high-end financial product launch conference, the system extracts data from the speaker's voice, generates a three-dimensional face model, and uses the model to generate facial expressions and lip animations frame by frame. Unlike conventional methods, the system dynamically adjusts skeletal motion parameters according to different parts of the speech content. For example, when introducing the core advantages of a product, the system will add a slight tilt of the head or a slight lift of the eyebrows to enhance the appeal of the speech. These adjusted three-dimensional dynamic parameters are mapped to a two-dimensional space to generate corresponding two-dimensional facial animations. The rendering process uses high-quality light and shadow processing technology to ensure that facial expressions and details can be clearly displayed in large-screen presentations, enhancing the overall effect of the product launch and the audience's sense of participation.

[0097] This embodiment combines voice data with a three-dimensional face model to accurately generate a two-dimensional facial animation that is synchronized with the voice, and ultimately generates a high-quality video of the person speaking. This ensures the authenticity and clarity of the video, and improves the user experience and information communication effect in multiple application scenarios, such as customer service, distance education, and financial product promotion.

[0098] In one embodiment, the above S202 includes:

[0099] S2021, extracting key speech features from the speech data, wherein the key speech features include phonemes, intonation, rhythm, and pause information, so as to drive the three-dimensional face model to generate corresponding facial expression changes;

[0100] S2022, using the key speech features to drive the three-dimensional face model, generating facial dynamic expression parameters and bone movement parameters frame by frame, so that the expression changes in the three-dimensional face model correspond to the key speech features in the speech data;

[0101] S2023, mapping the dynamic expression parameters and skeletal motion parameters of the generated three-dimensional face model to a two-dimensional space to generate a corresponding two-dimensional facial animation;

[0102] S2024, aligning the generated two-dimensional facial animation with the voice data in time axis, so that each frame of the animation matches the voice data in time, and making the lip animation consistent with the voice content through frame-by-frame adjustment.

[0103] In this embodiment, the key features extracted from the speech data include phonemes (such as pronunciation units), intonation (high and low fluctuations), rhythm (speed and beat), and pause information. These features are used to drive the three-dimensional face model to generate facial expression changes. For example, a high pitch may correspond to a wide mouth, and a fast rhythm may cause rapid movement of facial muscles. Using the extracted speech features, the system drives the three-dimensional face model to generate facial dynamic expression parameters and bone movement parameters frame by frame. These parameters define the movement mode and amplitude of each part of the face (such as mouth, eyes, eyebrows, etc.), so that the three-dimensional model can dynamically show the facial expression changes and bone movement of the character. This step ensures that the expression of the three-dimensional model is synchronized with the speech content to achieve a real and vivid effect. The generated three-dimensional dynamic expression parameters and bone movement parameters are mapped to a two-dimensional space. This is usually achieved through projection technology, projecting the dynamic changes of the three-dimensional model onto a two-dimensional plane to generate a two-dimensional facial animation. The two-dimensional animation retains all the expression details of the three-dimensional model and adapts it to the two-dimensional video output. In order to ensure that the generated two-dimensional animation is completely synchronized with the speech content, the system aligns the two-dimensional facial animation with the speech data. This process involves adjusting the animation frame by frame so that the lip movements accurately match the pronunciation of the speech and ensure the synchronization of the lip movements and speech, thereby improving the naturalness and authenticity of the animation.

[0104] In one specific implementation, in a virtual customer service system, the system first extracts key speech features from the user's voice, including phonemes, intonation, rhythm, and pause information. These features are used to drive the three-dimensional face model of the virtual customer service frame by frame to generate dynamic expressions and skeletal motion parameters. In this process, the system introduces an emotion recognition module to analyze the emotional features in the voice data, so that the expression of the virtual customer service not only reflects the voice content, but also reflects the user's emotional state. The generated three-dimensional dynamic expressions and skeletal motion parameters are mapped to a two-dimensional space to form a two-dimensional facial animation. Finally, the system aligns the generated two-dimensional animation with the voice data on the time axis to ensure that the lip shape is highly synchronized with the voice content, and adjusts frame by frame so that the final video shows a high degree of emotional resonance.

[0105] In other specific embodiments, in the video courses of the education platform, the system obtains the teacher's voice data and extracts phonemes, intonation, rhythm and pause information from it. These voice features drive the three-dimensional face model to generate facial expressions and skeletal motion parameters. In this embodiment, the system specifically introduces a knowledge point recognition module, which can identify key knowledge points in the voice and highlight these key contents when generating facial expressions, such as by increasing the degree of eye opening or increasing slight movements of the head to enhance the expression effect. After the generated three-dimensional dynamic expressions and skeletal motion parameters are mapped to the two-dimensional space, the system generates high-definition two-dimensional facial animations through frame-by-frame rendering, and aligns the time axis to synchronize the lip animation with the explanation of the knowledge points. The final generated video is more helpful for students to understand and remember.

[0106] This embodiment generates a two-dimensional facial animation by combining voice data with a three-dimensional face model, thereby achieving high-precision facial expressions and lip synchronization, and ensuring the authenticity and naturalness of the video content.

[0107] In one embodiment, in the above S30, generating a background video based on the background video sequence includes:

[0108] S301, using a pre-trained video restoration model to process the mask area in the background video sequence, repairing and completing the missing background information frame by frame, and generating a preliminary rough background video;

[0109] S302, calling the BLIP model to perform spatial feature analysis on each frame of the rough background video, and generating a spatial feature promt for guiding the image redrawing process;

[0110] S303, based on the generated spatial feature prompt, using the Stable Diffusion technology, redrawing the rough background video frame by frame;

[0111] S304, after redrawing frame by frame, applying super-resolution reconstruction technology to each frame of the background video to generate the background video.

[0112] In this embodiment, the system uses a pre-trained video inpainting model to process the masked areas in the background video sequence. The mask identifies the areas in the background that are missing or need to be inpainted. The system inpaints these areas frame by frame, fills in the missing background information, and generates a preliminary but relatively rough background video.

[0113] Based on the generated rough background video, the system calls the BLIP model to perform spatial feature analysis on each frame of the video. The BLIP model extracts the spatial features in the video and generates a spatial feature prompt for image redrawing. This prompt contains information such as the structure, texture, and lighting of each frame in the video, guiding the subsequent redrawing process.

[0114] Using the spatial feature prompt generated from the BLIP model, the system uses the Stable Diffusion technique to redraw the rough background video frame by frame. This step enhances the visual consistency and quality of the video by re-rendering the details of the background in each frame, making the background more natural and realistic.

[0115] After the frame-by-frame redrawing is completed, the system applies super-resolution reconstruction technology to each frame of the background video. By improving the resolution of the video, the background video is made clearer and more delicate, and finally a high-quality background video is generated.

[0116] In another specific embodiment, when generating an accident scene video during the insurance claim process, the system uses a pre-trained video repair model to repair the damaged areas in the video sequence. In this embodiment, the repair process specifically combines environmental information of the accident scene, such as weather conditions and lighting angles, to generate a more realistic background video. The system calls the BLIP model to perform spatial feature analysis on the repaired video, and the generated prompt not only contains spatial information, but also combines the unique environmental features of the scene to guide the redrawing process of the Stable Diffusion technology. During the redrawing process, the system refines the edges of objects in the background to make the outlines of objects in each frame clearer. Finally, the level of detail of the background video is enhanced through super-resolution reconstruction technology, and the generated video is more visually impactful and can effectively convey the real scene of the accident.

[0117] Through the above steps, this embodiment significantly improves the quality and consistency of the background video. The combination of video restoration, spatial feature analysis, frame-by-frame redrawing and super-resolution reconstruction technology makes the final generated background video have the characteristics of high resolution, rich details and strong visual appeal.

[0118] In one embodiment, in the above S304, after applying the super-resolution reconstruction technology to each frame of the background video, the process further includes:

[0119] S3041, extracting camera parameters from the person's speech video, and applying the camera parameters to the background video to match the visual effect of the person's speech video;

[0120] S3042, adjusting the focal length in the background video based on the focal length value in the camera parameter so that the focal length in the background video is consistent with that in the person's speech video;

[0121] S3043, adjusting the aperture size of the background video based on the aperture value in the camera parameter so that the aperture sizes of the background video and the person's speech video are consistent;

[0122] S3044, adjusting the focus of the background video based on the focus in the camera parameters so that the focus of the background video is consistent with that of the person's speech video;

[0123] S3045, identifying lighting conditions from the person's speech video, and adjusting lighting parameters of the background video based on the lighting conditions so that the background video and the person's speech video are consistent in lighting effects.

[0124] In this embodiment, the system extracts camera parameters from the speech video, including focal length, aperture value, focus point, etc. Camera parameters determine the visual characteristics of the video, such as depth of field, clarity, focus position, etc. Applying these parameters to the background video ensures that the foreground and background are rendered under the same camera settings, enhancing the realism and consistency of the video.

[0125] Based on the focal length value extracted from the person's speech video, the system adjusts the focal length in the background video. The focal length adjustment can change the visual effect of the background so that the background and the focal length in the person's video are consistent, thus ensuring that the visual relationship between the person and the background conforms to the perspective laws in physical reality.

[0126] The aperture value determines the depth of field and exposure of the image. Based on the extracted aperture value, the system adjusts the aperture size of the background video to match the aperture effect in the background video and the person's speech video. This ensures that the depth of field and clarity between the background and foreground are consistent, avoiding unnatural visual effects.

[0127] The system adjusts the focus position in the background video based on the focus point information in the camera parameters. This ensures that the focus in the background video is consistent with the focus in the person's speech video, avoiding mismatches in focus between the background and foreground, thereby enhancing the overall coherence of the video.

[0128] Identify lighting conditions from the person’s speech video and adjust the lighting parameters of the background video based on these conditions. This adjustment ensures lighting consistency between the background and the person, making the final video consistent in lighting and shadow effects, and improving the realism of the visual effect.

[0129] In one specific implementation, in a financial product promotion video, the system extracts camera parameters, including focal length, aperture value, and focus point, from the speaker's video after applying super-resolution reconstruction technology. This implementation particularly emphasizes the dynamic adjustment of focal length and aperture. The system dynamically adjusts the focal length and aperture of the background video according to different speech clips to match the visual effects of the speaker in each clip. This method not only ensures the consistency of the visual effects between the background and the foreground, but also makes the video appear more natural and smooth when switching between different scenes. The adjustment of lighting conditions also combines the different scene requirements of the promotional video. Through dynamic lighting simulation, the light and shadow effects of the background video are more coordinated with the changes in the foreground.

[0130] In another specific embodiment, in the insurance claim explanation video, the system uses super-resolution reconstruction technology to improve the resolution of the background video, extracts the camera parameters in the claims agent video, and applies them to the background video. In this embodiment, the system takes special consideration of the special lighting conditions at the accident scene, such as backlighting, shadows, etc. By identifying and simulating these lighting conditions, the system adjusts the lighting parameters in the background video so that the background video can truly reflect the lighting environment at the accident scene. At the same time, the focus adjustment is also based on key locations in the claims process (such as damaged parts of the vehicle), ensuring that these important areas are clearly visible in the video, enhancing the authenticity and persuasiveness of the claims video.

[0131] This embodiment combines super-resolution reconstruction technology and camera parameter adjustment to achieve highly consistent visual effects between the background and foreground. By adjusting the focal length, aperture, focus point and lighting conditions, the background video and the person's speech video are visually coordinated, which enhances the realism and professionalism of the video, and is particularly suitable for application scenarios such as finance, insurance and remote conferencing.

[0132] In one embodiment, the above S40 includes:

[0133] S401, setting the person's speech video and the background video as independent layers, so that the person's speech video and the background video occupy different video tracks respectively;

[0134] S402, adjusting the time axis of the person's speech video and the background video so that the starting point and the ending point of the two are consistent;

[0135] S403, placing the character speech video layer on the background video layer, so that the character speech video covers the background video;

[0136] S404, adjusting the transparency and blending mode of the person's speech video layer so that the fusion effect of the person's speech video and the background video meets the target effect;

[0137] S405, obtaining the target video by merging the character speech video with the background video.

[0138] In this embodiment, the system first sets the person's speech video and the background video as separate layers. The person's speech video is usually used as the foreground, while the background video is used as the background layer. Each layer occupies a different video track. This separation allows each layer to be adjusted individually, thereby ensuring that each video element is processed and presented more accurately.

[0139] To ensure that the person's speech video and the background video are completely synchronized in time, the system adjusts the time axis of the two to make their start and end points consistent. This operation ensures that when the video is merged, the foreground and background video clips are coordinated in time and avoid time misalignment.

[0140] The system places the person's speech video layer on top of the background video layer, so that the person's speech video covers the background video. You can adjust the layer order to ensure that the person's speech video is displayed in the foreground layer, thereby achieving the correct visual overlay effect and making the person clearly visible in front of the background.

[0141] The system further adjusts the transparency and blending mode of the person's speech video layer to achieve the ideal visual fusion effect. By adjusting the transparency, you can control the degree of fusion between the foreground person and the background, while the blending mode determines the interaction between the layers, such as soft light, overlay or multiply effects, so that the final generated video visual effect meets the expectations.

[0142] After the above steps are completed, the system merges the character speech video with the background video layer by layer to finally generate the target video. The target video perfectly integrates the foreground character and the background scene, presenting a high-quality visual effect.

[0143] In one specific implementation, when making a financial advertising video, the system first sets the speaker's video and the dynamic financial data background video as independent layers. Unlike conventional methods, this implementation pays special attention to the dynamic adjustment of the timeline. According to the rhythm and tone changes of the speech content, the system adjusts the playback speed of the background video to synchronize it with the content of the foreground video. For example, when the speaker mentions market fluctuations, the background video changes faster to enhance the visual impact. By dynamically adjusting the transparency of the layer, the system enables the data in the background to be clearly displayed through the foreground video at critical moments, thereby highlighting key information.

[0144] In another specific implementation, in the insurance claim explanation video, the system sets the agent's speech video and the accident scene video as independent layers. Unlike simple layer overlay, the system in this implementation adopts dynamic layer order adjustment technology. According to the changes in the explanation content, the system automatically adjusts the order of the foreground and background layers at a specific time point. For example, when explaining the accident, the system promotes the accident scene video to the foreground, making it the main perspective of the video and enhancing the visual impact. Subsequently, the system restores the agent's video to the foreground and fine-tunes its transparency to ensure that the background information does not excessively interfere with the explanation content.

[0145] This embodiment can generate a target video with high consistency and professional standards by merging the speech video of the person and the background video in layers and adjusting parameters such as timeline, transparency and blending mode. Whether it is financial publicity, virtual meetings, or insurance claims, it can ensure the natural integration of foreground characters and background scenes, and enhance the visual effect and information communication effect of the video.

[0146] The present invention also provides a video generation device based on layer separation, referring to Figure 2 , Figure 2 This is a functional module diagram of a preferred embodiment of a video generation device based on layer separation of the present invention. The video generation device based on layer separation includes:

[0147] A layer separation module, which obtains a portrait image containing a person image, and segments the portrait image to obtain a person foreground layer and a background layer;

[0148] An audio driving module, which obtains voice data and generates a character speech video based on the voice data and the character foreground layer, wherein the voice data is used to drive the facial expression and lip shape changes of the character;

[0149] A background video generation module, which copies the background layer frame by frame to generate a static background video sequence, and generates a background video based on the background video sequence;

[0150] The video synthesis module performs layer merging of the character speech video and the background video to generate a target video.

[0151] The specific implementation of the video generation device based on layer separation of the present invention is basically the same as the various embodiments of the video generation method based on layer separation described above, and will not be repeated here.

[0152] The present invention also provides a video generation device based on layer separation, such as Figure 3As shown, the video generation device based on layer separation may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0153] Those skilled in the art will understand that Figure 3 The hardware structure of the layer-separation-based video generating device shown in the figure does not constitute a limitation on the layer-separation-based video generating device, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0154] like Figure 3 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a video generation program based on layer separation. The operating system is a program for managing and controlling the video generation device and software resources based on layer separation, and supports the operation of the network communication module, the user interface module, the video generation program based on layer separation, and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

[0155] exist Figure 3 In the hardware structure of the layer-separation-based video generation device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the layer-separation-based video generation program stored in the memory 1005, and perform the same operation as the layer-separation-based video generation method.

[0156] The specific implementation of the video generation device based on layer separation of the present invention is basically the same as the various embodiments of the video generation method based on layer separation described above, and will not be repeated here.

[0157] In addition, an embodiment of the present invention further proposes a computer storage medium, on which a video generation program based on layer separation is stored. When the video generation program based on layer separation is executed by a processor, the steps of the video generation method based on layer separation as described above are implemented.

[0158] The specific implementation of the computer storage medium of the present invention is basically the same as the above-mentioned embodiments of the video generation method based on layer separation, and will not be repeated here.

Claims

1. A video generation method based on layer separation, characterized in that: The following steps are involved: Acquire a portrait image containing a person image, and segment the portrait image to obtain a person foreground layer and a background layer; Acquire voice data, and generate a character speech video based on the voice data and the character foreground layer, wherein the voice data is used to drive the facial expression and lip shape changes of the character; Copying the background layer frame by frame to generate a still background video sequence, and generating a background video based on the background video sequence; The character speech video and the background video are layer-merged to generate a target video.

2. The video generation method based on layer separation according to claim 1, characterized in that: Segmenting the portrait image to obtain a person foreground layer and a background layer includes: Inputting the portrait image into a pre-trained real-time portrait segmentation network, extracting a mask representing a person region from the portrait image, wherein the mask is used to identify pixel points belonging to the foreground of the person in the portrait image, and forming a binary image, wherein the foreground portion is a valid region and the background portion is an invalid region; Using the mask, the person region in the portrait image is separated from the overall image to generate an independent person foreground layer, wherein the person foreground layer only retains pixel information of the person; Based on the mask, the person area is removed from the portrait image, the background part is retained, and a background layer is generated. The background layer only contains background pixel information after the person is removed.

3. The video generation method based on layer separation according to claim 1, characterized in that: Generating a person speech video based on the voice data and the person foreground layer includes: The character foreground layer is used as a reference frame, and a three-dimensional face model is generated based on the reference frame using a three-dimensional modeling technology, wherein the three-dimensional face model describes the static geometric structure of the face and the position data of the key points of the skeleton; The three-dimensional face model is driven by the voice data to generate facial dynamic expression parameters and skeletal motion parameters frame by frame, and the generated dynamic expression parameters and skeletal motion parameters are mapped to a two-dimensional space to generate a two-dimensional facial animation for each frame; The generated two-dimensional facial animation is rendered frame by frame using a renderer, and a super-resolution reconstruction technique is applied to generate the character's speech video.

4. The video generation method based on layer separation as claimed in claim 3, characterized in that: The three-dimensional face model is driven by the voice data to generate facial dynamic expression parameters and bone motion parameters frame by frame, and the generated dynamic expression parameters and bone motion parameters are mapped to a two-dimensional space to generate a two-dimensional facial animation for each frame, including: Extracting key speech features from the speech data, wherein the key speech features include phonemes, intonation, rhythm, and pause information, so as to drive the three-dimensional face model to generate corresponding facial expression changes; Using the key speech features to drive the three-dimensional face model, generating facial dynamic expression parameters and bone movement parameters frame by frame, so that the expression changes in the three-dimensional face model correspond to the key speech features in the speech data; Mapping the dynamic expression parameters and bone motion parameters of the generated 3D face model into 2D space to generate corresponding 2D facial animation; The generated two-dimensional facial animation is time-aligned with the speech data so that each frame of the animation matches the speech data in time, and the lip animation is made consistent with the speech content through frame-by-frame adjustment.

5. The video generation method based on layer separation according to claim 1, characterized in that: Generating a background video based on the background video sequence includes: Using a pre-trained video restoration model to process the mask area in the background video sequence, repairing and completing the missing background information frame by frame, and generating a preliminary rough background video; Calling the BLIP model to perform spatial feature analysis on each frame of the rough background video, and generating a spatial feature promt for guiding the image redrawing process; Based on the generated spatial feature prompt, the rough background video is redrawn frame by frame using the Stable Diffusion technology; After redrawing frame by frame, a super-resolution reconstruction technique is applied to each frame of the background video to generate the background video.

6. The video generation method based on layer separation as claimed in claim 5, characterized in that: After applying super-resolution reconstruction technology to each frame of background video, it also includes: Extracting camera parameters from the person's speech video, and applying the camera parameters to the background video to match the visual effect of the person's speech video; Based on the focal length value in the camera parameters, adjusting the focal length in the background video so that the focal length in the background video is consistent with that in the person's speech video; Based on the aperture value in the camera parameters, the aperture size of the background video is adjusted so that the aperture sizes of the background video and the person's speech video are consistent; Based on the focus point in the camera parameters, the focus point of the background video is adjusted so that the focus of the background video and the focus of the person's speech video are consistent; Identify lighting conditions from the person's speech video, and adjust lighting parameters of the background video based on the lighting conditions to ensure that the background video and the person's speech video are consistent in lighting effects.

7. The video generation method based on layer separation according to claim 1, characterized in that: Merging the character speech video and the background video in layers to generate a target video includes: Set the character speech video and background video as independent layers, so that the character speech video and background video occupy different video tracks respectively; Adjust the timeline of the character's speech video and the background video so that the starting and ending points of the two are consistent; Place the character speech video layer above the background video layer so that the character speech video covers the background video; Adjust the transparency and blending mode of the character's speech video layer to make the fusion effect of the character's speech video and the background video meet the target effect; The target video is obtained by merging the person's speech video with the background video.

8. A video generation device based on layer separation, characterized in that: The video generation device based on layer separation comprises: A layer separation module, which obtains a portrait image containing a person image, and segments the portrait image to obtain a person foreground layer and a background layer; An audio driving module, which obtains voice data and generates a character speech video based on the voice data and the character foreground layer, wherein the voice data is used to drive the facial expression and lip shape changes of the character; A background video generation module, which copies the background layer frame by frame to generate a static background video sequence, and generates a background video based on the background video sequence; The video synthesis module performs layer merging of the character speech video and the background video to generate a target video.

9. A video generation device based on layer separation, characterized in that: The layer separation-based video generation device includes a memory, a processor, and a layer separation-based video generation program stored in the memory and executable on the processor. When the layer separation-based video generation program is executed by the processor, the steps of the layer separation-based video generation method as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The storage medium stores a video generation program based on layer separation, and when the video generation program based on layer separation is executed by a processor, the steps of the video generation method based on layer separation as described in any one of claims 1-7 are implemented.

Images