Video generation method and device, equipment, storage medium and program product

Abstract

Embodiments of the invention provide a video generation method and apparatus, a device, a storage medium and a program product. The method comprises the steps of obtaining to-be-processed data; the to-be-processed data comprises a target foreground video and a target scene image; the target scene image comprises a video background of a to-be-generated target video; the target foreground video comprises a target object; extracting a target object mask image, a target foreground image and a target camera pose from the target foreground video, and determining a target noise implicit code corresponding to the to-be-processed data; and inputting the target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise implicit code into a target video generation model to generate a target video corresponding to the to-be-processed data. Thus, the electronic equipment combines the target camera pose to realize automatic generation of the target video, and a mirror moving effect can be added in the target video, so that fusion of the foreground and the background of the target video is more natural and smooth, and a video generation effect is enhanced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video generation method, apparatus, device, storage medium, and program product. Background Art

[0002] With the continuous development of artificial intelligence generated content (AIGC) technology, content information in multiple modalities such as text, images, videos, and audio can gradually be automatically generated.

[0003] In related technologies, automatic video generation usually involves simply splicing and fusing the foreground and background before generation and playback. Due to the lack of camera movement effects such as lens control, the fusion of the foreground and background in scenes such as character video generation is not natural and smooth, resulting in poor video generation effects. Summary of the Invention

[0004] Multiple aspects of the present application provide a video generation method, apparatus, device, storage medium and program product, which can generate videos in combination with camera posture, and can add camera movement effects to the target video, so that the fusion of the foreground and background of the target video generated in the final generation is more natural and smooth, thereby enhancing the video generation effect.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, comprising:

[0006] Acquire data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target video to be generated; the target foreground video includes a target object;

[0007] Extracting a target object mask image, a target foreground image, and a target camera pose from the target foreground video, and determining a target noise hidden code corresponding to the data to be processed;

[0008] The target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise hidden code are input into a target video generation model to generate a target video corresponding to the data to be processed.

[0009] In a possible implementation, extracting the target object mask image, the target foreground image, and the target camera pose from the target foreground video includes:

[0010] Segmenting the target foreground video using a target segmentation extraction model to obtain the target object mask image and the target foreground image;

[0011] The target foreground video is detected and processed by the target camera detection model to obtain the target camera pose.

[0012] In one possible implementation, inputting the target scene image, the target object mask image, the target foreground image, the target camera pose, and the target noise hidden code into a target video generation model to generate a target video corresponding to the data to be processed includes:

[0013] Extracting target scene semantic features and target scene detail features corresponding to the target scene image through the target video generation model;

[0014] generating target image features corresponding to the data to be processed according to the target scene semantic features, the target scene detail features, and the background denoising features corresponding to the target noise latent code;

[0015] Feature processing and timing control are performed on the target image features and the target camera posture to generate a target video corresponding to the data to be processed.

[0016] In one possible implementation, the method further includes:

[0017] Acquire sample data; the sample data includes a sample foreground video and a sample scene image;

[0018] Determining, based on the sample foreground video, a sample mask image, a sample foreground image, and a sample camera pose corresponding to the sample scene image, and determining a sample noise hidden code corresponding to the sample data;

[0019] Iteratively training a preset image generation model according to the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code to obtain a target image generation model;

[0020] According to the sample mask image, the sample foreground image, the sample scene image, the sample noise hidden code, the sample camera pose and the target image generation model, the preset video generation model is iteratively trained to obtain the target video generation model.

[0021] In a possible implementation, iteratively training a preset image generation model based on the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code to obtain a target image generation model includes:

[0022] Extracting sample scene semantic features and sample scene detail features corresponding to the sample scene image;

[0023] Performing splicing processing on the sample mask image, the sample foreground image, and the sample noise hidden code to obtain a sample input tensor;

[0024] The preset image generation model is iteratively trained according to the sample scene semantic features, the sample scene detail features and the sample input tensor to obtain the target image generation model.

[0025] In a possible implementation, the sample foreground video includes a first sample foreground video, a second sample foreground video, and a third sample foreground video; the sample scene image includes a first sample scene image, a second sample scene image, and a third sample scene image;

[0026] The first sample foreground video and the first sample scene image are used to perform scene completion training on the preset image generation model; the second sample foreground video and the second sample scene image are used to perform scene transformation training on the preset image generation model; the third sample foreground video and the third sample scene image are used to perform background generation training on the preset image generation model.

[0027] In a possible implementation, the pixel value in the first sample foreground image corresponding to the first sample foreground video is a first value; the pixel value in the second sample foreground image corresponding to the second sample foreground video is the first value; and the pixel value in the second sample mask image corresponding to the second sample foreground video is a second value.

[0028] In a possible implementation, the iterative training of a preset video generation model based on the sample mask image, the sample foreground image, the sample scene image, the sample noise hidden code, the sample camera pose, and the target image generation model to obtain the target video generation model includes:

[0029] Inputting the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code into the preset video generation model to obtain sample image features; the preset video generation model includes the preset image generation model and a timing control module;

[0030] Based on the sample image features and the sample camera poses, the timing control module in the preset video generation model is iteratively trained to obtain the target video generation model.

[0031] In one possible implementation, the timing control module includes a camera pose encoder and a camera pose adapter; and the iterative training of the timing control module in the preset video generation model based on the sample image features and the sample camera pose to obtain the target video generation model includes:

[0032] Encoding the sample camera pose by the camera pose encoder to obtain sample camera pose features;

[0033] Fusing the sample camera pose features and the sample image features to obtain sample pose fusion features, and performing dimension conversion on the sample pose fusion features through the camera pose adapter to obtain sample video features;

[0034] The preset video generation model is iteratively trained according to the sample video features to obtain the target video generation model.

[0035] In a second aspect, an embodiment of the present application provides another video generation method, including:

[0036] In response to an interactive operation of a client, obtaining data to be processed corresponding to the interactive operation; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target person video to be generated; the target foreground video includes a target person;

[0037] Extracting a target person mask image, a target person foreground image, and a target camera pose from the target foreground video, and determining a target noise hidden code corresponding to the data to be processed;

[0038] The target scene image, the target person mask image, the target person foreground image, the target camera pose and the target noise hidden code are input into a target video generation model to generate a target person video corresponding to the data to be processed.

[0039] In a third aspect, an embodiment of the present application provides a video generation device, including:

[0040] An acquisition module is used to acquire data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target video to be generated; the target foreground video includes a target object;

[0041] A determination module is used to extract a target object mask image, a target foreground image, and a target camera pose from the target foreground video, and determine a target noise hidden code corresponding to the data to be processed;

[0042] A generation module is used to input the target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise hidden code into a target video generation model to generate a target video corresponding to the data to be processed.

[0043] In a possible implementation, the determining module is specifically configured to:

[0044] Segmenting the target foreground video using a target segmentation extraction model to obtain the target object mask image and the target foreground image;

[0045] The target foreground video is detected and processed by the target camera detection model to obtain the target camera pose.

[0046] In a possible implementation, the generating module is specifically configured to:

[0047] Extracting target scene semantic features and target scene detail features corresponding to the target scene image through the target video generation model;

[0048] generating target image features corresponding to the data to be processed according to the target scene semantic features, the target scene detail features, and the background denoising features corresponding to the target noise latent code;

[0049] Feature processing and timing control are performed on the target image features and the target camera posture to generate a target video corresponding to the data to be processed.

[0050] In one possible embodiment, the device is further used for:

[0051] Acquire sample data; the sample data includes a sample foreground video and a sample scene image;

[0052] Determining, based on the sample foreground video, a sample mask image, a sample foreground image, and a sample camera pose corresponding to the sample scene image, and determining a sample noise hidden code corresponding to the sample data;

[0053] Iteratively training a preset image generation model according to the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code to obtain a target image generation model;

[0054] According to the sample mask image, the sample foreground image, the sample scene image, the sample noise hidden code, the sample camera pose and the target image generation model, the preset video generation model is iteratively trained to obtain the target video generation model.

[0055] In one possible embodiment, the device is further used for:

[0056] Extracting sample scene semantic features and sample scene detail features corresponding to the sample scene image;

[0057] Performing splicing processing on the sample mask image, the sample foreground image, and the sample noise hidden code to obtain a sample input tensor;

[0058] The preset image generation model is iteratively trained according to the sample scene semantic features, the sample scene detail features and the sample input tensor to obtain the target image generation model.

[0059] In a possible implementation, the sample foreground video includes a first sample foreground video, a second sample foreground video, and a third sample foreground video; the sample scene image includes a first sample scene image, a second sample scene image, and a third sample scene image;

[0060] The first sample foreground video and the first sample scene image are used to perform scene completion training on the preset image generation model; the second sample foreground video and the second sample scene image are used to perform scene transformation training on the preset image generation model; the third sample foreground video and the third sample scene image are used to perform background generation training on the preset image generation model.

[0061] In a possible implementation, the pixel value in the first sample foreground image corresponding to the first sample foreground video is a first value; the pixel value in the second sample foreground image corresponding to the second sample foreground video is the first value; and the pixel value in the second sample mask image corresponding to the second sample foreground video is a second value.

[0062] In one possible embodiment, the device is further used for:

[0063] Inputting the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code into the preset video generation model to obtain sample image features; the preset video generation model includes the preset image generation model and a timing control module;

[0064] Based on the sample image features and the sample camera poses, the timing control module in the preset video generation model is iteratively trained to obtain the target video generation model.

[0065] In one possible implementation, the timing control module includes a camera pose encoder and a camera pose adapter; the device is further configured to:

[0066] Encoding the sample camera pose by the camera pose encoder to obtain sample camera pose features;

[0067] Fusing the sample camera pose features and the sample image features to obtain sample pose fusion features, and performing dimension conversion on the sample pose fusion features through the camera pose adapter to obtain sample video features;

[0068] The preset video generation model is iteratively trained according to the sample video features to obtain the target video generation model.

[0069] In a fourth aspect, an embodiment of the present application provides another video generation device, including:

[0070] An acquisition module is configured to, in response to an interactive operation of a client, acquire data to be processed corresponding to the interactive operation; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target person video to be generated; and the target foreground video includes a target person;

[0071] A determination module is used to extract a target person mask image, a target person foreground image, and a target camera pose from the target foreground video, and determine a target noise hidden code corresponding to the data to be processed;

[0072] A generation module is used to input the target scene image, the target person mask image, the target person foreground image, the target camera pose and the target noise hidden code into a target video generation model to generate a target person video corresponding to the data to be processed.

[0073] In a fifth aspect, an embodiment of the present application provides a video generating device, including: a memory and a processor;

[0074] The memory stores computer-executable instructions;

[0075] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video generation method described in any one of the first aspect or the second aspect.

[0076] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the video generation method described in either the first aspect or the second aspect.

[0077] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the video generation method shown in any one of the first aspect or the second aspect.

[0078] In an embodiment of the present application, an electronic device obtains data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target video to be generated; the target foreground video includes a target object; a target object mask image, a target foreground image, and a target camera pose are extracted from the target foreground video, and a target noise hidden code corresponding to the data to be processed is determined; the target scene image, the target object mask image, the target foreground image, the target camera pose, and the target noise hidden code are input into a target video generation model to generate a target video corresponding to the data to be processed. In the present application, an electronic device obtains a target foreground video including a target object and a target scene image, and then inputs the target scene image, the target noise hidden code, and the target object mask image, the target foreground image, and the target camera pose extracted from the target foreground video into the target video generation model, and finally obtains a target video with the target scene image as the video background. In this way, the electronic device realizes the automatic generation of the target video in combination with the target camera pose, can add a camera movement effect to the target video, so that the fusion of the foreground and background of the target video generated in the final step is more natural and smooth, thereby enhancing the video generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0080] Figure 1 A flowchart of a video generation method provided by an exemplary embodiment of the present application;

[0081] Figure 2 A flowchart of another video generation method provided by an exemplary embodiment of the present application;

[0082] Figure 3 A flowchart of a target video generation model training method provided by an exemplary embodiment of the present application;

[0083] Figure 4 A schematic diagram of a model training for a background generation task provided by an exemplary embodiment of the present application;

[0084] Figure 5 A schematic diagram of training a target video generation model provided by an exemplary embodiment of the present application;

[0085] Figure 6 A training diagram of a timing control module provided by an exemplary embodiment of the present application;

[0086] Figure 7 A flowchart of another video generation method provided by an exemplary embodiment of the present application;

[0087] Figure 8 A schematic structural diagram of a video generation device provided by an exemplary embodiment of the present application;

[0088] Figure 9 A schematic structural diagram of another video generating device provided by an exemplary embodiment of the present application;

[0089] Figure 10 A schematic structural diagram of a video generation device provided by an exemplary embodiment of the present application.

[0090] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards, and provide corresponding operation portals for users to choose to authorize or refuse.

[0092] The following is an explanation of the professional terms involved in this application:

[0093] Diffusion Model: A diffusion model is a generative model that gradually adds noise to raw data, creating a series of blurred samples. It then learns how to gradually recover the original data from the noise. Diffusion models are effective for generating images, audio, and text. Specific examples of diffusion models include the Stable Diffusion (SD) model.

[0094] Video Synthesis: Video synthesis involves using computer vision algorithms and models to create new, dynamic video sequences. Depending on the input conditions, video synthesis can be categorized into various types, such as text-based and image-based video.

[0095] Multi-task learning: Multi-task learning is a machine learning method that improves the model's generalization and learning efficiency by simultaneously training the model to solve multiple related tasks. In multi-task learning, models share hidden layers and parameters, allowing different tasks to learn from and enhance each other, thereby improving their respective predictive performance.

[0096] With the continuous development of AIGC technology, content information in various modalities, such as text, images, video, and audio, can be automatically generated. AIGC technology is also being applied in a wide range of scenarios. For example, in advertising, AIGC can provide businesses with the visual assets they need, reducing human resources while providing higher-quality materials. In filmmaking, AIGC technology can generate special effects scenes and automatically compose music for film footage. In game development, AIGC technology can automatically generate game elements such as characters and environment maps, providing game developers with more content options. Furthermore, AIGC technology can be applied in a variety of fields, including teaching assistance, article continuation, office document production, and marketing. AIGC shortens the time it takes for creators to innovate. Through simple tools, ideas can be quickly presented, significantly improving creative efficiency and lowering the technical barriers and difficulties of design.

[0097] In related technologies, AIGC-based video generation is also gradually developing, such as character video generation. Character video generation methods in related technologies usually focus on generating the character foreground. For example, structured control signals such as depth maps or skeleton maps are used to drive the movement of the character foreground, and then the character foreground and background are simply spliced ​​and fused to generate the video. This video generation method in related technologies lacks camera movement effects such as lens control, resulting in random and uncontrollable motion patterns in the generated video background, making the fusion between the foreground and the background more difficult. For example, it is impossible to achieve a natural and smooth fusion between the character foreground and the scenery background, resulting in poor video generation and the inability to achieve high-quality video generation.

[0098] In order to solve the above problems, the present application provides a video generation method, apparatus, device, storage medium and program product, wherein an electronic device obtains data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes the video background of the target video to be generated; the target foreground video includes a target object; a target object mask image, a target foreground image and a target camera pose are extracted from the target foreground video, and a target noise hidden code corresponding to the data to be processed is determined; the target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise hidden code are input into a target video generation model to generate a target video corresponding to the data to be processed. In the present application, the electronic device obtains a target foreground video including a target object and a target scene image, and then inputs the target scene image, the target noise hidden code and the target object mask image, the target foreground image and the target camera pose extracted from the target foreground video into the target video generation model, and finally obtains a target video with the target scene image as the video background. In this way, the electronic device realizes the automatic generation of the target video in combination with the target camera pose, can add a camera movement effect to the target video, so that the fusion of the foreground and background of the target video generated in the final step is more natural and smooth, thereby enhancing the video generation effect.

[0099] The technical solutions shown in this application are described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be repeated in different embodiments.

[0100] Figure 1 This is a flow chart of a video generation method provided by an exemplary embodiment of the present application. Figure 1 , the video generation method may include:

[0101] S101, obtaining data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target video to be generated; and the target foreground video includes a target object.

[0102] The execution subject of the embodiments of the present application can be an electronic device, or a video generation device provided in the electronic device. The video generation device can be implemented by software, or by a combination of software and hardware. For ease of understanding, the following description takes the execution subject as an electronic device as an example. The electronic device can specifically refer to a mobile terminal, a server, a cloud platform, a cloud, etc. The embodiments of the present application do not limit the specific type of electronic device.

[0103] The data to be processed may refer to the raw data received by the electronic device for video generation, and may specifically include a target foreground video and a target scene image. The target foreground video may refer to a video containing a target object, such as a person, animal, or other object, and is used to provide the foreground of the target object for the target video to be generated. The target scene image may refer to a picture of a specific scene, used to provide a background for the target video to be generated, and may specifically refer to an image of a snowy mountain scene, a beach scene, or a grassland scene.

[0104] In this step, when video generation is required, the user can upload the data to be processed through interactive operations in the electronic device. The electronic device responds to the user's interactive operations and obtains the data to be processed corresponding to the interactive operations. Subsequently, video generation can be performed based on the data to be processed.

[0105] S102: extracting a target object mask image, a target foreground image, and a target camera pose from the target foreground video, and determining a target noise hidden code corresponding to the data to be processed.

[0106] In the embodiment of the present application, the target object mask image may refer to a black and white mask image of the target object, such as a person mask image. The target foreground image may refer to an image including the foreground of the target object in the target foreground video. The target camera pose may refer to the motion state information of the camera in the target foreground video. The target camera pose may be represented by Plücker coding (a motion trajectory representation method in three-dimensional space) and other forms of representation. Of course, other representation forms may also be used, and the embodiment of the present application does not limit this. The target noise hidden code may refer to a hidden code superimposed with random noise, and the target noise hidden code is subsequently gradually denoised to obtain the target image, and then the target video is generated.

[0107] In practical video generation scenarios, background content and camera movement are crucial. For example, in advertising production, controllable background generation can create a higher-quality, product-specific presentation for video displays. For example, a down jacket ad could feature a snowy, icy background. Compared to manually changing the background, automatic background generation significantly saves resources and time. Furthermore, the camera movement of the scenery can help the video unfold more smoothly, enhancing its narrative and sense of space. Appropriate camera movement can also create a strong emotional impact and visual beauty in a video. For example, a pull-back camera movement can quickly bring a character into the scene, while a wrap-around camera movement can create a strong connection between the character and the scenery. Furthermore, camera movements such as camera rotation, camera tracking, camera forward movement, and camera panning each have their own distinct camera movement effects. Related video generation methods typically focus on foreground generation control without addressing camera movement control, resulting in poor video generation and an inability to achieve a natural, smooth integration of foreground and background.

[0108] In an embodiment of the present application, after acquiring the target foreground video and the target scene image, the electronic device can extract the target object mask image, the target foreground image, and the target camera pose from the target foreground video, and simultaneously determine the target noise hidden code, and subsequently generate the target video based on the target object mask image, the target foreground image, the target camera pose, and the target noise hidden code. In this way, the electronic device introduces the target camera pose in the target video generation process, and can control the camera movement effect of the generated video background to make it consistent with the camera movement mode of the video foreground, making the overall picture of the target video more harmonious and natural, and the fusion of the video foreground and video background more natural and smooth, thereby improving the video generation effect.

[0109] S103: Input the target scene image, target object mask image, target foreground image, target camera pose and target noise hidden code into the target video generation model to generate a target video corresponding to the data to be processed.

[0110] In an embodiment of the present application, the target video generation model may refer to a pre-trained model for video generation. The target video generation model may specifically include an image encoder, a reference network (Reference Net), a stable diffusion model (SD base model) of a U-net structure, a variational autoencoder (VAE), and a timing control module. Among them, the image encoder can be used to extract semantic information from the target scene image. The specific types of the image encoder may include an encoder based on contrastive language-image retraining (CLIP), an encoder based on convolutional neural networks (CNN), or an encoder based on a transformer. The reference network can be used to extract detailed features from the target scene image and enhance the authenticity and accuracy of the background in the target video that is finally generated. The SD base model can be the basic model of the target video generation model, which can be used to denoise the target noise hidden code to generate the target image and the final target video. The variational autoencoder can be used to obtain the potential features of the input image. The timing control module can be used to perform timing control and camera movement control on the video during the video generation process. It should be noted that the target video generation model can also include other network architectures and other modules, which can be flexibly configured based on actual needs, and the embodiments of this application do not limit this.

[0111] In this step, after determining the target object mask image, target foreground image, target camera pose and target noise hidden code, the electronic device can input the target scene image, target object mask image, target foreground image, target camera pose and target noise hidden code into a pre-trained target video generation model, which will be processed by the target video generation model to finally generate a target video corresponding to the data to be processed, thereby realizing automatic generation of the target video.

[0112] In an embodiment of the present application, an electronic device obtains data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target video to be generated; the target foreground video includes a target object; a target object mask image, a target foreground image, and a target camera pose are extracted from the target foreground video, and a target noise hidden code corresponding to the data to be processed is determined; the target scene image, the target object mask image, the target foreground image, the target camera pose, and the target noise hidden code are input into a target video generation model to generate a target video corresponding to the data to be processed. In the present application, an electronic device obtains a target foreground video including a target object and a target scene image, and then inputs the target scene image, the target noise hidden code, and the target object mask image, the target foreground image, and the target camera pose extracted from the target foreground video into the target video generation model, and finally obtains a target video with the target scene image as the video background. In this way, the electronic device realizes the automatic generation of the target video in combination with the target camera pose, can add a camera movement effect to the target video, so that the fusion of the foreground and background of the target video generated in the final step is more natural and smooth, thereby enhancing the video generation effect.

[0113] Based on the above embodiments, Figure 2 This is a flowchart of another video generation method provided by an exemplary embodiment of the present application. Figure 2 , the method may include:

[0114] S201, obtaining data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target video to be generated; and the target foreground video includes a target object.

[0115] S202. Segment the target foreground video using the target segmentation extraction model to obtain a target object mask image and a target foreground image; detect the target foreground video using the target camera detection model to obtain a target camera pose; and determine the target noise hidden code corresponding to the data to be processed.

[0116] In the embodiment of the present application, the target segmentation and extraction model may refer to a pre-trained model for performing image segmentation and foreground extraction on the target foreground video, specifically a semantic segmentation algorithm, a human matting algorithm, or an instance segmentation algorithm, etc. The target camera detection model may refer to a pre-trained model for performing camera motion state detection, specifically a monocular depth estimation algorithm or a camera motion recovery algorithm (VGG-based Structure from Motion, VGGSFM), etc. Of course, the target segmentation and extraction model and the target camera detection model may also be implemented using other algorithms or models, which can be flexibly selected based on actual needs, and the embodiment of the present application does not limit this.

[0117] In this step, after acquiring the data to be processed, including the target foreground video and the target scene image, the electronic device can segment the target foreground video using the target segmentation extraction model to obtain a target object mask image and a target foreground image. Furthermore, the electronic device can detect the target foreground video using the target camera detection model to determine the target camera pose corresponding to the target foreground video. Furthermore, the electronic device can also determine the target noise hidden code corresponding to the data to be processed. In this way, based on the target segmentation extraction model and the target camera detection model, the electronic device can accurately extract the target object mask image, the target foreground image, and the target camera pose, further improving the reliability and accuracy of subsequent target video generation.

[0118] S203: extracting target scene semantic features and target scene detail features corresponding to the target scene image through the target video generation model.

[0119] In the embodiments of the present application, the target scene semantic features may refer to the semantic features of the target scene image extracted by the image encoder. The target scene detail features may refer to the fine-grained features of the target scene image extracted by the reference network, such as the features of specific objects in the target scene image.

[0120] In related technologies, the video generation process typically involves aligning an image with text, and then using the image's semantic features as the data foundation for video generation. This method of video generation in related technologies loses the detailed features inherent in the image itself, failing to effectively capture the details of the scene, resulting in a loss of background detail in the generated video.

[0121] In an embodiment of the present application, the target video generation model includes an image encoder and a reference network. The image encoder can be used to extract the semantic features of the target scene in the target scene image, and the reference network can be used to extract the target scene detail features in the target scene image. This can enhance the accuracy and authenticity of the background generation in the target video, and the target video generation effect is more natural, which can meet the actual needs of users.

[0122] Specifically, the electronic device extracts target scene semantic features of the target scene image through an image encoder in the target video generation model, which is the process of aligning the image to text. For the input target scene image, the image encoder can determine a text description related to the target scene image. Furthermore, the electronic device can extract target scene detail features corresponding to the target scene image through a reference network in the target video generation model. The reference network can adopt the same U-Net structure as the stable diffusion (SD) primitive model, consisting of downsampling and upsampling layers. The target video generation model can first encode the target scene image into latent space scene features using a variational autoencoder, and then input these latent space scene features into the reference network. Based on the structural characteristics of the reference network, target scene detail features at multiple scales can be output through different reference network modules, where different target scene detail features correspond to different details in the target scene image. Because the structure of the reference network is consistent with that of the SD primitive model, the target video generation model can inject target scene detail features at different scales into corresponding modules in the SD primitive model. This paired information injection facilitates the target video generation model to learn and understand the correspondence between background and foreground.

[0123] S204: Generate target image features corresponding to the data to be processed based on the target scene semantic features, target scene detail features, and background denoising features corresponding to the target noise latent code.

[0124] In an embodiment of the present application, the background denoising feature may refer to the denoising feature vector output by the stable diffusion model during the denoising process. The target image feature may refer to the feature vector of the target image generated by the target video generation model, wherein the target image may refer to the image after background completion, that is, the image including the foreground of the target object and the background of the target scene image. Specifically, the electronic device can perform feature fusion processing and dimensional processing based on the attention mechanism (Attention) according to the target scene semantic features, the target scene detail features and the background denoising features corresponding to the target noise hidden code through the target video generation model to obtain the target image features corresponding to the target image.

[0125] S205: Perform feature processing and timing control on target image features and target camera posture to generate a target video corresponding to the data to be processed.

[0126] In an embodiment of the present application, after the electronic device generates the target image features through the target video generation model, it can perform feature processing and timing control on the target image features and the target camera posture. Specifically, the target camera posture is encoded by the camera posture encoder in the timing control module to obtain the target camera posture features, and then feature fusion, dimension conversion and other feature processing are performed on the target camera posture features and the target image features. At the same time, timing consistency control can be performed, and finally the target video corresponding to the data to be processed can be generated based on the output target video features.

[0127] In an embodiment of the present application, an electronic device obtains data to be processed including a target foreground video and a target scene image, and then inputs the extracted and determined target object mask image, target foreground image, target camera pose and target noise latent code into a target video generation model. The target video generation model first generates target image features based on the semantic features of the target scene, the detail features of the target scene and the background denoising features corresponding to the target noise latent code, and then generates a target video in combination with the target camera pose. This can ensure the accuracy of the target video generation, and at the same time achieve the camera control and timing unification in the target video, so that the fusion of the foreground and background in the target video is more natural and smooth, thereby improving the video generation effect.

[0128] Based on the above embodiments, Figure 3 This is a flowchart of a target video generation model training method provided by an exemplary embodiment of this application. Figure 3 , the target video generation model training method may include:

[0129] S301. Obtain sample data; the sample data includes a sample foreground video and a sample scene image.

[0130] S302: Determine a sample mask image, a sample foreground image, and a sample camera pose corresponding to the sample scene image according to the sample foreground video, and determine a sample noise hidden code corresponding to the sample data.

[0131] S303: Iteratively train the preset image generation model according to the sample mask image, the sample foreground image, the sample scene image, and the sample noise latent code to obtain a target image generation model.

[0132] In the embodiment of the present application, the target video generation model adopts a two-stage training method during the training process. The electronic device first learns the spatial position of the foreground and background of the characters in the image domain, that is, iteratively trains the preset image generation model to obtain the target image generation model; then, the image model is expanded into a video model, and the video timing consistency and camera movement are learned in the video domain, that is, iteratively trains the preset video generation model to obtain the target video generation model. In this way, the target video generation model ultimately obtained by the electronic device through the two-stage training can achieve accurate and natural video background generation.

[0133] Specifically, the electronic device may obtain sample data, which may include sample foreground videos and sample scene images. The sample data may refer to pre-configured training samples. The electronic device may then extract, based on the sample foreground videos, a sample mask image, a sample foreground image, and a sample camera pose corresponding to the sample scene image, and simultaneously determine a sample noise latent code. During the first phase of training, the electronic device may iteratively train a preset image generation model based on the sample mask image, sample foreground image, sample scene image, and sample noise latent code to obtain a target image generation model.

[0134] In a possible implementation, step S303 may be implemented as follows:

[0135] Extract the sample scene semantic features and sample scene detail features corresponding to the sample scene image; splice the sample mask image, sample foreground image and sample noise latent code to obtain a sample input tensor; iteratively train the preset image generation model based on the sample scene semantic features, sample scene detail features and sample input tensor to obtain a target image generation model.

[0136] In the embodiment of the present application, the preset image generation model may include an image encoder, a reference network, a U-Net network (i.e., an SD base model), and a variational autoencoder. During the training of the preset image generation model, the electronic device may extract sample scene semantic features corresponding to the sample scene image using an image encoder such as CLIP, and may also extract sample scene detail features of the sample scene image based on the reference network.

[0137] Specifically, the video foreground (such as character foreground information, etc.) in the target video finally generated in the embodiment of the present application is determined based on the target foreground video. Therefore, the target object mask image, target foreground image, etc. extracted based on the target foreground video are one of the input conditions of the target video generation model. In this way, the electronic device can complete the background area while keeping the character foreground information unchanged, thereby realizing the automatic generation of the video background. On this basis, in the training process of the target video generation model, the foreground information fusion method is of high importance for the model to retain the foreground information, and a reasonable foreground information fusion method can also avoid the damage or loss of foreground information to a certain extent. In an embodiment of the present application, in the process of training the preset image generation model, the electronic device can perform channel expansion on the stable diffusion base model, and splice the sample mask image, the sample foreground image and the sample noise hidden code (Contact, the circle C in the figure of the present application is the feature splicing fusion) to obtain a sample input tensor. For example, the 4-channel sample foreground image, the 4-channel sample noise hidden code and the single-channel sample mask image can be spliced ​​in the feature dimension to form a 9-channel sample input tensor, and then this sample input tensor can be input into the U-Net network for noise reduction and iterative training. Among them, the sample noise hidden code is used to generate the final image, and the sample mask image helps the model identify which areas need to be kept and which areas need to be filled. Through this image splicing method, the electronic device can more accurately understand the relationship between the foreground and the background during the training of the preset image generation model, thereby achieving a more natural background completion.

[0138] It should be noted that the iterative training process of the preset image generation model can perform parameter adjustment and iterative training based on factors such as the preset loss function, the deviation value between noises, and the target number of iterations. It can be flexibly set based on actual needs, and the embodiments of the present application do not limit this.

[0139] In addition, in the embodiment of the present application, during the training process of the target video generation model, the electronic device can adopt a multi-task learning approach, performing training for scene completion tasks and scene transformation tasks simultaneously with background generation tasks, thereby enhancing the target video generation model's ability to generate related scenes and multi-view scenes based on contextual information. In one possible implementation, the sample foreground video includes a first sample foreground video, a second sample foreground video, and a third sample foreground video; the sample scene image includes a first sample scene image, a second sample scene image, and a third sample scene image;

[0140] Among them, the first sample foreground video and the first sample scene image are used to perform scene completion training on the preset image generation model; the second sample foreground video and the second sample scene image are used to perform scene transformation training on the preset image generation model; the third sample foreground video and the third sample scene image are used to perform background generation training on the preset image generation model.

[0141] In an embodiment of the present application, during the first phase of training of the target video generation model, the electronic device can employ a multi-task learning approach, simultaneously training the background generation task and the scene completion task and the scene transformation task. Background generation can refer to automatically generating background information based on foreground information; scene completion can refer to automatically generating related scenes to adapt to camera movements to present different environmental content; and scene transformation can refer to multi-perspective learning and presentation of scenes, enabling the display of scenes from different perspectives during camera movements.

[0142] In a possible implementation, the pixel value in the first sample foreground image corresponding to the first sample foreground video is the first value; the pixel value in the second sample foreground image corresponding to the second sample foreground video is the first value; and the pixel value in the second sample mask image corresponding to the second sample foreground video is the second value.

[0143] For example, Figure 4 A schematic diagram of a model training for a background generation task provided in an exemplary embodiment of the present application. Figure 5 A training diagram of a target video generation model provided by an exemplary embodiment of the present application. Figure 4 as well as Figure 5 , the two-stage training process of the target video generation model in the embodiment of the present application is described in detail:

[0144] In an embodiment of the present application, different sample data can be used for the background generation task, the scene completion task, and the scene transformation task. The scene completion task training of the preset image generation model can use a first sample foreground video and a first sample scene image. The first sample foreground video can refer to a foreground video after a portion of the scene is erased, and the first sample scene image can refer to a scene image after a portion of the scene is erased. Furthermore, the pixel values ​​of the first sample foreground image extracted from the first sample foreground video are a first value, i.e., a pixel value of 0. Subsequently, the first sample foreground image with all 0 pixel values, the first sample mask image extracted from the first sample foreground video, and the sample noise hidden code can be concatenated to obtain a sample input tensor. This sample input tensor and the first sample scene image are then input into the preset image generation model for iterative training. This allows the scene completion task training to correspond to the inputs of the background generation task and the scene transformation task, facilitating the joint learning of various tasks. The sample mask image is used to allow the preset image generation model to identify the areas that need to be completed and maintained, and the sample noise hidden code is used to generate the final completion result. The preset image generation model can extract the sample scene semantic features and sample scene detail features of the first sample scene image through the image encoder and the reference network, and then perform scene completion during the training process. In this way, by training the preset image generation model on the scene completion task, the electronic device can enable the model to have the ability to generate related scenes. The model can understand the surrounding environment content based on the given image information, providing content guarantee for the subsequent character background video generation. During the second stage of training, with the introduction of camera posture, the model can further understand the changes in camera movement and scenery, thereby ensuring the accuracy of target video generation and optimizing the target video generation effect.

[0145] The scene transformation task can be trained in both the first and second stages. During the first stage of training, the goal of the scene transformation task is to enable the preset image generation model to generate images of a given scene image from any other perspective. During the second stage of training, the goal of the scene transformation task is to train the model to generate a corresponding scene video with a camera movement effect given a scene image and a corresponding sample camera pose. In this way, through scene transformation in the image and video domains, the electronic device can enable the target video generation model obtained by training to have the ability to generate multi-perspective scenes.

[0146] Specifically, the sample data of the scene change task uses a second sample foreground video and a second sample scene image. The second sample foreground video may refer to a foreground video whose pixel values ​​are all the second value (i.e., 1). The pixel values ​​of the second sample foreground image extracted from the second sample foreground video are all the first value (i.e., 0), and the pixel values ​​of the second sample mask image extracted from the second sample foreground video are all the second value. Figure 5 As shown, during the scene change task training process in the first phase, the preset image generation model can extract the sample scene semantic features and sample scene detail features of the second sample scene image through the image encoder and the reference network respectively. At the same time, the electronic device can splice the second sample mask image with all pixel values ​​1 and the second sample foreground image with all pixel values ​​0 and the sample noise hidden code to obtain a sample input tensor, and then the sample scene semantic features, sample scene detail features and sample input tensor can be input into the U-Net model to iteratively train the preset image generation model. Among them, during the scene change task training process, the second sample mask image with all pixel values ​​1 is used to enable the preset image generation model to change all pixel values ​​of the image to generate images of different perspectives, and the second sample foreground image with all pixel values ​​0 is used to correspond to the input of the background generation training character, so that the background generation task, the scenery completion task, and the scene change task can be trained simultaneously in the first phase, thereby improving the model training efficiency.

[0147] During the second stage of scene change task training, the preset image generation model can be used to learn camera control. The electronic device can input the second sample scene image into the image encoder and the reference network to extract the sample scene semantic features and sample scene detail features, and can input the sample mask video (including multiple second sample mask images) with pixel values ​​all being 1 and the sample foreground video (including multiple second sample foreground images) with pixel values ​​all being 0 and the sample noise hidden code into the U-Net network for training. In addition, the electronic device also introduces sample camera poses to iteratively train the preset video generation model, so that during the second stage of training, the scene change task, background generation task and camera control learning can be trained simultaneously, so that the target video generation model finally trained can have the ability to generate multi-perspective scenes, and improve the naturalness and smoothness of the fusion of the foreground and background of the target video.

[0148] For the training of background generation task, the electronic device can use the third sample foreground video and the third sample scene image to perform background generation task training. It should be noted that during the multi-task training of the electronic device for background generation task, scene completion task and scene transformation task, sample data of the same scene can be used, or sample data of different scenes can be used, which is not limited in the embodiment of the present application. Figure 4 、 Figure 5 As shown, during the training process of the background generation task, the preset image generation model extracts sample scene semantic features and sample scene detail features of the third sample scene image through an image encoder and a reference network, respectively. The electronic device can then concatenate the third sample mask image, the third sample foreground image, and the sample noise latent code extracted from the third sample foreground image to obtain a sample input tensor. The preset image generation model can be iteratively trained based on the sample scene semantic features, the sample scene detail features, and the sample input tensor to ultimately obtain a target image generation model. During the second stage of training, a target video generation model can be obtained.

[0149] In the first stage of multi-task training, Figure 4 As shown, the attention mechanism processing process in the U-Net or SD base model can include: in the preset image generation model training process, the reference attention layer (ReferenceAttention) connects the sample scene detail feature (Scene Ref Embedding) with the sample denoising feature (BackgroundDenoising Feature) to obtain a fusion feature, and then performs a self-attention processing (Self Attention) on the fusion feature; then, in the cross attention layer (Cross Attention), the result of the self-attention processing is cross-attended with the sample scene semantic feature (Scene Clip Embedding), and finally the target image feature is output, based on which the target image can be generated.

[0150] S304: Iteratively train the preset video generation model according to the sample mask image, the sample foreground image, the sample scene image, the sample noise hidden code, the sample camera pose, and the target image generation model to obtain a target video generation model.

[0151] In an embodiment of the present application, after the first stage, i.e., the training in the image domain, is completed, the electronic device can further iteratively train the preset video generation model based on the sample mask image, the sample foreground image, the sample scene image, and the sample camera pose, combined with the parameters in the target image generation model, to obtain the target video generation model. Since the preset video generation model adds structures such as a timing control module on the basis of the preset image generation model, these newly added structures do not destroy the overall architecture of the original preset image generation model, and the electronic device can directly use the pre-trained parameters in the target image generation model to initialize the preset video generation model. In this way, the electronic device can maintain the capabilities of the original target image generation model, greatly reducing the difficulty of training the preset video generation model and improving the efficiency of model training.

[0152] In a possible implementation, step S304 may be implemented as follows:

[0153] The sample mask image, sample foreground image, sample scene image and sample noise hidden code are input into the preset video generation model to obtain the sample image features; the preset video generation model includes a preset image generation model and a timing control module; based on the sample image features and the sample camera pose, the timing control module in the preset video generation model is iteratively trained to obtain the target video generation model.

[0154] In the embodiment of the present application, during the second phase of training, the electronic device can iteratively train a preset video generation model based on the sample foreground image, the sample scene image, the sample noise hidden code, and the sample camera pose, while also combining the training parameters in the target image generation model to ultimately obtain a target video generation model. This preset video generation model adds modules such as timing control to the preset image generation model, ensuring natural and smooth video generation.

[0155] Specifically, the SD base model is used as an image generation model, and its network architecture can be directly migrated to the background generation task. However, since the SD base model lacks a module for modeling video timing, if video generation training is performed directly based on the network architecture of the SD base model, the generated video will inevitably show jitter and inconsistent content between frames. The complexity of video generation lies in the need to not only process the characteristics of the image's spatiotemporal dimensions, but also to ensure the coherence and consistency between video frames. Therefore, in an embodiment of the present application, during the second-stage training process of the electronic device, the preset video generation model not only has the image generation capability of the SD base model, but also introduces new structures such as a timing control module. That is, without destroying the SD base model, the preset video generation model can include a plug-and-play timing control module. The newly added timing control module does not destroy the overall architecture of the original preset image generation model. During the second-stage training process, the electronic device can directly use the pre-trained parameters in the target image generation model to initialize the preset video generation model, thereby ensuring that the preset video generation model has the capabilities of the target image generation model and improving the model training efficiency.

[0156] During the second phase of model training, the electronic device focuses on training the timing control module for the preset video generation model. The timing control module can include a two-layer attention mechanism structure that can effectively model the time dimension of the video, while the parameters of the other modules of the preset video generation model are consistent with those in the target image generation model and remain fixed. In this way, during the first phase of model training, the electronic device can use multi-task learning to train the model's scene completion, scene transformation, and background generation capabilities; during the second phase of model training, the electronic device focuses on training the timing control module, which can improve the model's ability to generate continuous video.

[0157] In a possible implementation, the training of the timing control module in the preset video generation model can be implemented in the following manner:

[0158] The sample camera pose is encoded by the camera pose encoder to obtain sample camera pose features; the sample camera pose features and the sample image features are fused to obtain sample pose fusion features, and the sample pose fusion features are dimensionally transformed by the camera pose adapter to obtain sample video features; the preset video generation model is iteratively trained according to the sample video features to obtain the target video generation model.

[0159] In an embodiment of the present application, the timing control module may include modules such as a camera pose encoder and a camera pose adapter corresponding to the camera movement control training, so that the camera pose can be combined with the attention mechanism structure to enhance the camera movement effect of the video while ensuring the consistency of video generation.

[0160] In the related art, existing video generation models do not explicitly pay attention to the motion relationship between the foreground and background. The automatically generated video mainly relies on the motion information implicit in the video foreground for camera movement, which may lead to a disharmony in the motion pattern between the scene background and the character foreground. For example, when providing some character foreground videos in which the camera follows the character's movement, since the size and position of the character in the foreground video do not change significantly, the motion information contained therein is relatively weak. This will cause the existing video generation model to be unable to effectively understand the motion state of the camera following. Therefore, the generated video may appear static, or the camera movement pattern it exhibits is relatively random and lacks coherence, resulting in poor video generation effect.

[0161] In an embodiment of the present application, the timing control module in the preset video generation model may include a camera pose encoder and a camera pose adapter, wherein the camera pose encoder is used to encode the camera pose to obtain a camera pose feature that can accurately reflect the position and motion state of the camera; the camera pose adapter can be used to perform dimensional conversion on the feature to ensure the accuracy of video generation. In this way, the electronic device integrates the camera pose encoder and the camera pose adapter with the attention structure in the timing control module, so that the target video generation model can fully consider the motion trajectory of the camera when generating the video.

[0162] For example, Figure 6 A training diagram of a timing control module provided by an exemplary embodiment of the present application. Figure 5 、 Figure 6 As shown, the timing control module includes an attention structure, a camera pose encoder, and a camera pose adapter. The camera pose encoder encodes the input sample camera poses to obtain multi-scale sample camera pose features; the camera pose adapter fuses sample image features with multi-scale camera pose features, thereby enabling the target video generation model to control the video camera movement. Each timing control module structure in the SD base model includes a camera pose adapter with a two-layer linear layer structure, which can determine the input and output size of the camera pose adapter based on the dimension of the video features. Specifically, during the second-stage training process, the preset video generation model can be based on the timing control module. First, the sample image features and camera pose features are concatenated together in the feature dimension to form sample pose fusion features. The sample pose fusion features are then input into the camera pose adapter for feature fusion and dimensionality transformation, outputting sample video features. Subsequent iterative training can be performed based on the sample video features and combined with a loss function to ultimately obtain the target video generation model. In this way, the training process of the timing control module can enable the target video generation model to control the camera pose during the video generation process, and can realize different types of camera movement control through different camera poses. The camera's sample camera pose is processed by a multi-scale camera pose encoder (such as Plücker coding), and feature fusion and dimensionality transformation are performed through multiple plug-and-play camera pose adapters, which can realize the control of camera movement during the video generation process.

[0163] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0164] In an embodiment of the present application, the electronic device can realize the natural fusion of the foreground in the target foreground video and the background in the target scene image according to the data to be processed, that is, the target foreground video and the target scene image, and finally generate the target video. This process is not just a simple image superposition, but requires the generation of a harmonious and unified target video so that the target object in the target foreground video and the background can be naturally integrated. This process requires the model to understand the spatial position relationship between the foreground and the background, that is, it needs to identify the relative position of the outline of the foreground target object and the background elements, and it needs to generate the video based on the camera posture. The process also requires appropriate expansion of the background, that is, scene completion and scene transformation (perspective transformation), because when the camera moves, some scenes will move out of the picture, and some scenes will move into the picture, so the model needs to have the ability to generate related scenes in the scene.

[0165] In an embodiment of the present application, the electronic device introduces a control signal of the camera posture during the video generation process, which can explicitly control the camera movement effect of the generated background video, making it consistent with the camera movement mode of the foreground, making the overall picture harmonious and natural; and by using a reference network to obtain fine-grained scene features in the reference image, the accuracy and authenticity of the background generation are enhanced, and the video generation effect is more natural and more in line with the actual needs of the user. In addition, during the model training process, the electronic device first learns the spatial position of the foreground and background in the image domain, and then expands the image model into a video model, and learns video timing consistency and camera movement in the video domain. Through a two-stage training strategy, the model can achieve realistic background generation; and a multi-task learning strategy is adopted during the training process, which enhances the model's ability to generate related scenes and multi-perspective scenes based on contextual information. While training the background generation task, scene completion and scene transformation are learned, so that the model has better scene generation capabilities. The target video generation model finally obtained can realize the camera movement control of the video background while automatically generating the video background, thereby improving the video generation effect.

[0166] Based on the above embodiments, Figure 7 FIG. 1 is a flow chart of another video generation method provided by an exemplary embodiment of the present application. Figure 7 As shown, the video generation method includes:

[0167] S701. In response to an interactive operation of a client, obtain data to be processed corresponding to the interactive operation; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target person video to be generated; and the target foreground video includes a target person.

[0168] S702: extracting a target person mask image, a target person foreground image, and a target camera pose from the target foreground video, and determining a target noise hidden code corresponding to the data to be processed.

[0169] S703: Input the target scene image, target person mask image, target foreground image, target camera pose and target noise hidden code into the target video generation model to generate the target person video corresponding to the data to be processed.

[0170] In an embodiment of the present application, the electronic device responds to the interactive operation of the client and obtains the data to be processed corresponding to the interactive operation; the data to be processed includes the target foreground video and the target scene image; the target scene image includes the video background of the target person video to be generated; the target foreground video includes the target person; the target person mask image, the target person foreground image and the target camera pose are extracted from the target foreground video, and the target noise hidden code corresponding to the data to be processed is determined; the target scene image, the target person mask image, the target foreground image, the target camera pose and the target noise hidden code are input into the target video generation model to generate the target person video corresponding to the data to be processed. In this way, the electronic device realizes the automatic generation of the target person video based on the combination of the target camera pose, can add the camera movement effect to the target person video, so that the fusion of the person foreground and the scene background of the target person video generated in the final generation is more natural and smooth, thereby enhancing the person video generation effect.

[0171] Figure 8 For a structural diagram of a video generation device provided by an exemplary embodiment of the present application, see Figure 8 , the video generating device includes:

[0172] The acquisition module 81 is used to acquire data to be processed; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of the target video to be generated; the target foreground video includes a target object;

[0173] A determination module 82 is configured to extract a target object mask image, a target foreground image, and a target camera pose from the target foreground video, and determine a target noise hidden code corresponding to the data to be processed;

[0174] The generation module 83 is used to input the target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise hidden code into the target video generation model to generate the target video corresponding to the data to be processed.

[0175] In a possible implementation, the determination module 82 is specifically configured to:

[0176] Through the target segmentation extraction model, the target foreground video is segmented to obtain the target object mask image and the target foreground image;

[0177] The target foreground video is detected and processed through the target camera detection model to obtain the target camera pose.

[0178] In a possible implementation, the generating module 83 is specifically configured to:

[0179] Through the target video generation model, the target scene semantic features and target scene detail features corresponding to the target scene image are extracted;

[0180] Generate target image features corresponding to the data to be processed based on the target scene semantic features, target scene detail features, and background denoising features corresponding to the target noise latent code;

[0181] Perform feature processing and timing control on the target image features and target camera pose to generate the target video corresponding to the data to be processed.

[0182] In one possible implementation, the device 80 is further configured to:

[0183] Acquire sample data; the sample data includes a sample foreground video and a sample scene image;

[0184] According to the sample foreground video, determine the sample mask image, sample foreground image and sample camera pose corresponding to the sample scene image, and determine the sample noise hidden code corresponding to the sample data;

[0185] According to the sample mask image, sample foreground image, sample scene image and sample noise latent code, the preset image generation model is iteratively trained to obtain the target image generation model;

[0186] According to the sample mask image, sample foreground image, sample scene image, sample noise hidden code, sample camera pose and target image generation model, the preset video generation model is iteratively trained to obtain the target video generation model.

[0187] In one possible implementation, the device 80 is further configured to:

[0188] Extracting sample scene semantic features and sample scene detail features corresponding to the sample scene image;

[0189] The sample mask image, sample foreground image and sample noise hidden code are concatenated to obtain the sample input tensor;

[0190] According to the semantic features of the sample scene, the detailed features of the sample scene and the sample input tensor, the preset image generation model is iteratively trained to obtain the target image generation model.

[0191] In a possible implementation, the sample foreground video includes a first sample foreground video, a second sample foreground video, and a third sample foreground video; the sample scene image includes a first sample scene image, a second sample scene image, and a third sample scene image;

[0192] The first sample foreground video and the first sample scene image are used to perform scene completion training on the preset image generation model; the second sample foreground video and the second sample scene image are used to perform scene transformation training on the preset image generation model; the third sample foreground video and the third sample scene image are used to perform background generation training on the preset image generation model.

[0193] In a possible implementation, the pixel value in the first sample foreground image corresponding to the first sample foreground video is the first value; the pixel value in the second sample foreground image corresponding to the second sample foreground video is the first value; and the pixel value in the second sample mask image corresponding to the second sample foreground video is the second value.

[0194] In one possible implementation, the device 80 is further configured to:

[0195] Inputting the sample mask image, the sample foreground image, the sample scene image and the sample noise hidden code into a preset video generation model to obtain sample image features; the preset video generation model includes a preset image generation model and a timing control module;

[0196] Based on the sample image features and sample camera poses, the timing control module in the preset video generation model is iteratively trained to obtain the target video generation model.

[0197] In one possible implementation, the timing control module includes a camera pose encoder and a camera pose adapter; the device is further configured to:

[0198] Encoding the sample camera pose by the camera pose encoder to obtain sample camera pose features;

[0199] Fusing the sample camera pose features and the sample image features to obtain sample pose fusion features, and performing dimension conversion on the sample pose fusion features through the camera pose adapter to obtain sample video features;

[0200] The preset video generation model is iteratively trained according to the sample video features to obtain the target video generation model.

[0201] The video generating device 80 provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0202] Figure 9 For a structural diagram of a video generation device provided by an exemplary embodiment of the present application, see Figure 9 , the video generating device 90 includes:

[0203] An acquisition module 91 is configured to acquire, in response to an interactive operation of a client, data to be processed corresponding to the interactive operation; the data to be processed includes a target foreground video and a target scene image; the target scene image includes a video background of a target person video to be generated; and the target foreground video includes a target person.

[0204] A determination module 92 is configured to extract a target person mask image, a target person foreground image, and a target camera pose from the target foreground video, and determine a target noise hidden code corresponding to the data to be processed;

[0205] The generation module 93 is used to input the target scene image, the target person mask image, the target foreground image, the target camera pose and the target noise hidden code into the target video generation model to generate the target person video corresponding to the data to be processed.

[0206] The video generating device 90 provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0207] Figure 10 A schematic diagram of a video generation device provided by an exemplary embodiment of the present application is shown in FIG. Figure 10 The video generation device 100 may include a processor 1001 and a memory 1002. Exemplarily, the processor 1001 and the memory 1002 are interconnected via a bus 1003.

[0208] Memory 1002 stores computer-executable instructions;

[0209] The processor 1001 executes the computer-executable instructions stored in the memory 1002 , so that the processor 1001 performs the video generation method as shown in the above method embodiment.

[0210] Accordingly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the video generation method of the above method embodiment.

[0211] Accordingly, an embodiment of the present application may also provide a computer program product, including a computer program. When the computer program is executed by a processor, it can implement the video generation method shown in the above method embodiment.

[0212] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0213] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0216] In a typical configuration, a computing device includes one or more processors, input / output interfaces, network interfaces, and memory.

[0217] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0218] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0219] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0220] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A video generation method, characterized in that: include: Get the data to be processed; The data to be processed includes a target foreground video and a target scene image; The target scene image includes a video background of a target video to be generated; The target foreground video includes a target object; Extracting a target object mask image, a target foreground image, and a target camera pose from the target foreground video, and determining a target noise hidden code corresponding to the data to be processed; The target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise hidden code are input into a target video generation model to generate a target video corresponding to the data to be processed.

2. The method according to claim 1, characterized in that The step of extracting a target object mask image, a target foreground image, and a target camera pose from the target foreground video includes: Segmenting the target foreground video using a target segmentation extraction model to obtain the target object mask image and the target foreground image; The target foreground video is detected and processed by the target camera detection model to obtain the target camera pose.

3. The method according to claim 1, characterized in that The step of inputting the target scene image, the target object mask image, the target foreground image, the target camera pose, and the target noise hidden code into a target video generation model to generate a target video corresponding to the data to be processed includes: Extracting target scene semantic features and target scene detail features corresponding to the target scene image through the target video generation model; generating target image features corresponding to the data to be processed according to the target scene semantic features, the target scene detail features, and the background denoising features corresponding to the target noise latent code; Feature processing and timing control are performed on the target image features and the target camera posture to generate a target video corresponding to the data to be processed.

4. The method according to claim 1, wherein The method further comprises: Acquire sample data; the sample data includes a sample foreground video and a sample scene image; Determining, based on the sample foreground video, a sample mask image, a sample foreground image, and a sample camera pose corresponding to the sample scene image, and determining a sample noise hidden code corresponding to the sample data; Iteratively training a preset image generation model according to the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code to obtain a target image generation model; According to the sample mask image, the sample foreground image, the sample scene image, the sample noise hidden code, the sample camera pose and the target image generation model, the preset video generation model is iteratively trained to obtain the target video generation model.

5. The method according to claim 4, characterized in that The iterative training of a preset image generation model according to the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code to obtain a target image generation model includes: Extracting sample scene semantic features and sample scene detail features corresponding to the sample scene image; Performing splicing processing on the sample mask image, the sample foreground image, and the sample noise hidden code to obtain a sample input tensor; The preset image generation model is iteratively trained according to the sample scene semantic features, the sample scene detail features and the sample input tensor to obtain the target image generation model.

6. The method according to claim 5, characterized in that The sample foreground videos include a first sample foreground video, a second sample foreground video, and a third sample foreground video; the sample scene images include a first sample scene image, a second sample scene image, and a third sample scene image; The first sample foreground video and the first sample scene image are used to perform scene completion training on the preset image generation model; The second sample foreground video and the second sample scene image are used to perform scene change training on the preset image generation model; The third sample foreground video and the third sample scene image are used to perform background generation training on the preset image generation model.

7. The method according to claim 6, characterized in that The pixel value in the first sample foreground image corresponding to the first sample foreground video is the first value; the pixel value in the second sample foreground image corresponding to the second sample foreground video is the first value; The pixel value in the second sample mask image corresponding to the second sample foreground video is a second value.

8. The method according to any one of claims 4 to 7, characterized in that The method of iteratively training a preset video generation model according to the sample mask image, the sample foreground image, the sample scene image, the sample noise hidden code, the sample camera pose, and the target image generation model to obtain the target video generation model includes: Inputting the sample mask image, the sample foreground image, the sample scene image, and the sample noise hidden code into the preset video generation model to obtain sample image features; the preset video generation model includes the preset image generation model and a timing control module; Based on the sample image features and the sample camera poses, the timing control module in the preset video generation model is iteratively trained to obtain the target video generation model.

9. The method according to claim 8, characterized in that The timing control module includes a camera pose encoder and a camera pose adapter; the timing control module in the preset video generation model is iteratively trained based on the sample image features and the sample camera pose to obtain the target video generation model, including: Encoding the sample camera pose by the camera pose encoder to obtain sample camera pose features; Fusing the sample camera pose features and the sample image features to obtain sample pose fusion features, and performing dimension conversion on the sample pose fusion features through the camera pose adapter to obtain sample video features; The preset video generation model is iteratively trained according to the sample video features to obtain the target video generation model.

10. A video generation method, characterized in that: include: In response to an interactive operation of a client, obtaining data to be processed corresponding to the interactive operation; The data to be processed includes a target foreground video and a target scene image; The target scene image includes a video background of a target person video to be generated; The target foreground video includes a target person; Extracting a target person mask image, a target person foreground image, and a target camera pose from the target foreground video, and determining a target noise hidden code corresponding to the data to be processed; The target scene image, the target person mask image, the target person foreground image, the target camera pose and the target noise hidden code are input into a target video generation model to generate a target person video corresponding to the data to be processed.

11. A video generating device, characterized in that: include: An acquisition module is used to obtain data to be processed; The data to be processed includes a target foreground video and a target scene image; The target scene image includes a video background of a target video to be generated; The target foreground video includes a target object; A determination module is used to extract a target object mask image, a target foreground image, and a target camera pose from the target foreground video, and determine a target noise hidden code corresponding to the data to be processed; A generation module is used to input the target scene image, the target object mask image, the target foreground image, the target camera pose and the target noise hidden code into a target video generation model to generate a target video corresponding to the data to be processed.

12. A video generating device, characterized in that: include: memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the video generation method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the video generation method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a computer, implements the video generation method according to any one of claims 1 to 10.

Images