Video generation method, and training method and device of video generation model

Abstract

The invention provides a video generation method, a training method and device of a video generation model, electronic equipment, a storage medium, a program product and an intelligent agent, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes of digital human, artificial intelligence-based content generation and the like. The specific implementation scheme is as follows: acquiring constraint information and an original video; based on the noise weight indicating the noise adding content, noise is added to the original video, a noise video is obtained, the noise weight is determined based on a starting time step, and the starting time step indicates the starting point of the denoising process; and based on the constraint information serving as the video content constraint and the starting time step serving as the denoising process constraint, denoising the noise video to obtain a target video, the video content of the target video being matched with the constraint information.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to scenarios such as digital humans and AI-based content generation. Specifically, it relates to video generation methods, training methods for video generation models, devices, electronic devices, storage media, program products, and intelligent agents. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI-based video generation has become a hot topic in the technology field. This technology has changed traditional video production methods and is widely used in various fields. However, there is still room for exploration in how to improve the quality of generated videos. Summary of the Invention

[0003] This disclosure provides a video generation method, a training method for a video generation model, an apparatus, an electronic device, a storage medium, a program product, and an intelligent agent.

[0004] According to one aspect of this disclosure, a video generation method is provided, comprising: acquiring constraint information and an original video; adding noise to the original video based on a noise weight indicating the noise content to obtain a noisy video, wherein the noise weight is determined based on a start time step, the start time step indicating the starting point of a denoising process; and denoising the noisy video based on the constraint information as a constraint on video content and the start time step as a constraint on the denoising process to obtain a target video, wherein the video content of the target video matches the constraint information.

[0005] According to another aspect of this disclosure, a method for training a video generation model is provided, comprising: acquiring a sample target video and sample constraint information; adding noise to the sample target video based on noise weights used to indicate the noise content to obtain a sample noisy video, wherein the noise weights are determined based on a sample start time step, and the sample start time step indicates the starting point of a denoising sub-process characterized by a sample time step sub-sequence; denoising the sample noisy video using the video generation model based on the sample constraint information as video content constraints and the sample time step sequence as denoising process constraints to obtain a sample denoised video; and training the video generation model based on the sample denoised video and the sample target video to obtain a trained video generation model.

[0006] According to another aspect of this disclosure, a video generation apparatus is provided, comprising: an acquisition module for acquiring constraint information and an original video; a noise addition module for adding noise to the original video based on a noise weight indicating the noise content to obtain a noisy video, wherein the noise weight is determined based on a start time step, the start time step indicating the starting point of a denoising process; and a video generation module for denoising the noisy video based on the constraint information as a constraint on video content and the start time step as a constraint on the denoising process to obtain a target video, wherein the video content of the target video matches the constraint information.

[0007] According to another aspect of this disclosure, a training apparatus for a video generation model is provided, comprising: a sample acquisition module for acquiring sample target video and sample constraint information; a sample noise addition module for adding noise to the sample target video based on noise weights indicating the noise content, to obtain a sample noise video, wherein the noise weights are determined based on a sample start time step, and the sample start time step indicates the starting point of a denoising sub-process characterized by a sample time step sub-sequence; a video generation module for denoising the sample noise video using the video generation model based on the sample constraint information as video content constraints and the sample time step sequence as denoising process constraints, to obtain a sample denoised video; and a training module for training the video generation model based on the sample denoised video and the sample target video, to obtain a trained video generation model.

[0008] According to another aspect of this disclosure, an intelligent agent is provided, comprising: an input module for receiving input information; a processing module for obtaining output information by calling a large model to execute the method described above based on the input information received by the input module; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This illustration schematically shows an exemplary system architecture to which video generation methods and apparatus can be applied according to embodiments of the present disclosure;

[0015] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 3 A schematic diagram illustrating the generation of a target video according to an embodiment of the present disclosure is shown.

[0017] Figure 4 This illustration schematically shows a process diagram of intermediate iteration denoising according to an embodiment of the present disclosure;

[0018] Figure 5 A schematic diagram illustrating feature updates according to an embodiment of the present disclosure is shown.

[0019] Figure 6A The illustration schematically shows a process diagram for determining audio features according to an embodiment of the present disclosure;

[0020] Figure 6B The illustration shows a flowchart of determining facial expression features according to an embodiment of the present disclosure;

[0021] Figure 7 A flowchart illustrating a training method for a video generation model according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 8 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 9 A block diagram of a training apparatus for a video generation model according to an embodiment of the present disclosure is shown schematically;

[0024] Figure 10 A schematic diagram illustrating the structure of an intelligent agent according to embodiments of the present disclosure is shown; and

[0025] Figure 11 A block diagram of an electronic device suitable for implementing a video generation method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] The creation of digital humans based on Artificial Intelligence (AI) has seen rapid development and is already being applied in various fields, such as virtual anchors, brand ambassadors, intelligent customer service, guidance, explanation, consultation, healthcare, education, and industrial applications. AI digital humans can provide 24 / 7 services including consultation, business processing, and live streaming, thereby improving service capabilities and reducing service costs.

[0028] However, some technical problems still need to be solved in AI-based video generation scenarios. For example, the video generation capability is poor, and it is difficult to achieve both generation effect and generation efficiency at the same time.

[0029] In view of this, the present disclosure provides a video generation method, comprising: acquiring constraint information and an original video; adding noise to the original video based on a noise weight indicating the noise content to obtain a noisy video, wherein the noise weight is determined based on a start time step, the start time step indicating the starting point of the denoising process; and denoising the noisy video based on the constraint information as a constraint on the video content and the start time step as a constraint on the denoising process to obtain a target video, wherein the video content of the target video matches the constraint information.

[0030] By utilizing the embodiments provided in this disclosure, the noise content added to the original video is controlled based on the starting time step, and the starting point of the denoising process is also controlled using the starting time step. This allows for improved denoising efficiency by reducing the number of denoising steps, while simultaneously improving the alignment of the target video and the original video in terms of the macroscopic structure of objects by utilizing the macroscopic structural information of objects contained in the noisy video. This reduces denoising errors generated in the early denoising stages, thereby improving the denoising effect. Thus, both the quality and efficiency of target video generation are simultaneously achieved.

[0031] Figure 1 The illustration schematically depicts an exemplary system architecture to which video generation methods and apparatus can be applied according to embodiments of the present disclosure.

[0032] It is important to note that Figure 1The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. They do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture to which the video generation method and apparatus can be applied may include a terminal device. However, the terminal device can implement the video generation method and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0036] Optionally, the user can record an original video with audio using the first terminal device 101. The original video and audio, as constraint information, are then sent to the server 105 via the first terminal device 101.

[0037] Server 105 may be a server that provides various services, such as a back-end management server that supports the content browsed by the user using the first terminal device 101 (for example only).

[0038] Optionally, server 105 may execute the video generation method provided in this embodiment of the present disclosure based on the obtained constraint information and the original video to obtain a target video whose video content matches the constraint information, and send it to the second terminal device 102 and the third terminal device 103 through network 104 so as to display the target video on the second terminal device 102 and the third terminal device 103.

[0039] It should be understood that Figure 1 The number of first, second, and third terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of first, second, and third terminal devices, networks, and servers can be included.

[0040] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0041] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0042] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0043] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically.

[0044] like Figure 2 As shown, the method includes operations S210~S230.

[0045] Operate S210 to obtain constraint information and raw video.

[0046] In operation S220, noise is added to the original video based on the noise weight indicating the noise content, resulting in a noisy video.

[0047] In operation S230, the noisy video is denoised based on the constraint information that serves as a constraint on the video content and the starting time step that serves as a constraint on the denoising process, to obtain the target video.

[0048] Raw video can refer to unprocessed video. For example, a video shot by a user.

[0049] Constraint information, also known as guiding information, is used to constrain the video content of the generated target video so that the video content of the target video matches the constraint information.

[0050] For example, constraint information may include audio, which constrains the lip movements of the target object in the target video, ensuring that the lip movements of the target object in the final generated target video match the audio. Alternatively, constraint information may include a reference image, whose reference object constrains the image of the target object in the target video, ensuring that the image of the target object in the final generated target video matches the image of the reference object.

[0051] Noise can refer to unnecessary or redundant interference information present in video frames.

[0052] The diffusion principle can be used to process the original video by adding noise, resulting in a noisy video. Then, the noisy video is denoised to generate a target video whose content matches the constraint information.

[0053] Multi-round iterative denoising can be performed. The time step can be used to indicate the current denoising round. For example, a time step of 1 indicates the first round of denoising. Multi-round iterative denoising can include a sequence of multiple time steps, such as 1 to 500. The time step sequence can indicate the entire denoising process.

[0054] Setting a start time step allows you to re-indicate the starting point of the denoising process, instead of starting with the start time step of the entire denoising process, such as 1.

[0055] If the starting time step is specified as 100, the denoising process used for video generation is a sub-process of the full denoising process. The time steps of the denoising process are sub-sequences of the time step sequence, for example, the time steps include 100 to 500.

[0056] Noise weights are determined based on the start time step, which indicates the starting point of the denoising process. Noise weights, indicating the amount of noise to be added, are determined based on the start time step, and noise is added to the original video based on these weights. This ensures that the noise-added video matches the starting point of the denoising process indicated by the start time step, improving the compatibility between the data to be processed and the processing operation.

[0057] For example, a full denoising process with time step 1 can target a video with 100% noise content. A denoising process with time step 100 can target a video with 80% noise content.

[0058] In the entire denoising process utilizing the diffusion principle, the early denoising stages, such as those with time steps 1 to 100, have very high noise levels. The goal of denoising is to reconstruct the macroscopic structure of the image, such as object pose, object identity, and object contours. After completing the early denoising stages, the video has already had its macroscopic structural information reconstructed through the denoising processing of these early stages. Therefore, by controlling the amount of original information contained in the noisy video that is also present in the original video, and enabling the rapid identification and learning of macroscopic structural information in the noisy video, denoising can begin in the middle stage of the entire denoising process. This skips the early denoising stages that reconstruct macroscopic structural information, thus avoiding denoising errors caused by the early denoising stages and ensuring alignment of the target video with the original video in terms of object contours.

[0059] By utilizing the embodiments provided in this disclosure, the noise content added to the original video is controlled based on the starting time step, and the starting point of the denoising process is also controlled using the starting time step. This allows for improved denoising efficiency by reducing the number of denoising steps, while simultaneously improving the alignment of the target video and the original video in terms of the macroscopic structure of objects by utilizing the macroscopic structural information of objects contained in the noisy video. This reduces denoising errors generated in the early denoising stages, thereby improving the denoising effect. Thus, both the quality and efficiency of target video generation are simultaneously achieved.

[0060] The preceding text provided an overview of how to generate the target video. The following text will explain how to utilize constraint information to generate the target video.

[0061] According to embodiments of this disclosure, constraint information may include at least one of the following: reference image, facial expression information, descriptive information, and audio.

[0062] Optionally, the matching of the video content of the target video with the constraint information may include at least one of the following: the facial image of the target object in the target video matches the facial image of the reference object in the reference image; the expression of the target object matches the expression information; the video theme of the target video matches the descriptive information; and the lip movements of the target object match the audio.

[0063] Using audio as constraint information can be applied in scenarios where audio is used to drive the target object in the target video to speak audio, so that the lip movements are aligned with the audio content.

[0064] Using reference images as constraint information can be applied in digital human video generation scenarios. The generation of virtual target objects in the target video is driven by the reference objects in the reference images, so that the identity information and facial features of the virtual target objects in the target video are consistent with the identity information and facial features of the reference objects.

[0065] Furthermore, the constraints can be further enriched using facial expression information, descriptive information, etc., to enhance the richness and diversity of the generated target video. For example, using facial expression information as a constraint allows control over the consistency between the facial expression of the target object and the facial expression information in the target video. Descriptive information can be used as a constraint to constrain the target video as a whole. For instance, if descriptive information is used to constrain the posture and head movements of the target object, then the posture and head movements of the target object in the target video will be semantically identical to those indicated by the descriptive information. As another example, if descriptive information is used to constrain the background of the target video, then the background of the target video will be identical to the background indicated by the descriptive information.

[0066] The following will be through Figure 3 This section explains how constraint information constrains the video content of the target video.

[0067] Figure 3A schematic diagram illustrating the generation of a target video according to an embodiment of the present disclosure is shown.

[0068] like Figure 3 As shown, constraint information may include a reference image, which includes a reference object. Description information may include text descriptions such as "add star background". Audio may include "news voice". The original video includes the original object; noise is added to the original video to obtain a noisy video.

[0069] like Figure 3 As shown, the lip movements of the target object in the target video are aligned with the audio, the facial features of the target object are consistent with those of the reference object, and the background of the target video matches the descriptive information. This results in a video narrated by a virtual digital human.

[0070] According to the video generation method provided in this disclosure, the generation of virtual target objects can be controlled by constraint information to depict the target video from different aspects and dimensions, thereby making the target object's posture, actions, lip movements, expressions, etc. consistent with expectations, and thus improving the interest and controllability of the target video.

[0071] For example, when the constraint information includes a reference image, the maskless video generation method provided in this disclosure improves the digital human driving effect of the video generation model, ensuring a high degree of restoration of pose and identity information. Furthermore, it can control the starting point of the denoising process, skipping the early stages of denoising. When the head region of the reference object in the reference image included in the constraint information is pasted back to the head region of the target object in the target video, edge alignment is achieved, avoiding fake faces and segmentation issues. This solves the problem of poor detail rendering in virtual characters.

[0072] The following section will provide a detailed explanation of how to use the start time step and constraint information to denoise noisy videos.

[0073] According to embodiments of this disclosure, for example, Figure 2 The operation S230 shown, which denoises a noisy video based on constraint information that serves as a constraint on the video content and a starting time step that serves as a constraint on the denoising process, to obtain a target video, may include: extracting a time step subsequence from the time step sequence that indicates the entire denoising process based on the starting time step; and iteratively denoising the noisy video based on the constraint features of the constraint information and the time step subsequence to obtain the target video.

[0074] A time step sequence can include the first time step, the second time step, the third time step, ..., the Nth time step. N can be greater than or equal to 500. Each time step can be represented by a number such as an integer like 1, 2, 3, ..., N, or by a fraction between 0 and 1. It can also be represented by text such as Timestep1, Timestep2, ..., TimestepN. The key is that the sequence must be temporally continuous.

[0075] The starting time step can be any time step in the time step sequence, such as the third time step. A subsequence of time steps can be extracted from the time step sequence, starting from the starting time step and ending at the last time step, such as a subsequence including the third time step, ..., the Nth time step. This subsequence of time steps can then be used to control the iterative denoising process of the noisy video to obtain the target video.

[0076] According to embodiments of this disclosure, a time step sub-sequence can be extracted from the time step sequence indicating the entire denoising process. This simplifies the determination of the time step sub-sequence constraining the denoising process while improving the flexibility of determining the time step sequence, based on a standard denoising process.

[0077] The above section explained how to determine the time step sequence. The following section will explain in detail how to use the time step sequence for iterative denoising.

[0078] According to embodiments of this disclosure, the noisy video is iteratively denoised based on the constraint features of the constraint information and the target time step sequence to obtain the target video, which may include at least one denoising step as follows:

[0079] Based on constraint features and features determined at the current time step, denoising is performed on multiple previous round video segment features to obtain multiple current round denoised video segment features. Feature updates are then performed on these multiple current round denoised video segment features to obtain multiple current round video segment features.

[0080] Multiple features of the previous video segments are obtained by performing at least one denoising operation on multiple noisy video segments obtained by splitting the noisy video into frames. The current time step is the time step in the time step subsequence.

[0081] When the constraint information includes a reference image, descriptive information, audio, and facial expression information, features can be extracted from each of these components separately to obtain image features of the reference image, descriptive features of the descriptive information, facial expression features of the facial expression information, and audio features of the audio. Alternatively, image features, descriptive features, facial expression features, and audio features can all be used as constraint features. Furthermore, feature fusion can be performed on image features, descriptive features, facial expression features, and audio features to obtain the constraint features.

[0082] The following will be explained through examples and in conjunction with Figure 4 This paper introduces an iterative denoising method for noisy videos.

[0083] Noisy video can be split into frames to obtain multiple temporally consecutive noisy video segments. For example, a first noisy video segment, a second noisy video segment, and a third noisy video segment. Each noisy video segment includes multiple noisy video frames. Feature extraction is performed on the first, second, and third noisy video segments respectively to obtain the features of the first, second, and third noisy video segments.

[0084] We can extract features from constraint information to obtain constraint features.

[0085] The time step subsequence includes the i-th time step as the starting time step. Taking the i-th time step as an example, feature extraction is performed on the i-th time step to obtain the i-th time step features.

[0086] Based on the constraint features and the features at the i-th time step, the first noisy video segment features, the second noisy video segment features, and the third noisy video segment features are denoised sequentially to obtain the first denoised video segment features, the second denoised video segment features, and the third denoised video segment features in the i+1 current round.

[0087] Update the features of the first denoised video segment, the second denoised video segment, and the third denoised video segment in the current round (i+1) to obtain the features of the first video segment, the second video segment, and the third video segment in the current round (i+1).

[0088] Once the features of multiple video segments have been processed, it is determined that the denoising process for the round corresponding to the starting time step has been completed, and the next round of denoising can be executed.

[0089] Figure 4 The schematic diagram illustrates a process flow diagram of intermediate iteration denoising according to an embodiment of the present disclosure.

[0090] like Figure 4As shown, after multiple iterations, taking the current time step 441 as an example, feature extraction is performed on the current time step 441, which indicates the current iteration round, to obtain the current time step feature 442. Based on the constraint feature 432 of constraint information 431 and the current time step feature 442, denoising is performed on the previous round's first video segment feature 411, the previous round's second video segment feature 412, and the previous round's third video segment feature 413, respectively, to obtain the current round's first denoised video segment feature 421, the current round's second denoised video segment feature 422, and the current round's third denoised video segment feature 423. The current round's first denoised video segment feature 421, the current round's second denoised video segment feature 422, and the current round's third denoised video segment feature are updated using feature 423 to obtain the current round's first video segment feature, the current round's second video segment feature, and the current round's third video segment feature.

[0091] Optionally, if the current time step is the last time step in a time step subsequence, the denoising process is determined to be complete, and the target video is obtained based on the features of multiple current round video segments. For example, the target video can be obtained by concatenating the features of multiple current round video segments. However, this is not the only option. The target video can also be obtained by post-processing the features of multiple current round video segments. Post-processing operations include at least one of the following: decoding, fully connected processing, linear processing, and concatenation.

[0092] According to the denoising process provided in this embodiment, multiple video frame features in the video segment features can be processed simultaneously, fully combining the temporal correlation of multiple video frame features in the video segment features. In addition, after denoising, the denoised video segment features are updated, thereby improving the temporal continuity of multiple video segments and avoiding problems such as action jumps or discontinuities, thereby solving the problem of poor temporal continuity of virtual characters.

[0093] The following section will explain the feature update process in detail.

[0094] According to embodiments of this disclosure, feature updates are performed on multiple current-round denoised video segment features to obtain multiple current-round video segment features. This may include: determining a first sub-segment feature of the preceding noisy video segment and a second sub-segment feature of the following noisy video segment from the multiple current-round denoised video segment features; fusing the first and second sub-segment features to obtain fused sub-segment features; and updating the first and second sub-segment features based on the fused sub-segment features to obtain multiple current-round video segment features.

[0095] After splitting the noisy video into multiple frames, each of the resulting noisy video segments contains boundary video frames. These boundary video frames need to be aligned at the beginning and end to achieve a smooth transition in the final target video.

[0096] Feature fusion can be performed on the frame features of two adjacent boundary video frames, and the fused features can be used for updating. However, this is not the only option. To avoid skipping or discontinuity issues in the target video, feature fusion can be performed on the sub-segment features of two adjacent boundary video segments, and the fused features can be used for updating. For example, for two adjacent noisy video segments that are temporally consecutive, a preceding noisy video segment and a following noisy video segment, feature fusion can be performed on the first sub-segment features of the preceding noisy video segment and the second sub-segment features of the following noisy video segment, and the fused features can be used for updating.

[0097] The following will be through Figure 5 This section explains how to use the features of the first and second sub-segments for feature updates.

[0098] Figure 5 A schematic diagram illustrating feature updates according to an embodiment of the present disclosure is shown.

[0099] like Figure 5 As shown, the noisy video is split into frames to obtain multiple noisy video segments that are sequentially consecutive. For example, this includes the first noisy video segment, the second noisy video segment, and the third noisy video segment.

[0100] Each noisy video segment may include multiple temporally consecutive noisy video sub-segments. For example, the first noisy video segment includes noisy video sub-segment 511 (1-1), noisy video sub-segment 512 (1-2), and noisy video sub-segment 513 (1-3). The second noisy video segment includes noisy video sub-segment 521 (2-1), noisy video sub-segment 522 (2-2), and noisy video sub-segment 523 (2-3). The third noisy video segment includes noisy video sub-segment 531 (3-1), noisy video sub-segment 532 (3-2), and noisy video sub-segment 533 (3-3).

[0101] like Figure 5 As shown, the 2nd-3rd noisy video sub-segment 523 can be the preceding noisy video sub-segment in the noisy video, and the 3rd-1st noisy video sub-segment 531 can be the following noisy video sub-segment in the noisy video. The first sub-segment feature 523-1 can be the feature corresponding to the 2nd-3rd noisy video sub-segment 523, and the second sub-segment feature 531-1 can be the feature corresponding to the 3rd-1st noisy video sub-segment 531.

[0102] like Figure 5As shown, the first sub-segment feature 523-1 and the second sub-segment feature 531-1, which are sequentially continuous, can be fused to obtain the fused sub-segment feature 500-1. The first sub-segment feature 523-1 and the second sub-segment feature 531-1 are then replaced using the fused sub-segment feature to complete the update.

[0103] Optionally, the features corresponding to the 1st-3rd noisy video sub-segments and the features corresponding to the 2nd-1st noisy video sub-segment can be fused and updated. This completes the transition between features of multiple adjacent sub-segments, achieving a smooth connection in the temporal dimension.

[0104] Based on relevant examples, when the current time step is the last time step, the features of multiple current-round denoised video segments can be used as multiple current-round video segment features. These multiple current-round video segment features are then decoded separately to obtain multiple decoded video segments. These multiple decoded video segments are then concatenated to obtain the target video.

[0105] Compared to the related example, by updating the features of multiple denoised video segments, the features of multiple video segments can be smoothly connected in the time dimension, ensuring a smooth transition between multiple video segments in the target video at the splicing point and avoiding problems such as jumps or discontinuities.

[0106] Optionally, feature fusion of the first sub-segment features and the second sub-segment features to obtain fused sub-segment features may include: performing feature weighted fusion of the first sub-segment features and the second sub-segment features based on the first weight vector and the second weight vector set for each of the first sub-segment features and the second sub-segment features to obtain fused sub-segment features.

[0107] See Figure 5 As shown, both the first and second weight vectors can include multiple weights equal to the number J of video frames in the noisy video segment, where J is greater than or equal to 2. The first weight vector can include 1-wj, where j can range from 1 to J. The second weight vector can include wj. The weights in the first weight vector and the weights at corresponding positions in the second weight vector sum to 1, and wj decreases or increases by the same value.

[0108] According to embodiments of this disclosure, feature fusion using the above method can further eliminate discontinuities at video segment boundaries and improve the continuity of video segments by adjusting weights.

[0109] The previous section explained how to perform feature updates. The following section will explain how to determine the starting time step.

[0110] According to embodiments of this disclosure, when performing such Figure 2Prior to the operation S220 shown, the video generation method may further include: determining a start time step from a plurality of predetermined start time steps.

[0111] Multiple predetermined start time steps can be evaluated using evaluation videos to determine the evaluation results for each predetermined start time step. Based on the evaluation results for each predetermined start time step, a start time step is determined from among the multiple predetermined start time steps.

[0112] As analyzed above, using a starting time step to add noise to the original video and indicating the starting point of the denoising process achieves a dual improvement in denoising efficiency and effect. Correspondingly, the evaluation result can characterize at least one of the following evaluation metrics: denoising effect metric and denoising efficiency metric. Preferably, the evaluation result characterizes both the denoising effect metric and the denoising efficiency metric.

[0113] According to embodiments of this disclosure, the effectiveness of the determined start time step is improved by quantifying the evaluation results of multiple pre-screened predetermined start times.

[0114] Optionally, the number of denoising iterations to be executed can be determined based on a predetermined start time step and time step sequence. Based on the number of denoising iterations, efficiency evaluation results related to denoising efficiency metrics can be determined.

[0115] The evaluation video can also be denoised to obtain the target evaluation video. Based on the target evaluation video and the evaluation video, the effect evaluation results related to the denoising effect indicators are obtained.

[0116] The following section will explain how to determine the performance evaluation indicators.

[0117] According to embodiments of this disclosure, evaluating multiple predetermined starting time steps using an evaluation video to determine the evaluation results for each of the predetermined starting time steps may include: adding noise to the evaluation video using initial noise weights to obtain a noisy evaluation video; denoising the noisy evaluation video based on constraint information as video content constraints and predetermined starting time steps as constraints for the denoising process to obtain a target evaluation video; and obtaining the evaluation result for each predetermined starting time step based on the target evaluation video and the evaluation video.

[0118] The initial noise weights are determined based on a predetermined start time step. After adding noise to the evaluation video, the noisy evaluation video is denoised according to the evaluation time step subsequence indicated by the predetermined start time step to obtain the target evaluation video.

[0119] The noise assessment video is a video with added noise compared to the assessment video. The target assessment video, obtained by denoising the noise assessment video, aims to restore the video information masked by the noise. Therefore, based on the target assessment video and the assessment video, the assessment result at a predetermined starting time step can include: determining the matching result between the video content of the target assessment video and the video content of the assessment video. Based on the matching result, the effect assessment result is determined.

[0120] The evaluation results can be obtained based on the results of the effectiveness evaluation and the efficiency evaluation, for example, by weighted summation.

[0121] According to embodiments of this disclosure, denoising of noisy videos is used as an evaluation method to determine the evaluation result, which matches the actual application process. Thus, based on the quantitative evaluation result, the quantitative accuracy and effectiveness of the evaluation result are further improved by ensuring that the evaluation operation is consistent with the operation used in the actual application.

[0122] The previous section explained how to determine the starting time step and how to use it for noise reduction. The following section will explain how to identify noisy video.

[0123] According to embodiments of this disclosure, for example, Figure 2 The illustrated operation S220, which adds noise to the original video based on noise weights used to indicate the noise content, to obtain a noisy video, may include: performing data transformation on the starting time step to obtain noise weights; obtaining target noise based on the noise weights and the base noise; and fusing the target noise with the original video to obtain the noisy video.

[0124] The starting time step can be normalized to complete the data transformation. For example, the number 100 representing the starting time step can be converted into a fraction between 0 and 1. However, this is not the only option. The starting time step, represented in words, can also be converted into numerical data to complete the data transformation. For example, the 100th time step can be converted into the numerical value 100.

[0125] Preferably, the textual representation of the start time step can be converted into a numerical representation. Then, the fractions that are not between 0 and 1 are normalized to obtain numerical values ​​represented by fractions between 0 and 1, thereby obtaining the noise weight.

[0126] Using the above method to transform the data of the starting time step can improve the quantification of noise weights. In addition, it can also avoid the problem that the noise content added to the noisy video is of different dimensions due to different units, which leads to poor noise reduction effect.

[0127] The base noise can refer to fixed noise. By using the base noise and noise weights, the noise to be added to the original video can be determined simply and flexibly.

[0128] Optionally, evaluation videos of different video types can be used to evaluate multiple predetermined starting time steps to determine the starting time step that matches the video type. This establishes a mapping relationship between video type and starting time step. Based on the video type of the original video and the mapping relationship, the starting time step that matches the original video is determined. This allows for the rapid and flexible determination of the starting time step that matches the video type.

[0129] The process of fusing the target noise with the original video to obtain a noisy video can include: weighted summation of the pixel values ​​of the target noise and the original video to obtain the noisy video.

[0130] Alternatively, the noisy video can be obtained using the following formula (1).

[0131] ;Formula (1)

[0132] in, This represents noisy video frames within a noisy video. This represents the original video frames in the original video. Indicates the fundamental noise. Indicates noise weight, .

[0133] According to embodiments of this disclosure, data transformation is performed on the starting time step to obtain noise weights. This quantifies and unifies the dimensions of the starting time step, facilitating the quantification of the noise content to be added to the original video. Furthermore, using the base noise as a foundation, the target noise is obtained based on the noise weights and the base noise, improving the flexibility and convenience of noise addition.

[0134] The above section explained how to identify noisy video. The following section will explain the special processing of audio, facial expression, and other information within the constraint information.

[0135] Figure 6A The illustration shows a schematic diagram of a process for determining audio features according to an embodiment of the present disclosure.

[0136] like Figure 6A As shown, when the constraint features include audio features, a general feature extraction tool can be used to extract general features from the audio to obtain the first audio feature. A specialized feature extraction tool can then be used to extract specialized features from the audio to obtain the second audio feature. Finally, the first and second audio feature sequences are fused to obtain the final audio feature.

[0137] like Figure 6AAs shown, the fusion method may include using the concat fusion function to fuse the first audio feature and the second audio feature sequence, and then performing cross-attention processing to obtain speech features.

[0138] Both general-purpose and specialized feature extraction tools can refer to deep learning networks for audio feature extraction. Unlike general-purpose tools, specialized feature extraction tools are applicable to the same language type as the audio. Multiple pre-defined specialized feature extraction tools can be trained for different language types. From these pre-defined tools, a specialized audio feature extraction tool that matches the language type is selected.

[0139] According to embodiments of this disclosure, audio features are extracted by using a combination of dedicated feature extraction tools and general feature extraction tools. This enhances speech generalization capabilities while highlighting audio features related to language type, thereby improving the generalization and relevance of audio features and ultimately improving the consistency between the driven lip movements and the audio.

[0140] Figure 6B The illustration shows a schematic flowchart of determining facial expression features according to an embodiment of the present disclosure.

[0141] like Figure 6B As shown, when the constraint features include facial expression features, the image facial expression information of the reference image is determined. Feature extraction is performed on the image facial expression information to obtain image facial expression features. Emotional features are extracted from the audio. Based on the image facial expression features and emotional features, the facial expression features are determined.

[0142] Image recognition or classification can be performed on the reference image to determine its facial expression information. Emotional features can then be extracted from the audio to obtain emotional features. Feature fusion of the image facial expression features and emotional features is then performed, using a method similar to the fusion of the first and second audio features, which will not be elaborated upon here.

[0143] According to embodiments of this disclosure, image facial features are fused with emotional features in audio to achieve facial expression control, which can control the speaker's facial expressions more linearly and flexibly without sounding stiff or unnatural.

[0144] Optionally, a video generation model can be used to denoise the noisy video. The video generation model can include, but is not limited to, diffusion models, and can also include large models based on diffusion models. For example, using a large model for denoising can leverage the high processing power of the large model to increase the number of video frames in the noisy video clip. This improves the generation efficiency of the target video and increases its length.

[0145] The following section will provide a detailed explanation of how to train the video generation model so that it can be better applied to the video generation method provided in the above embodiments.

[0146] Figure 7 A flowchart illustrating a training method for a video generation model according to an embodiment of the present disclosure is shown.

[0147] like Figure 7 As shown, the method includes operations S710 to S740.

[0148] The S710 is used to acquire sample target videos and sample constraint information.

[0149] In operation S720, noise is added to the sample target video based on noise weights used to indicate the noise content, resulting in a sample noisy video. The noise weights are determined based on the sample start time step, which indicates the starting point of the denoising sub-process characterized by the sample time step sub-sequence.

[0150] When operating the S730, based on the sample constraint information that serves as a constraint on the video content and the sample time step sequence that serves as a constraint on the denoising process, the video generation model is used to denoise the sample noisy video to obtain the sample denoised video.

[0151] When operating the S740, the video generation model is trained based on the sample denoised video and the sample target video to obtain the trained video generation model.

[0152] It should be noted that the starting time step and constraint information involved in the above embodiments differ from the sample starting time step and sample constraint information provided in this embodiment in that they are named differently, but their meanings and processing methods are the same. Further details will not be provided thereafter.

[0153] In the training method of the video generation model provided in this embodiment, the information covered in the sample noise video can be adapted to the denoising stage corresponding to the sample time step sequence by controlling the original information content of the sample target video in the sample noise video, thereby improving the training efficiency of the video generation model and improving the training effect.

[0154] According to embodiments of this disclosure, for example, Figure 7The illustrated operation S720 adds noise to a sample target video based on a noise weight used to indicate the noise content, resulting in a sample noisy video. This includes: adding noise to a first sample target video based on a first noise weight, resulting in a first sample noisy video. The first noise weight is determined based on a first starting time step, which indicates the starting point of the denoising sub-process represented by the first time step sub-sequence of the sample. Adding noise to a second sample target video based on a second noise weight, resulting in a second sample noisy video, where the second noise weight is determined based on a second starting time step, which indicates the starting point of the denoising sub-process represented by the second time step sub-sequence of the sample. A sample noisy video is determined based on the first and second sample noisy videos. The temporal variation intensity of the video content in the first sample target video is less than that in the second sample target video, the first time step sub-sequence of the sample is earlier temporally than the second time step sub-sequence of the sample, and the noise content of the first sample noisy video is higher than that of the second sample noisy video.

[0155] Different types of training samples can be determined based on different sample time step sequences, thereby adapting to the training requirements of different denoising stages, thus improving training efficiency and training effect.

[0156] There are no restrictions on the types of target videos included in the sample. For example, it can include a first target video and a second target video, but it is not limited to two types and can include more target videos, as long as they can match the segmented sample time step sequence.

[0157] For example, the denoising stage can be divided into an early denoising stage and a mid-to-late denoising stage. The early denoising stage matches the sample time step sequence, such as time step 1-100, while the mid-to-late denoising stage matches the sample time step sequence, such as time step 101-500. The early denoising stage can assign the first sample target video. The mid-to-late denoising stage can assign the second sample target video.

[0158] Optionally, the first sample target video can be further assigned to strictly paired samples and non-strictly paired samples. Strictly paired samples and non-strictly paired samples can be distinguished by actions such as posture. In the first sample target video, the posture, identity of the person, and background of the sample object remain largely unchanged in time, with only lip movements changing; this can be considered a strictly paired sample. Non-strictly paired data can be sample objects whose posture changes in time, but the magnitude or intensity of the change is less than a predetermined threshold.

[0159] Utilizing the first sample target video ensures consistency in pose and identity, providing a strong supervisory signal for the video generation model's learning. Applied to early denoising training in diffusion denoising, it leverages the high noise content of this stage to primarily learn and construct the macroscopic structure of the image, such as facial pose and identity information. By ensuring the temporal stability of the pose, background, and identity information in the first sample target video, the video generation model learns consistent structural information, focusing more on mastering the overall structural information.

[0160] The second sample target video can be a video in which the intensity of the pose change of the sample object in time exceeds a predetermined threshold. For example, head movements, temporal movements, etc., have a temporal change amplitude greater than the predetermined threshold, or there are various lighting and backgrounds with a change intensity greater than the predetermined threshold.

[0161] By using the second sample target video for mid-to-late stage denoising training in diffusion denoising, the low noise content and reduced requirements for pose consistency in this denoising stage can be leveraged to generate precise lip movements to match the target audio and refine training requirements such as facial texture and details, allowing the video generation model to focus on detailed characterization training.

[0162] According to embodiments of this disclosure, for a video generation model with diffusion denoising, the diffusion denoising principle of the video generation model is utilized and combined with the task requirements of different diffusion denoising stages. Different types of sample target videos are set for training. By adaptively adjusting the added noise content and the sample time step sequence, the training stage is refined into multiple denoising stages. This allows the video generation model to deeply learn the instructions of each denoising stage, thereby improving the training effect.

[0163] Based on a first noise weight, noise is added to the first sample target video to obtain a first sample noisy video. This process may include: performing data transformation on the first sample's starting time step to obtain a first noise weight; obtaining a first target noise based on the first noise weight and the base noise; and fusing the first target noise with the sample target video to obtain the first sample noisy video.

[0164] Based on the second noise weight, noise is added to the second sample target video to obtain the second sample noisy video. This may include: performing data transformation on the starting time step of the second sample to obtain the second noise weight; obtaining the second target noise based on the second noise weight and the basic noise; and fusing the second target noise with the sample target video to obtain the second sample noisy video.

[0165] The above section explained the target video for the sample. The following section will explain a special example of sample constraint information.

[0166] According to embodiments of this disclosure, sample constraint information may include a sample reference image.

[0167] When the sample constraint information includes a sample reference image, for example... Figure 7 The operation S710 shown, which obtains sample constraint information, may include: obtaining an initial sample reference image; and masking the sample reference objects in the initial sample reference image based on the mask ratio corresponding to the sample time step subsequence to obtain the sample reference image.

[0168] During training, masking the sample reference objects, especially the lips, can guide the video generation model to better learn lip movements from the audio of the sample constraint information, thereby improving the generation effect of the spoken video and enhancing the matching between lip movements and audio.

[0169] Optionally, it can be adapted to the training process by dividing it into different training denoising stages, or different mask ratios can be configured for different sample time step sequences to improve the training effect of the video generation model from the perspective of adjusting constraint information.

[0170] According to embodiments of this disclosure, masking a sample reference object of an initial sample reference image based on a mask ratio corresponding to a sample time step sequence to obtain a sample reference image may include: masking a sample reference object of the initial sample reference image based on a first mask ratio corresponding to a first sample time step sequence to obtain a first sample reference image; masking a sample reference object of the initial sample reference image based on a second mask ratio corresponding to a second sample time step sequence to obtain a second sample reference image; and finally, obtaining a sample reference image based on the first and second sample reference images.

[0171] To align with early and mid-to-late stage denoising training, a first mask ratio and a second mask ratio can be used. The first mask ratio is larger than the second mask ratio. For example, in early denoising training, the mask ratio is set to 0.8, and in mid-to-late stage, it is set to 0.5 to 0.2. This ensures that the interaction between lip movements and audio is forcibly established during the early denoising training process. Once this relationship is established and stabilized, it is gradually relaxed in the mid-to-late stage to guide the video generation model to learn appearance features such as lip texture.

[0172] According to embodiments of this disclosure, the above method can be adapted to the training requirements of different denoising stages from the perspective of adjusting constraint information, thereby improving the training effect of the video generation model.

[0173] According to embodiments of this disclosure, for example, Figure 7The operation S740 shown, which trains a video generation model based on the sample denoised video and the sample target video to obtain a trained video generation model, may include: determining the video frame matching between the sample denoised video and the sample target video, and adjusting the parameters of the video generation model based on the matching results to obtain the trained video generation model. However, it is not limited to this. Stream matching can also be used to determine the denoising speed based on the noise content and the sample target video. The parameters of the video generation model can then be adjusted based on the denoising speed to obtain the trained video generation model.

[0174] According to embodiments of this disclosure, for example, Figure 7 The operation S730 shown, based on sample constraint information as video content constraints and sample time step sequences as denoising process constraints, uses a video generation model to denoise the sample noisy video to obtain a sample denoised video. This can include: extracting a sample time step sequence from the sample time step sequence indicating the entire denoising process based on the sample starting time step; and iteratively denoising the sample noisy video based on the sample constraint features of the sample constraint information and the sample time step sequences to obtain the sample denoised video.

[0175] According to embodiments of this disclosure, based on sample constraint features and sample time step sequences of sample constraint information, iterative denoising is performed on the noisy sample video to obtain a denoised sample video, including at least one denoising step as follows:

[0176] Based on sample constraint features and sample current time step features determined by the current time step, denoising processing is performed on multiple previous round sample video segment features to obtain multiple current round sample denoised video segment features. These multiple previous round sample video segment features are obtained by performing denoising at least once on multiple sample noisy video segments obtained by splitting the sample noisy video into frames. The current time step is the time step in the time step subsequence. Feature updates are then performed on each of the multiple current round sample denoised video segment features to obtain multiple current round sample video segment features.

[0177] According to embodiments of this disclosure, feature updates are performed on multiple current-round sample denoised video segment features to obtain multiple current-round sample video segment features. This includes: determining, from the multiple current-round sample denoised video segment features, a first sample sub-segment feature of an earlier sample noisy video segment and a second sample sub-segment feature of a later sample noisy video segment, wherein temporally consecutive earlier and later sample noisy video segments belong to two adjacent sample noisy video segments. Feature fusion is performed on the first sample sub-segment feature and the second sample sub-segment feature to obtain a fused sample sub-segment feature. Based on the fused sample sub-segment feature, the first sample sub-segment feature and the second sample sub-segment feature are updated respectively to obtain multiple current-round sample video segment features.

[0178] According to embodiments of this disclosure, sample constraint information includes sample audio.

[0179] According to embodiments of this disclosure, the training method for the video generation model may further include: performing feature extraction processing on sample audio using a general feature extraction tool and a dedicated feature extraction tool, respectively, to obtain sample first audio features and sample second audio features, wherein the language type applicable to the dedicated feature extraction tool is the same as the language type of the sample audio. The sample first audio features and sample second audio feature sequences are then fused to obtain sample audio features.

[0180] According to embodiments of this disclosure, sample constraint information includes sample facial expression information. Sample constraint features include sample facial expression features.

[0181] According to embodiments of this disclosure, the training method for the video generation model may further include: determining sample image facial expression information of a sample reference image; extracting features from the sample image facial expression information to obtain sample image facial expression features; extracting sample emotional features from sample audio; and determining sample facial expression features based on the sample image facial expression features and the sample emotional features.

[0182] Figure 8 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0183] like Figure 8 As shown, the video generation device 800 includes: an acquisition module 810, a noise-adding module 820, and a video generation module 830.

[0184] The acquisition module 810 is used to acquire constraint information and the original video.

[0185] The noise addition module 820 is used to add noise to the original video based on the noise weight indicating the noise content, so as to obtain a noisy video. The noise weight is determined based on the start time step, which indicates the starting point of the denoising process.

[0186] The video generation module 830 is used to denoise a noisy video based on constraint information that serves as a constraint on the video content and the starting time step that serves as a constraint on the denoising process, so as to obtain a target video, wherein the video content of the target video matches the constraint information.

[0187] According to embodiments of this disclosure, the video generation module includes: a cropping submodule and an iterative denoising submodule.

[0188] The extraction submodule is used to extract a time step subsequence from the time step sequence indicating the entire denoising process, based on the starting time step.

[0189] The iterative denoising submodule is used to iteratively denoise noisy videos based on constraint features and time step sequences of constraint information to obtain the target video.

[0190] According to embodiments of this disclosure, the iterative denoising submodule includes at least the following units: a denoising unit and an update unit.

[0191] The denoising unit is used to denoise multiple previous round video segment features based on constraint features and current time step features determined by the current time step, respectively, to obtain multiple current round denoised video segment features. The multiple previous round video segment features are obtained by splitting the noisy video into frames and performing denoising on multiple noisy video segments at least once. The current time step is the time step in the time step subsequence.

[0192] The update unit is used to update the features of multiple current round denoised video segments respectively, so as to obtain multiple current round video segment features.

[0193] According to embodiments of this disclosure, the updating unit includes: a fragment determination subunit, a fusion subunit, and an updating subunit.

[0194] The segment determination subunit is used to determine the first sub-segment feature of the preceding noisy video segment and the second sub-segment feature of the following noisy video segment from multiple current round denoised video segment features. The preceding and following noisy video segments that are sequentially consecutive belong to two adjacent noisy video segments.

[0195] The fusion subunit is used to fuse the features of the first sub-segment and the features of the second sub-segment to obtain the fused sub-segment features.

[0196] The update sub-unit is used to update the features of the first sub-segment and the features of the second sub-segment based on the fused sub-segment features, so as to obtain multiple current round video segment features.

[0197] According to embodiments of this disclosure, the noise addition module includes: a conversion submodule, a noise determination submodule, and a noise addition submodule.

[0198] The transformation submodule is used to transform the data at the starting time step to obtain noise weights.

[0199] The noise determination submodule is used to obtain the target noise based on the noise weights and the base noise.

[0200] The noise-adding submodule is used to fuse the target noise with the original video to obtain a noisy video.

[0201] According to embodiments of this disclosure, the video generation apparatus further includes an evaluation module and a time step determination module.

[0202] The evaluation module is used to evaluate multiple predetermined start time steps using the evaluation video and determine the evaluation results for each of the multiple predetermined start time steps. The evaluation results represent at least one of the following evaluation metrics: denoising effect metric and denoising efficiency metric.

[0203] The time step determination module is used to determine the starting time step from multiple predetermined starting time steps based on the evaluation results of each of the multiple predetermined starting time steps.

[0204] According to embodiments of this disclosure, the evaluation module includes: an evaluation noise-addition submodule, an evaluation noise reduction module, and an evaluation result determination submodule.

[0205] The evaluation noise-adding submodule is used to add noise to the evaluation video using initial noise weights to obtain a noise evaluation video, wherein the initial noise weights are determined based on a predetermined start time step.

[0206] The evaluation denoising module is used to denoise the noisy evaluation video based on constraint information that serves as a constraint on the video content and a predetermined start time step that serves as a constraint on the denoising process, so as to obtain the target evaluation video.

[0207] The evaluation result determination submodule is used to obtain the evaluation result at a predetermined starting time step based on the target evaluation video and the evaluation video.

[0208] According to embodiments of this disclosure, the constraint information includes at least one of the following: reference image, facial expression information, descriptive information, and audio.

[0209] Matching the video content of the target video with the constraint information includes at least one of the following: the facial image of the target object in the target video matches the facial image of the reference object in the reference image; the expression of the target object matches the expression information; the video theme of the target video matches the descriptive information; and the lip movements of the target object match the audio.

[0210] According to embodiments of this disclosure, the constraint features include audio features.

[0211] According to embodiments of this disclosure, the video generation apparatus further includes: an audio feature extraction module and an audio feature determination module.

[0212] The audio feature extraction module is used to extract features from the audio using a general feature extraction tool and a special feature extraction tool, respectively, to obtain the first audio feature and the second audio feature. The special feature extraction tool is applicable to the same language type as the audio.

[0213] The audio feature determination module is used to fuse the first audio feature and the second audio feature sequence to obtain audio features.

[0214] According to embodiments of this disclosure, the constraint features include facial expression features.

[0215] According to embodiments of this disclosure, the video generation apparatus further includes: an image expression determination module, an expression feature extraction module, an emotion feature extraction module, and an expression feature determination module.

[0216] The image expression determination module is used to determine the image expression information of the reference image.

[0217] The facial expression feature extraction module is used to extract features from image facial expression information to obtain image facial expression features.

[0218] The emotion feature extraction module is used to extract emotion features from audio.

[0219] The facial expression feature determination module is used to determine facial expression features based on image facial expression features and emotion features.

[0220] Figure 9 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0221] like Figure 9 As shown, the training device 900 for the video generation model includes: a sample acquisition module 910, a sample noise addition module 920, a video generation module 930, and a training module 940.

[0222] The sample acquisition module 910 is used to acquire the target video and sample constraint information.

[0223] The sample noise-adding module 920 is used to add noise to the sample target video based on noise weights that indicate the noise content, to obtain a sample noise video. The noise weights are determined based on the sample start time step, which indicates the starting point of the denoising sub-process characterized by the sample time step sub-sequence.

[0224] The video generation module 930 is used to denoise the sample noisy video using a video generation model based on the sample constraint information that serves as a constraint on the video content and the sample time step sequence that serves as a constraint on the denoising process, so as to obtain the sample denoised video.

[0225] Training module 940 is used to train the video generation model based on the sample denoised video and the sample target video to obtain the trained video generation model.

[0226] According to embodiments of this disclosure, the sample noise-adding module includes: a first sample noise-adding submodule, a second sample noise-adding submodule, and a sample video determination submodule.

[0227] The first sample noise-adding submodule is used to add noise to the first sample target video based on the first noise weight to obtain the first sample noisy video. The first noise weight is determined based on the first starting time step, and the first starting time step indicates the starting point of the denoising sub-process characterized by the first time step sub-sequence of the sample.

[0228] The second sample noise-adding submodule is used to add noise to the second sample target video based on the second noise weight to obtain the second sample noisy video. The second noise weight is determined based on the second starting time step, which indicates the starting point of the denoising sub-process characterized by the second time step sub-sequence of the sample.

[0229] The sample video determination submodule is used to determine the sample noise video based on the first sample noise video and the second sample noise video.

[0230] Among them, the temporal variation intensity of the video content in the first sample target video is less than that in the second sample target video, the first time step sequence of the sample is earlier than the second time step sequence of the sample, and the noise content of the first sample noisy video is higher than that of the second sample noisy video.

[0231] According to embodiments of this disclosure, the sample constraint information includes a sample reference image;

[0232] According to embodiments of this disclosure, the sample acquisition module includes: an initial sample acquisition submodule and a mask submodule.

[0233] The initial sample acquisition submodule is used to acquire the initial sample reference image.

[0234] The masking submodule is used to mask the sample reference object of the initial sample reference image based on the masking ratio corresponding to the sample time step sequence, so as to obtain the sample reference image.

[0235] According to embodiments of this disclosure, the mask submodule includes: a first mask unit, a second mask unit, and a sample image acquisition unit.

[0236] The first masking unit is used to mask the sample reference object of the initial sample reference image based on the first masking ratio corresponding to the first sample time step sequence, so as to obtain the first sample reference image.

[0237] The second masking unit is used to mask the sample reference object of the initial sample reference image based on the second masking ratio corresponding to the second sample time step sequence, so as to obtain the second sample reference image.

[0238] The sample image acquisition unit is used to obtain a sample reference image based on the first sample reference image and the second sample reference image.

[0239] The first mask ratio is greater than the second mask ratio.

[0240] Figure 10 A schematic block diagram of a smart agent according to an embodiment of the present disclosure is shown.

[0241] In embodiments of this disclosure, such as Figure 10 As shown, the AI ​​agent 1000 may include an input module 1010, a processing module 1020, and an output module 1030.

[0242] Input module 1010 is used to receive input information.

[0243] The processing module 1020 is used to obtain output information based on the input information received by the input module by calling a large model to execute the video generation method provided in the embodiments of this disclosure, or by calling a large model to execute the training method of the video generation model provided in the embodiments of this disclosure.

[0244] Output module 1030 is used to output the output information obtained by the processing module.

[0245] According to embodiments of this disclosure, the input module 1010 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 1000 can understand and process. The input module 1010 is the primary link for the AI ​​agent 1000 to interact with the outside world, enabling the AI ​​agent 1000 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0246] In the example, the input module 1010 can input the constraint information described above, the original video, the sample constraint information, and the sample target video, etc.

[0247] In the example, processing module 1020 is the core support for the AI ​​agent 1000's ability to handle complex tasks. Processing module 1020 can execute the video generation method and the video generation model training method described above.

[0248] In the example, the performance of the processing module 1020 is closely related to the large model on which the AI ​​agent 1000 is based. To fully leverage the capabilities of the large model, the internal structure of the processing module 1020 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0249] In the example, after the AI ​​agent 1000 acquires the original video and constraint information, the processing module 1020 can use the video generation model to process the original video and constraint information to obtain the target video, and then pass the target video to the output module 1030.

[0250] Understandably, while large models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. Once the AI ​​agent 1000 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to create weather forecasts.

[0251] In the example, output module 1030 can output the target video or the trained video generation model described above.

[0252] The AI ​​agent 1000 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.

[0253] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0254] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0255] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0256] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0257] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0258] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded into random access memory (RAM) 1103 from storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0259] Multiple components in device 1100 are connected to input / output (I / O) interface 1105, including: input unit 1106, such as a keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as a disk, optical disk, etc.; and communication unit 1109, such as a network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0260] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as video generation methods or video generation model training methods. For example, in some embodiments, the video generation method or video generation model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the video generation method or video generation model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform a video generation method or a training method for a video generation model.

[0261] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0262] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0263] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0264] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0265] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0266] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0267] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0268] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: Obtain constraint information and the original video; Based on noise weights indicating the amount of noise to be added, noise is added to the original video to obtain a noisy video. The noise weights are determined based on a starting time step, which indicates the starting point of the denoising process. Based on the constraint information that serves as a constraint on video content and the starting time step that serves as a constraint on the denoising process, the noisy video is denoised to obtain a target video, wherein the video content of the target video matches the constraint information.

2. The method according to claim 1, wherein, The step of denoising the noisy video based on the constraint information (as a constraint on video content) and the starting time step (as a constraint on the denoising process) to obtain the target video includes: Based on the aforementioned starting time step, a time step sub-sequence is extracted from the time step sequence indicating the entire denoising process; and Based on the constraint features of the constraint information and the time step sequence, the noisy video is iteratively denoised to obtain the target video.

3. The method according to claim 2, wherein, The method of iteratively denoising the noisy video based on the constraint features of the constraint information and the time step sequence to obtain the target video includes at least one denoising step as follows: Based on the constraint features and the current time step features determined based on the current time step, denoising processing is performed on multiple previous round video segment features to obtain multiple current round denoised video segment features. These multiple previous round video segment features are obtained by performing at least one denoising operation on multiple noisy video segments obtained by splitting the noisy video into frames. The current time step is the time step in the time step subsequence. The features of the current round of denoised video segments are updated respectively to obtain the features of the current round of video segments.

4. The method according to claim 3, wherein, The step of updating the features of multiple current round denoised video segments to obtain multiple current round video segment features includes: From the features of multiple current round denoised video segments, determine the first sub-segment feature of the preceding noise video segment and the second sub-segment feature of the following noise video segment in the noise video, wherein the preceding noise video segment and the following noise video segment that are sequentially consecutive in time belong to two adjacent noise video segments. The first sub-segment features and the second sub-segment features are fused to obtain fused sub-segment features; and Based on the fused sub-segment features, the first sub-segment features and the second sub-segment features are updated respectively to obtain multiple current round video segment features.

5. The method according to any one of claims 1 to 4, wherein, The process of adding noise to the original video based on noise weights used to indicate the noise content, resulting in a noisy video, includes: The noise weights are obtained by performing data transformation on the starting time step; Based on the noise weights and the baseline noise, the target noise is obtained; and The target noise is fused with the original video to obtain the noisy video.

6. The method according to any one of claims 1 to 5, further comprising: The evaluation video is used to evaluate multiple predetermined start time steps, and the evaluation results for each of the predetermined start time steps are determined. The evaluation results represent at least one of the following evaluation metrics: denoising effect metric, denoising efficiency metric; and The starting time step is determined from the plurality of predetermined starting time steps based on the evaluation results of each of the plurality of predetermined starting time steps.

7. The method according to claim 6, wherein, The process of evaluating multiple predetermined start time steps using evaluation videos and determining the evaluation results for each of the predetermined start time steps includes: Noise is added to the evaluation video using an initial noise weight to obtain a noise evaluation video, wherein the initial noise weight is determined based on the predetermined start time step; Based on the constraint information serving as video content constraints and the predetermined start time step serving as denoising process constraints, the noise evaluation video is denoised to obtain the target evaluation video; and Based on the target evaluation video and the evaluation video, the evaluation result at the predetermined starting time step is obtained.

8. The method according to any one of claims 1 to 7, wherein, The constraint information includes at least one of the following: reference image, facial expression information, descriptive information, and audio; The matching of the video content of the target video with the constraint information includes at least one of the following: the facial image of the target object in the target video matches the facial image of the reference object in the reference image; the expression of the target object matches the expression information; the video theme of the target video matches the description information; and the lip movements of the target object match the audio.

9. The method according to claim 2, wherein, The constraint features include audio features; The method further includes: Audio features are extracted using both a general-purpose feature extraction tool and a specialized feature extraction tool to obtain first and second audio features, wherein the specialized feature extraction tool is applicable to the same language type as the audio. The first audio feature and the second audio feature sequence are fused to obtain the audio feature.

10. The method according to claim 2 or 9, wherein, The constraint features include facial expression features; The method further includes: Determine the image expression information of the reference image; Feature extraction is performed on the image expression information to obtain image expression features; Extracting emotional features from audio; and The facial expression features are determined based on the image expression features and the emotion features.

11. A method for training a video generation model, comprising: Obtain the target video and sample constraint information; Noise is added to the sample target video based on noise weights used to indicate the noise content, resulting in a sample noisy video. The noise weights are determined based on the sample start time step, which indicates the starting point of the denoising sub-process characterized by the sample time step sub-sequence. Based on the sample constraint information that serves as a constraint on video content and the sample time step sequence that serves as a constraint on the denoising process, the sample noisy video is denoised using a video generation model to obtain a sample denoised video. as well as Based on the denoised sample video and the target sample video, the video generation model is trained to obtain the trained video generation model.

12. The method according to claim 11, wherein, The process of adding noise to the sample target video based on noise weights used to indicate the noise content, to obtain a sample noisy video, includes: Based on the first noise weight, noise is added to the first sample target video to obtain the first sample noisy video, wherein the first noise weight is determined based on the first starting time step, and the first starting time step indicates the starting point of the denoising sub-process characterized by the first time step sub-sequence of the sample. Based on a second noise weight, noise is added to the second sample target video to obtain a second sample noisy video. The second noise weight is determined based on a second starting time step, which indicates the starting point of the denoising sub-process characterized by the second time step sub-sequence of the sample. The sample noise video is determined based on the first sample noise video and the second sample noise video; Wherein, the temporal variation intensity of the video content in the first sample target video is less than that in the second sample target video, the first time step sequence of the sample is earlier than the second time step sequence of the sample, and the noise content of the first sample noisy video is higher than that of the second sample noisy video.

13. The method according to claim 11 or 12, wherein, The sample constraint information includes a sample reference image; Obtain sample constraint information, including: Obtain the initial sample reference image; and Based on the mask ratio corresponding to the sample time step sequence, the sample reference object of the initial sample reference image is masked to obtain the sample reference image.

14. The method according to claim 13, wherein, The step of masking the sample reference object of the initial sample reference image based on the mask ratio corresponding to the sample time step subsequence to obtain the sample reference image includes: Based on the first mask ratio corresponding to the first sample time step subsequence, the sample reference object of the initial sample reference image is masked to obtain the first sample reference image; Based on the second mask ratio corresponding to the second sample time step sequence, the sample reference object of the initial sample reference image is masked to obtain the second sample reference image; and The sample reference image is obtained based on the first sample reference image and the second sample reference image; The first mask ratio is greater than the second mask ratio.

15. A video generation apparatus, comprising: The acquisition module is used to acquire constraint information and the original video. A noise-adding module is used to add noise to the original video based on noise weights indicating the noise content, resulting in a noisy video. The noise weights are determined based on a starting time step, which indicates the starting point of the denoising process. A video generation module is used to denoise the noisy video based on the constraint information that serves as a constraint on the video content and the starting time step that serves as a constraint on the denoising process, to obtain a target video, wherein the video content of the target video matches the constraint information.

16. A training device for a video generation model, comprising: The sample acquisition module is used to acquire the target video and sample constraint information. A sample noise-adding module is used to add noise to the sample target video based on a noise weight used to indicate the noise content, thereby obtaining a sample noise video, wherein the noise weight is determined based on the sample start time step, and the sample start time step indicates the starting point of the denoising sub-process characterized by the sample time step sub-sequence. The video generation module is used to denoise the sample noisy video using a video generation model based on the sample constraint information (which serves as video content constraints) and the sample time step sequence (which serves as denoising process constraints), to obtain a sample denoised video; and The training module is used to train the video generation model based on the sample denoised video and the sample target video to obtain the trained video generation model.

17. An intelligent agent, comprising: The input module is used to receive input information; The processing module is configured to obtain output information by calling a large model to execute the method of any one of claims 1 to 14 based on the input information received by the input module; An output module is used to output the output information obtained by the processing module.

18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.

Images