Video generation method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a visual representation, a camera motion representation and a human motion representation based on an original video sequence; fusing the camera motion representation and the human body motion representation by using a decoupling fusion strategy to obtain a fused representation; according to the fusion representation, updating the visual representation; and generating a target video sequence based on the updated visual representation. According to the method, the camera motion signal and the human body motion signal can be effectively integrated, the temporal and spatial variation characteristics after the camera motion signal and the human body motion signal are combined are effectively processed, fine-grained control over camera motion, human body motion and joint interaction of the camera motion and the human body motion is achieved, and the quality of a video generated by artificial intelligence is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] AIGC (Artificial Intelligence Generated Content) video generation technology refers to the generation of video content through artificial intelligence algorithms, such as the generation of still images, the generation of motion of objects in videos, and the synchronous generation of audio and visual content.

[0003] In recent years, AIGC research on motion control in video generation technology has introduced different types of motion signals to achieve more precise controllability in video generation, especially for object motion. Research on motion control aims to enrich the possible scenarios for motion control. However, most current research focuses only on controlling one type of motion (e.g., camera motion or human motion), which limits the scope of controllability in video generation.

[0004] For example, human-centered motion control remains a key challenge in video generation, especially in scenes like the iconic Grammy "Glambot" moment, where simultaneous control of camera movement and human pose is required. Current methods struggle to effectively coordinate object and camera motion. Summary of the Invention

[0005] This application provides a video generation method, apparatus, electronic device, storage medium, and program product to solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a video generation method. The method includes: acquiring visual representation, camera motion representation, and human motion representation based on an original video sequence; fusing the camera motion representation and the human motion representation using a decoupling fusion strategy to obtain a fused representation; updating the visual representation according to the fused representation; and generating a target video sequence based on the updated visual representation.

[0007] Secondly, embodiments of this application provide a video generation apparatus. The apparatus includes: a feature extraction module for acquiring visual representations, camera motion representations, and human motion representations based on an original video sequence; a decoupling fusion module for fusing the camera motion representations and the human motion representations using a decoupling fusion strategy to obtain a fused representation; a visual update module for updating the visual representations according to the fused representations; and a video generation module for generating a target video sequence based on the updated visual representations.

[0008] Thirdly, embodiments of this application provide an electronic device. The electronic device includes a memory, one or more processors, and one or more application programs. The one or more application programs are stored in the memory and configured to execute the methods provided in embodiments of this application when invoked by one or more processors.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium stores program code configured to execute the method provided in embodiments of this application when invoked by a processor.

[0010] Fifthly, embodiments of this application provide a computer program product, which, when invoked by a processor, causes the processor to execute the method provided in embodiments of this application.

[0011] The video generation method of this application has the following technical effects: it uses camera motion representation (token) and human motion representation to process camera motion and human motion, and uses a decoupling fusion strategy to fuse the camera motion representation and the human motion representation. This can effectively integrate camera motion signals and human motion signals, effectively process the spatiotemporal variation characteristics of the combination of camera motion signals and human motion signals, realize fine-grained control of camera motion, human motion, and joint interaction of camera motion and human motion, and improve the quality of AI-generated videos. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a model framework diagram of the TokenMotion model provided in one embodiment of this application; Figure 2 This is a flowchart illustrating a video generation method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating step S120 provided in an embodiment of this application; Figure 4 This is a structural block diagram of a video generation apparatus provided in one embodiment of this application; Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] Video diffusion models have made significant progress in text-to-video and image-to-video generation. Recent research has further enhanced the controllability of generation by incorporating external control signals such as depth maps, subject appearance, and motion. Among these, human-centered motion control has become a key challenge in the field of video generation. Taking the iconic Grammy "Glambot" moment as an example, such scenes require simultaneous joint control of dramatic camera movements and changes in human posture. The importance of this capability stems from the explosive growth of AI-generated content and its enormous potential in creative production fields (from movies to popular short videos). However, this crucial aspect remains largely unexplored, and current research only offers limited solutions.

[0016] First, most current research on motion control focuses only on one aspect of object motion control or camera control, lacking the ability to achieve joint control. Second, although a few studies have attempted to explore joint control of camera and object motion, they still have limitations in human-centered motion control due to overly simplified motion representations and inadequate motion integration methods. For example, MotionCtrl (a video generation technique) uses object-level keypoints, while Direct-A-Video (a text-to-video generation framework) and MotionBooth (a text-to-video generation framework that allows customization and control of object and camera motion) use bounding boxes to represent object trajectories. These coarse representations fail to capture local motion, especially subtle pose changes. Furthermore, these studies directly integrate two motion signals without addressing their potential interactions, and motion conflicts exist in complex joint control scenarios, indicating that human-centered motion control still faces unresolved challenges.

[0017] A key challenge lies in effectively integrating camera control and human motion control. Intuitively, for any pair of video frames, the motion of each pixel is generated by different combinations of camera and human motion. The spatial and temporal variability of this motion synthesis necessitates a unified modeling of both motion signals within the same representation space, along with a specialized design to capture their interaction. However, current methods lack such a specialized design. Furthermore, current joint control methods employ UNet (a convolutional neural network) as the backbone, which incorporates object motion and camera motion into spatial and temporal modeling respectively, resulting in suboptimal joint encoding performance. While the Diffusion Transformer (DiT) model appears to be a promising alternative, a DiT-based joint control model has not yet been proposed. Additionally, recent research on camera control has revealed that previous motion coding techniques require customized adjustments to function effectively with DiT-based models, a challenge that may be further exacerbated in joint motion control.

[0018] This application proposes a video generation method to address the aforementioned challenges. The video generation method of this application designs a DiT-based video diffusion model (referred to as TokenMotion in this paper, the model framework of TokenMotion is as follows...). Figure 1 As shown, this method enables fine-grained control of three basic human-centered motion control scenarios (① camera control only; ② human motion control only; ③ combined control of both), addressing the key challenge of integrating camera and human motion in AI-generated video content. TokenMotion processes camera and human motion through a decoupling and fusion strategy, effectively integrating the two motion signals using a dynamic mask that senses the human body. This effectively handles the spatiotemporal variations inherent in the combination of camera and human motion signals. Furthermore, to address the issue of decreased control precision in DiT-based frameworks, this application introduces motion segmentation technology, encoding motion into fixed-length sequences. The video generation method of this application demonstrates superior generalization capabilities in both text-to-video and image-to-video paradigms for human-centered motion control. The video generation method of this application will be described in more detail below.

[0019] The video generation method in this application can be applied to a video generation device or electronic device, and the video generation device can be deployed in an electronic device. An electronic device refers to a device with Bluetooth connectivity, communication capabilities, computing power, and a display screen. Electronic devices can include, but are not limited to, smartphones, tablets, laptops, desktop computers, smart home appliances, wearable devices, in-vehicle computers, and virtual reality devices. Electronic devices can communicate with other electronic devices or servers via wired or wireless networks. A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server can be used to provide services for applications running on electronic devices.

[0020] See Figure 2 , Figure 2 This is a schematic flowchart of a video generation method provided in an embodiment of this application. The video generation method may include steps S110 to S140.

[0021] Step S110: Based on the original video sequence, obtain visual representation, camera motion representation, and human motion representation.

[0022] It should be noted that in this application, the term "representation" in visual representation, camera motion representation, and human motion representation refers to "token." A visual representation (i.e., a visual token) is the basic unit used for encoding and decoding video information. A camera motion representation (i.e., a camera motion token) is the basic unit used for encoding and decoding camera motion signals in video. A human motion representation (i.e., a human motion token) is the basic unit used for encoding and decoding human motion signals in video.

[0023] This application can convert raw video sequences into visual representations using a three-dimensional causal variational autoencoder (CausalVAE). CausalVAE is a technique that introduces causal relationships into the decoupled representation domain, achieving interpretable latent variables (i.e.,...) through a neural structural causal model. The technology. Specifically, the three-dimensional causal variational autoencoder will convert the shape into... The original video sequence was converted to a length of visual representation sequence ,in, , and These represent the height, width, and length of the video, respectively. and These are the spatial compression ratio and the temporal compression ratio, respectively.

[0024] In some embodiments, such as Figure 1 As shown, after the 3D causal variational autoencoder transforms the original video sequence into visual representations (one-dimensional vectors), these visual representations, along with positional encoding, text cue encoding, and timestamps, are input into N stacked 3D self-attention layers (for denoising), thereby learning clearer visual representations from the original visual representations. Furthermore, the visual representations from the 3D self-attention layers are further processed by feedforward layers to obtain more complex visual representations, which helps the model better understand the visual features in the video.

[0025] This application can extract camera motion trajectories from original video sequences; parameterize the camera motion trajectories using the Plucker Embedding method to obtain camera trajectory parameters; and compress the camera trajectory parameters spatially and temporally using a block module to obtain a camera motion representation.

[0026] Plück embedding is a mathematical method for representing lines in three-dimensional space. It is implemented using Plück coordinates (a homogeneous coordinate system), which maps lines in three-dimensional space to six-dimensional space. Each line can be defined by the two points or two planes it contains, thus representing it as a point with six degrees of freedom in Plück coordinates.

[0027] In this application, Plück embedding is used to represent camera motion. For the original video sequence, its camera motion consists of a series of camera parameters, each containing an intrinsic parameter matrix. and an extrinsic matrix ,in and These represent the rotation and translation components, respectively. To effectively anchor the raw values ​​of the camera motion within pixel coordinates, this application treats the camera motion signal as camera rays emanating from the camera center, given certain... and In this case, Plück coordinates are used. Mathematically, for any pixel in the f-th frame... The ray that corresponds to it in Plück coordinates It can be represented as: This application parameterizes the camera motion trajectory into camera trajectory parameters. The camera trajectory parameters maintain dimensional consistency with the spatiotemporal structure of the video.

[0028] The block modules of this application (such as...) Figure 1The camera encoder and human pose encoder in the model are used to perform latent spatial transformation on the model input. The block module includes: 2D convolutional layers and 3D causal convolutional layers. The 2D convolutional layers are used for spatial compression of the data, and the 3D causal convolutional layers are used for temporal compression of the data. The compression ratio of the block module is the same as that of the 3D causal variational autoencoder, as shown above. The 3D causal variational autoencoder is used to convert the original video sequence into a visual representation.

[0029] Specifically, the spatial dimension of the camera trajectory parameters can be compressed to [a specific value] using a two-dimensional convolutional layer in the block module. Then, a 3D causal convolutional layer is used to compress the temporal dimension of the camera trajectory parameters to [a specific value]. Finally, the camera motion characterization is obtained. Space compression ratio of modular units and time compression ratio Spatial compression ratio with 3D causal variational autoencoder and time compression ratio Maintain consistency.

[0030] To ensure the generated video captures subtle human dynamics, the representation space must possess sufficient expressive power to encode these nuanced kinematic changes. Therefore, this application employs a full-body pose estimation model to obtain human pose information from the original video sequence. The human pose information calculated by the full-body pose estimation model effectively captures subtle changes in facial expressions and nuanced differences in human posture, exhibiting robustness in both single-person and multi-person scenarios. Specifically, the full-body pose estimation model estimates the full-body pose of the human body in the original video sequence, including the body, feet, face, and hands. The full-body pose estimation model can be an efficient whole-body pose estimation model (Effective Whole-body Pose Estimation with Two-stage Distillation, or DWPose for short).

[0031] After obtaining human posture information, such as Figure 1 As shown, human posture information can be compressed in space and time through the above-mentioned block modules to obtain human motion representation.

[0032] In this application, the use of the aforementioned segmented modules to process camera trajectory parameters and human pose information enables the rapid and effective extraction of camera trajectory cues and human pose cues. The generated human motion representation, camera motion representation, and visual representation maintain dimensional consistency, thereby enabling seamless integration in subsequent processing stages. This is beneficial for maintaining temporal synchronization and spatial correspondence between camera motion and human motion in the subsequently generated video.

[0033] Step S120: Use a decoupling fusion strategy to fuse the camera motion representation and the human motion representation to obtain a fused representation.

[0034] After encoding the camera motion representation and the human motion representation separately, the focus can be on teaching the model the interaction between these two types of motion. Intuitively, the model should first project the learned camera motion representation globally into the entire latent space, and then project the human motion representation only into the human-related regions, while ensuring that these localized human poses are consistent with the established camera viewpoint.

[0035] like Figure 1 As shown, before step S120, camera motion representation and human motion representation can be processed separately using parallel self-attention layers. By processing the flattened camera motion representation and human motion representation using parallel self-attention layers, global dependencies can be learned to maintain motion consistency across frames.

[0036] Compared to the global characteristics of camera motion, human motion has local characteristics, requiring an implicit decoupling method to determine their distinct domains. This application achieves this decoupling through a masking strategy, aiming to isolate the human motion representation only to human-related regions. Furthermore, by employing a dynamic mask to relax the learning of isolating human motion representations to human-related regions, this dynamic mask combines learnable components with hard human pose priors.

[0037] See Figure 3 , Figure 3 This is a flowchart illustrating step S120 according to an embodiment of this application. Step S120 may include steps S121 to S123.

[0038] Step S121: Process the camera motion representation through dimensionality reduction and normalization layers to obtain the attention mask for camera motion.

[0039] Taking the nth layer as an example, given the camera motion representation on the nth attention block... This application first applies a dimensionality reduction layer, followed by a normalization layer, to obtain the learnable part of the motion mask, namely the attention mask for camera motion. : Step S122: Process the human motion representation through a dimensionality reduction layer and a normalization layer to obtain the attention mask for human motion.

[0040] Taking the nth layer as an example, given the human motion representation on the nth attention block This application first applies a dimensionality reduction layer, followed by a normalization layer, to obtain the learnable part of the motion mask, namely the attention mask for human motion. : Step S123: Based on the attention mask of camera motion and the attention mask of human motion, fuse the camera motion representation and the human motion representation to obtain the fused representation.

[0041] In some embodiments, the attention masks for camera motion and human motion can be compared using SoftMax in the manner shown in the following expression, and the camera motion representation and human motion representation can be fused accordingly based on the comparison results: In other embodiments, such as Figure 1 As shown, prior values ​​of human posture can be obtained first based on human motion representation. Then, the prior values ​​of human posture are... Attention mask added to human motion In this process, a new attention mask for human motion is obtained. Finally, the attention masks for camera motion and new human motion can be compared using SoftMax, as shown in the following expression, and the camera motion representation and human motion representation can be fused accordingly based on the comparison results: Among them, prior values ​​of human posture are obtained based on human motion representation. This can include: adjusting the size of the human motion representation to match the latent dimension of the nth attention block; then dilating the human motion representation to outline the region related to the human body; and finally converting the human motion representation into a binary mask to obtain the prior human pose value. .

[0042] Step S130: Update the visual representation based on the fused representation.

[0043] Taking the nth block as an example, this application uses the fusion representation of the nth attention block through a cross-attention mechanism. Visual representation on the nth attention block Updates are performed. To effectively enable the model to learn the interaction between camera motion and human motion, this application restricts cross-attention to operate only on the fused representation, rather than on either the camera motion representation or the human motion representation alone. This forces the model to separate motion from the fused representation, aligning it with both motions (i.e., camera motion and human motion) to minimize the distance between the target video and the reference video during training. Specifically, through three learnable linear layers, the visual representation on the nth attention block... Projected as a query The fusion representation of the nth attention block It is then converted into the corresponding key. Sum For query ,key Sum By learning the cross-attention mechanism, the visual representation on the nth attention block can be updated. For example, updating the visual representation based on the fused representation can be represented by the following expression.

[0044] In some embodiments, to maintain an optimal balance between motion controllability and video generation fidelity, a low-rank adaptation layer (LoRA layer) can be incorporated before updating the visual representation using the learned motion information. Specifically, the visual representation on the nth attention block is updated through three learnable linear layers. Projected as a query The fusion representation of the nth attention block It is then converted into the corresponding key. Sum For query ,key Sum By learning the cross-attention mechanism and then processing the results from the cross-attention learning through a low-rank adaptation layer, the visual representation on the nth attention block can be updated. For example, after incorporating the low-rank adaptation layer, updating the visual representation based on the fused representation can be expressed as the following expression.

[0045] Step S140: Generate the target video sequence based on the updated visual representation.

[0046] A video diffusion model can be used to generate target video sequences based on updated visual representations.

[0047] This application is as follows Figure 1The TokenMotion architecture shown enables fine-grained control while preserving the basic quality characteristics of the generated target video sequence.

[0048] Steps S110 to S140 have the following technical effects: camera motion representation (token) and human motion representation are used to process camera motion and human motion, and a decoupling fusion strategy is used to fuse the camera motion representation and the human motion representation. This can effectively integrate camera motion signals and human motion signals, effectively process the spatiotemporal variation characteristics of the combination of camera motion signals and human motion signals, realize fine-grained control of camera motion, human motion, and joint interaction of camera motion and human motion, and improve the quality of AI-generated videos.

[0049] It is understood that this application Figure 1 The TokenMotion technology shown can be applied to a variety of scenarios to make video creation and editing on smartphones more intuitive and powerful: 1. Better selfie videos: TokenMotion technology intelligently handles camera shake and body movements, helping to stabilize and enhance videos that require movement while shooting, such as vlogs or dance videos filmed while walking.

[0050] 2. Smoother motion shots: When filming others in action (such as sports, dancing, etc.), TokenMotion technology can help balance camera movement and subject tracking, potentially transforming shaky amateur footage into more professional-looking videos.

[0051] 3. Creative Controls: Users may be able to adjust or optimize camera movement and subject motion after recording, without needing professional editing software to correct awkward movements or camera shake.

[0052] 4. Resource efficiency: By representing actions as tokens, TokenMotion technology is more efficient than traditional video processing technologies, which is especially important for mobile devices with limited processing power.

[0053] To evaluate the impact of the proposed TokenMotion, experiments were conducted on the base model CogvideoX, and the results are shown in Table 1. The base model outperformed the proposed method on all three metrics, with a particularly significant difference in performance on the Fréchet Video Distance (FVD) metric. This indicates that the introduced motion signal significantly contributes to improving the spatiotemporal consistency of the generated video. The FVD metric is used to evaluate the quality of generated video; it measures the realism and coherence of the video by comparing the distribution differences between the generated and real videos in the deep feature space. FVD extracts video features based on a pre-trained 3D convolutional neural network (such as I3D) and calculates the Frechet distance between the feature distributions of the generated and real videos. A smaller Frechet distance indicates higher quality generated video.

[0054] To demonstrate that the motion signal is effectively encoded, this application replaces the block module in TokenMotion with the ControlNet module, which generates a larger-dimensional motion representation with the same dimension as the hidden state. Without the proposed block module, the model's FVD score is lower, indicating that this model variant exhibits poorer motion controllability during generation.

[0055] To demonstrate the effectiveness of the decoupled fusion strategy proposed in this application, experiments were conducted where the joint modeling of camera motion and human motion was set to direct addition. As shown in Table 1, without the decoupled fusion strategy of this application, the model struggles to maintain both frame quality and spatiotemporal consistency when generating video, exhibiting the worst performance across all three metrics. This indicates that a specially designed module is needed to enable the model to handle the interaction between these two motion signals.

[0056] To demonstrate the effectiveness of introducing explicit pose guidance into the mask, this application conducted experiments on a model variant that uses only learnable masks to decouple human and camera motion. As shown in Table 1, although the FVD metric decreases slightly when using a hybrid mask, the performance of the Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID, where Inception is a deep learning model) metrics is significantly better with the design of this application. LPIPS is a metric used to measure the perceptual similarity between images, evaluating perceptual differences based on a deep learning model; a smaller LPIPS value indicates a smaller perceptual difference between the two images. FID is a metric used to evaluate the quality of images generated by a generative model; it quantifies the distance between the generated and real images by comparing the distribution differences in feature space between them; a lower FID value indicates a more similar generated image to the real image.

[0057] See Figure 4 , Figure 4 This is a structural block diagram of a video generation apparatus provided in one embodiment of this application. The video generation apparatus 100 includes a feature extraction module 110, a decoupling and fusion module 120, a visual update module 130, and a video generation module 140.

[0058] The feature extraction module 110 is used to: acquire visual representations, camera motion representations, and human motion representations based on the original video sequence.

[0059] The decoupling fusion module 120 is used to: fuse the camera motion representation and the human motion representation using a decoupling fusion strategy to obtain a fused representation.

[0060] The visual update module 130 is used to update the visual representation based on the fused representation.

[0061] The video generation module 140 is used to: generate a target video sequence based on the updated visual representation.

[0062] In some embodiments, the decoupling fusion module 120 is further configured to: process the camera motion representation through a dimensionality reduction layer and a normalization layer to obtain an attention mask for camera motion; process the human motion representation through a dimensionality reduction layer and a normalization layer to obtain an attention mask for human motion; and fuse the camera motion representation and the human motion representation according to the attention mask for camera motion and the attention mask for human motion to obtain a fused representation.

[0063] In some embodiments, the decoupling fusion module 120 is further configured to: before fusing the camera motion representation and the human motion representation using a decoupling fusion strategy to obtain the fused representation, process the camera motion representation and the human motion representation separately through parallel self-attention layers.

[0064] In some embodiments, the feature extraction module 110 is used to: extract the camera motion trajectory from the original video sequence; parameterize the camera motion trajectory using the Plück embedding method to obtain camera trajectory parameters; and compress the camera trajectory parameters spatially and temporally using a block module to obtain a camera motion representation.

[0065] In some embodiments, the feature extraction module 110 is used to: obtain human posture information in the original video sequence using a whole-body posture estimation model; and compress the human posture information spatially and temporally using a block-segmentation module to obtain a human motion representation.

[0066] Those skilled in the art will clearly understand that the video generation apparatus 100 provided in the embodiments of this application can implement the video generation method provided in the embodiments of this application. The specific working process of the above apparatus and modules can be found in the corresponding process of the video generation method in the embodiments of this application, and will not be repeated here.

[0067] In the embodiments provided in this application, the coupling, direct coupling, or communication connection between the modules shown or discussed may be indirect coupling or communication coupling through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms. The embodiments of this application do not limit this.

[0068] Furthermore, in the seven embodiments of this application, the functional modules can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules, and this application embodiment does not impose any restrictions on this.

[0069] See Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device 200 may include a memory 210 and a processor 220. The memory 210 stores an application program, which, when invoked by the processor 220, causes the processor 220 to execute the video generation method in this embodiment of the application.

[0070] Processor 220 may include one or more processing cores. Processor 220 connects to various parts within the electronic device 200 using various interfaces and lines, and is used to run or execute instructions, programs, code sets, or instruction sets stored in memory 210, as well as to call and execute data stored in memory 210, performing various functions and processing data of the electronic device 200. Optionally, processor 220 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 220 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 220, but implemented separately through a communication chip.

[0071] The memory 210 may include random access memory (RAM) or read-only memory (ROM). The memory 210 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 210 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may store data created by the electronic device 200 during use.

[0072] This application also provides a computer-readable storage medium. The computer-readable storage medium stores program code that, when invoked by a processor, causes the processor to execute the video generation method described above in the embodiments of this application.

[0073] Computer-readable storage media can be electronic storage devices such as flash memory, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), hard disks, or ROM. Optionally, computer-readable storage media include non-volatile computer-readable storage medium (Non-TCRSM). The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed in an appropriate form.

[0074] This application also provides a computer program product, which includes a computer program that, when invoked by a processor, causes the processor to execute the method provided in the embodiments of this application.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video generation method, characterized in that, include: Based on the original video sequence, visual representation, camera motion representation, and human motion representation are obtained. A decoupling fusion strategy is used to fuse the camera motion representation and the human motion representation to obtain a fused representation; Update the visual representation based on the fused representation; Based on the updated visual representation, a target video sequence is generated.

2. The method according to claim 1, characterized in that, The decoupling fusion strategy is used to fuse the camera motion representation and the human motion representation to obtain a fused representation, including: The camera motion representation is processed by dimensionality reduction and normalization layers to obtain the attention mask for the camera motion. The human motion representation is processed by dimensionality reduction and normalization layers to obtain the attention mask for human motion. The camera motion representation and the human motion representation are fused based on the attention mask of the camera motion and the attention mask of the human motion to obtain a fused representation.

3. The method according to claim 1 or 2, characterized in that, Before fusing the camera motion representation and the human motion representation using a decoupling fusion strategy to obtain the fused representation, the method further includes: The camera motion representation and the human motion representation are processed separately through parallel self-attention layers.

4. The method according to claim 1, characterized in that, Based on the original video sequence, obtain camera motion representation, including: Extract camera motion trajectories from the original video sequence; The camera motion trajectory is parameterized using the Plück embedding method to obtain camera trajectory parameters; The camera trajectory parameters are compressed spatially and temporally by a block module to obtain a camera motion representation.

5. The method according to claim 1, characterized in that, Based on the original video sequence, human motion representation is obtained, including: A full-body pose estimation model is used to obtain human pose information from the original video sequence; The human posture information is compressed spatially and temporally by a segmentation module to obtain a human motion representation.

6. The method according to claim 4 or 5, characterized in that, The segmentation module includes a two-dimensional convolutional layer and a three-dimensional causal convolutional layer. The two-dimensional convolutional layer is used to compress the data spatially, and the three-dimensional causal convolutional layer is used to compress the data temporally. The compression ratio of the segmentation module is the same as that of the three-dimensional causal variational autoencoder. The three-dimensional causal variational autoencoder is used to convert the original video sequence into the visual representation.

7. A video generation apparatus, characterized in that, include: The feature extraction module is used to obtain visual representations, camera motion representations, and human motion representations based on the original video sequence; The decoupling fusion module is used to fuse the camera motion representation and the human motion representation using a decoupling fusion strategy to obtain a fused representation; A visual update module is used to update the visual representation based on the fused representation; A video generation module is used to generate a target video sequence based on the updated visual representation.

8. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores an application program that, when invoked by the processor, causes the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, and the product includes a computer program, which, when invoked by a processor, causes the program code to cause the processor to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when invoked by a processor, causes the processor to perform the method as described in any one of claims 1-6.

Images