Video generation method, apparatus, device, and storage medium

Abstract

The application provides a video generation method, device and equipment and a storage medium. The video generation method comprises the following steps: in response to a video generation request, constructing initial cache data according to first input data corresponding to the video generation request and multi-modal input data corresponding to the video generation request; performing geometric calibration according to second input data input by the video generation request and the initial cache data to obtain calibrated cache data; performing fusion rendering according to the initial cache data and the calibrated cache data to obtain fusion rendering data; and performing video generation processing according to the fusion rendering data and target constraint data to obtain a target generated video. The generated video can eliminate long-term memory dependence and the cumulative error of traditional monocular depth estimation, and can effectively eliminate alignment error in the video generation process, thereby improving the display effect of the generated video.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a video generation method, apparatus, device, and storage medium. Background Technology

[0002] Currently, with the rapid development of video generation technology, more and more users are using video generation technology to generate video content. However, existing video generation technologies are prone to motion trajectory deviations, object disappearances or deformations during the video generation process, and rely on dense input views, which can easily produce artifacts in sparse input or dynamic scenes, resulting in poor image quality of the generated video content. Summary of the Invention

[0003] This application provides a video generation method, apparatus, device, and storage medium, aiming to solve the technical problem of poor image quality of video content generated by existing video generation technologies.

[0004] On one hand, embodiments of this application provide a video generation method, which includes the following steps: In response to a video generation request, initial cache data is constructed based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. Geometric calibration is performed based on the second input data from the video generation request and the initial cache data to obtain calibration cache data; The initial cache data and the calibration cache data are used for fusion rendering to obtain fusion rendering data; Based on the fused rendering data and target constraint data, video generation processing is performed to obtain the target generated video.

[0005] In one possible implementation of this application, the step of constructing initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request includes: Depth estimation is performed on the first input data corresponding to the video generation request to obtain depth image data. The depth image data is then back-projected onto the target three-dimensional space to obtain an initial cached point cloud. Obtain the multimodal input data corresponding to the video generation request, and construct an initial cache matrix based on the multimodal input data and the initial cache point cloud; The initial cache point cloud and the initial cache matrix are determined as the initial cache data corresponding to the video generation request.

[0006] In one possible implementation of this application, the step of performing geometric calibration based on the second input data from the video generation request input and the initial cache data to obtain calibration cache data includes: Obtain the second input data corresponding to the video generation request, and generate the cache data to be processed corresponding to the second input data; Determine the projection error function corresponding to the cached data to be processed and the initial cache matrix; The projection calibration data corresponding to the cached data to be processed is calculated based on the cached data to be processed, the initial cached data, and the projection error function. The projection calibration data includes a depth scaling factor and a calibration translation amount. Geometric calibration is performed on the cache data to be processed based on the projection calibration data to obtain calibration cache data.

[0007] In one possible implementation of this application, the step of performing fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data includes: The initial cache data and the calibration cache data are combined to obtain the fused data to be processed; The fusion data to be processed is subjected to feature map pooling based on the fusion encoder and the target feature dimension to obtain fusion rendering data.

[0008] In one possible implementation of this application, the step of performing video generation processing based on the fused rendering data and target constraint data to obtain the target generated video includes: Obtain the motion loss function from the target constraint data, and use the motion loss function to constrain the fused rendering data to obtain constrained rendering data; The constrained rendering data is segmented into video blocks. The video segmentation blocks are autoregressively generated using an autoregressive generation function and a trajectory constraint function in the target constraint data to obtain the target generated video.

[0009] In one possible implementation of this application, the step of using an autoregressive generation function and a trajectory constraint function in the target constraint data to perform autoregressive generation on the video segmentation block to obtain the target generated video includes: The video segmentation blocks are used to generate video, resulting in a video to be processed. The trajectory constraint is applied to the video to be processed and generated based on the trajectory constraint function in the target constraint data to obtain the constrained generated video. The rendering detection function and the constraint-generated video are used to determine the rendering anomaly index. If the rendering anomaly index is less than a preset index threshold, the constraint-generated video is determined as the target generated video.

[0010] In one possible implementation of this application, constraining the fused rendering data using the motion loss function to obtain constrained rendering data includes: Noise embedding is performed on the fused rendering data based on the target mask matrix and target noise data to obtain noise rendering data; The motion loss function is used to constrain the noise rendering data to obtain constrained rendering data.

[0011] On the other hand, this application provides a video generation apparatus, the video generation apparatus comprising: The cache generation module is configured to respond to a video generation request and construct initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. The input calibration module is configured to perform geometric calibration based on the second input data input from the video generation request and the initial cache data to obtain calibration cache data; The data fusion module is configured to perform fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data; The video generation module is configured to perform video generation processing based on the fused rendering data and target constraint data to obtain the target generated video.

[0012] On the other hand, this application also provides a video generation device, the video generation device comprising: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the video generation method.

[0013] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the video generation method.

[0014] In this application, in response to a video generation request, initial cache data is constructed based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. Geometric calibration is performed based on the second input data from the video generation request and the initial cache data to obtain calibration cache data. Fusion rendering is performed based on the initial cache data and the calibration cache data to obtain fused rendering data. Video generation processing is performed based on the fused rendering data and target constraint data to obtain the target generated video. This enables the construction of initial cache data representing a 3D cache infrastructure and a temporal-spatial 3D cache structure based on pre-input first image data and multimodal data during video generation. Upon receiving new second input data, the second input data is converted into a corresponding 3D cache structure. The converted second input data is then calibrated with the initial cache data to obtain calibration cache data that seamlessly integrates with the initial cache data. This calibration cache data and the initial cache data are then fused from multiple perspectives to obtain fused rendering data. Finally, the fused rendering data and target constraint data are used for autoregressive video generation to obtain the target generated video that meets various constraints and has aligned motion trajectories. This enables the generated video to eliminate long-term memory dependency and the accumulated error of traditional monocular depth estimation, and effectively eliminates alignment errors in the video generation process, thereby improving the image quality of the generated video. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating a scenario of the video generation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating one embodiment of the video generation method in this application. Figure 3 A flowchart illustrating one embodiment of the video generation method provided in this application; Figure 4 A flowchart illustrating an embodiment of motion constraints in the video generation method provided in this application; Figure 5 A schematic diagram of the structure of one embodiment of the video generation apparatus provided in this application; Figure 6 This is a schematic diagram of the structure of one embodiment of the video generation device provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0019] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0020] Currently, with the rapid development of video generation technology, more and more users are using video generation technology to generate video content. However, existing video generation technologies are prone to motion trajectory deviations, object disappearances or deformations during the video generation process, and rely on dense input views, which can easily produce artifacts in sparse input or dynamic scenes, resulting in poor image quality of the generated video content.

[0021] Based on this, this application proposes a video generation method, apparatus, device, and computer-readable storage medium to solve the technical problem of poor image quality of video content generated by existing video generation technologies.

[0022] The video generation method in this embodiment of the invention is applied to a video generation device, which is configured in a video generation equipment. The video generation equipment includes one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the video generation method. The video generation equipment can be a smart terminal, such as a mobile phone, tablet computer, network device, and smart computer. Optionally, the video generation equipment can also be a server or a service cluster composed of multiple servers.

[0023] like Figure 1 As shown, Figure 1 This is a schematic diagram of a video generation method according to an embodiment of the present application. The video generation scenario in this embodiment includes a video generation device 100 (the video generation device 100 integrates a video generation apparatus), and the video generation device 100 is equipped with a computer-readable storage medium corresponding to the video generation method to execute the steps of the video generation method.

[0024] Understandable Figure 1 The video generation device in the video generation method scenario shown, or the devices included in the video generation device, does not constitute a limitation on the embodiments of the present invention. That is, the number or type of the video generation device in the video generation method scenario, or the number or type of devices included in each device, does not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of the present invention.

[0025] In this embodiment of the invention, the video generation device 100 is mainly used to: respond to a video generation request and construct initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request; Geometric calibration is performed based on the second input data from the video generation request and the initial cache data to obtain calibration cache data; The initial cache data and the calibration cache data are used for fusion rendering to obtain fusion rendering data; Based on the fused rendering data and target constraint data, video generation processing is performed to obtain the target generated video.

[0026] The video generation device 100 in this embodiment of the invention can be an independent video generation device, such as a mobile phone, tablet computer, network device, server and smart computer, or a video generation network or video generation cluster composed of multiple video generation devices.

[0027] This application provides a video generation method, apparatus, device, and computer-readable storage medium, which will be described in detail below.

[0028] It will be understood by those skilled in the art that Figure 1 The application environment shown is only one application scenario related to the solution of this application and does not constitute a limitation on the application scenario of this application. Other application environments may include more than one application scenario. Figure 1 The number of video generation devices shown, or the video generation network connectivity relationships, for example... Figure 1 Only one video generating device is shown in the diagram. It is understood that the scenario of this video generating method may also include one or more video generating devices, which are not specifically limited here. The video generating device 100 may also include a memory for storing input data and other data.

[0029] It should be noted that, Figure 1 The schematic diagram of the video generation method shown is merely an example. The scenarios of the video generation method described in this embodiment are intended to more clearly illustrate the technical solutions of this embodiment and do not constitute a limitation on the technical solutions provided by this embodiment.

[0030] Based on the scenarios described above for video generation methods, various embodiments of the video generation method disclosed in this invention are proposed.

[0031] like Figure 2 As shown, Figure 2 This is a flowchart illustrating one embodiment of the video generation method in this application. The video generation method includes the following steps 201 to 204: 201. In response to the video generation request, construct initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request; The video generation method in this embodiment is applied to a video generation device. The type and number of video generation devices are not specifically limited. That is, the video generation device can be one or more smart terminals or servers. In a specific embodiment, the video generation device is any smart terminal capable of video generation, such as a smart computer or a smartphone.

[0032] Optionally, the video generation request is an operation request that drives the video generation device to generate a target video based on the input data corresponding to the video generation request (which can be any one or more of single-frame image data, multi-frame image data, and dynamic video data). Optionally, the video generation request can be actively triggered by the user, for example, by clicking the video generation button provided by the video generation device (which can be a physical button or a virtual button control provided on a designated video generation interface). Optionally, in other embodiments, the video generation request can also be automatically triggered by an application or other interactive object.

[0033] Optionally, the first input data is image data information input by the target interactive object to the video generation device. This first input data is used for depth estimation and back-projection onto the target 3D space to generate an initial cached point cloud representing the 3D cache infrastructure. This records the position and color information of each pixel in the dynamic scene, thereby constructing an editable scene memory to maintain scene consistency across frames during subsequent video generation. Optionally, in a specific embodiment, the first input data can be a monocular input image or a multi-view input image.

[0034] Optionally, the multimodal input data refers to input information corresponding to various input formats, such as single-frame image data, multi-frame image data, and dynamic video data, inputted by the target interactive object to the video generation device. This first input data is used to construct an initial buffer matrix representing the temporal-spatial three-dimensional buffer structure.

[0035] Optionally, after receiving a video generation request, the video generation device acquires the first input data and the multimodal input data corresponding to the video generation request, and constructs a representational explicit spatiotemporal three-dimensional cache using the first input data and the multimodal input data to construct the initial cache data corresponding to the editable scene memory.

[0036] Optionally, the video generation device performs depth estimation on the first input data corresponding to the video generation request to obtain depth image data, and back-projects the depth image data onto the target three-dimensional space to obtain an initial cached point cloud, so as to record the position and color information of each pixel in the dynamic scene, thereby constructing an editable scene memory.

[0037] Optionally, in one specific embodiment, the initial cached point cloud is calculated as follows:

[0038] , ,

[0039] in, This is the initial cached point cloud corresponding to the first input data. The coordinates of the first input data projected onto the target 3D space. The first input data is projected onto the target 3D space to obtain the corresponding color information. The focal length corresponding to the first input data. , , This refers to the pixel position information projected onto the camera coordinate system. , , This refers to the depth coordinate information corresponding to the depth image data.

[0040] Optionally, the video generation device further acquires the multimodal input data corresponding to the video generation request, and constructs an initial cache matrix based on the number of frames and viewpoints of the multimodal input data. That is, the video generation device acquires multimodal input data representing various input formats such as single-frame image data / multi-frame image data and dynamic video data in the video generation request, and constructs an initial cache matrix using the multimodal input data and the corresponding initial cache point cloud. This organizes the multimodal input data from different times and angles into a hierarchical, time-space three-dimensional cache matrix arranged in chronological order to obtain the initial cache matrix. Optionally, in a specific embodiment, the expression for the initial cache matrix is: Each initial cache matrix element is , where l is the number of frames and v is the number of viewpoints.

[0041] Optionally, after obtaining the initial cache point cloud and the initial cache matrix, the video generation device determines the initial cache point cloud and the initial cache matrix as the initial cache data corresponding to the video generation request.

[0042] 202. Perform geometric calibration based on the second input data of the video generation request and the initial cache data to obtain calibration cache data; Optionally, after acquiring the initial cache data and the second input data that is later than the first input data and the multimodal input data, the video generation device also performs geometric calibration based on the second input data and the initial cache data to generate calibration cache data. This enables the generation of a cache point cloud and a cache matrix to be processed corresponding to the new second input data as cache data to be processed, and the alignment of the cache data to be processed with the original initial cache data to obtain calibration cache data. This allows the calibration cache data and the initial cache data to be seamlessly connected, facilitating subsequent steps to fuse and generate video data.

[0043] Optionally, the second input data is image or video input data that was input later than the first input data and the multimodal input data.

[0044] Optionally, the calibration cache data is the three-dimensional cache data obtained after geometrically calibrating the cache data to be processed corresponding to the second input data. This calibration cache data can be accurately integrated with the generated initial cache data.

[0045] Optionally, geometric calibration processing refers to the operation of minimizing the projection error between the second input data and the cached data to be processed and the initial cached data.

[0046] Optionally, the video generation device acquires the second input data corresponding to the video generation request and generates the cached data to be processed corresponding to the second input data. The cached data to be processed is depth image data obtained by depth estimation of the second input data.

[0047] Optionally, after acquiring the cached data to be processed, the video generation device determines the projection error function corresponding to the cached data to be processed and the initial cached data. The projection error function is a function algorithm that calculates the projection error between the cached data to be processed and the initial cache matrix.

[0048] Optionally, the video generation device calculates the projection calibration data corresponding to the cached data to be processed based on the cached data to be processed, the initial cache matrix, and the projection error function. That is, the video generation device inputs the cached data to be processed and the initial cache data into the projection error function, and solves the projection error function using the least squares method to obtain the projection calibration data. The projection calibration data includes a depth scaling factor and a calibration translation amount. Specifically, the projection calibration data represents the calibration parameter data obtained when the projection error between the cached data to be processed and the initial cache data is minimized. Optionally, in a specific embodiment, the expression for the projection calibration data is as follows:

[0049] Where s is the depth scaling factor and t is the translation amount. For cached data to be processed, This is the depth image data corresponding to the initial cached data.

[0050] Optionally, after acquiring the projection calibration data, the video generation device performs geometric calibration on the buffer data to be processed based on the projection calibration data to obtain calibration buffer data. That is, the video generation device uses the projection calibration data to align the buffer data to be processed with the initial buffer point cloud and initial buffer matrix corresponding to the initial buffer data to obtain calibration buffer data that can be accurately incorporated into the initial buffer point cloud and initial buffer matrix.

[0051] 203. Perform fusion rendering based on the initial cache data and the calibration cache data to obtain fusion rendering data; Optionally, after acquiring the calibration cache data, the video generation device also performs multi-view fusion rendering based on the initial cache data and the calibration cache data to obtain fusion rendering data.

[0052] Optionally, the fused rendering data is feature data obtained by multi-view fused rendering of the initial cache data and calibration cache data.

[0053] Optionally, after acquiring the calibration cache data, the video generation device combines the initial cache data and the calibration cache data to obtain the fused data to be processed. Specifically, the video generation device can perform data combination processing such as stitching the initial cache data and calibration cache data according to time sequence or number of viewpoints to obtain the fused data to be processed. The fused data to be processed is the fused data obtained by combining the calibration cache data and the initial cache data.

[0054] Optionally, the video generation device also pre-configures a fusion encoder for rendering the fused data to be processed, so as to perform feature map pooling processing on the fused data to be processed according to the fusion encoder and the target feature dimension to obtain fused rendering data. That is, the video generation device uses the fusion encoder to perform feature pooling processing on the fused data to be processed corresponding to multiple viewpoints according to the target feature dimension to obtain fused rendering data. Optionally, in a specific embodiment, the fusion encoder is a VAE encoder.

[0055] Optionally, in one specific embodiment, the calculation method for the fused rendering data is as follows:

[0056] in, To integrate rendering data, ~ The image data corresponding to each viewpoint in the data to be fused is... Let E be the target feature dimension, and E be the fusion encoder.

[0057] 204. Perform video generation processing based on the fused rendering data and target constraint data to obtain the target generated video.

[0058] Optionally, after acquiring the fused rendering data, in order to improve the conformity of the motion state of objects in subsequent videos to real physical laws, the video generation device also performs video generation processing based on the fused rendering data and target constraint data to obtain the target generated video. That is, the video generation device uses the target constraint data to constrain the fused rendering data and the generated video data to obtain the target generated video.

[0059] Optionally, the target constraint data is constraint data used to add specified constraint information during the video generation process of the fused rendered video. Optionally, in one specific embodiment, the target constraint data includes a motion loss function and a trajectory constraint function. The motion loss function is constraint data used to control the rate of change of features in adjacent frames to maintain consistency with the actual motion. The trajectory constraint function is constraint data used to apply trajectory constraints to editable objects in the video, aligning the target projection position of the editable object with the control trajectory.

[0060] Optionally, the video generation device can obtain the motion loss function from the target constraint data and use this motion loss function to constrain the fused rendering data to obtain constrained rendering data. That is, after obtaining the motion loss function, the video generation device adds the motion loss function to the fused rendering data in the diffusion model to control the feature change rate of adjacent frames to be consistent with the real motion, thereby obtaining constrained rendering data.

[0061] Optionally, the expression for the motion loss function is as follows:

[0062] in, Let be the motion loss function. For temporal gradient operators, Fpred represents the motion feature data in the fused rendering data. These are the actual motion characteristics.

[0063] Optionally, after acquiring the constrained rendering data, the video generation device performs video segmentation on the constrained rendering data to obtain video segmentation blocks. That is, the video generation device divides the constrained rendering data into several overlapping blocks, each overlapping block containing at least one or more frames of rendered image data, thus obtaining video segmentation blocks. Optionally, in a specific embodiment, after acquiring the video segmentation blocks, the video generation device further constrains the video segmentation blocks using a trajectory constraint function to generate the target generated video. Optionally, the video segmentation blocks are generated in the following way:

[0064] in, This refers to video segmentation blocks.

[0065] Optionally, after acquiring the video segmentation block, the video generation device uses an autoregressive generation function and a trajectory constraint function in the target constraint data to generate a video from the video segmentation block, thereby obtaining the target generated video.

[0066] Optionally, in one specific embodiment, after acquiring the generated video from the video segmentation blocks, the video generation device further optimizes the generated video using a multi-resolution optimization strategy to obtain the target generated video. This multi-resolution optimization strategy includes a low-resolution coarse-tuning strategy, a high-resolution fine-tuning strategy, and a stability optimization strategy. The low-resolution coarse-tuning strategy involves adjusting the resolution at 320... The optimization strategy involves coarsely adjusting the overall structure, camera motion, and 3D buffer logic of the generated video under parameters of 576×1024 resolution and 100k training steps. The high-resolution fine-tuning strategy refines texture, edges, and geometric details under parameters of 576×1024 resolution and 100k training steps. The stability tuning strategy optimizes the stability of the video content under parameters of a 56-frame long sequence and 10k training steps. After completing the adjustments corresponding to the previous optimization strategy, the video generation device lowers the learning rate based on the learning rate coefficient. Optionally, in a specific embodiment, this learning rate coefficient is 0.5.

[0067] In this embodiment, the video generation device responds to a video generation request by constructing initial cache data based on the first input data and multimodal input data corresponding to the video generation request; performing geometric calibration based on the second input data from the video generation request and the initial cache data to obtain calibration cache data; performing fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data; and performing video generation processing based on the fused rendering data and target constraint data to obtain the target generated video. This enables the device to construct initial cache data representing a 3D cache infrastructure and a temporal-spatial 3D cache structure based on pre-input first image data and multimodal data during video generation. Upon receiving new second input data, the device converts the second input data into a corresponding 3D cache structure, calibrates the converted second input data with the initial cache data to obtain calibration cache data that seamlessly integrates with the initial cache data, fuses the calibration cache data and the initial cache data from multiple perspectives to obtain fused rendering data, and performs autoregressive video generation based on the fused rendering data and target constraint data to obtain the target generated video that meets various constraints and has motion trajectory alignment. This enables the generated video to eliminate long-term memory dependency and the accumulated error of traditional monocular depth estimation, and effectively eliminates alignment errors in the video generation process, thereby improving the image quality of the generated video.

[0068] like Figure 3 As shown, Figure 3 This is a flowchart illustrating one embodiment of the video generation method provided in this application. Figure 3In the illustrated embodiment, the video generation method further includes steps 301 to 303: 301. Generate video from the video segmentation blocks to obtain the video to be processed; 302. Apply trajectory constraints to the video to be processed based on the trajectory constraint function in the target constraint data to obtain a constrained generated video; 303. Use the rendering detection function and the constraint-generated video to determine the rendering anomaly index. If the rendering anomaly index is less than a preset index threshold, determine the constraint-generated video as the target generated video.

[0069] Based on the above embodiments, in this embodiment, after obtaining the video segmentation block, the video generation device uses an autoregressive generation function and a trajectory constraint function in the target constraint data to perform autoregressive generation on the video segmentation block to obtain the target generated video.

[0070] Optionally, the video generation device generates video from the video segments to obtain the video to be processed. That is, the video generation device performs spatiotemporal alignment (i.e., temporal and spatial alignment) on each video segment, and renders and composites the aligned video segments to obtain the video to be processed.

[0071] Optionally, after acquiring the video to be processed, the video generation device, in order to improve the trajectory accuracy of editable objects in the video, also applies trajectory constraints to the video based on the trajectory constraint function in the target constraint data, resulting in a constrained generated video. That is, the video generation device uses the trajectory constraint function to apply trajectory constraints to the video to be processed, resulting in a constrained generated video. Optionally, the trajectory constraint can be implemented as follows:

[0072] Optionally, the video generation device also utilizes the rendering detection function and the constrained generated video to determine rendering anomaly indicators, and uses these indicators to evaluate the video generation quality of the constrained generated video. Optionally, the rendering anomaly indicator is calculated as follows:

[0073] in, To render abnormal indicators.

[0074] Optionally, if the rendering anomaly index is greater than a preset index threshold, a partial regeneration of the constraint-generated video is triggered to generate an updated constraint-generated video, until the rendering anomaly index corresponding to the constraint-generated video is less than the preset index threshold.

[0075] Optionally, if the video generation device determines that the rendering anomaly index is less than a preset index threshold, the constrained generated video is identified as the target generated video. In this embodiment, the video generation device generates a video to be processed by processing the video segmentation blocks; it applies trajectory constraints to the video to be processed according to the trajectory constraint function in the target constraint data to obtain a constrained generated video; it uses the rendering detection function and the constrained generated video to determine the rendering anomaly index; if the rendering anomaly index is less than a preset index threshold, the constrained generated video is identified as the target generated video. This achieves trajectory constraints on the video content and calculation of the rendering anomaly index during the video generation process, and evaluates the generation quality through the rendering anomaly index, thereby improving the video generation quality.

[0076] like Figure 4 As shown, Figure 4 This is a flowchart illustrating an embodiment of the video generation method for performing motion constraints provided in this application. In this embodiment, the video generation method includes steps 301 to 302: 301. Noise embedding is performed on the fused rendering data according to the target mask matrix and target noise data to obtain noise rendering data; 302. Constrain the noise rendering data using the motion loss function to obtain constrained rendering data.

[0077] Based on the above embodiments, in this embodiment, after acquiring the fused rendering data, in order to improve the conformity of the motion state of objects in the subsequent video to the laws of reality, the video generation device also performs video generation processing based on the fused rendering data and the target constraint data to obtain the target generated video.

[0078] Optionally, the video generation device embeds noise into the fused rendering data based on the target mask matrix and target noise data to obtain noise rendering data. That is, the video generation device applies controllable noise to the uncovered areas in the fused rendering data using the target mask matrix and target noise data to obtain noise rendering data. Optionally, the noise rendering data generation method is as follows:

[0079] in, Rendering data for noise. To fuse the rendering data, M is the target mask matrix. For the target noise data, where, , The noise standard deviation decreases with the number of iterations.

[0080]

[0081] Optionally, after acquiring the noise rendering data, the video generation device also uses the motion loss function to constrain the noise rendering data to obtain constrained rendering data, and uses the constrained rendering data to generate the target and the video.

[0082] In this embodiment, the video generation device embeds noise into the fused rendering data based on the target mask matrix and target noise data to obtain noisy rendering data; then, it constrains the noisy rendering data using the motion loss function to obtain constrained rendering data. This enables controllable noise addition to uncovered areas in the fused rendering data, thereby improving the generation quality of subsequent video.

[0083] To better implement the video generation method in the embodiments of this application, based on the video generation method, the embodiments of this application also provide a video generation apparatus, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of one embodiment of the video generation apparatus provided in this application. Specifically, the video generation apparatus 500 includes: The cache generation module 501 is configured to respond to a video generation request and construct initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. The input calibration module 502 is configured to perform geometric calibration based on the second input data input in the video generation request and the initial cache data to obtain calibration cache data; The data fusion module 503 is configured to perform fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data; The video generation module 504 is configured to perform video generation processing based on the fused rendering data and target constraint data to obtain the target generated video.

[0084] In one possible implementation of this embodiment, the video generation device constructs initial buffer data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request, including: Depth estimation is performed on the first input data corresponding to the video generation request to obtain depth image data. The depth image data is then back-projected onto the target three-dimensional space to obtain an initial cached point cloud. Obtain the multimodal input data corresponding to the video generation request, and construct an initial cache matrix based on the multimodal input data and the initial cache point cloud; The initial cache point cloud and the initial cache matrix are determined as the initial cache data corresponding to the video generation request.

[0085] In one possible implementation of this embodiment, the video generation device performs geometric calibration based on the second input data from the video generation request input and the initial buffer data to obtain calibration buffer data, including: Obtain the second input data corresponding to the video generation request, and generate the cache data to be processed corresponding to the second input data; Determine the projection error function corresponding to the cached data to be processed and the initial cache matrix; The projection calibration data corresponding to the cached data to be processed is calculated based on the cached data to be processed, the initial cached data, and the projection error function. The projection calibration data includes a depth scaling factor and a calibration translation amount. Geometric calibration is performed on the cache data to be processed based on the projection calibration data to obtain calibration cache data.

[0086] In one possible implementation of this embodiment, the video generation device performs fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data, including: The initial cache data and the calibration cache data are combined to obtain the fused data to be processed; The fusion data to be processed is subjected to feature map pooling based on the fusion encoder and the target feature dimension to obtain fusion rendering data.

[0087] In one possible implementation of this embodiment, the video generation device performs video generation processing based on the fused rendering data and target constraint data to obtain the target generated video, including: Obtain the motion loss function from the target constraint data, and use the motion loss function to constrain the fused rendering data to obtain constrained rendering data; The constrained rendering data is segmented into video blocks. The video segmentation blocks are autoregressively generated using an autoregressive generation function and a trajectory constraint function in the target constraint data to obtain the target generated video.

[0088] In one possible implementation of this embodiment, the video generation device uses an autoregressive generation function and a trajectory constraint function in the target constraint data to perform autoregressive generation on the video segmentation blocks to obtain the target generated video, including: The video segmentation blocks are used to generate video, resulting in a video to be processed. The trajectory constraint is applied to the video to be processed and generated based on the trajectory constraint function in the target constraint data to obtain the constrained generated video. The rendering detection function and the constraint-generated video are used to determine the rendering anomaly index. If the rendering anomaly index is less than a preset index threshold, the constraint-generated video is determined as the target generated video.

[0089] In one possible implementation of this embodiment, the video generation device uses the motion loss function to constrain the fused rendering data to obtain constrained rendering data, including: Noise embedding is performed on the fused rendering data based on the target mask matrix and target noise data to obtain noise rendering data; The motion loss function is used to constrain the noise rendering data to obtain constrained rendering data.

[0090] In this embodiment, the video generation device responds to a video generation request by constructing initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request; performing geometric calibration based on the second input data input by the video generation request and the initial cache data to obtain calibration cache data; performing fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data; and performing video generation processing based on the fused rendering data and target constraint data to obtain the target generated video. This enables the device to construct initial cache data representing a three-dimensional cache infrastructure and a temporal-spatial three-dimensional cache structure based on pre-input first image data and multimodal data during the video generation process. Upon receiving new second input data, the device converts the second input data into a corresponding three-dimensional cache structure, calibrates the converted second input data with the initial cache data to obtain calibration cache data that seamlessly integrates with the initial cache data, fuses the calibration cache data and the initial cache data from multiple perspectives to obtain fused rendering data, and performs autoregressive video generation based on the fused rendering data and target constraint data to obtain the target generated video that meets various constraint conditions and has motion trajectory alignment. This enables the generated video to eliminate long-term memory dependency and the accumulated error of traditional monocular depth estimation, and effectively eliminates alignment errors in the video generation process, thereby improving the image quality of the generated video.

[0091] This invention also provides a video generation device, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of one embodiment of the video generation device provided in this application.

[0092] The video generation device integrates any of the video generation apparatuses provided in the embodiments of the present invention, and the video generation device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured by the processor to perform the steps of the video generation method described in any of the embodiments of the video generation method described above.

[0093] Specifically, a video generation device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will understand that... Figure 6 The video generation device structure shown does not constitute a limitation on the video generation device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 601 is the control center of the video generation device. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 602, and by calling data stored in the memory 602, thereby providing overall monitoring of the video generation device. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601.

[0094] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and video generation by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the video generation device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0095] The video generation device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0096] The video generation device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0097] Although not shown, the video generation device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the video generation device loads the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 runs the applications stored in the memory 602 to realize various functions, as follows: In response to a video generation request, initial cache data is constructed based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. Geometric calibration is performed based on the second input data from the video generation request and the initial cache data to obtain calibration cache data; The initial cache data and the calibration cache data are used for fusion rendering to obtain fusion rendering data; Based on the fused rendering data and target constraint data, video generation processing is performed to obtain the target generated video.

[0098] Therefore, embodiments of the present invention provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. A computer program is stored thereon, which is loaded by a processor to execute the steps in any of the video generation methods provided in the embodiments of the present invention. For example, the computer program loaded by the processor can execute the following steps: In response to a video generation request, initial cache data is constructed based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. Geometric calibration is performed based on the second input data from the video generation request and the initial cache data to obtain calibration cache data; The initial cache data and the calibration cache data are used for fusion rendering to obtain fusion rendering data; Based on the fused rendering data and target constraint data, video generation processing is performed to obtain the target generated video.

[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0100] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0101] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0102] The above provides a detailed description of a video generation method provided by the embodiments of this application. Specific embodiments have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A video generation method, characterized in that, The video generation method includes: In response to a video generation request, initial cache data is constructed based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. Geometric calibration is performed based on the second input data from the video generation request and the initial cache data to obtain calibration cache data; The initial cache data and the calibration cache data are used for fusion rendering to obtain fusion rendering data; Based on the fused rendering data and target constraint data, video generation processing is performed to obtain the target generated video.

2. The video generation method according to claim 1, characterized in that, The step of constructing initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request includes: Depth estimation is performed on the first input data corresponding to the video generation request to obtain depth image data. The depth image data is then back-projected onto the target three-dimensional space to obtain an initial cached point cloud. Obtain the multimodal input data corresponding to the video generation request, and construct an initial cache matrix based on the multimodal input data and the initial cache point cloud; The initial cache point cloud and the initial cache matrix are determined as the initial cache data corresponding to the video generation request.

3. The video generation method according to claim 1, characterized in that, The step of performing geometric calibration based on the second input data from the video generation request and the initial cache data to obtain calibration cache data includes: Obtain the second input data corresponding to the video generation request, and generate the cache data to be processed corresponding to the second input data; Determine the projection error function corresponding to the cached data to be processed and the initial cached data; The projection calibration data corresponding to the cached data to be processed is calculated based on the cached data to be processed, the initial cached data, and the projection error function. The projection calibration data includes a depth scaling factor and a calibration translation amount. Geometric calibration is performed on the cache data to be processed based on the projection calibration data to obtain calibration cache data.

4. The video generation method according to claim 1, characterized in that, The step of performing fusion rendering based on the initial cache data and the calibration cache data to obtain fusion rendering data includes: The initial cache data and the calibration cache data are combined to obtain the fused data to be processed; The fusion data to be processed is subjected to feature map pooling based on the fusion encoder and the target feature dimension to obtain fusion rendering data.

5. The video generation method according to claim 1, characterized in that, The step of performing video generation processing based on the fused rendering data and target constraint data to obtain the target generated video includes: Obtain the motion loss function from the target constraint data, and use the motion loss function to constrain the fused rendering data to obtain constrained rendering data; The constrained rendering data is segmented into video blocks. The video segmentation blocks are autoregressively generated using an autoregressive generation function and a trajectory constraint function in the target constraint data to obtain the target generated video.

6. The video generation method according to claim 5, characterized in that, The step of using an autoregressive generation function and a trajectory constraint function in the target constraint data to perform autoregressive generation on the video segmentation blocks to obtain the target generated video includes: The video segmentation blocks are used to generate video, resulting in a video to be processed. The trajectory constraint is applied to the video to be processed and generated based on the trajectory constraint function in the target constraint data to obtain the constrained generated video. The rendering detection function and the constraint-generated video are used to determine the rendering anomaly index. If the rendering anomaly index is less than a preset index threshold, the constraint-generated video is determined as the target generated video.

7. The video generation method according to claim 5, characterized in that, The step of constraining the fused rendering data using the motion loss function to obtain constrained rendering data includes: Noise embedding is performed on the fused rendering data based on the target mask matrix and target noise data to obtain noise rendering data; The motion loss function is used to constrain the noise rendering data to obtain constrained rendering data.

8. A video generation apparatus, characterized in that, The video generation device includes: The cache generation module is configured to respond to a video generation request and construct initial cache data based on the first input data corresponding to the video generation request and the multimodal input data corresponding to the video generation request. The input calibration module is configured to perform geometric calibration based on the second input data input from the video generation request and the initial cache data to obtain calibration cache data; The data fusion module is configured to perform fusion rendering based on the initial cache data and the calibration cache data to obtain fused rendering data; The video generation module is configured to perform video generation processing based on the fused rendering data and target constraint data to obtain the target generated video.

9. A video generation device, characterized in that, The video generation device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the video generation method according to any one of claims 1 to 7.

Images