Video generation method, video generation model training method, computing device, and storage medium

Abstract

Embodiments of the present specification provide a video generation method, a video generation model training method, a computing device and a storage medium, wherein the video generation method comprises: determining an initial video containing a target object, an initial image containing an associated object, and a target collection track, wherein the associated object is an object to be worn by the target object; generating intermediate object data corresponding to the target object according to the initial video and the initial image, wherein the intermediate object data is used to represent the motion change of the target object wearing the associated object in the three-dimensional space dimension and the time dimension corresponding to the initial video; rendering the intermediate object data based on the target collection track to obtain an intermediate video corresponding to the target object; and inputting the initial video, the initial image, the target collection track and the intermediate video into a video generation model to obtain a target video corresponding to the target object, wherein the target video displays the target object wearing the associated object based on the target collection track.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to video generation methods, video generation model training methods, computing devices, and storage media. Background Technology

[0002] With the development of e-commerce, virtual try-on technology has emerged, allowing users to virtually try on clothes through online websites. Early virtual try-on technology primarily focused on image-based virtual try-on, using techniques such as human body analysis, pose estimation, clothing deformation, and image generation to integrate the clothing the user wants to try on onto their image, thus achieving online virtual try-on.

[0003] However, image-based virtual try-on technology can only provide users with a static preview experience of clothing, and the clothing try-on images usually rely on user-provided images of people, making it difficult to guarantee stable image quality. The try-on effect is relatively limited and fails to meet user needs, thus impacting the user experience. Therefore, an effective technical solution is urgently needed to address these issues. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a video generation method. One or more embodiments of this specification also relate to a video generation apparatus, a video generation model training method, a video generation model training apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a video generation method is provided, comprising: Determine the initial video containing the target object, the initial image containing the associated object, and the target acquisition trajectory, wherein the associated object is the object to be worn by the target object; Based on the initial video and the initial image, intermediate object data corresponding to the target object is generated, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; Based on the target acquisition trajectory, the intermediate object data is rendered to obtain the intermediate video corresponding to the target object; The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model to obtain the target video corresponding to the target object. The target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0006] According to a second aspect of the embodiments of this specification, a video generation apparatus is provided, comprising: The first determining module is configured to determine an initial video containing a target object, an initial image containing an associated object, and a target acquisition trajectory, wherein the associated object is the object to be worn by the target object; The first generation module is configured to generate intermediate object data corresponding to the target object based on the initial video and the initial image, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; The first rendering module is configured to render the intermediate object data based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object; The first input module is configured to input the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain a target video corresponding to the target object, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0007] According to a third aspect of the embodiments of this specification, a video generation model training method is provided, comprising: The initial video of the sample containing the sample object, the initial image of the sample containing the sample-associated object, and the sample acquisition trajectory are determined, wherein the sample-associated object is the object to be worn by the sample object; Based on the initial sample video and the initial sample image, intermediate sample object data corresponding to the sample object is generated, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and temporal dimension corresponding to the initial sample video; Based on the sample acquisition trajectory, the intermediate object data of the sample is rendered to obtain the intermediate video of the sample object; The video generation model to be trained is obtained by using the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video.

[0008] According to a fourth aspect of the embodiments of this specification, a video generation model training apparatus is provided, comprising: The second determining module is configured to determine the initial video of the sample containing the sample object, the initial image of the sample containing the sample associated object, and the sample acquisition trajectory, wherein the sample associated object is the object to be worn by the sample object; The second generation module is configured to generate intermediate sample object data corresponding to the sample object based on the initial sample video and the initial sample image, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial sample video. The second rendering module is configured to render the intermediate object data of the sample based on the sample acquisition trajectory, so as to obtain the intermediate video of the sample corresponding to the sample object. The training module is configured to train the video generation model to be trained based on the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video, so as to obtain the trained video generation model.

[0009] According to a fifth aspect of the embodiments of this specification, another video generation method is provided, including: Receive a video generation request sent by the client and determine the target acquisition trajectory corresponding to the video generation request, wherein the video generation request carries an initial video containing the target object and an initial image containing an associated object, the associated object being the object to be worn by the target object; Based on the initial video and the initial image, intermediate object data corresponding to the target object is generated, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; Based on the target acquisition trajectory, the intermediate object data is rendered to obtain the intermediate video corresponding to the target object; The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model to obtain the target video corresponding to the target object, and the target video is sent to the client. The target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0010] According to a sixth aspect of the embodiments of this specification, another video generation apparatus is provided, comprising: The receiving module is configured to receive a video generation request sent by a client and determine the target acquisition trajectory corresponding to the video generation request. The video generation request carries an initial video containing the target object and an initial image containing an associated object, wherein the associated object is the object to be worn by the target object. The third generation module is configured to generate intermediate object data corresponding to the target object based on the initial video and the initial image, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; The third rendering module is configured to render the intermediate object data based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object; The second input module is configured to input the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain a target video corresponding to the target object, and send the target video to the client, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0011] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0012] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0013] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0014] One embodiment of this specification provides a video generation method, comprising: determining an initial video containing a target object, an initial image containing an associated object, and a target acquisition trajectory, wherein the associated object is an object to be worn by the target object; generating intermediate object data corresponding to the target object based on the initial video and the initial image, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and temporal dimension corresponding to the initial video; rendering the intermediate object data based on the target acquisition trajectory to obtain an intermediate video corresponding to the target object; and inputting the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain a target video corresponding to the target object, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory. In the above method, when generating the target video of the target object wearing associated objects, a target acquisition trajectory can be added. After generating intermediate object data of the target object wearing associated objects based on the initial video and initial image, the intermediate object data is rendered based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object. Furthermore, the target video of the target object wearing associated objects is obtained by inputting the initial video, initial image, target acquisition trajectory, and intermediate video into the video generation model. The target video is generated based on the target acquisition trajectory, and multiple perspectives of the target object wearing associated objects can be displayed in the target video according to the target acquisition trajectory, thereby achieving a better display of the try-on effect, ensuring the quality and stability of the target video generation, and further meeting the user's try-on needs and try-on experience. Attached Figure Description

[0015] Figure 1 This is an architecture diagram of a video generation system provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a video generation method provided in one embodiment of this specification; Figure 3 This is a schematic diagram illustrating the alignment of a human body and a scene in a video generation method provided in one embodiment of this specification; Figure 4 This is a schematic diagram illustrating the generation of dynamic object data in a video generation method provided in one embodiment of this specification; Figure 5 This is a schematic diagram of a reference branch and a generation branch in a video generation method provided in one embodiment of this specification; Figure 6 This is a flowchart illustrating the processing steps of a video generation method provided in one embodiment of this specification. Figure 7 This is a schematic diagram of the structure of a video generation device provided in one embodiment of this specification; Figure 8 This is a flowchart illustrating a video generation model training method provided in one embodiment of this specification; Figure 9 This is a schematic diagram of the structure of a video generation model training device provided in one embodiment of this specification; Figure 10 This is a flowchart of another video generation method provided in one embodiment of this specification; Figure 11 This is a schematic diagram of another video generation device provided in one embodiment of this specification; Figure 12 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0016] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0017] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0018] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0019] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] The technical solutions provided in this application can employ deep learning models with relatively large parameter scales. However, this large model is merely an example; this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based language models (LM) or multimodal models (MM). First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] 4D: Four-Dimensional, which adds time as a fourth dimension to the three-dimensional space (length, width, and height).

[0022] SMPL-X: Skinned Multi-Person Linear model – Xpressive, is a parametric human body model that can represent the structure of the human body, hands, and face with a small number of pose and shape parameters.

[0023] DIT: DiffusionTransformer, an advanced generative artificial intelligence architecture that combines the powerful sequence modeling capabilities of the Transformer model with the high-quality image and video generation capabilities of the diffusion model.

[0024] Diffusion model: The diffusion model is a type of generative deep learning model. Its basic idea is to first gradually add noise to image or video data, and then train the model to gradually recover the target image or video from the noise.

[0025] Video virtual try-on: Video virtual try-on is an extension of virtual try-on in a video setting. The input usually includes a video of a person and an image of the target clothing, and the output is a dynamic try-on video of a person wearing the target clothing in the video.

[0026] Camera-controlled video virtual try-on: Camera-controlled video virtual try-on refers to the ability to generate virtual try-on videos from new perspectives based on a specified capture trajectory, rather than being limited to the fixed shooting angle of the original input video. For example, users can view the effect of clothing on the body from the front, side, back, surround view, or zoomed view.

[0027] 3D Gaussian Representation: 3D Gaussian representation is a method of representing 3D scenes or objects. It uses a large number of Gaussian primitives with attributes such as position, scale, rotation direction, transparency, and color to represent 3D content. Compared to traditional meshes or sparse point clouds, 3D Gaussian representation can better express fine-grained appearance textures and supports real-time rendering.

[0028] Point cloud: A point cloud is a data set composed of a large number of three-dimensional points. Each point typically contains spatial coordinates and may also contain information such as color and confidence level. Point clouds can be used to represent the structure of a three-dimensional scene.

[0029] To address the aforementioned technical problems, this specification provides a video generation method, and also relates to a video generation apparatus, a video generation model training method, a video generation model training apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0030] See Figure 1 , Figure 1 This specification illustrates an architecture diagram of a video generation system according to one embodiment of the present specification. The video generation system may include a client 100 and a server 200. Client 100 is used to display a graphical user interface to the user and, in response to the user's operation on the graphical user interface, sends a video generation request to server 200. Server 200, in response to the video generation request, determines an initial video containing the target object, an initial image containing an associated object, and a target acquisition trajectory, wherein the associated object is the object to be worn by the target object; generates intermediate object data corresponding to the target object based on the initial video and the initial image, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; renders the intermediate object data based on the target acquisition trajectory to obtain an intermediate video corresponding to the target object; inputs the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain a target video corresponding to the target object, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory; and sends the target video to client 100. Client 100 is also used to receive target videos sent by server 200.

[0031] The graphical user interface (GUI) can display a video generation interface where users can upload initial videos and images. In practical applications, the GUI can also display an e-commerce platform's apparel product interface, where users can select the corresponding apparel product and upload an initial video to generate a target video of the user trying on the apparel.

[0032] The video generation system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In the video generation scenario, server 200 is used to provide video generation services between multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0033] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In a video generation scenario, users can publish a data stream to server 200 through client 100, server 200 can generate a target video based on the data stream, and push the target video to other clients that have established communication.

[0034] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0035] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on a computing device and depends on the device or certain apps on the device to run. The computing device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured on the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0036] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0037] It is worth noting that the video generation method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the video generation method provided in the embodiments of this specification. In other embodiments, the video generation method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0038] See Figure 2 , Figure 2 A flowchart of a video generation method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0039] Step 202: Determine the initial video containing the target object, the initial image containing the associated object, and the target acquisition trajectory, wherein the associated object is the object to be worn by the target object.

[0040] Specifically, the video generation method provided in the embodiments of this specification can be used on the server side.

[0041] In practical applications, this video generation method can be used to generate clothing try-on videos for e-commerce platforms. On these platforms, users can purchase clothing or accessories online. To ensure a positive user experience, this method can be used to generate try-on videos of the clothing or accessories, allowing users to view the effects online and further enhancing their purchasing experience. For example, it can generate try-on videos of tops, pants, hats, etc.

[0042] The target object can be understood as the person to be tried on, such as a user who wants to buy clothing or wear accessories. The initial video containing the target object can be understood as a video of the person to be tried on, such as a video shot from the front of the target object. The associated object can be understood as the object to be worn by the target object, such as a product that the user wants to buy. For example, if the user wants to buy a top, then the associated object can be the top. The initial image containing the associated object can be an image of the associated object, which can be obtained from the product images displayed on the product display interface of the e-commerce platform.

[0043] The target acquisition trajectory can be understood as the acquisition trajectory corresponding to the specified target video to be generated. This acquisition trajectory can be understood as the movement trajectory of the image acquisition device, that is, the path the image acquisition device moves in three-dimensional space. This acquisition trajectory can be determined based on the viewing angle and position of the image acquisition device. The initial video containing the target object can be obtained by the image acquisition device based on the initial acquisition trajectory. For example, when capturing the initial video containing the target object, the image acquisition device can be moved away from the target object after shooting from its front, then its viewing angle can be turned towards the target object, and its position can be moved to the left to complete the capture of the target object, resulting in the initial video containing the target object. The initial acquisition trajectory corresponding to this initial video is the movement trajectory of the image acquisition device's position and viewing angle during the initial video capture process. Therefore, the target acquisition trajectory could be, for example, the movement trajectory of the image acquisition device facing the target object and circling the target object. Image acquisition devices can include, but are not limited to, cameras, glasses equipped with cameras, mobile phones, and other devices.

[0044] Understandably, the initial acquisition trajectory is the actual acquisition trajectory when capturing the initial video, while the target acquisition trajectory is a new acquisition trajectory specified for the generated target video. The target acquisition viewpoint and target acquisition position corresponding to this target acquisition trajectory can be acquisition viewpoints and positions that do not exist in the initial acquisition trajectory. For example, if the initial video is captured from the front of the target object, and the specified target acquisition trajectory indicates that the generated target video can show the back of the target object, then the back viewpoint corresponding to the target acquisition trajectory does not exist in the initial acquisition trajectory. The video generation method provided in the embodiments of this specification can generate target videos corresponding to target acquisition trajectories that have different initial acquisition viewpoints and initial acquisition positions than the initial acquisition trajectory. Understandably, the acquisition viewpoint and acquisition position refer to the viewpoint and position during the image acquisition device's shooting process.

[0045] Based on this, the initial video obtained by capturing the target object based on the initial acquisition trajectory can be determined, as well as the initial image of the associated object containing the target object to be worn and the specified target acquisition trajectory.

[0046] Furthermore, users can send a video generation request to the server through the client. This video generation request can carry an initial video of the user (i.e., the target object) and an initial image of the associated object.

[0047] In practical applications, the client can display the product display page of an e-commerce platform through the display interface. This product display page can show the product image of the product to be purchased (such as a short-sleeved shirt) and a video generation control for the product. The user can click the video generation control through the client and upload an initial video containing the user. The client can respond to the click operation by generating a video generation request for the product based on the initial video and sending it to the server. The server can receive the video generation request, obtain the initial video and the initial image containing the product carried in the video generation request, and determine the target acquisition trajectory corresponding to the video generation request.

[0048] In one optional embodiment of this specification, the target acquisition trajectory may be selected by the user from multiple candidate acquisition trajectories provided by the e-commerce platform, or it may be generated based on the user-specified viewpoint information and / or location information to be generated.

[0049] The viewpoint information to be generated can be understood as the acquisition viewpoint information corresponding to the target video specified by the user, and the location information to be generated can be understood as the acquisition location information corresponding to the target video specified by the user.

[0050] Based on this, the method also includes: The system sends multiple candidate acquisition trajectories corresponding to the video generation request to the client and receives the target acquisition trajectory sent by the client, wherein the target acquisition trajectory is any one of the multiple candidate acquisition trajectories; or Based on the viewpoint information and / or location information to be generated carried in the video generation request, the target acquisition trajectory corresponding to the video generation request is generated.

[0051] For example, the server can send multiple predefined candidate acquisition trajectories corresponding to the video generation request to the client. The client's display interface can show the multiple candidate acquisition trajectories to the user and display the reference video corresponding to each candidate acquisition trajectory. The user can select any one of the multiple candidate acquisition trajectories as the target acquisition trajectory. In response to the user's selection operation on the multiple candidate acquisition trajectories, the client sends the target acquisition trajectory selected by the user to the server.

[0052] Alternatively, users can specify the viewpoint information and / or location information to be generated. For example, a user can input the instruction "I want to generate a video that shows my try-on effect from the front, side and back". Then, the corresponding viewpoint information and / or location information to be generated can be determined based on the instruction information, and the target acquisition trajectory can be generated based on the viewpoint information and / or location information to be generated.

[0053] In summary, determining the corresponding target acquisition trajectory by sending video generation requests based on the client facilitates the subsequent generation of target videos based on the target acquisition trajectory, which can meet users' needs for different perspectives of clothing try-on effects and further improve the user experience.

[0054] Step 204: Based on the initial video and the initial image, generate intermediate object data corresponding to the target object, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video.

[0055] Specifically, when the target object is a person, the intermediate object data can include the three-dimensional spatial structure and the changes in human motion in the time dimension, as well as the background content corresponding to the background point cloud data, so as to realize the fusion of the associated object and the target object in the initial video in a unified world coordinate system.

[0056] In practical applications, intermediate object data can be a renderable 4D proxy, where 4D can be used to represent that the proxy simultaneously contains three-dimensional spatial structure and changes in human motion over time.

[0057] In one optional embodiment of this specification, generating intermediate object data corresponding to the target object based on the initial video and the initial image includes: The target object in the initial video is subjected to object pose extraction to obtain the target pose sequence corresponding to the target object; Based on the initial video and the initial image, generate three-dimensional object data corresponding to the target object, wherein the three-dimensional object data is used to represent a three-dimensional virtual image of the target object wearing the associated object; Based on the target pose sequence and the 3D object data, generate dynamic object data corresponding to the target object; Based on the background point cloud data corresponding to the initial video and the dynamic object data, intermediate object data corresponding to the target object is generated.

[0058] The target pose sequence can be understood as a sequence of motion parameters of the target object that changes over time in the initial video. This target pose sequence can be used to represent the body posture and motion state of the target object in each video frame of the initial video. That is to say, the target pose sequence can include the pose parameters of the target object in each video frame of the initial video. For example, the target object is standing in the first video frame of the initial video, raises its left foot forward in the second video frame of the initial video, and lands its left foot in the third video frame of the initial video. The sequence of motion parameters corresponding to these three video frames can characterize the leg-raising action of the target object.

[0059] Three-dimensional object data can be understood as a three-dimensional human figure.

[0060] Dynamic object data can be understood as a dynamic human figure that changes over time by continuously deforming the three-dimensional object data based on the target pose sequence.

[0061] Based on this, each video frame of the initial video can be determined, and the body pose and motion state of the target object in each video frame can be extracted to obtain the target pose sequence corresponding to the target object. Furthermore, the target object contained in the initial video and related objects contained in the initial image can be fused to generate 3D object data corresponding to the target object. This 3D object data and the target pose sequence are then fused to generate dynamic object data corresponding to the target object. Finally, based on the background point cloud data and dynamic object data corresponding to the initial video, intermediate object data corresponding to the target object is generated.

[0062] In summary, by generating intermediate object data corresponding to the target object, it is easier to generate intermediate videos based on the target acquisition trajectory using this intermediate object data. This provides stable references such as human body structure, clothing position, background relationship, and image acquisition device motion information for subsequent target video generation, thereby further improving the quality of the generated target video.

[0063] In one optional embodiment of this specification, the step of extracting the object pose of the target object in the initial video to obtain the target pose sequence corresponding to the target object includes: Perform 3D scene reconstruction on the initial video to obtain dynamic point cloud data corresponding to the initial video; The dynamic point cloud data is segmented to obtain the background point cloud data corresponding to the initial video and the object point cloud data corresponding to the target object. Perform object pose estimation on the target object in the initial video to obtain the initial pose sequence corresponding to the target object; Based on the object point cloud data, the initial pose sequence is mapped to the reconstructed scene corresponding to the background point cloud data to obtain the target pose sequence corresponding to the target object.

[0064] The dynamic point cloud data corresponding to the initial video can include an ordered set of static point clouds on the time axis. This set of static point clouds can be understood as the point cloud data corresponding to each video frame of the initial video. The point cloud data can be understood as a data set composed of a large number of points with precise coordinates in three-dimensional space. This data set can also contain the color information or reflection intensity information of each point.

[0065] Background point cloud data can be understood as the point cloud data corresponding to the background in the initial video. Background point cloud data is used to represent the scene environment in the initial video. Object point cloud data can be understood as the point cloud data corresponding to the target object in the initial video, i.e., human body point cloud data. Object point cloud data is used to assist in the subsequent alignment of the human body and the scene.

[0066] The initial pose sequence can be understood as the pose sequence before alignment, and the target pose sequence can be understood as the pose sequence after alignment. In practical applications, the pose sequence can be the SMPL-X human parameter sequence.

[0067] Based on this, when reconstructing a 3D scene from the initial video, each video frame of the initial video is determined, and the depth information (e.g., a depth map), image acquisition device parameter information, and confidence information of each video frame are estimated. Then, based on the depth information, image acquisition device parameter information, and confidence information of each video frame, each video frame is back-projected into the 3D space to obtain the point cloud data corresponding to each video frame, thereby obtaining the dynamic point cloud data corresponding to the initial video.

[0068] The dynamic point cloud data corresponding to the initial video can be segmented into background point cloud data and object point cloud data based on human body segmentation. In practical applications, when segmenting the dynamic point cloud data, an initial pose sequence can be used to match the dynamic point cloud data. Points in the dynamic point cloud data whose distance from the initial pose sequence is less than a preset distance threshold are identified as object point cloud data, and points whose distance from the initial pose sequence is greater than the preset distance threshold are identified as background point cloud data. Alternatively, the dynamic point cloud data can be segmented based on a deep learning model or a neural network model to directly obtain background point cloud data and object point cloud data. Alternatively, the dynamic point cloud data can be segmented based on background clipping and motion information. Based on multiple video frames of the initial video, point cloud data with the same position in multiple video frames can be identified as background point cloud data, and point cloud data with changing positions in multiple video frames can be identified as object point cloud data. This specification does not limit the embodiments in this way.

[0069] The target object in the initial video can be estimated to obtain a set of initial pose sequences that change over time. Since the dynamic point cloud data obtained from the above 3D scene reconstruction is located in the world coordinate system, while the initial pose sequence is usually located in the camera coordinate system, the two are not consistent in scale, rotation and translation. Based on this, the initial pose sequence can be mapped to the reconstructed scene corresponding to the background point cloud data based on the object point cloud data to obtain the target pose sequence corresponding to the target object.

[0070] In practical applications, human pose estimation is performed on the target object in the initial video to obtain a set of SMPL-X human parameter sequences that vary over time. These SMPL-X human parameter sequences describe the body pose and motion state of the target object in each frame of the initial video, providing a basis for subsequent motion control of the 3D human figure. When mapping the initial pose sequence to the reconstructed scene corresponding to the background point cloud data based on the object point cloud data, the object point cloud data (i.e., human point cloud data) can be used as geometric anchor points. The similarity transformation parameters from the SMPL-X human model to the world coordinate system corresponding to the SMPL-X human parameter sequence can be calculated. These similarity transformation parameters may include scale parameters, rotation parameters, and translation parameters. Based on these similarity transformation parameters, the SMPL-X human parameter sequence is mapped to the reconstructed scene corresponding to the background point cloud data, ensuring that the human motion and background space are consistent, thus obtaining the target pose sequence after the human body and scene are aligned. See also Figure 3 , Figure 3 A schematic diagram illustrating the alignment of a human body and a scene in a video generation method according to an embodiment of this specification is shown.

[0071] In practice, when using the object point cloud data (i.e. human point cloud data) as geometric anchor points to calculate the similarity transformation parameters from the SMPL-X human model to the world coordinate system corresponding to the SMPL-X human parameter sequence, the initial pose of the SMPL-X human model can be obtained by initial estimation based on the object point cloud data. The initial pose, shape, and initialized global similarity transformation parameters of the SMPL-X human model are then iteratively optimized based on a preset energy function until a precisely aligned SMPL-X human model and its similarity transformation parameters in the world coordinate system are output.

[0072] In summary, by using object point cloud data as geometric anchor points to align the human body with the scene, the scale uncertainty in the 3D scene reconstruction process based on the initial video is avoided, ensuring that the subsequently generated human figure can be stably placed in the reconstructed 3D scene.

[0073] In one optional embodiment of this specification, generating the 3D object data corresponding to the target object based on the initial video and the initial image includes: The target video frame corresponding to the target object is determined from multiple candidate video frames corresponding to the initial video; Based on the target video frame and the initial image, generate two-dimensional object data corresponding to the target object; The two-dimensional object data is transformed to obtain the three-dimensional object data corresponding to the target object.

[0074] In this context, multiple candidate video frames can be understood as each video frame in the initial video, and the target video frame can be understood as the candidate video frame that meets the image clarity criteria. Meeting the image clarity criteria can be understood as the target object contained in the candidate video frame satisfying image clarity conditions; for example, the human body of the target object in the candidate video frame may have good visibility and minimal occlusion. Two-dimensional object data can be understood as a two-dimensional image containing a target object wearing related objects. This two-dimensional object data can serve as a reference image to demonstrate the appearance effect of the target object wearing the related objects.

[0075] Based on this, after obtaining the aligned target pose sequence, 3D image data of the target object wearing associated objects can be further constructed. Keyframes with good human visibility and minimal occlusion of the target object can be selected from multiple candidate video frames corresponding to the initial video as the target video frame. The target object in this target video frame is then fused with the associated object in the initial image to generate a 2D image of the target object wearing associated objects. This 2D image is then dimensionally transformed into 3D object data.

[0076] In practical applications, 3D object data can be based on a 3D human figure using a 3D Gaussian representation. A 3D Gaussian representation consists of a large number of Gaussian units with attributes of position, scale, rotation direction, transparency, and color. This better preserves the target object's clothing texture, color, outline, and human information, providing superior appearance representation and facilitating subsequent video generation from multiple perspectives. For example, in the field of try-on video generation, the person in the target video frame and the clothing to be tried on in the initial image can be fused to generate a 2D image of the person wearing the clothing, which can then be converted into a 3D Gaussian representation of the dressed human figure.

[0077] In summary, by constructing 3D object data of the target object wearing related objects, it is easier to perform multi-view rendering in the future.

[0078] In one optional embodiment of this specification, when generating dynamic object data corresponding to the target object based on the target pose sequence and the 3D object data, the target pose sequence can be used to drive the 3D object data to undergo continuous deformation. See [link to documentation]. Figure 4 , Figure 4The diagram illustrates the generation of dynamic object data in a video generation method according to an embodiment of this specification. A linear blending skinning method can be used to transform a standard-pose 3D Gaussian human figure (i.e., 3D object data) to the human pose (i.e., target pose sequence) corresponding to each video frame of the initial video, thereby obtaining time-varying dynamic object data.

[0079] When generating intermediate object data corresponding to the target object based on the background point cloud data corresponding to the initial video and the dynamic object data, the dynamic object data and the background point cloud data can be placed in a unified world coordinate system to form a renderable 4D proxy (i.e., intermediate object data). This renderable 4D proxy can be rendered according to the initial acquisition trajectory corresponding to the initial video, or according to a new target acquisition trajectory specified by the user, such as surround, zoom in, zoom out, left shift, right shift, top view, or bottom view acquisition trajectories. Through this 4D proxy, rendering results such as intermediate video and intermediate mask information under the target acquisition trajectory can be obtained, thereby providing stable information such as human body structure, clothing position, background relationships, and image acquisition device motion information.

[0080] Step 206: Based on the target acquisition trajectory, render the intermediate object data to obtain the intermediate video corresponding to the target object.

[0081] Specifically, the generated renderable 4D proxy can be rendered based on the target acquisition trajectory, and the obtained rendering result is the intermediate video corresponding to the target object.

[0082] Step 208: Input the initial video, the initial image, the target acquisition trajectory, and the intermediate video into the video generation model to obtain the target video corresponding to the target object, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0083] The video generation model can be used to process video generation tasks. Such models may include deep learning models, neural network models, large language models, etc., but this specification does not limit the specific implementation. Optionally, the video generation model can be implemented based on a diffusion model.

[0084] For example, an initial video, an initial image, a target acquisition trajectory, and an intermediate video can be input into a video generation model. The model is then processed to obtain a target video corresponding to the target object. This target video can be generated based on the target acquisition trajectory and can display the target object of the wearable associated object according to the target viewpoint and target position corresponding to the target acquisition trajectory.

[0085] Furthermore, the video generation model includes a reference branch and a generation branch; The step of inputting the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain the target video corresponding to the target object includes: The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model. Based on the reference branch, the initial video and the initial image are processed to obtain reference information. Based on the generated branch, the reference information, the intermediate video, and the target acquisition trajectory are processed to obtain the target video corresponding to the target object.

[0086] The reference branch can be used to extract relevant information from the initial video and initial image as reference information, which can be used to assist in the generation of the target video. The generation branch can be used to generate the target video based on the target acquisition trajectory.

[0087] Based on this, after inputting the initial video, initial image, target acquisition trajectory, and intermediate video into the video generation model, information can be extracted from the initial video and initial image based on the reference branch to obtain reference information. This reference information is then injected into the generation branch. Based on the generation branch and the reference information, the intermediate video and target acquisition trajectory are processed to obtain the target video corresponding to the target object.

[0088] In practical applications, the reference branch can be used to extract information such as the appearance of people, clothing textures, and background details from the initial video and / or initial image. The generation branch can be used to generate the target video under the target acquisition trajectory. This video generation model can include a cross-view reference adapter, which can inject the reference information extracted from the reference branch into the generation branch through an attention mechanism. This allows the target video generated by the video generation model to maintain the consistency of clothing color, pattern, material, outline, and person identity under the new viewpoint (i.e., the new viewpoint corresponding to the target acquisition trajectory). This effectively alleviates the problems of clothing texture loss, identity drift, and insufficient background details caused by the difference between the target acquisition trajectory and the initial acquisition trajectory, and improves the ability to preserve the person's identity and clothing details under the new viewpoint.

[0089] Further, the step of processing the initial video and the initial image based on the reference branch to obtain reference information includes: Based on the reference branch, the object to be replaced in the initial video corresponding to the target object is masked to obtain reference mask information; Semantic extraction is performed on the initial image and the initial video to obtain reference image information and reference text information. The reference image information is obtained based on the initial image, and the reference text information is used to describe the associated object information corresponding to the associated object and the scene information corresponding to the initial video. The reference mask information, the reference image information, and the reference text information are used as the reference information.

[0090] The target object and the associated object to be replaced can be understood as the associated object that the target object in the initial video is already wearing and needs to be replaced. For example, if the person in the initial video is wearing a white shirt and wants to try on a yellow shirt, then the associated object to be replaced is the white shirt worn by the person in the initial video. In the final target video, the white shirt will be replaced with the yellow shirt that the person wants to try on.

[0091] Reference image information can be understood as related object information extracted from the initial image. This related object information may include color, material, pattern, and style information of the related objects. Reference text information may include scene information extracted from the initial video and related object information extracted from the initial image. The scene information may be a semantic description of the background and scene in the initial video, and the related object information may be a semantic description of the related objects.

[0092] Based on this, in the reference branch of the video generation model, the associated objects already worn by the target object in the initial video that need to be replaced can be masked to obtain reference mask information. Furthermore, semantic extraction can be performed on the initial image based on the image encoder to obtain reference image information, and semantic extraction can be performed on the initial video and initial image based on the text encoder to obtain reference text information. Using the reference mask information, reference image information, and reference text information as reference information facilitates subsequent injection into the generation branch.

[0093] In practical applications, multimodal semantic conditions can be introduced into the reference branch to assist in the video generation process of the generation branch. The image encoder can be used to extract color, material, pattern and style information from the initial image (such as a clothing image) as reference image information. The text encoder can be used to extract high-level semantic descriptions of related objects (such as clothing) and scenes from the initial video (which can be the target video frame of the initial video) and the initial image as reference text information. This reference image information and reference text information can be injected into the generation branch, making the generated target video more stable in terms of related object semantics, scene content and overall appearance.

[0094] In one optional embodiment of this specification, the video generation model includes a diffusion unit; The step of processing the reference information, the intermediate video, and the target acquisition trajectory based on the generated branch to obtain the target video corresponding to the target object includes: Based on the generated branch, the intermediate video is masked according to the target acquisition trajectory to obtain intermediate mask information, wherein the intermediate mask information is used to characterize the background information of the viewpoint and position corresponding to the target acquisition trajectory in the intermediate video; Based on the diffusion unit, the intermediate mask information, the reference mask information, the reference image information, and the reference text information are processed to obtain the target video corresponding to the target object.

[0095] In practical applications, a diffusion unit can include multiple DIT blocks, and each DIT block can include a self-attention layer, a cross-attention layer, and a feedforward network. The feedforward network can be understood as a fully connected layer.

[0096] In the generation branch, based on the target acquisition trajectory, the portion of the intermediate video excluding the background information corresponding to the viewpoint and position of the target acquisition trajectory can be masked to obtain intermediate mask information. Understandably, the viewpoint and position corresponding to the target acquisition trajectory are not present in the initial acquisition trajectory, so the background information corresponding to these viewpoints and positions in the intermediate video needs to be supplemented. This intermediate mask information is used to indicate valid human and clothing areas, i.e., valid target and associated object areas.

[0097] The video generation model also includes a video encoder, which can input intermediate video to obtain rendering prior features. Intermediate mask information, rendering prior features, and latent noise variables (i.e., target noise) can be fused and input into a diffusion unit for progressive denoising to generate the target video. By introducing this rendering prior, explicit constraints can be applied to the human pose, clothing position, background spatial relationships, and target acquisition trajectory of the target object during the target video generation process, avoiding problems such as human structure drift, clothing boundary jitter, and inaccurate image acquisition device motion.

[0098] Furthermore, reference mask information can be injected into the self-attention layer of the diffusion unit via a self-attention mechanism, while reference image and reference text information can be injected into the cross-attention layer of the diffusion unit via a cross-attention mechanism. The data is then processed through a feedforward network and passed through multiple DIT blocks to finally obtain the target video.

[0099] See Figure 5 , Figure 5 A schematic diagram of a reference branch and a generation branch is shown in a video generation method according to an embodiment of this specification. Figure 5As shown, in the reference branch, the cross-view reference adapter can include multiple adapter blocks. Figure 5 The diagram illustrates the l-th adapter block, the (l+1)-th adapter block, and the i-th DIT block, where Q is the query, K is the key, V is the value, O is the fused feature obtained after self-attention or cross-attention processing, Fl can represent the input of the l-th adapter, and Fi can represent the input of the i-th DIT block. The features output by the l-th adapter block can be input to the self-attention layer in the i-th DIT block. In practical applications, the number of adapter blocks can be less than the number of DIT blocks.

[0100] For one of the adapter blocks, the input reference information can be processed by a self-attention mechanism to obtain the reference fusion feature corresponding to the reference information. The reference fusion feature is then input into the next adapter block for self-attention processing. The reference fusion feature is then injected into one of the DIT blocks in the generation branch. In the DIT block, the reference fusion feature and intermediate mask information are processed by a self-attention mechanism to obtain intermediate fusion features. After injecting reference image information and reference text information, the intermediate fusion feature, reference image information, and reference text information are then processed by a cross-attention mechanism. After passing through the feedforward network, the information is input into the next DIT block, and so on until the target video is obtained.

[0101] In the generation branch, the input features Fi (i.e., rendering priors) of the DIT block are linearly projected to generate Qi, Ki, and Vi. Then, through a self-attention mechanism, global relationships are modeled on the input itself, outputting the intermediate fusion feature On. This preserves the powerful image generation prior knowledge of the pre-trained model, ensuring basic image quality and semantic reasonableness.

[0102] In the reference branch, the reference information Fl is input into the adapter block l. After self-attention processing, key information in the conditions is extracted step by step, and the reference fusion feature Ol is output. The reference fusion feature Ol is then injected into the generation branch. In the generation branch, the intermediate fusion feature On and the reference fusion feature Ol are further fused and then processed by the subsequent cross-attention mechanism.

[0103] In one optional embodiment of this specification, the video generation model further includes an encoder, a decoder, a segmentation unit, an image encoder, and a text encoder; Before processing the intermediate mask information, the reference mask information, the reference image information, and the reference text information based on the diffusion unit to obtain the target video corresponding to the target object, the method further includes: Based on the encoder, the intermediate mask information is encoded to obtain intermediate mask features; Based on the block unit, target noise is added to the intermediate mask features and the blocks are processed to obtain multiple intermediate mask feature blocks; Based on the encoder, the reference mask information is encoded to obtain reference mask features; Based on the block unit, the reference mask features are divided into blocks to obtain multiple reference mask feature blocks; Based on the image encoder, the reference image information is encoded to obtain reference image features; Based on the text encoder, the reference text information is encoded to obtain reference text features; The process of processing the intermediate mask information, the reference mask information, the reference image information, and the reference text information based on the diffusion unit to obtain the target video corresponding to the target object includes: Based on the diffusion unit, attention mechanism processing is performed on the plurality of intermediate mask feature blocks, the plurality of reference mask feature blocks, the reference image features, and the reference text features to obtain target fusion features; Based on the segmentation unit and the decoder, the target fusion features are decoded to obtain the target video corresponding to the target object.

[0104] Here, the encoder can be understood as a video encoder, and the target noise can be understood as random noise, such as Gaussian noise.

[0105] For example, in the generation branch, the video encoder can encode the intermediate video and intermediate mask information to obtain intermediate mask features (which may include the rendering prior features mentioned above). Based on the block unit, target noise is added to the intermediate mask features and the blocks are processed to obtain multiple intermediate mask feature blocks. These multiple intermediate mask feature blocks are used as rendering prior input diffusion units.

[0106] In the reference branch, the video encoder encodes the masked video (the video obtained by masking the initial video) and reference mask information to obtain reference mask features. These features are then divided into blocks based on the block unit, resulting in multiple reference mask feature blocks. These multiple reference mask feature blocks are input into the cross-view reference adapter. The cross-view reference adapter processes these blocks using a self-attention mechanism to obtain reference fusion features, which are then injected into the self-attention layer of the diffusion unit. In this self-attention layer, the reference fusion features and multiple intermediate mask feature blocks are processed using a self-attention mechanism to obtain intermediate fusion features, which are then input into the cross-attention layer.

[0107] Furthermore, in the reference branch, reference image information can be extracted from the initial image using an image encoder and then encoded to obtain reference image features. Similarly, reference text information can be extracted from the initial video and initial image using a text encoder and then encoded to obtain reference text features. These reference image and text features are then input into the cross-attention layer of the diffusion unit for processing. In the cross-attention layer, the intermediate fusion features, reference image features, and reference text features are processed using a cross-attention mechanism to obtain the target fusion features. Then, the target fusion features are subjected to dimensionality upscaling, non-linear activation, and dimensionality reduction based on a feedforward network to obtain the processed target fusion features. Finally, the processed target fusion features are decoded using a block unit and a decoder to obtain the target video corresponding to the target object.

[0108] In summary, by combining the above-mentioned reference branches and generation branches, the target video is generated, and the final generated target video can maintain the texture and shape of the associated objects, maintain the continuity of the target object's human movements, and conform to the target acquisition trajectory specified by the user.

[0109] Furthermore, in the video generation method provided in the embodiments of this specification, in order to obtain a video generation model with video generation capabilities, the video generation model to be trained can be trained until a fully trained video generation model is obtained. Based on this, the method further includes: The initial video of the sample containing the sample object, the initial image of the sample containing the sample-associated object, and the sample acquisition trajectory are determined, wherein the sample-associated object is the object to be worn by the sample object; Based on the initial sample video and the initial sample image, intermediate sample object data corresponding to the sample object is generated, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and temporal dimension corresponding to the initial sample video; Based on the sample acquisition trajectory, the intermediate object data of the sample is rendered to obtain the intermediate video of the sample object; The video generation model to be trained is obtained by using the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video.

[0110] For example, during the training of the video generation model, the initial sample video containing the sample object, the initial sample image containing the sample associated object, and the sample acquisition trajectory can be obtained as training data. Based on the initial sample video and the initial sample image, intermediate sample object data corresponding to the sample object is generated, and the intermediate sample object data is rendered based on the sample acquisition trajectory to obtain the intermediate sample video corresponding to the sample object. The video generation model to be trained is trained based on the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video until a video generation model that meets the training stopping condition is obtained.

[0111] The training stopping condition may be that the number of training iterations reaches a preset threshold and / or the model loss value reaches a preset loss value threshold. The training method for the video generation model to be trained may be supervised training or self-supervised training, and the embodiments in this specification do not limit this.

[0112] In practical applications, the model parameters of the cross-view reference adapter, self-attention layer, and cross-attention layer in the video generation model to be trained can be adjusted.

[0113] Understandably, the process of generating intermediate sample object data, obtaining intermediate sample video, and the process of the video generation model to be trained processing the initial sample video, initial sample image, sample acquisition trajectory, and intermediate sample video are similar to those described above, and will not be repeated in the embodiments of this specification.

[0114] In summary, by training the video generation model, the video generation capability of the video generation model can be realized.

[0115] Furthermore, the sample acquisition trajectory includes a first sample acquisition trajectory and a second sample acquisition trajectory, wherein the first sample acquisition trajectory is the acquisition trajectory for capturing the initial video of the sample; The step of rendering the intermediate object data of the sample based on the sample acquisition trajectory to obtain the intermediate video of the sample object includes: Based on the first acquisition trajectory of the sample, the intermediate object data of the sample is rendered to obtain the first intermediate video of the sample corresponding to the sample object; Based on the second acquisition trajectory of the sample, the intermediate object data of the sample is rendered to obtain the second intermediate video of the sample corresponding to the sample object; The step of training the video generation model to be trained based on the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video to obtain the trained video generation model includes: Based on the initial video of the sample, the initial image of the sample, the first acquisition trajectory of the sample, and the first intermediate video of the sample, the video generation model to be trained is trained to obtain the intermediate video generation model; Based on the initial video of the sample, the initial image of the sample, the second acquisition trajectory of the sample, and the second intermediate video of the sample, the intermediate video generation model is fine-tuned to obtain a trained video generation model.

[0116] For example, the training process of the video generation model can be divided into two training phases. The first training phase is the pre-training phase, where the video generation model is trained based on the first acquisition trajectory of the initial sample video to obtain an intermediate video generation model. This intermediate model learns to naturally generate associated objects into the target video, ensuring consistency in the texture of the associated objects, the characters' movements, and the video's temporal sequence. In the first training phase, a progressive training method is adopted, moving from low-resolution initial sample videos to high-resolution initial sample videos, to improve training stability and the ability to preserve details of associated objects. That is, in the first training phase, the first initial sample video can be sampled first. After training based on the above process, the second initial sample video is sampled for further training, and so on. The resolution of the second initial sample video is higher than that of the first initial sample video, meaning that the resolution of the initial sample video in the next training round is higher than that of the initial sample video in the current training round.

[0117] The second training phase is the fine-tuning phase, which can fine-tune the intermediate video generation model based on the second sample acquisition trajectory, which is different from the first sample acquisition trajectory. A renderable 4D agent and the second sample acquisition trajectory are introduced. The intermediate sample videos rendered by the renderable 4D agent under different second sample acquisition trajectories are used as structural priors to train the intermediate video generation model. This allows the intermediate video generation model to learn to maintain the consistency of human body structure, deformation of associated objects, texture of associated objects and spatial relationship with the background from a new perspective.

[0118] Understandably, since it is difficult to collect real multi-view try-on videos on a large scale, a renderable 4D agent can be reconstructed from real monocular videos, and multi-view training samples under different sample second acquisition trajectories can be synthesized.

[0119] Furthermore, in the first training phase, the model parameters of the self-attention layer and cross-attention layer in the video generation model can be adjusted, and in the second training phase, the model parameters of the cross-view reference adapter, self-attention layer, and cross-attention layer in the video generation model can be adjusted.

[0120] In practical applications, taking virtual try-on video generation as an example, the first training stage is pre-training of the basic video virtual try-on model. Based on the pre-trained video generation model, training is performed on ordinary monocular video virtual try-on data, enabling the video generation model to learn to naturally generate the clothing to be tried on into the character video while maintaining consistency in clothing texture, task actions, and video temporal sequence. This stage adopts a progressive training method from low resolution to high resolution to improve training stability and the ability to preserve clothing details. The second training stage is fine-tuning of camera-controlled video virtual try-on. In this stage, a renderable 4D agent and sample second acquisition trajectory are introduced. The intermediate sample videos rendered by the renderable 4D agent under different sample second acquisition trajectories are used as structural priors to further train the video generation model, enabling the video generation model to learn to maintain consistency in human body structure, clothing deformation, clothing texture, and background spatial relationships from new perspectives.

[0121] For the trained video generation model, during the testing phase, the user can input a video of a person, an image of the target clothing, and a target acquisition trajectory. The model can reconstruct the background scene and human movements using the video, generating a 3D Gaussian human figure wearing the target clothing. A sequence of human movements then drives the movement of this 3D Gaussian human figure, constructing a renderable 4D proxy. Subsequently, the renderable 4D proxy can be rendered according to the target acquisition trajectory, obtaining the rendered intermediate video and intermediate mask information. The intermediate video, intermediate mask information, target clothing image, reference image (the reference image of the user wearing the target clothing, i.e., the aforementioned 2D object data), and semantic conditions are then input into the video generation model to generate a target video that conforms to the target acquisition trajectory. The final output target video maintains the texture and pattern of the target clothing, preserves the continuity of the person's movements, and strictly conforms to the user-specified target acquisition trajectory.

[0122] In summary, through the training in the two stages described above, the video generation model learns the ability to generate videos from new perspectives.

[0123] In summary, the video generation method described above generates a target video that can be displayed according to the user-specified target acquisition trajectory. It supports free-view fitting effects such as surround viewing, zooming in and out, and side / rear viewing. A renderable 4D proxy unifies the human body wearing the target clothing, human movements, background scene, and acquisition trajectory into a single spatiotemporal representation, providing stable human body structure, clothing position, background relationships, and motion constraints of the image acquisition device for subsequent video generation. Based on the renderable 4D proxy, the rendered intermediate video is used as a structural prior input to the video generation model, ensuring that the target video maintains consistency with the target acquisition trajectory, human movement, and background structure while generating a realistic feel. Through a cross-view reference adapter, information such as the person's identity, clothing texture, and background details from reference images or the initial video are injected into the generation process, improving the ability to preserve clothing color, patterns, materials, outlines, and person's identity from new perspectives.

[0124] In summary, the above method, when generating the target video of the target object wearing associated objects, can incorporate a target acquisition trajectory. After generating intermediate object data of the target object wearing associated objects based on the initial video and initial image, the intermediate object data is rendered based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object. Furthermore, the target video of the target object wearing associated objects is obtained by inputting the initial video, initial image, target acquisition trajectory, and intermediate video into the video generation model. Since the target video is generated based on the target acquisition trajectory, it can display multiple perspectives of the target object wearing associated objects based on the target acquisition trajectory, thereby achieving a better display of the try-on effect, ensuring the quality and stability of the target video generation, and further meeting the user's try-on needs and experience.

[0125] The following is in conjunction with the appendix Figure 6 Taking the video generation method provided in this specification as an example of its application in generating try-on videos, the video generation method will be further explained. Among other things, Figure 6 The present specification illustrates a process flowchart of a video generation method according to an embodiment, which specifically includes the following steps.

[0126] Step 602: Reconstruct the data based on the initial video (i.e., the input video) to obtain dynamic point cloud data (world coordinate system) and initial pose sequence (camera coordinate system).

[0127] Step 604: Using human point cloud data as geometric anchor points, map the initial pose sequence to the reconstructed scene corresponding to the background point cloud data to obtain the target pose sequence.

[0128] Step 606: Sample the initial video to obtain the target video frame (i.e., key frame), and combine the initial image containing the target clothing with the target video frame to generate a reference image (i.e., two-dimensional object data).

[0129] Step 608: Generate a virtual human figure (i.e., 3D object data) based on a normalized 3D Gaussian based image from the reference image.

[0130] Step 610: Drive the 3D object data to continuously deform using the target pose sequence to obtain dynamic object data that changes over time, and generate a renderable 4D proxy based on the dynamic object data and background point cloud data.

[0131] Step 612: Render the renderable 4D proxy based on the target acquisition trajectory to obtain the rendered video (i.e., intermediate video), and perform masking on the rendered video to obtain the rendering mask information (i.e., intermediate mask information).

[0132] Understandably, during the training phase of the video generation model, the renderable 4D proxy can also be rendered based on the initial acquisition trajectory to obtain the rendered video.

[0133] Step 614: Use the rendered video and rendering mask information as the rendering prior input video encoder and block unit (i.e. blocker contained in the block unit), and add target noise to the block unit to obtain multiple intermediate mask feature blocks.

[0134] Step 616: Perform masking processing on the initial video based on the reference image to obtain the masked video and mask information (i.e., reference mask information).

[0135] Step 618: Input the reference mask information and mask video into the video encoder and the block unit to obtain multiple reference mask feature blocks.

[0136] Step 620: Perform self-attention mechanism processing on multiple reference mask feature blocks based on cross-view reference adapter to obtain reference fusion features, and inject the reference fusion features into the self-attention layer in the DIT block.

[0137] Step 622: Input multiple intermediate mask feature blocks into the DIT block, process the multiple intermediate mask feature blocks and reference fused features based on the self-attention layer using the self-attention mechanism to obtain intermediate fused features, and input the intermediate fused features into the cross-attention layer.

[0138] Step 624: Extract reference image information and reference text information from the initial image and initial video respectively based on the image encoder and text encoder, and encode them to obtain reference image features and reference text features. Input the reference image features and reference text features into the cross attention layer in the DIT block.

[0139] The reference image information and reference text information can be used as multimodal semantic conditions. For example, the reference text information could be "a model is walking in an orange down jacket, the background features are city streets, natural lighting, outdoor scene...".

[0140] Step 626: Based on the cross-attention layer, the intermediate fusion features, reference image features, and reference text features are processed by the cross-attention mechanism to obtain the target fusion features.

[0141] Step 628: Perform dimensionality upscaling, nonlinear activation, and dimensionality reduction on the target fusion features based on the feedforward network to obtain the processed target fusion features.

[0142] Step 630: Based on the block unit (i.e. the deblocker contained in the block unit) and the decoder, decode the processed target fusion features to obtain the target video (i.e. the fitting video) corresponding to the target object.

[0143] Understandably, after performing the above processing based on the first DIT block, the processing result is input into the second DIT block and the above processing is repeated until the final target fusion feature is obtained.

[0144] In the above method, when generating the target video of the target object wearing associated objects, a target acquisition trajectory can be added. After generating intermediate object data of the target object wearing associated objects based on the initial video and initial image, the intermediate object data is rendered based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object. Furthermore, the target video of the target object wearing associated objects is obtained by inputting the initial video, initial image, target acquisition trajectory, and intermediate video into the video generation model. The target video is generated based on the target acquisition trajectory, and multiple perspectives of the target object wearing associated objects can be displayed in the target video according to the target acquisition trajectory, thereby achieving a better display of the try-on effect, ensuring the quality and stability of the target video generation, and further meeting the user's try-on needs and try-on experience.

[0145] Corresponding to the above method embodiments, this specification also provides embodiments of a video generation apparatus. Figure 7 A schematic diagram of a video generation apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The first determining module 702 is configured to determine an initial video containing a target object, an initial image containing an associated object, and a target acquisition trajectory, wherein the associated object is the object to be worn by the target object; The first generation module 704 is configured to generate intermediate object data corresponding to the target object based on the initial video and the initial image, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video. The first rendering module 706 is configured to render the intermediate object data based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object; The first input module 708 is configured to input the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain a target video corresponding to the target object, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0146] In an optional embodiment, the first generation module 704 is further configured to: The target object in the initial video is subjected to object pose extraction to obtain the target pose sequence corresponding to the target object; Based on the initial video and the initial image, generate three-dimensional object data corresponding to the target object, wherein the three-dimensional object data is used to represent a three-dimensional virtual image of the target object wearing the associated object; Based on the target pose sequence and the 3D object data, generate dynamic object data corresponding to the target object; Based on the background point cloud data corresponding to the initial video and the dynamic object data, intermediate object data corresponding to the target object is generated.

[0147] In an optional embodiment, the first generation module 704 is further configured to: Perform 3D scene reconstruction on the initial video to obtain dynamic point cloud data corresponding to the initial video; The dynamic point cloud data is segmented to obtain the background point cloud data corresponding to the initial video and the object point cloud data corresponding to the target object. Perform object pose estimation on the target object in the initial video to obtain the initial pose sequence corresponding to the target object; Based on the object point cloud data, the initial pose sequence is mapped to the reconstructed scene corresponding to the background point cloud data to obtain the target pose sequence corresponding to the target object.

[0148] In an optional embodiment, the first generation module 704 is further configured to: The target video frame corresponding to the target object is determined from multiple candidate video frames corresponding to the initial video; Based on the target video frame and the initial image, generate two-dimensional object data corresponding to the target object; The two-dimensional object data is transformed to obtain the three-dimensional object data corresponding to the target object.

[0149] In one optional embodiment, the video generation model includes a reference branch and a generation branch; The first input module 708 is further configured as follows: The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model. Based on the reference branch, the initial video and the initial image are processed to obtain reference information. Based on the generated branch, the reference information, the intermediate video, and the target acquisition trajectory are processed to obtain the target video corresponding to the target object.

[0150] In an optional embodiment, the first input module 708 is further configured to: Based on the reference branch, the object to be replaced in the initial video corresponding to the target object is masked to obtain reference mask information; Semantic extraction is performed on the initial image and the initial video to obtain reference image information and reference text information. The reference image information is obtained based on the initial image, and the reference text information is used to describe the associated object information corresponding to the associated object and the scene information corresponding to the initial video. The reference mask information, the reference image information, and the reference text information are used as the reference information.

[0151] In one optional embodiment, the video generation model includes a diffusion unit; The first input module 708 is further configured as follows: Based on the generated branch, the intermediate video is masked according to the target acquisition trajectory to obtain intermediate mask information, wherein the intermediate mask information is used to characterize the background information of the viewpoint and position corresponding to the target acquisition trajectory in the intermediate video; Based on the diffusion unit, the intermediate mask information, the reference mask information, the reference image information, and the reference text information are processed to obtain the target video corresponding to the target object.

[0152] In one optional embodiment, the video generation model further includes an encoder, a decoder, a chunking unit, an image encoder, and a text encoder; The first input module 708 is further configured as follows: Based on the encoder, the intermediate mask information is encoded to obtain intermediate mask features; Based on the block unit, target noise is added to the intermediate mask features and the blocks are processed to obtain multiple intermediate mask feature blocks; Based on the encoder, the reference mask information is encoded to obtain reference mask features; Based on the block unit, the reference mask features are divided into blocks to obtain multiple reference mask feature blocks; Based on the image encoder, the reference image information is encoded to obtain reference image features; Based on the text encoder, the reference text information is encoded to obtain reference text features; Based on the diffusion unit, attention mechanism processing is performed on the plurality of intermediate mask feature blocks, the plurality of reference mask feature blocks, the reference image features, and the reference text features to obtain target fusion features; Based on the segmentation unit and the decoder, the target fusion features are decoded to obtain the target video corresponding to the target object.

[0153] In an optional embodiment, the device further includes a training module configured to: The initial video of the sample containing the sample object, the initial image of the sample containing the sample-associated object, and the sample acquisition trajectory are determined, wherein the sample-associated object is the object to be worn by the sample object; Based on the initial sample video and the initial sample image, intermediate sample object data corresponding to the sample object is generated, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and temporal dimension corresponding to the initial sample video; Based on the sample acquisition trajectory, the intermediate object data of the sample is rendered to obtain the intermediate video of the sample object; The video generation model to be trained is obtained by using the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video.

[0154] In one optional embodiment, the sample acquisition trajectory includes a first sample acquisition trajectory and a second sample acquisition trajectory, wherein the first sample acquisition trajectory is the acquisition trajectory for capturing the initial video of the sample; The training module is further configured as follows: Based on the first acquisition trajectory of the sample, the intermediate object data of the sample is rendered to obtain the first intermediate video of the sample corresponding to the sample object; Based on the second acquisition trajectory of the sample, the intermediate object data of the sample is rendered to obtain the second intermediate video of the sample corresponding to the sample object; Based on the initial video of the sample, the initial image of the sample, the first acquisition trajectory of the sample, and the first intermediate video of the sample, the video generation model to be trained is trained to obtain the intermediate video generation model; Based on the initial video of the sample, the initial image of the sample, the second acquisition trajectory of the sample, and the second intermediate video of the sample, the intermediate video generation model is fine-tuned to obtain a trained video generation model.

[0155] In summary, in the above-described device, when generating a target video of a target object wearing an associated object, a target acquisition trajectory can be added. After generating intermediate object data of the target object wearing associated object based on the initial video and initial image, the intermediate object data is rendered based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object. Furthermore, the target video of the target object wearing associated object is obtained by inputting the initial video, initial image, target acquisition trajectory, and intermediate video into the video generation model. Since the target video is generated based on the target acquisition trajectory, multiple perspectives of the target object wearing associated object can be displayed in the target video according to the target acquisition trajectory, thereby achieving a better display of the try-on effect, ensuring the quality and stability of the target video generation, and further meeting the user's try-on needs and try-on experience.

[0156] The above is an illustrative scheme of a video generation apparatus according to this embodiment. It should be noted that the technical solution of this video generation apparatus and the technical solution of the video generation method described above belong to the same concept. For details not described in detail in the technical solution of the video generation apparatus, please refer to the description of the technical solution of the video generation method described above.

[0157] Corresponding to the above method embodiments, this specification also provides a video generation model training method, see [link to documentation]. Figure 8 , Figure 8 A flowchart of a video generation model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0158] Step 802: Determine the initial video of the sample containing the sample object, the initial image of the sample containing the sample-associated object, and the sample acquisition trajectory, wherein the sample-associated object is the object to be worn by the sample object; Step 804: Based on the initial sample video and the initial sample image, generate intermediate sample object data corresponding to the sample object, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial sample video; Step 806: Based on the sample acquisition trajectory, render the intermediate object data of the sample to obtain the intermediate video of the sample object; Step 808: Train the video generation model to be trained based on the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video to obtain the trained video generation model.

[0159] The above is an illustrative scheme of a video generation model training method according to this embodiment. It should be noted that the technical solution of this video generation model training method belongs to the same concept as the technical solution of the video generation method described above. Details not described in detail in the technical solution of the video generation model training method can be found in the description of the technical solution of the video generation method described above.

[0160] Corresponding to the above method embodiments, this specification also provides a video generation model training device. Figure 9 A schematic diagram of a video generation model training apparatus according to one embodiment of this specification is shown. Figure 9 As shown, the device includes: The second determining module 902 is configured to determine the initial video of the sample containing the sample object, the initial image of the sample containing the sample associated object, and the sample acquisition trajectory, wherein the sample associated object is the object to be worn by the sample object; The second generation module 904 is configured to generate intermediate sample object data corresponding to the sample object based on the initial sample video and the initial sample image, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial sample video. The second rendering module 906 is configured to render the intermediate object data of the sample based on the sample acquisition trajectory, so as to obtain the intermediate video of the sample object. The training module 908 is configured to train the video generation model to be trained based on the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video, so as to obtain the trained video generation model.

[0161] The above is an illustrative scheme of a video generation model training device according to this embodiment. It should be noted that the technical solution of this video generation model training device and the technical solution of the video generation method described above belong to the same concept. For details not described in detail in the technical solution of the video generation model training device, please refer to the description of the technical solution of the video generation method described above.

[0162] Corresponding to the above method embodiments, this specification also provides another video generation method, see [link to documentation]. Figure 10 , Figure 10 A flowchart of another video generation method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0163] Step 1002: Receive a video generation request sent by the client and determine the target acquisition trajectory corresponding to the video generation request, wherein the video generation request carries an initial video containing the target object and an initial image containing an associated object, and the associated object is the object to be worn by the target object; Step 1004: Based on the initial video and the initial image, generate intermediate object data corresponding to the target object, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; Step 1006: Based on the target acquisition trajectory, render the intermediate object data to obtain the intermediate video corresponding to the target object; Step 1008: Input the initial video, the initial image, the target acquisition trajectory, and the intermediate video into the video generation model to obtain the target video corresponding to the target object, and send the target video to the client, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0164] In an optional embodiment, determining the target acquisition trajectory corresponding to the video generation request includes: The system sends multiple candidate acquisition trajectories corresponding to the video generation request to the client and receives the target acquisition trajectory sent by the client, wherein the target acquisition trajectory is any one of the multiple candidate acquisition trajectories; or Based on the viewpoint information and / or location information to be generated carried in the video generation request, the target acquisition trajectory corresponding to the video generation request is generated.

[0165] The above is an illustrative scheme of a video generation method according to this embodiment. It should be noted that the technical solution of this video generation method belongs to the same concept as the technical solution of the video generation method described above. For details not described in detail in the technical solution of the video generation method, please refer to the description of the technical solution of the video generation method described above.

[0166] Corresponding to the above method embodiments, this specification also provides another video generation apparatus. Figure 11 A schematic diagram of another video generation apparatus provided in one embodiment of this specification is shown. For example... Figure 11 As shown, the device includes: The receiving module 1102 is configured to receive a video generation request sent by a client and determine the target acquisition trajectory corresponding to the video generation request. The video generation request carries an initial video containing a target object and an initial image containing an associated object, wherein the associated object is the object to be worn by the target object. The third generation module 1104 is configured to generate intermediate object data corresponding to the target object based on the initial video and the initial image, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video. The third rendering module 1106 is configured to render the intermediate object data based on the target acquisition trajectory to obtain the intermediate video corresponding to the target object; The second input module 1108 is configured to input the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain a target video corresponding to the target object, and send the target video to the client, wherein the target video displays the target object wearing the associated object based on the target acquisition trajectory.

[0167] The above is an illustrative scheme of a video generation apparatus according to this embodiment. It should be noted that the technical solution of this video generation apparatus and the technical solution of the video generation method described above belong to the same concept. For details not described in detail in the technical solution of the video generation apparatus, please refer to the description of the technical solution of the video generation method described above.

[0168] Figure 12 A structural block diagram of a computing device 1200 according to an embodiment of this specification is shown. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0169] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0170] In one embodiment of this specification, the above-described components of the computing device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0171] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1200 can also be a mobile or stationary server.

[0172] The processor 1220 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0173] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above method.

[0174] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0175] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the method described above belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the method described above.

[0176] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0177] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of the above method.

[0178] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0179] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0180] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0181] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0182] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A video generation method, comprising: Determine the initial video containing the target object, the initial image containing the associated object, and the target acquisition trajectory, wherein the associated object is the object to be worn by the target object; Based on the initial video and the initial image, intermediate object data corresponding to the target object is generated, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; Based on the target acquisition trajectory, the intermediate object data is rendered to obtain the intermediate video corresponding to the target object; The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model to obtain the target video corresponding to the target object. The target video displays the target object wearing the associated object based on the target acquisition trajectory.

2. The method according to claim 1, wherein generating intermediate object data corresponding to the target object based on the initial video and the initial image comprises: The target object in the initial video is subjected to object pose extraction to obtain the target pose sequence corresponding to the target object; Based on the initial video and the initial image, generate three-dimensional object data corresponding to the target object, wherein the three-dimensional object data is used to represent a three-dimensional virtual image of the target object wearing the associated object; Based on the target pose sequence and the 3D object data, generate dynamic object data corresponding to the target object; Based on the background point cloud data corresponding to the initial video and the dynamic object data, intermediate object data corresponding to the target object is generated.

3. The method according to claim 2, wherein extracting the object pose of the target object in the initial video to obtain the target pose sequence corresponding to the target object includes: Perform 3D scene reconstruction on the initial video to obtain dynamic point cloud data corresponding to the initial video; The dynamic point cloud data is segmented to obtain the background point cloud data corresponding to the initial video and the object point cloud data corresponding to the target object. Perform object pose estimation on the target object in the initial video to obtain the initial pose sequence corresponding to the target object; Based on the object point cloud data, the initial pose sequence is mapped to the reconstructed scene corresponding to the background point cloud data to obtain the target pose sequence corresponding to the target object.

4. The method according to claim 2, wherein generating the three-dimensional object data corresponding to the target object based on the initial video and the initial image comprises: The target video frame corresponding to the target object is determined from multiple candidate video frames corresponding to the initial video; Based on the target video frame and the initial image, generate two-dimensional object data corresponding to the target object; The two-dimensional object data is transformed to obtain the three-dimensional object data corresponding to the target object.

5. The method according to any one of claims 1-4, wherein the video generation model comprises a reference branch and a generation branch; The step of inputting the initial video, the initial image, the target acquisition trajectory, and the intermediate video into a video generation model to obtain the target video corresponding to the target object includes: The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model. Based on the reference branch, the initial video and the initial image are processed to obtain reference information. Based on the generated branch, the reference information, the intermediate video, and the target acquisition trajectory are processed to obtain the target video corresponding to the target object.

6. The method according to claim 5, wherein processing the initial video and the initial image based on the reference branch to obtain reference information includes: Based on the reference branch, the object to be replaced in the initial video corresponding to the target object is masked to obtain reference mask information; Semantic extraction is performed on the initial image and the initial video to obtain reference image information and reference text information, wherein the reference image information is obtained based on the initial image, and the reference text information is used to describe the associated object information corresponding to the associated object and the scene information corresponding to the initial video; The reference mask information, the reference image information, and the reference text information are used as the reference information.

7. The method according to claim 6, wherein the video generation model includes a diffusion unit; The step of processing the reference information, the intermediate video, and the target acquisition trajectory based on the generated branch to obtain the target video corresponding to the target object includes: Based on the generated branch, the intermediate video is masked according to the target acquisition trajectory to obtain intermediate mask information, wherein the intermediate mask information is used to characterize the background information of the viewpoint and position corresponding to the target acquisition trajectory in the intermediate video; Based on the diffusion unit, the intermediate mask information, the reference mask information, the reference image information, and the reference text information are processed to obtain the target video corresponding to the target object.

8. The method according to claim 7, wherein the video generation model further comprises an encoder, a decoder, a segmentation unit, an image encoder, and a text encoder; Before processing the intermediate mask information, the reference mask information, the reference image information, and the reference text information based on the diffusion unit to obtain the target video corresponding to the target object, the method further includes: Based on the encoder, the intermediate mask information is encoded to obtain intermediate mask features; Based on the block unit, target noise is added to the intermediate mask features and the blocks are processed to obtain multiple intermediate mask feature blocks; Based on the encoder, the reference mask information is encoded to obtain reference mask features; Based on the block unit, the reference mask features are divided into blocks to obtain multiple reference mask feature blocks; Based on the image encoder, the reference image information is encoded to obtain reference image features; Based on the text encoder, the reference text information is encoded to obtain reference text features; The process of processing the intermediate mask information, the reference mask information, the reference image information, and the reference text information based on the diffusion unit to obtain the target video corresponding to the target object includes: Based on the diffusion unit, attention mechanism processing is performed on the plurality of intermediate mask feature blocks, the plurality of reference mask feature blocks, the reference image features, and the reference text features to obtain target fusion features; Based on the segmentation unit and the decoder, the target fusion features are decoded to obtain the target video corresponding to the target object.

9. The method according to any one of claims 1-4, further comprising: The initial video of the sample containing the sample object, the initial image of the sample containing the sample-associated object, and the sample acquisition trajectory are determined, wherein the sample-associated object is the object to be worn by the sample object; Based on the initial sample video and the initial sample image, intermediate sample object data corresponding to the sample object is generated, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and temporal dimension corresponding to the initial sample video; Based on the sample acquisition trajectory, the intermediate object data of the sample is rendered to obtain the intermediate video of the sample object; The video generation model to be trained is obtained by using the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video.

10. The method according to claim 9, wherein the sample acquisition trajectory includes a first sample acquisition trajectory and a second sample acquisition trajectory, and the first sample acquisition trajectory is the acquisition trajectory for capturing the initial video of the sample; The step of rendering the intermediate object data of the sample based on the sample acquisition trajectory to obtain the intermediate video of the sample corresponding to the sample object includes: Based on the first acquisition trajectory of the sample, the intermediate object data of the sample is rendered to obtain the first intermediate video of the sample corresponding to the sample object; Based on the second acquisition trajectory of the sample, the intermediate object data of the sample is rendered to obtain the second intermediate video of the sample corresponding to the sample object; The step of training the video generation model to be trained based on the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video to obtain the trained video generation model includes: Based on the initial video of the sample, the initial image of the sample, the first acquisition trajectory of the sample, and the first intermediate video of the sample, the video generation model to be trained is trained to obtain the intermediate video generation model; Based on the initial video of the sample, the initial image of the sample, the second acquisition trajectory of the sample, and the second intermediate video of the sample, the intermediate video generation model is fine-tuned to obtain a trained video generation model.

11. A method for training a video generation model, comprising: The initial video of the sample containing the sample object, the initial image of the sample containing the sample-associated object, and the sample acquisition trajectory are determined, wherein the sample-associated object is the object to be worn by the sample object; Based on the initial sample video and the initial sample image, intermediate sample object data corresponding to the sample object is generated, wherein the intermediate sample object data is used to characterize the motion changes of the sample object wearing the sample associated object in the three-dimensional spatial dimension and temporal dimension corresponding to the initial sample video; Based on the sample acquisition trajectory, the intermediate object data of the sample is rendered to obtain the intermediate video of the sample object; The video generation model to be trained is obtained by using the initial sample video, the initial sample image, the sample acquisition trajectory, and the intermediate sample video.

12. A video generation method, comprising: Receive a video generation request sent by the client and determine the target acquisition trajectory corresponding to the video generation request, wherein the video generation request carries an initial video containing the target object and an initial image containing an associated object, the associated object being the object to be worn by the target object; Based on the initial video and the initial image, intermediate object data corresponding to the target object is generated, wherein the intermediate object data is used to characterize the motion changes of the target object wearing the associated object in the three-dimensional spatial dimension and time dimension corresponding to the initial video; Based on the target acquisition trajectory, the intermediate object data is rendered to obtain the intermediate video corresponding to the target object; The initial video, the initial image, the target acquisition trajectory, and the intermediate video are input into the video generation model to obtain the target video corresponding to the target object, and the target video is sent to the client. The target video displays the target object wearing the associated object based on the target acquisition trajectory.

13. The method according to claim 12, wherein determining the target acquisition trajectory corresponding to the video generation request includes: The system sends multiple candidate acquisition trajectories corresponding to the video generation request to the client and receives the target acquisition trajectory sent by the client, wherein the target acquisition trajectory is any one of the multiple candidate acquisition trajectories; or Based on the viewpoint information and / or location information to be generated carried in the video generation request, the target acquisition trajectory corresponding to the video generation request is generated.

14. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

16. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images