Method and device for generating video, electronic equipment and computer program product

Abstract

The invention relates to a method and device for generating a video, electronic equipment and a computer program product. The method includes obtaining a visual marker for generating an image frame in a video. The method further includes obtaining a control marker for constraining position information of the object in the image frame. Further, the method includes generating an image frame in the video based on the visual marker and the control marker, where an object in the image frame satisfies the location information. In this way, the content of the generated image frame can be constrained by using the control mark, thereby improving the motion effect of the generated video, improving the degree of matching between the generated video and the user's requirements, and improving the user experience.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims priority to a patent application with the patent application number 202410132635.0 filed on January 30, 2024. The disclosure of this application is considered to be part of this patent application and is incorporated into this patent application by reference. Technical Field

[0003] This disclosure generally relates to the field of artificial intelligence, and more particularly, to methods, devices, electronic devices, and computer program products for generating videos. Background Art

[0004] Text-guided video generation is a technique that uses text descriptions to guide the generation of video content. In such a video generation task, the model receives text descriptions in the form of natural language and generates image frames corresponding to the text based on these descriptions, and then combines these image frames into a video. One of the key challenges of this task is to establish the association between the text description and the video content, including understanding the objects, actions, spatio-temporal relationships, etc. in the text description, and then converting this information into a series of image frames.

[0005] Motion control refers to controlling the movement of objects, scenes, and cameras in the generated video, for example, through text descriptions. For example, the text description may include information about the movement of an object or a person, so it is necessary to control the object or person in the generated video to move according to the text description. In the related art, machine learning models are usually used to implement motion control in video generation tasks. Summary of the Invention

[0006] In a first aspect of an embodiment of the present disclosure, a method for generating a video is provided. The method includes obtaining visual markers for generating image frames in the video. The method further includes obtaining control markers for constraining the position information of objects in the image frames. In addition, the method further includes generating image frames in the video based on the visual markers and the control markers, where the objects in the image frames satisfy the position information.

[0007] In a second aspect of an embodiment of the present disclosure, a device for generating a video is provided. The device includes a visual marker obtaining module configured to obtain visual markers for generating image frames in the video. The device further includes a control marker obtaining module configured to obtain control markers for constraining the position information of objects in the image frames. In addition, the device further includes a video image generating module configured to generate image frames in the video based on the visual markers and the control markers, where the objects in the image frames satisfy the position information.

[0008] In a third aspect of the embodiments of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement a method for generating a video. The method includes obtaining a visual marker for generating an image frame in the video. The method further includes obtaining a control marker for constraining the position information of an object in the image frame. Additionally, the method further includes generating an image frame in the video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.

[0009] In a fourth aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause the machine to implement a method for generating a video. The method includes obtaining a visual marker for generating an image frame in the video. The method further includes obtaining a control marker for constraining the position information of an object in the image frame. Additionally, the method further includes generating an image frame in the video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.

[0010] The Summary of the Invention section is provided to introduce a selection of concepts in a simplified form, which will be further described in the Detailed Description below. The Summary of the Invention section is not intended to identify the key features or main features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent. In the drawings, the same or similar reference numerals denote the same or similar elements, where:

[0012] Figure 1 A schematic diagram of an example environment in which multiple embodiments of the present disclosure can be implemented is shown;

[0013] Figure 2 A flowchart of a method for generating a video according to some embodiments of the present disclosure is shown;

[0014] Figure 3 A schematic diagram of an example architecture for generating a video according to some embodiments of the present disclosure is shown;

[0015] Figure 4 A schematic diagram of an example of generating training data from an existing video dataset according to some embodiments of the present disclosure is shown;

[0016] Figure 5A schematic diagram showing an example process of a self - alignment operation according to some embodiments of the present disclosure;

[0017] Figure 6 A flowchart showing an example process of a multi - stage training process according to some embodiments of the present disclosure;

[0018] Figure 7 A schematic diagram showing an example process of generating a video by generating a hard bounding box and expanding it into a soft bounding box in the case of providing a hard bounding box in an end frame according to some embodiments of the present disclosure;

[0019] Figure 8 A schematic diagram showing an example process of generating a video by generating a hard bounding box and expanding it into a soft bounding box in the case of providing a motion trajectory of an object according to some embodiments of the present disclosure;

[0020] Figure 9 A schematic diagram showing an example in which multiple bounding boxes are provided in a start frame and a bounding box near the right boundary is provided in an end frame according to some embodiments of the present disclosure;

[0021] Figure 10 A schematic diagram showing an example in which an object bounding box and a motion trajectory and a bounding box of another object are provided in a start frame and a bounding box of the other object is provided in an end frame according to some embodiments of the present disclosure;

[0022] Figure 11 A block diagram showing a device for generating a video according to some embodiments of the present disclosure; and

[0023] Figure 12 A block diagram showing a device capable of implementing multiple embodiments of the present disclosure. Detailed implementation manners

[0024] It can be understood that all data related to users involved in this technical solution should be obtained and used after obtaining user authorization. This means that in this technical solution, if it is necessary to use the personal information of users, before obtaining these data, the explicit consent and authorization of users are required. Otherwise, relevant data collection and use will not be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly observed in the process of data collection, use, and storage, and necessary technical means and measures should be taken to ensure the data security of users and ensure the safe use of data.

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not used to limit the protection scope of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "comprising" and its like should be understood as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects unless otherwise specified. There may also be other explicit and implicit definitions hereinafter.

[0027] With the development of video generation technology, some video generation models can generate videos based on text prompts or key image frames. As an example of a video generation model, the video diffusion model is an extension of the image diffusion model, which incorporates the architecture of the U-shaped network (U-Net) model in the image model and adds a temporal layer that helps generate multiple image frames. The text-to-video (T2V) diffusion model is generally the cornerstone of various forms of video generation models with constraints. In the text-to-video diffusion model, image frames can be created based on text descriptions, and then videos can be generated based on the text descriptions and pre-generated image frames. This approach enables the video generation model to use static images as references, thereby being able to focus on the dynamic aspects of video generation and thus improving the quality of the generated videos.

[0028] It should be understood that for the sake of convenience, some embodiments herein are described by taking the video diffusion model with a U-Net architecture as an example, but it is not intended to limit the specific architecture of the video generation model. The solution provided by the present disclosure can be applied to any video generation model that generates visual tokens and generates image frames based on the visual tokens.

[0029] In some usage scenarios, users hope to provide information about the movement mode for the objects in the generated video by inputting text descriptions. For example, the user can provide a reference image of a building and then input the text description "tilt the camera upward to show the top of the building". At this time, the user hopes that in the generated video, the camera gradually raises the lens from the perspective of shooting the ground and finally shoots the top of the building. However, in the related art, although it is possible to generate a video with high picture quality and slow camera movement based on the reference image and text description provided by the user, the model cannot well understand the user's requirements for the movement mode of the objects in the video, so the top of the building cannot be accurately shown in the generated video.

[0030] In addition, in some usage scenarios, when the user's requirements for the movement mode are very precise, it is difficult to accurately describe the desired picture in words. For example, when the user hopes that there are two little dogs running towards the camera in the video, one white little dog gets closer and closer to the camera and runs to the center of the picture, and at this time this white little dog fills one-third of the picture. At the same time, the other black little dog also gets closer and closer to the camera, but the direction it runs is towards the toy beside the camera, so it gradually deviates from the center of the picture and finally disappears from the right side of the picture. For ordinary users, it is very difficult to accurately describe such movement requirements, so the desired video cannot be generated.

[0031] For this reason, embodiments of the present disclosure provide a solution for generating a video. In this solution, the user can provide a bounding box for constraining the position and size of an object in the video. The video generation model can obtain a set of visual tokens for generating image frames and control tokens for the provided bounding box, and then generate image frames based on the visual tokens and the control tokens. In the generated image frames, the object in the video will appear within the bounding box.

[0032] In this way, the video generation model can use the control tokens of the bounding box to understand the position and size that the user expects the target object to move to, so that it can use the control tokens to constrain the content of the generated image frames. In this way, the user can simply and accurately express the desired movement mode of the object. In addition, compared with only providing text descriptions or reference images, this method can improve the matching degree between the generated video and the user's requirements, thereby improving the user experience.

[0033] Figure 1 The schematic diagram of an example environment 100 in which multiple embodiments of the present disclosure can be implemented is shown. As Figure 1As shown, environment 100 includes a video generation model 102, which is also referred to as a base model herein. It can be a video diffusion model that generates videos based on text descriptions or key image frames as described above, or other machine learning models that use neural network technology to generate videos. Environment 100 also includes a motion control module 104, which can be combined with the video generation model 102 in the form of a plug-in to enhance the motion control ability of the video generation model 102. As Figure 1 shown, the video generation model 102 can generate visual tokens 112-1, 112-2, …, 112-N (collectively referred to as visual tokens 112). The visual tokens 112 are a set of vectors generated based on information such as text descriptions, reference images, etc., and include information about the image to be generated.

[0034] As Figure 1 shown, environment 100 includes bounding boxes 106 and 108, which are used to constrain the position and size of an object in the image frame to be generated. In the embodiments of the present disclosure, the term "object" can be an independent object (e.g., a little dog), a part of an independent object (e.g., a human hand), or multiple objects combined together (e.g., a person riding a horse). In addition, the term "motion" can be the motion of an object relative to the camera (or lens), or the motion of the camera relative to the object. For example, when the bounding boxes 106 or 108 indicate the position and size of an object after motion, an object with autonomous motion ability (e.g., a little dog, a car, etc.) can move itself to the specified position and be presented as the specified size, while an object without autonomous motion ability (e.g., a rock, a building, etc.) can be presented at the specified position in the frame with the specified size by moving the camera. In some embodiments, bounding boxes such as bounding boxes 106 and 108 are rectangular boxes, and two types of bounding boxes can be implemented - hard bounding boxes and soft bounding boxes. A hard bounding box is used to specify a specific position and a specific size of an object, which means that in the generated image frame, the object is generated at the coordinates specified by the hard bounding box (e.g., the center coordinates of the bounding box), and the size of the object corresponds to the size of the hard bounding box. A soft bounding box is used to specify a position range and a size range of an object, which means that in the generated image frame, the object is generated within the range enclosed by the soft bounding box, and the size of the object does not exceed this range.

[0035] In environment 100, bounding box 106 is processed into control marker 116, and bounding box 108 is processed into control marker 118. Thus, control markers 116 and 118 respectively include motion control information of the objects corresponding to bounding boxes 106 and 108. Then, motion control module 104 can generate new visual markers 114-1, 114-2, …, 114-N (collectively referred to as visual markers 114) based on visual marker 112, control marker 116, and control marker 118. In this way, the motion control information provided by bounding boxes 106 and 108 can be included in visual markers 114. Then, video generation model 102 can generate image frame 120 based on visual markers 114. In image frame 120, two puppies move from the positions in the previous image frame 122 to the positions specified by bounding boxes 106 and 108, and the size of the white puppy corresponds to bounding box 106, and the size of the black puppy corresponds to bounding box 108. Then, multiple image frames such as image frames 120 and 122 can form video 124.

[0036] In this way, motion control module 104 can use control markers 116 and 118 of bounding boxes 106 and 108 to provide motion control information for video generation model 102, so as to be able to improve the motion effect of the generated video, increase the matching degree between the video and the user's requirements, and enhance the user experience.

[0037] Figure 2 The flowchart of method 200 for generating a video according to some embodiments of the present disclosure is shown. As Figure 2 shown, at block 202, method 200 can obtain visual markers for generating image frames in the video. For example, in Figure 1 the environment 100 shown, motion control module 104 can obtain visual marker 112, which is generated by video generation model 102 based on information such as text descriptions or reference images. It contains the visual information of the image frame to be generated and is used to generate image frame 120.

[0038] At block 204, method 200 can obtain control markers for constraining the position information of the objects in the image frame. The position information can be information associated with positions such as bounding boxes, contours, coordinate values, coordinate ranges, etc. For example, in Figure 1 the environment 100 shown, motion control module 104 can obtain control markers 116 and 118, which are generated based on bounding boxes 106 and 108. Thus, control markers 116 and 118 contain the motion control information of bounding boxes 106 and 108. Bounding boxes 106 and 108 can constrain the positions and sizes of the objects in the image frame to be generated, thus playing a role in controlling the motion of the objects.

[0039] At block 206, method 200 may generate an image frame in a video based on a visual marker and a control marker, where the object in the image frame satisfies the position information. For example, in Figure 1 the illustrated environment 100, the motion control module 102 may generate a visual marker 114 based on the visual marker 112, the control marker 116, and the control marker 118, and then the video generation model 102 may generate an image frame 120 based on the visual marker 114. In the image frame 120, two puppies appear within the ranges constrained by the bounding boxes 106 and 108. Depending on different embodiments, the positions of the two puppies may exactly correspond to the positions of the bounding boxes 106 and 108, or the positions of the two puppies may also be within the ranges constrained by the bounding boxes 106 and 108.

[0040] In this way, method 200 can utilize the control marker generated based on the bounding box to understand the position where the user expects the target object to move to, so as to be able to use the control marker to constrain the content of the generated image frame. In this way, the user can simply and accurately express the expected object movement manner, and compared with only providing a text description or a reference image, this manner can improve the movement effect of the generated video, improve the matching degree between the generated video and the user's requirements, and thus can improve the user experience.

[0041] In some embodiments, the position information is a bounding box, and in order to obtain the control marker of the bounding box, the coordinates of the bounding box in the image frame may be determined, and the control marker may be generated based on the coordinates. In some embodiments, supporting the use of multiple bounding boxes to constrain the movement manners of multiple objects is supported. In these embodiments, an object identifier for the bounding box may be generated based on the color of the bounding box, and the control marker may be generated based on the coordinates and the object identifier. In some embodiments, both a hard bounding box (also referred to as the first type of bounding box in this article) and a soft bounding box (also referred to as the second type of bounding box in this article) may be supported. In these embodiments, the type of the bounding box may be determined, where the type of the bounding box includes a hard bounding box that constrains the specific position and specific size of the object to be generated and a soft bounding box that constrains the position range and size range of the object to be generated, and the control marker may be generated based on the coordinates, the object identifier, and the type.

[0042] In some embodiments, in response to the type of the bounding box being a hard bounding box, the center position of the object coincides with the center position of the bounding box, and the size of the object corresponds to the size of the bounding box. In some embodiments, in response to the type of the bounding box being a soft bounding box, the center position of the object is within the bounding box, and the size of the object does not exceed the bounding box. In some embodiments, multiple embeddings can be generated based on coordinates, object identifiers, and types, and based on the multiple embeddings, control tokens can be generated by utilizing a multi-layer perceptron. In some embodiments, a second set of visual tokens can be generated based on a first set of visual tokens and control tokens, where the number of visual tokens in the first set of visual tokens is the same as the number of visual tokens in the second set of visual tokens.

[0043] Figure 3 FIG. 4 shows a schematic diagram of an example architecture 300 for generating a video according to some embodiments of the present disclosure. As Figure 3 shown, the architecture 300 includes a spatial self-attention layer 302, a multi-layer perceptron 308, a motion control module 318, and a spatial cross-attention layer 322. The spatial self-attention layer 302 and the spatial cross-attention layer 322 can be, for example, modules in a video diffusion model (e.g., Figure 1 the video generation model 102 in can be a video diffusion model) based on a three-dimensional (3D) U-Net architecture. The video diffusion model can iteratively predict noise vectors in a noisy video input, thereby gradually converting pure Gaussian noise into high-quality video frames. The 3D U-Net is composed of alternating convolutional blocks and attention blocks. Each block contains two components, namely, a spatial component that processes each image frame as a separate image, and a temporal component that facilitates information exchange between image frames. In each attention block, the spatial component typically includes a self-attention layer followed by a cross-attention layer, and the cross-attention layer is used to regulate the generation of the video based on text prompts. The motion control module is inserted between these two attention layers, enabling the model to manage motion control in video generation.

[0044] As Figure 3As shown, the architecture 300 inserts a motion control module 318 between the spatial self-attention layer 302 and the spatial cross-attention layer 322 of the original video diffusion model. The spatial self-attention layer 302 receives frame-level visual tokens 304 and generates visual tokens 306-1, 306-2, …, 306-N (collectively referred to as visual tokens 306) based on the frame-level visual tokens 304. The motion control module 318 receives the visual tokens 306 and control tokens 316-1, 316-2, …, 316-N (collectively referred to as control tokens 316) as inputs and outputs visual tokens 320-1, 320-2, …, 320-N (collectively referred to as visual tokens 320). Among them, each control token in the control tokens 316 corresponds to a corresponding object (or bounding box). Since the control tokens 316 include the motion control information provided by the bounding box, the newly generated visual tokens 320 will also contain the motion control information provided by the bounding box. Then, the visual tokens 320 are input into the spatial cross-attention layer 322, and the spatial cross-attention layer 322 can generate frame-level visual tokens 326 based on the visual tokens 320 and text tokens 324-1, 324-2, …, 324-N (collectively referred to as text tokens 324). Then, the video diffusion model can generate an image frame based on the frame-level visual tokens 326. In order not to change the original structure of the spatial cross-attention layer 322, the number of visual tokens 306 can be kept the same as that of the visual tokens 320. In this way, the parameters of the original video diffusion model (including the spatial self-attention layer 302 and the spatial cross-attention layer 322) can be fixed during the training phase, and only the parameters of the motion control module 318 are adjusted, so as to avoid retraining caused by modifying the structure of the video diffusion model, thus saving costs and avoiding the decrease in the accuracy of the original video diffusion model caused by retraining.

[0045] In the architecture 300, if v is used to represent the frame-level visual tokens 304 of the image frame to be generated, h text is used to represent the sequence of text tokens 324, and h box is used to represent the sequence of control tokens 316, then the enhanced spatial attention block can be described by the following equations (1)-(3):

[0046] v = v + SelfAttn(v) (1)

[0047] v = v + TS(SelfAttn([v, h box )) (2)

[0048] v = v + CrossAttn(v, h text ) (3)

[0049] Where TS(·) represents a token selection operation that specifically considers visual tokens, SelfAttn represents the spatial self-attention layer 302, and CrossAttn represents the spatial cross-attention layer 322.

[0050] In the architecture 300, the number of control tokens 316 depends on the number of bounding boxes that the video generation model supports to coexist in an image frame, and the control tokens 316 correspond one-to-one with the bounding boxes. For example, if the video generation model only supports a bounding box for one object in the image frame, the number of control tokens 316 is 1; if the video generation model supports five bounding boxes for five objects in the image frame, the number of control tokens 316 is 5. If the video generation model supports providing five bounding boxes in an image frame, but only needs to control the movement of two objects (i.e., only two bounding boxes are provided) in the video to be generated, learnable specific tokens can be used to fill the vacant three control tokens. In the architecture 300, the text tokens 324 are not necessary, that is, if the user does not provide a text description of the video to be generated, learnable tokens can be used to fill the vacant text tokens.

[0051] As Figure 3 shown, to generate the control tokens 316, the coordinates 310 of the bounding box 328, the unique object identifier 312 for identifying the bounding box 328 (or the object corresponding to the bounding box 328), and the bounding box type 314 can be determined. Each control token 316 can be defined by the following equation (4):

[0052] t b = MLP(Fourier([b loc ,b id ,b flag )) (4)

[0053] Where b loc represents a 4D vector containing the upper-left and lower-right coordinates of the bounding box (i.e., the coordinates 310), and is normalized between 0 and 1. b id represents the object identifier 312, which is used to identify and link the bounding boxes between individual image frames. b flag represents the bounding box type 314. For example, 1 represents a hard bounding box, and 0 represents a soft bounding box. In addition, Fourier represents the Fourier embedding operation, and MLP represents the multi-layer perceptron. In this way, the multi-layer perceptron can be used to make the control tokens include higher-level and more abstract semantic features, thereby improving the performance of the motion control module 318 and the motion effect of the generated image frames.

[0054] In some embodiments, b id, where each object corresponds to a bounding box with a unique color, such that b id is a vector with 3D RGB values normalized to between 0 and 1. b loc , b id and b flag are concatenated into a vector and corresponding embeddings are generated via a Fourier embedding operation. Then, this embedding is input into a multi-layer perceptron 308, thereby generating a control token 316. By using RGB values to generate object identifiers, corresponding bounding boxes can be generated in the image frame based on the object identifiers during the training phase, thus facilitating the alignment between the generated bounding boxes and the ground truth bounding boxes and improving the effect of model training.

[0055] When using Fourier embedding to encode b loc , b id and b flag , it can be ensured that all input dimensions are scaled to between 0 and 1. For any given input x within this range, its Fourier embedding is defined by the following equation (5):

[0056] Fourier(x) = [cos(x·100 0 / 8 ),..., cos(x·100 7 / 8 ), sin(x·100 0 / 8 ),..., sin(x·100 7 / 8 )] (5)

[0057] In some embodiments, the Fourier embeddings of each input can be combined to generate an overall embedding with a dimension of 128. Then, these embeddings can be processed by a multi-layer perceptron. This multi-layer perceptron can have three hidden layers, each with a dimension of 512. Then, the output control token can be adjusted to match the dimension of the visual token (i.e., 1024).

[0058] It should be understood that although generating the control token 316 based on the coordinates 310 of the bounding box 328, the object identifier 312, and the bounding box type 314 is shown in the architecture 300, the object identifier 312 and the bounding box type 314 are not necessary in some embodiments. For example, in some embodiments, if only bounding boxes of a specific type (e.g., hard bounding boxes) are supported, the control token 316 can be generated only based on the coordinates 310. In some embodiments, if only multiple specific types of bounding boxes are supported, the control token 316 can be generated only based on the coordinates 310 and the object identifier 312.

[0059] In this way, the motion control module 318 can provide precise motion control information for the original video diffusion model, thereby improving the effect of the generated image frames and enabling the object to move according to the motion mode expected by the user. In addition, since the inserted motion control module 318 does not change the structure and parameters of the original video diffusion model, the architecture 300 can reuse the capabilities of the trained video diffusion model, thus improving the motion control of the objects in the video while ensuring the picture quality of the generated video.

[0060] In the training phase, to obtain the training dataset, training data that meets the conditions can be obtained from existing publicly accessible video datasets. For example, each video in the existing video dataset can be evaluated, and the embeddings of its starting frame and ending frame can be compared. If the cosine similarity between the embeddings of the starting frame and the ending frame is lower than a predetermined threshold, it indicates that obvious object motion or camera motion is exhibited in the video, and thus this data can be put into the dataset to form a selected dataset.

[0061] For the videos in the selected dataset, the starting frame of the video can be obtained, and an existing model can be used to generate a description of the content of the starting frame. Then, noun phrases (such as young people, white shirts, etc.) can be extracted from these descriptions and used as object prompts. Then, these object prompts can be used to identify the rectangular bounding boxes that enclose the object in the starting frame. Then, these bounding boxes can be tracked and propagated in all the image frames of the video, and thus a large number of objects enclosed by the bounding boxes can be obtained.

[0062] During the training process, the video can be randomly cropped according to a specific aspect ratio, and then all the bounding boxes are projected onto the cropped area. If a bounding box is completely outside the cropped area, such a bounding box can be projected as a line segment (or a bounding box approximating a line segment) along the boundary of the cropped area. Figure 4 A schematic diagram of an example 400 of generating training data from an existing video dataset according to some embodiments of the present disclosure is shown. As Figure 4 shown, the example 400 includes an image 402, and the image 402 includes identified bounding boxes 404, 406, and 408. Since the size of the image 402 is wider than the size required by the video generation model, the image 402 can be cropped according to the size required by the video generation model to obtain a cropped area 412. As Figure 4As shown, the bounding box 404 is completely within the cropping region 412, so no additional operations are required for it. A part of the bounding box 406 is outside the cropping region 412, so the bounding box 406 can be cropped to retain only the part within the cropping region 412, that is, the bounding box 416. In addition, the bounding box 408 is completely outside the cropping region 412, so it can be projected onto the bounding box 418 at the boundary of the cropping region 412. The bounding box 418 can be regarded as a line segment or a rectangular box with a very small width, and the height of the bounding box 418 is related to the height of the bounding box 408. In the training dataset, the bounding box 418 can represent an object entering from outside the image frame or an object moving from within the image frame to outside the image frame.

[0063] In this way, training samples that can be used to train the motion control module of the present disclosure can be generated from the existing training dataset, solving the problem of lack of training data corresponding to the method for generating videos provided by the embodiments of the present disclosure. And the training data generated in this way has good diversity, thus being able to improve the training effect.

[0064] In some embodiments, the objects in the video can be annotated in three steps. In the first step, dynamic video segments can be selected by comparing the start frame and the end frame of each 4-second video segment in the dataset. In some embodiments, these image frames can be processed and the cosine similarity of their feature embeddings in the average pooling layer can be calculated. Video segments with a similarity score lower than 0.65 will be retained for further processing. In the second step, for each selected video segment, a three-sentence description of the video content can be created, and then the noun phrases in these descriptions can be identified. Since most of these phrases are abstract nouns rather than specific object names, these noun phrases can be filtered and only the phrases representing specific object names can be retained. Subsequently, these filtered noun phrases can be processed to identify the initial bounding boxes in the start frame of the video segment. Then, these bounding boxes can be tracked in subsequent frames. For each detected object, some bounding boxes can be provided, one for each image frame in the video segment. Then, objects for which no bounding box can be detected or the detection confidence is lower than the threshold in some image frames can be excluded, so that the successfully tracked objects can be used as the ground truth data for training.

[0065] During training, in some embodiments, the motion control module can be trained by fixing the parameters of the base model while adjusting the parameters of the motion control module. In some embodiments, the motion control module can be trained by applying a self-alignment operation, which includes: generating an identification image frame based on the target bounding boxes in the training dataset, where the identification image frame includes identification bounding boxes that identify the objects constrained by the target bounding boxes, and training the motion control module by aligning the identification bounding boxes with the target bounding boxes. In some embodiments, the loss between the identification bounding boxes and the target bounding boxes can be determined, and the motion control module can be trained by making the loss satisfy a predetermined condition.

[0066] Figure 5 A schematic diagram of an example process 500 of the self-alignment operation according to some embodiments of the present disclosure is shown. Process 500 can train a model to generate bounding boxes of different colors for each encoded object in each image frame and specify the colors in the control tokens of the objects. This method can decompose the problems of the association between the bounding boxes and the objects and maintaining temporal consistency among multiple image frames into two more manageable tasks: generating bounding boxes with the correct colors for each object and aligning these boxes with the bounding boxes in each image frame for providing motion control information. In this way, it can be ensured that the bounding boxes of the same color always surround the same object in different image frames. For hard bounding boxes, the model only needs to generate the bounding boxes at the specified coordinates, while for soft bounding boxes, the bounding boxes can be generated within the specified regions. The self-aligned bounding boxes can be used as an intermediate representation, and the model can follow the constraints provided by the target bounding boxes to guide the generation of these self-aligned bounding boxes, thereby being able to guide the generation of visual objects. After completing the training phase of performing the self-alignment operation, the same dataset can be continued to be used to further train the model to eliminate the bounding boxes in the generated image frames.

[0067] As Figure 5 shown, when generating image frames 502, 512, 522, 532, and 542, process 500 can predict the bounding boxes 504, 506, 514, 516, 524, 526, 534, and 536 surrounding each controlled object and draw these bounding boxes into the corresponding image frames as part of the image frames. Among them, the bounding boxes that identify the same object are represented by the same color, and the bounding boxes that identify different objects are represented by different colors. For example, the bounding boxes 504, 514, 524, and 534 that identify the same object are all represented by white, the bounding boxes 506, 516, 526, and 536 that identify the same object are all represented by black, and the bounding boxes 504 and 506 that identify different objects are represented by different colors.

[0068] In this way, the self-alignment operation can effectively associate the bounding boxes with the objects and maintain temporal consistency across multiple frames. Additionally, the model can quickly learn to stop generating visible bounding boxes, but the alignment ability of these bounding boxes still remains. Thus, the self-alignment operation can help the model establish an appropriate internal representation.

[0069] In some embodiments, a first training dataset including hard bounding boxes can be obtained, and based on the first training dataset, a motion control module can be trained by applying the self-alignment operation. In some embodiments, a second training dataset can be generated by converting a portion of the bounding boxes in the first training dataset into soft bounding boxes, and based on the second training dataset, a motion control module can be trained by applying the self-alignment operation. In some embodiments, based on the second training dataset, a motion control module can be trained by utilizing the loss for the base model without applying the self-alignment operation.

[0070] Figure 6 The flowchart of an example process 600 of a multi-stage training process according to some embodiments of the present disclosure is shown. As Figure 6 shown, in block 602 which describes the first stage, process 600 can use all of the provided ground truth bounding boxes as hard bounding boxes to train the model. Since the motion control of hard bounding boxes is easier to learn than that of soft bounding boxes, this stage can be used as a preliminary stage to establish the model's initial understanding of coordinates and object identifiers.

[0071] [[ID=ID=12]]In block 604 which describes the second stage, process 600 can replace a portion of the hard bounding boxes with soft bounding boxes to train the model. For example, 80% of the hard bounding boxes can be replaced with soft bounding boxes. Process 600 can independently and randomly expand the hard bounding boxes in the up, down, left, and right directions (not exceeding the boundaries of the image frame), and use the expanded bounding boxes as soft bounding boxes. In the first stage and the second stage of block 602 and block 604, the self-alignment operation can be applied for training.

[0072] In block 606 which describes the third stage, process 600 can use the training data in the second stage to train the model, but does not perform the self-alignment operation. In this way, the first stage and the second stage can effectively enable the model to have the ability to handle hard bounding boxes and soft bounding boxes, and in the third stage, image frames without self-aligned bounding boxes can be generated. This can not only improve the training efficiency of the model, but also make full use of the self-alignment operation to enhance the training effect of the model. [[ID=ID=17]]

[0073] [[ID=ID=18]] Figure 7A schematic diagram of an example process 700 for generating a video by generating a hard bounding box and expanding it into a soft bounding box in the case of providing a hard bounding box in an end frame according to some embodiments of the present disclosure is shown. During the inference phase, the user can identify the bounding boxes only in a small number of image frames (e.g., in the start frame and the end frame). As Figure 7 shown, the example process 700 can identify a bounding box 704 in the start frame 702 and a bounding box 734 in the end frame 732, indicating that the object identified by the bounding box 704 moves from the position where the bounding box 704 is located to the position specified by the bounding box 734, and the size of the object in the end frame 732 should correspond to the size of the bounding box 734. To provide more stable motion control, the model can insert soft bounding boxes into the intermediate frames 712 and 722. In some embodiments, a linear interpolation between the user-specified bounding boxes 704 and 734 can be applied to the intermediate frames 712 and 722 to generate hard bounding boxes 714 and 724. Then, the process 700 can appropriately expand the hard bounding boxes 714 and 724 to generate soft bounding boxes 716 and 726. In this way, the process 700 can generate an intermediate frame 712 with a soft bounding box 716 and an intermediate frame 722 with a soft bounding box 726.

[0074] In this way, it can be ensured that the object generally follows the expected trajectory during the motion, and at the same time, the soft bounding box can also bring various variations to the model, thereby improving the diversity of the generated video.

[0075] Figure 8 A schematic diagram of an example process for generating a video by generating a hard bounding box and expanding it into a soft bounding box in the case of providing the motion trajectory of an object according to some embodiments of the present disclosure is shown. As Figure 8 shown, the user provides a bounding box 804 and a motion trajectory 808 in the start frame 802. The process 800 can generate hard bounding boxes 814, 824, and 834 in the subsequent image frames 812, 822, and 832 by interpolation and along the motion trajectory 808. For example, the hard bounding boxes 814, 824, and 834 can be generated by generating hard bounding boxes with the center positions on the motion trajectory. Then, the process 800 can expand the hard bounding boxes 814, 824, and 834 to generate soft bounding boxes 816, 826, and 836. In this way, the object identified by the bounding box 804 can move along the motion trajectory 808 to within the range defined by the soft bounding box 836 in the image frame 832, and in the image frames 812 and 822, the object can remain within the ranges defined by the soft bounding boxes 816 and 826.

[0076] In this way, the user can perform more precise motion control by specifying the motion trajectory of the object and can achieve effects that cannot be expressed by text description.

[0077] Figure 9 A schematic diagram of example 900 is shown that provides multiple bounding boxes in a starting frame and a bounding box near the right boundary in an ending frame according to some embodiments of the present disclosure. As Figure 9 shown, in example 900, a starting frame 902 and a text description 908 (i.e., "The puppy runs towards the camera") are provided, where the starting frame 902 includes a white puppy and a black puppy. In example 900, the starting frame 902 includes a black bounding box 904 that identifies the white puppy, and a white bounding box 906 that identifies the black puppy. Additionally, the ending frame 912 includes a black hard bounding box 914 and a white hard bounding box 916, where the hard bounding box 916 is near the right boundary of the ending frame 912 and has a very narrow width (e.g., less than a threshold width), which indicates that the black puppy needs to run out of the frame from the right side before the ending frame.

[0078] In example 900, the model generates an image frame sequence 920, where the two puppies gradually run towards the camera, and the white puppy runs to the position specified by the hard bounding box 914 in the ending frame, and the size of the white puppy corresponds to the hard bounding box 914. Additionally, the black puppy runs out of the right boundary in the ending frame, and in the image frame where the black puppy runs out of the right boundary, its size corresponds to the hard bounding box 916.

[0079] Figure 10 A schematic diagram of example 1000 is shown that provides an object bounding box and a motion trajectory and a bounding box of another object in a starting frame and a bounding box of the other object in an ending frame according to some embodiments of the present disclosure. As Figure 10 shown, in example 1000, a starting frame 1002 and a text description 1010 (i.e., "A person is throwing a frisbee") are provided, where the starting frame 1002 includes a person and a frisbee. In example 1000, the starting frame 1002 includes a white hard bounding box 1004 that identifies the person, a black hard bounding box 1006 that identifies the frisbee, and a motion trajectory 1008 that identifies the motion path of the frisbee.

[0080] In example 1000, the model generates an image frame sequence 1020, where the person is located at the position specified by the hard bounding box 1014 in the ending frame, and the size of the person corresponds to the hard bounding box 1014. Additionally, the frisbee is thrown by the person in the image frame sequence 1020, flies along the motion trajectory 1008, and finally flies back to the position of the end point indicated by the motion trajectory 1008.

[0081] As can be seen, the motion control module can achieve precise motion control based on the bounding box. It can manage the motion of foreground and background objects and modify their poses by adjusting the smaller components of larger objects. In addition, in the scenario of generating a video conditioned on an image, the user can easily select an object by drawing a hard bounding box around the object. Compared with text - or language - based control, this vision - based method is easier to operate. Moreover, for intermediate frames lacking user - provided bounding boxes, the motion control module can generate soft bounding boxes through algorithms to approximate the motion trajectory. These soft bounding boxes can be constructed based on the bounding boxes in the user - specified start and end frames, or can be constructed based on the user - specified bounding boxes and motion trajectories, thus enabling precise motion control and improving the user experience.

[0082] Figure 11 FIG. shows a block diagram of an apparatus 1100 for generating a video according to some embodiments of the present disclosure. As Figure 11 shown, the apparatus 1100 includes a visual marker acquisition module 1102 configured to acquire visual markers for generating image frames in the video. The apparatus 1100 further includes a control marker acquisition module 1104 configured to acquire control markers for constraining the position information of objects in the image frames. In addition, the apparatus 1100 further includes a video image generation module 1106 configured to generate image frames in the video based on the visual markers and the control markers, wherein the objects in the image frames satisfy the position information.

[0083] It can be understood that by using the apparatus 1100 of the present disclosure, at least one of the many advantages achievable by the method or process described above can be realized. For example, the apparatus 1100 can use control markers generated based on the bounding box to understand the position where the user expects the target object to move, so as to be able to use the control markers to constrain the content of the generated image frames. In this way, the user can simply and accurately express the desired object motion mode, and compared with only providing a text description or a reference image, this method can improve the motion effect of the generated video, increase the matching degree between the generated video and the user's requirements, and thus improve the user experience.

[0084] Figure 12 FIG. shows a block diagram of a device 1200 capable of implementing multiple embodiments of the present disclosure. The device 1200 may be the device or apparatus described in the embodiments of the present disclosure. As Figure 12As shown, device 1200 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 1201, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1202 or computer program instructions loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of device 1200 can also be stored. The CPU / GPU 1201, ROM 1202, and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204. Although not shown in Figure 12 , device 1200 may also include a coprocessor.

[0085] Multiple components in device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a magnetic disk, an optical disc, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0086] Each of the methods or processes described above can be executed by the CPU / GPU 1201. For example, in some embodiments, the method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the CPU / GPU 1201, one or more steps or actions of the methods or processes described above can be executed.

[0087] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of the present disclosure.

[0088] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0089] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0090] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages and conventional procedural programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0091] These computer - readable program instructions can be provided to a processing unit of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that, when the instructions are executed by the processing unit of the computer or other programmable data - processing apparatus, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. The computer - readable program instructions can also be stored in a computer - readable storage medium, and these instructions cause a computer, a programmable data - processing apparatus, and / or other devices to work in a particular manner, so that the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0092] The computer - readable program instructions can also be loaded onto a computer, other programmable data - processing apparatus, or other devices so that a series of operation steps are executed on the computer, other programmable data - processing apparatus, or other devices to produce a computer - implemented process, and thus the instructions executed on the computer, other programmable data - processing apparatus, or other devices implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

[0094] Some example implementations of the present disclosure are listed below.

[0095] Example 1. A method for generating a video, comprising:

[0096] obtaining a visual marker for generating an image frame in the video;

[0097] obtaining a control marker for constraining the position information of an object in the image frame; and

[0098] generating the image frame in the video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.

[0099] Example 2. The method according to Example 1, wherein the position information is a bounding box, and obtaining the control marker for constraining the position information of the object in the image frame comprises:

[0100] determining the coordinates of the bounding box in the image frame; and

[0101] generating the control marker based on the coordinates.

[0102] Example 3. The method according to Examples 1-2, wherein generating the control marker based on the coordinates comprises:

[0103] generating an object identifier for the bounding box based on the color of the bounding box; and

[0104] generating the control marker based on the coordinates and the object identifier.

[0105] Example 4. The method according to Examples 1-3, wherein generating the control marker based on the coordinates and the object identifier comprises:

[0106] Determine the type of the bounding box, where the type includes a first type that restricts a specific position and a specific size of an object to be generated and a second type that restricts a position range and a size range of the object to be generated; and

[0107] Generate the control token based on the coordinates, the object identifier, and the type.

[0108] Example 5. The method according to Examples 1-4, wherein generating the image frame in the video based on the visual token and the control token includes:

[0109] In response to the type of the bounding box being the first type, a central position of the object coincides with a central position of the bounding box, and a size of the object corresponds to a size of the bounding box; and

[0110] In response to the type of the bounding box being the second type, a central position of the object is within the bounding box, and a size of the object does not exceed the bounding box.

[0111] Example 6. The method according to Examples 1-5, wherein generating the control token based on the coordinates, the object identifier, and the type includes:

[0112] Generate a plurality of embeddings based on the coordinates, the object identifier, and the type; and

[0113] Generate the control token based on the plurality of embeddings by using a multi-layer perceptron.

[0114] Example 7. The method according to Examples 1-6, wherein the visual token is a first set of visual tokens, and generating the image frame in the video based on the visual token and the control token includes:

[0115] Generate a second set of visual tokens based on the first set of visual tokens and the control token, wherein a number of visual tokens in the first set of visual tokens is the same as a number of visual tokens in the second set of visual tokens.

[0116] Example 8. The method according to Examples 1-7, wherein the first set of visual tokens and the image frame are generated by a base model, the second set of visual tokens is generated by a motion control module, and the method further includes:

[0117] Train the motion control module by fixing parameters of the base model while adjusting parameters of the motion control module.

[0118] Example 9. The method according to Examples 1-8, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation includes:

[0119] Generating an identification image frame based on the target bounding box in the training dataset, where the identification image frame includes an identification bounding box identifying the object constrained by the target bounding box; and

[0120] Training the motion control module by aligning the identification bounding box with the target bounding box.

[0121] Example 10. The method according to Examples 1-9, wherein training the motion control module by aligning the identification bounding box with the target bounding box includes:

[0122] Determining the loss between the identification bounding box and the target bounding box; and

[0123] Training the motion control module by making the loss satisfy a predetermined condition.

[0124] Example 11. The method according to Examples 1-10, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further includes:

[0125] Obtaining a first training dataset, the first training dataset including bounding boxes of the first type; and

[0126] Based on the first training dataset, training the motion control module by applying the self-alignment operation.

[0127] Example 12. The method according to Examples 1-11, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further includes:

[0128] Generating a second training dataset by converting a part of the bounding boxes in the first training dataset into bounding boxes of the second type; and

[0129] Based on the second training dataset, training the motion control module by applying the self-alignment operation.

[0130] Example 13. The method according to Examples 1-12, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further includes:

[0131] Based on the second training dataset, training the motion control module by using the loss for the base model without applying the self-alignment operation.

[0132] Example 14. A device for generating a video, comprising:

[0133] A visual marker acquisition module, configured to acquire visual markers for generating image frames in the video;

[0134] A control marker acquisition module, configured to acquire control markers for constraining the position information of objects in the image frames; and

[0135] A video image generation module, configured to generate the image frames in the video based on the visual markers and the control markers, wherein the objects in the image frames satisfy the position information.

[0136] Example 15. The device according to Example 14, wherein the position information is a bounding box, and acquiring the control markers for constraining the position information of objects in the image frames includes:

[0137] A coordinate determination module, configured to determine the coordinates of the bounding box in the image frame; and

[0138] A coordinate usage module, configured to generate the control markers based on the coordinates.

[0139] Example 16. The device according to Examples 14-15, wherein generating the control markers based on the coordinates includes:

[0140] An identifier generation module, configured to generate an object identifier for the bounding box based on the color of the bounding box; and

[0141] An identifier usage module, configured to generate the control markers based on the coordinates and the object identifier.

[0142] Example 17. The device according to Examples 14-16, wherein generating the control markers based on the coordinates and the object identifier includes:

[0143] A type determination module, configured to determine the type of the bounding box, wherein the type includes a first type that constrains the specific position and specific size of the object to be generated and a second type that constrains the position range and size range of the object to be generated; and

[0144] A type usage module, configured to generate the control markers based on the coordinates, the object identifier, and the type.

[0145] Example 18. The device according to Examples 14-17, wherein generating the image frames in the video based on the visual markers and the control markers includes:

[0146] In response to the type of the bounding box being the first type, the center position of the object coincides with the center position of the bounding box, and the size of the object corresponds to the size of the bounding box; and

[0147] In response to the type of the bounding box being the second type, the center position of the object is within the bounding box, and the size of the object does not exceed the bounding box.

[0148] Example 19. The apparatus according to Examples 14-18, wherein generating the control tag based on the coordinates, the object identifier, and the type includes:

[0149] An embedding generation module configured to generate a plurality of embeddings based on the coordinates, the object identifier, and the type; and

[0150] An embedding usage module configured to generate the control tag based on the plurality of embeddings by utilizing a multi-layer perceptron.

[0151] Example 20. The apparatus according to Examples 14-19, wherein the visual tag is a first set of visual tags, and generating the image frame in the video based on the visual tag and the control tag includes:

[0152] A visual tag generation module configured to generate a second set of visual tags based on the first set of visual tags and the control tag, wherein the number of visual tags in the first set of visual tags is the same as the number of visual tags in the second set of visual tags.

[0153] Example 21. The apparatus according to Examples 14-20, wherein the first set of visual tags and the image frame are generated by a base model, the second set of visual tags is generated by a motion control module, and the apparatus further includes:

[0154] A motion control training module configured to train the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model.

[0155] Example 22. The apparatus according to Examples 14-21, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation includes:

[0156] An identification frame generation module configured to generate an identification image frame based on a target bounding box in a training dataset, the identification image frame including an identification bounding box identifying an object constrained by the target bounding box; and

[0157] A bounding box alignment module configured to train the motion control module by aligning the identification bounding box with the target bounding box.

[0158] Example 23. The apparatus according to Examples 14-22, wherein training the motion control module by aligning the identification bounding box with the target bounding box includes:

[0159] A loss determination module configured to determine a loss between the identification bounding box and the target bounding box; and

[0160] A loss utilization module configured to train the motion control module by making the loss satisfy a predetermined condition.

[0161] Example 24. The apparatus according to Examples 14-23, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further includes:

[0162] A first training set acquisition module configured to acquire a first training data set, the first training data set including bounding boxes of the first type; and

[0163] A first training set utilization module configured to train the motion control module based on the first training data set by applying the self-alignment operation.

[0164] Example 25. The apparatus according to Examples 14-24, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further includes:

[0165] A second training set acquisition module configured to generate a second training data set by converting a part of the bounding boxes in the first training data set into bounding boxes of the second type; and

[0166] A second training set utilization module configured to train the motion control module based on the second training data set by applying the self-alignment operation.

[0167] Example 26. The apparatus according to Examples 14-25, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further includes:

[0168] A third training set utilization module configured to train the motion control module based on the second training data set by utilizing the loss for the base model without applying the self-alignment operation.

[0169] Example 27. An electronic device, comprising:

[0170] A processor; and

[0171] A memory coupled to the processor, the memory having instructions stored therein that, when executed by the processor, cause the electronic device to perform actions, the actions including:

[0172] Obtain a visual marker for generating an image frame in the video;

[0173] Obtain a control marker for constraining the position information of an object in the image frame; and

[0174] Generate the image frame in the video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.

[0175] Example 28. The apparatus according to Example 27, wherein obtaining the control marker for constraining the bounding box of an object in the image frame includes:

[0176] Determine the coordinates of the bounding box in the image frame; and

[0177] Generate the control marker based on the coordinates.

[0178] Example 29. The apparatus according to Examples 27-28, wherein generating the control marker based on the coordinates includes:

[0179] Generate an object identifier for the bounding box based on the color of the bounding box; and

[0180] Generate the control marker based on the coordinates and the object identifier.

[0181] Example 30. The apparatus according to Examples 27-29, wherein generating the control marker based on the coordinates and the object identifier includes:

[0182] Determine the type of the bounding box, wherein the type includes a first type that constrains the specific position and specific size of an object to be generated and a second type that constrains the position range and size range of the object to be generated; and

[0183] Generate the control marker based on the coordinates, the object identifier, and the type.

[0184] Example 31. The apparatus according to Examples 27-30, wherein generating the image frame in the video based on the visual marker and the control marker includes:

[0185] In response to the type of the bounding box being the first type, the center position of the object coincides with the center position of the bounding box, and the size of the object corresponds to the size of the bounding box; and

[0186] In response to the type of the bounding box being the second type, the center position of the object is within the bounding box, and the size of the object does not exceed the bounding box.

[0187] Example 32. The apparatus according to Examples 27 - 31, wherein generating the control tag based on the coordinates, the object identifier, and the type includes:

[0188] generating a plurality of embeddings based on the coordinates, the object identifier, and the type; and

[0189] generating the control tag based on the plurality of embeddings by using a multi - layer perceptron.

[0190] Example 33. The apparatus according to Examples 27 - 32, wherein the visual tag is a first set of visual tags, and generating the image frame in the video based on the visual tag and the control tag includes:

[0191] generating a second set of visual tags based on the first set of visual tags and the control tag, wherein the number of visual tags in the first set of visual tags is the same as the number of visual tags in the second set of visual tags.

[0192] Example 34. The apparatus according to Examples 27 - 33, wherein the first set of visual tags and the image frame are generated by a base model, the second set of visual tags is generated by a motion control module, and the apparatus further includes:

[0193] training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module.

[0194] Example 35. The apparatus according to Examples 27 - 34, wherein the motion control module is trained by applying a self - alignment operation, and the self - alignment operation includes:

[0195] generating an identification image frame based on a target bounding box in a training dataset, the identification image frame including an identification bounding box identifying an object constrained by the target bounding box; and

[0196] training the motion control module by aligning the identification bounding box with the target bounding box.

[0197] Example 36. The apparatus according to Examples 27 - 35, wherein training the motion control module by aligning the identification bounding box with the target bounding box includes:

[0198] determining a loss between the identification bounding box and the target bounding box; and

[0199] Train the motion control module by making the loss meet a predetermined condition.

[0200] Example 37. The apparatus according to any one of Examples 27-36, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further comprises:

[0201] Obtain a first training dataset, the first training dataset including bounding boxes of the first type; and

[0202] Based on the first training dataset, train the motion control module by applying the self-alignment operation.

[0203] Example 38. The apparatus according to any one of Examples 27-37, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further comprises:

[0204] Generate a second training dataset by converting a part of the bounding boxes in the first training dataset into bounding boxes of the second type; and

[0205] Based on the second training dataset, train the motion control module by applying the self-alignment operation.

[0206] Example 39. The apparatus according to any one of Examples 27-38, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further comprises:

[0207] Based on the second training dataset, train the motion control module by utilizing the loss for the base model without applying the self-alignment operation.

[0208] The embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technologies in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a video, comprising: Determining, via user input, an object controlled by motion in a starting image frame; Obtaining, via user input, position information for constraining the motion of the object, wherein the position information includes a motion trajectory that indicates a motion path of the object; Generating, by a video generation model having a motion control module, a plurality of image frames of the video based on the starting image frame and the motion trajectory, wherein a motion path of the object in the plurality of image frames satisfies the motion trajectory.

2. The method according to claim 1, wherein the object and the position information are identified using one color, the starting image frame further includes another object controlled by motion and another position information for constraining the motion of the another object, and the another object and the another position information are identified using another color.

3. The method according to claim 1, wherein the motion trajectory further indicates an end point of the object's motion, and in an end image frame among the plurality of image frames, the object moves to the end point.

4. The method according to claim 1, wherein generating the plurality of image frames of the video based on the starting image frame and the motion trajectory includes: Generating a visual marker based on the starting image frame, the visual marker indicating visual information of an image frame to be generated; Generating a control marker based on the position information, the control marker indicating motion control information for the object; And Generating, by the video generation model, an image frame of the video based on the visual marker and the control marker.

5. The method according to claim 4, wherein generating the image frame of the video based on the visual marker and the control marker includes: Generating an object identifier for the motion trajectory based on a color of the motion trajectory; And Generating the control marker based on the object identifier.

6. The method according to claim 4, wherein the visual marker is a first set of visual markers, and generating the image frame of the video based on the visual marker and the control marker includes: Generating a second set of visual markers based on the first set of visual markers and the control marker, wherein a number of visual markers in the first set of visual markers is the same as a number of visual markers in the second set of visual markers.

7. The method according to claim 6, wherein the first set of visual markers and the image frame are generated by a base model, the second set of visual markers is generated by the motion control module, and the method further includes: Training the motion control module by fixing parameters of the base model while adjusting parameters of the motion control module.

8. A method for generating a video, comprising: Determining, via user input, an object controlled by motion in a starting image frame; Obtaining, via user input, position information for constraining the motion of the object; Generating a visual marker based on the starting image frame, the visual marker indicating visual information of an image frame to be generated; Generate a control marker based on the position information, where the control marker indicates motion control information for the object; and Generate the image frame in the video based on the visual marker and the control marker through a video generation model with a motion control module, where the object in the image frame satisfies the position information.

9. The method according to claim 8, wherein the position information includes a motion trajectory, and the motion trajectory indicates the motion path of the object.

10. The method according to claim 9, wherein the object and the position information are identified using one color, and the starting image frame further includes another object to be motion-controlled and another position information for constraining the motion of the other object, and the other object and the other position information are identified using another color.

11. The method according to claim 8, wherein the position information is a bounding box, and generating the control marker based on the position information includes: Determine the coordinates of the bounding box in the image frame; and Generate the control marker based on the coordinates.

12. The method according to claim 11, wherein generating the control marker based on the coordinates includes: Generate an object identifier for the bounding box based on the color of the bounding box; and Generate the control marker based on the coordinates and the object identifier.

13. The method according to claim 12, wherein generating the control marker based on the coordinates and the object identifier includes: Determine the type of the bounding box, where the type includes a first type that constrains the specific position and specific size of the object to be generated and a second type that constrains the position range and size range of the object to be generated; and Generate the control marker based on the coordinates, the object identifier, and the type.

14. The method according to claim 13, wherein generating the image frame in the video based on the visual marker and the control marker includes: In response to the type of the bounding box being the first type, the center position of the object coincides with the center position of the bounding box, and the size of the object corresponds to the size of the bounding box; and In response to the type of the bounding box being the second type, the center position of the object is within the bounding box, and the size of the object does not exceed the bounding box.

15. The method according to claim 14, wherein generating the control marker based on the coordinates, the object identifier, and the type includes: Generate a plurality of embeddings based on the coordinates, the object identifier, and the type; and Generate the control marker based on the plurality of embeddings by using a multi-layer perceptron.

16. The method according to claim 14, wherein the visual marker is a first set of visual markers, and generating the image frame in the video based on the visual marker and the control marker includes: Generate a second set of visual markers based on the first set of visual markers and the control markers, wherein the number of visual markers in the first set of visual markers is the same as the number of visual markers in the second set of visual markers.

17. The method according to claim 16, wherein the first set of visual markers and the image frame are generated by a base model, the second set of visual markers are generated by the motion control module, and the method further comprises: Training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module.

18. The method according to claim 17, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation comprises: Generating an identification image frame based on the target bounding box in the training data set, wherein the identification image frame includes an identification bounding box identifying the object constrained by the target bounding box; And Training the motion control module by aligning the identification bounding box with the target bounding box.

19. The method according to claim 18, wherein training the motion control module by aligning the identification bounding box with the target bounding box comprises: Determining the loss between the identification bounding box and the target bounding box; And Training the motion control module by making the loss satisfy a predetermined condition.

20. The method according to claim 18, wherein training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module further comprises: Obtaining a first training data set, the first training data set including bounding boxes of the first type; And Based on the first training data set, training the motion control module by applying the self-alignment operation.

21. The method according to claim 20, wherein training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module further comprises: Generating a second training data set by converting a part of the bounding boxes in the first training data set into bounding boxes of the second type; And Based on the second training data set, training the motion control module by applying the self-alignment operation.

22. The method according to claim 21, wherein training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module further comprises: Based on the second training data set, training the motion control module by utilizing the loss for the base model without applying the self-alignment operation.

23. A method for generating training data, comprising: Obtaining image frames of an existing video, the size of the image frames being larger than the size required by a video generation model, and the image frames including bounding boxes; Obtaining a cropped region by cropping the image frames based on the size required by the video generation model, wherein the bounding boxes are completely outside the cropped region; Obtaining line segments by projecting the bounding boxes to the boundary of the cropped region; And Training the video generation model using the line segments.

24. An electronic device, comprising: a processor; and a memory coupled to the processor, the memory having instructions stored therein, which when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 23.

Images