Method, apparatus, electronic device and computer program product for generating video
By obtaining and utilizing the visual and control marks generated by bounding boxes, the problem that the video generation model is difficult to accurately realize the user's motion intentions is solved, and the accuracy and user experience of video generation are improved.
Patent Information
- Application Number
- CN202410132635.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-01-30
AI Technical Summary
The existing video generation model is difficult to accurately understand the user's requirements for the movement of objects in the video, which makes the generated video unable to accurately realize the user's intentions, especially when the movement of objects is complex or has high precision requirements.
By obtaining the bounding box provided by the user to constrain the position and size of the object, visual marks and control marks are generated, and video frames are generated based on these marks using the motion control module to ensure that the object moves accurately in the video.
Improve the matching degree between video generation and user requirements, enhance user experience, and ensure that objects move accurately in the video in the way they expect.
Smart Images

Figure CN117979177B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to the field of artificial intelligence, and more specifically, to methods, apparatuses, electronic devices, and computer program products for generating videos. Background Art
[0002] Text-guided video generation is a technique that uses text descriptions to guide the generation of video content. In this video generation task, the model receives text descriptions in natural language and generates image frames corresponding to the text based on these descriptions, which are then combined into a video. One of the key challenges of this task is to establish the association between the text description and the video content, including understanding the objects, actions, spatiotemporal relationships, etc. in the text description, and then converting this information into a series of image frames.
[0003] Motion control refers to controlling the movement of objects, scenes, and cameras in the generated video, for example, through text descriptions. For example, the text description may include information about the movement of objects or characters, so it is necessary to control the objects or characters in the generated video to move according to the text description. In the related art, machine learning models are usually used to implement motion control in video generation tasks. Summary of the invention
[0004] In a first aspect of an embodiment of the present disclosure, a method for generating a video is provided. The method includes obtaining a visual marker for generating an image frame in the video. The method also includes obtaining a control marker for constraining position information of an object in the image frame. In addition, the method also includes generating an image frame in the video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.
[0005] In a second aspect of an embodiment of the present disclosure, a device for generating a video is provided. The device includes a visual marker acquisition module configured to acquire visual markers for generating an image frame in a video. The device also includes a control marker acquisition module configured to acquire control markers for constraining position information of an object in the image frame. In addition, the device also includes a video image generation module configured to generate an image frame in a video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.
[0006] In a third aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for generating a video. The method includes obtaining a visual marker for generating an image frame in a video. The method also includes obtaining a control marker for constraining position information of an object in the image frame. In addition, the method also includes generating an image frame in a video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.
[0007] In a fourth aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine executable instructions, which, when executed, cause a machine to implement a method for generating a video. The method includes obtaining a visual marker for generating an image frame in a video. The method also includes obtaining a control marker for constraining position information of an object in the image frame. In addition, the method also includes generating an image frame in a video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information.
[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0010] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;
[0011] Figure 2 A flowchart of a method for generating a video according to some embodiments of the present disclosure is shown;
[0012] Figure 3 A schematic diagram showing an example architecture for generating a video according to some embodiments of the present disclosure;
[0013] Figure 4 A schematic diagram showing an example of generating training data from an existing video dataset according to some embodiments of the present disclosure;
[0014] Figure 5A schematic diagram showing an example process of a self-alignment operation according to some embodiments of the present disclosure;
[0015] Figure 6 A flow chart illustrating an example process for a multi-stage training process according to some embodiments of the present disclosure;
[0016] Figure 7 A schematic diagram showing an example process of generating a video by generating a hard bounding box and extending it into a soft bounding box when a hard bounding box in an end frame is provided according to some embodiments of the present disclosure;
[0017] Figure 8 A schematic diagram showing an example process of generating a video by generating a hard bounding box and expanding it into a soft bounding box when a motion trajectory of an object is provided according to some embodiments of the present disclosure;
[0018] Fig. 9 A schematic diagram showing an example of providing a plurality of bounding boxes at a start frame and providing a bounding box close to a right boundary at an end frame according to some embodiments of the present disclosure;
[0019] Fig.10 A schematic diagram showing an example of providing an object bounding box and a motion trajectory and a bounding box of another object at a start frame and providing the bounding box of the other object at an end frame according to some embodiments of the present disclosure;
[0020] Fig.11 A block diagram showing an apparatus for generating a video according to some embodiments of the present disclosure; and
[0021] Fig.12 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0022] It is understood that all user-related data involved in this technical solution should be obtained and used after the user's authorization. This means that in this technical solution, if the user's personal information needs to be used, the user's explicit consent and authorization are required before obtaining this data, otherwise the relevant data will not be collected and used. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly observed during the collection, use and storage of data, and necessary technologies and measures should be taken to protect the user's data security and ensure the safe use of data.
[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0024] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.
[0025] With the development of video generation technology, some video generation models can generate videos based on text prompts or key image frames. As an example of a video generation model, the video diffusion model is an extension of the image diffusion model, which incorporates the architecture of the U-Net model in the image model and adds a temporal layer that helps generate multiple image frames. Text-to-video (T2V) diffusion models are often the cornerstone of various forms of constrained video generation models. In the text-to-video diffusion model, image frames can be created based on text descriptions, and then videos can be generated based on the text descriptions and pre-generated image frames. This approach enables the video generation model to use static images as references, thereby focusing on the dynamic aspects of video generation, and therefore improving the quality of the generated video.
[0026] It should be understood that, for ease of explanation, some embodiments herein are described using a video diffusion model with a U-Net architecture as an example, but are not intended to limit the specific architecture of the video generation model. The solution provided by the present disclosure can be applied to any video generation model that generates visual markers and generates image frames based on the visual markers.
[0027] In some usage scenarios, users want to provide information about the movement of objects in the generated video by entering text descriptions. For example, a user can provide a reference image of a building, and then enter the text description "tilt the camera upwards and reveal the top of the building". At this time, the user hopes that in the generated video, the camera gradually raises the lens upward from the perspective of shooting the ground and finally shoots the top of the building. However, in the related art, although it is possible to generate a video with high picture quality and slow lens movement based on the reference image and text description provided by the user, the model cannot well understand the user's requirements for the movement of objects in the video, and thus cannot accurately reveal the top of the building in the generated video.
[0028] In addition, in some usage scenarios, when the user's requirements for the movement method are very precise, it is difficult to accurately describe the desired picture in words. For example, when the user wants two puppies to run towards the camera in the video, one of the white puppies is getting closer and closer to the camera and runs to the center of the screen. At this time, the white puppy fills one-third of the screen. At the same time, the other black puppy is also getting closer and closer to the camera, but it is running towards the toy next to the camera, so it deviates more and more from the center of the screen and eventually disappears from the right side of the screen. For ordinary users, it is very difficult to accurately describe such movement requirements, and thus it is impossible to generate the desired video.
[0029] To this end, an embodiment of the present disclosure provides a solution for generating a video. In this solution, a user can provide a bounding box for constraining the position and size of an object in a video, and a video generation model can obtain a set of visual tags (tokens) and control tags for the provided bounding box for generating an image frame, and then generate an image frame based on the visual tags and the control tags. In the generated image frame, the object in the video will appear within the bounding box.
[0030] In this way, the video generation model can use the control mark of the bounding box to understand the position and size that the user expects the target object to move to, so that the control mark can be used to constrain the content of the generated image frame. In this way, the user can simply and accurately express the desired object movement. In addition, compared with only providing text descriptions or reference images, this method can improve the match between the generated video and the user's requirements, thereby improving the user experience.
[0031] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1As shown, the environment 100 includes a video generation model 102, which is also referred to as a base model in this article. It can be the video diffusion model that generates videos based on text descriptions or key image frames as described above, or it can be other machine learning models for generating videos implemented using neural network technology. The environment 100 also includes a motion control module 104, which can be combined with the video generation model 102 in the form of a plug-in to enhance the motion control capability of the video generation model 102. Figure 1 As shown, the video generation model 102 can generate visual tags 112-1, 112-2, ..., 112-N (collectively referred to as visual tags 112), which are a set of vectors generated based on information such as text descriptions, reference images, etc., including information of the image to be generated.
[0032] like Figure 1 As shown, environment 100 includes bounding boxes 106 and 108, which are used to constrain the position and size of objects in the video in the image frame to be generated. In an embodiment of the present disclosure, the term "object" can be an independent object (e.g., a puppy), a part of an independent object (e.g., a human hand), or multiple objects combined together (e.g., a person riding a horse). In addition, the term "motion" can be the movement of an object relative to a camera (or lens), or the movement of a camera relative to an object. For example, when bounding boxes 106 or 108 indicate the position and size of an object after movement, an object with autonomous motion ability (e.g., a puppy, a car, etc.) can move to a specified position by itself and be presented as a specified size, while an object that does not move autonomously (e.g., a rock, a building, etc.) can be presented to a specified position in the picture with a specified size by moving the camera. In some embodiments, bounding boxes such as bounding boxes 106 and 108 are rectangular boxes, and two types of bounding boxes can be implemented-hard bounding boxes and soft bounding boxes. The hard bounding box is used to specify a specific position and a specific size of an object, which means that in the generated image frame, the object is generated at the coordinates specified by the hard bounding box (for example, the center coordinates of the bounding box), and the size of the object corresponds to the size of the hard bounding box. The soft bounding box is used to specify the position range and size range of the object, which means that in the generated image frame, the object is generated within the range defined by the soft bounding box, and the size of the object does not exceed the range.
[0033] In the environment 100, the bounding box 106 is processed into the control mark 116, and the bounding box 108 is processed into the control mark 118, so the control marks 116 and 118 include motion control information of the objects corresponding to the bounding boxes 106 and 108, respectively. Then, the motion control module 104 can generate new visual marks 114-1, 114-2, ..., 114-N (collectively referred to as visual marks 114) based on the visual mark 112, the control mark 116, and the control mark 118. In this way, the visual mark 114 can include the motion control information provided by the bounding boxes 106 and 108. Then, the video generation model 102 can generate an image frame 120 based on the visual mark 114. In the image frame 120, the two puppies move from the position in the previous image frame 122 to the position specified by the bounding boxes 106 and 108, and the size of the white puppy corresponds to the bounding box 106, and the size of the black puppy corresponds to the bounding box 108. Multiple image frames, such as image frames 120 and 122 , may then form video 124 .
[0034] In this way, the motion control module 104 can use the control marks 116 and 118 of the bounding boxes 106 and 108 to provide motion control information for the video generation model 102, thereby improving the motion effect of the generated video, improving the match between the video and the user's requirements, and enhancing the user experience.
[0035] Figure 2 FIG. 2 is a flow chart of a method 200 for generating a video according to some embodiments of the present disclosure. Figure 2 As shown, at block 202, method 200 may obtain visual markers for generating image frames in a video. Figure 1 In the environment 100 shown, the motion control module 104 can obtain a visual marker 112 , which is generated by the video generation model 102 based on information such as text description or reference images, contains visual information of the image frame to be generated and is used to generate an image frame 120 .
[0036] At block 204, method 200 may obtain a control mark for constraining position information of an object in an image frame. The position information may be information associated with a position, such as a bounding box, an outline, a coordinate value, a coordinate range, etc. For example, Figure 1 In the environment 100 shown, the motion control module 104 can obtain control marks 116 and 118, which are generated based on the bounding boxes 106 and 108, so the control marks 116 and 118 contain motion control information of the bounding boxes 106 and 108. The bounding boxes 106 and 108 can constrain the position and size of the object in the image frame to be generated, thereby playing a role in controlling the motion of the object.
[0037] At block 206, method 200 may generate an image frame in the video based on the visual marker and the control marker, wherein the object in the image frame satisfies the position information. Figure 1 In the environment 100 shown, the motion control module 102 may generate a visual marker 114 based on the visual marker 112, the control marker 116, and the control marker 118, and then the video generation model 102 may generate an image frame 120 based on the visual marker 114. In the image frame 120, two puppies appear within the range constrained by the bounding boxes 106 and 108. Depending on different embodiments, the positions of the two puppies may correspond exactly to the positions of the bounding boxes 106 and 108, or the positions of the two puppies may be within the range constrained by the bounding boxes 106 and 108.
[0038] In this way, method 200 can use the control mark generated based on the bounding box to understand the position to which the user expects the target object to move, so that the control mark can be used to constrain the content of the generated image frame. In this way, the user can simply and accurately express the desired object movement mode, and compared with only providing text descriptions or reference images, this method can improve the motion effect of the generated video and improve the matching degree between the generated video and the user's requirements, thereby improving the user experience.
[0039] In some embodiments, the position information is a bounding box, and in order to obtain a control mark of the bounding box, the coordinates of the bounding box in the image frame can be determined, and the control mark is generated based on the coordinates. In some embodiments, the use of multiple bounding boxes to constrain the movement of multiple objects is supported. In these embodiments, an object identifier for the bounding box can be generated based on the color of the bounding box, and the control mark is generated based on the coordinates and the object identifier. In some embodiments, hard bounding boxes (also referred to as first type of bounding boxes herein) and soft bounding boxes (also referred to as second type of bounding boxes herein) can be supported at the same time. In these embodiments, the type of the bounding box can be determined, wherein the type of the bounding box includes a hard bounding box that constrains a specific position and a specific size of the object to be generated and a soft bounding box that constrains the position range and size range of the object to be generated, and the control mark is generated based on the coordinates, the object identifier, and the type.
[0040] In some embodiments, in response to the type of the bounding box being a hard bounding box, the center position of the object is consistent with the center position of the bounding box, and the size of the object corresponds to the size of the bounding box. In some embodiments, in response to the type of the bounding box being a soft bounding box, the center position of the object is within the bounding box, and the size of the object does not exceed the bounding box. In some embodiments, multiple embeddings can be generated based on coordinates, object identifiers, and types, and control marks can be generated based on the multiple embeddings by using a multilayer perceptron. In some embodiments, a second set of visual markers can be generated based on the first set of visual markers and the control markers, wherein the number of visual markers in the first set of visual markers is the same as the number of visual markers in the second set of visual markers.
[0041] Figure 3 Schematic diagram of an example architecture 300 for generating video according to some embodiments of the present disclosure is shown. Figure 3 As shown, the architecture 300 includes a spatial self-attention layer 302, a multi-layer perceptron 308, a motion control module 318, and a spatial cross-attention layer 322. The spatial self-attention layer 302 and the spatial cross-attention layer 322 can be, for example, a video diffusion model based on a three-dimensional (3D) U-Net architecture (e.g., Figure 1 The video generation model 102 in can be a module in a video diffusion model). The video diffusion model can iteratively predict the noise vector in the noisy video input, thereby gradually converting pure Gaussian noise into high-quality video frames. The 3D U-Net consists of alternating convolution blocks and attention blocks. Each block contains two components, namely, a spatial component that processes each image frame as a separate image, and a temporal component that facilitates information exchange between image frames. In each attention block, the spatial component typically includes a self-attention layer followed by a cross-attention layer, which is used to adjust the generation of the video based on text prompts. The motion control module is inserted between the two attention layers so that the model can manage motion control in video generation.
[0042] like Figure 3As shown, the architecture 300 inserts a motion control module 318 between the spatial self-attention layer 302 and the spatial cross-attention layer 322 of the original video diffusion model. The spatial self-attention layer 302 receives the frame-level visual tag 304 and generates visual tags 306-1, 306-2, ..., 306-N (collectively referred to as visual tags 306) based on the frame-level visual tag 304. The motion control module 318 receives the visual tag 306 and the control tag 316-1, 316-2, ..., 316-N (collectively referred to as control tags 316) as input, and outputs visual tags 320-1, 320-2, ..., 320-N (collectively referred to as visual tags 320). Among them, each control tag in the control tag 316 corresponds to a corresponding object (or bounding box). Since the control tag 316 includes the motion control information provided by the bounding box, the newly generated visual tag 320 will also include the motion control information provided by the bounding box. Then, the visual marker 320 is input to the spatial cross-attention layer 322, which can generate a frame-level visual marker 326 based on the visual marker 320 and the text markers 324-1, 324-2, ..., 324-N (collectively referred to as text markers 324). Then, the video diffusion model can generate image frames based on the frame-level visual markers 326. In order not to change the original structure of the spatial cross-attention layer 322, the number of visual markers 306 and visual markers 320 can be kept the same. In this way, the parameters of the original video diffusion model (including the spatial self-attention layer 302 and the spatial cross-attention layer 322) can be fixed during the training stage, and only the parameters of the motion control module 318 can be adjusted, thereby avoiding retraining caused by modifying the structure of the video diffusion model, thereby saving costs and avoiding a decrease in the accuracy of the original video diffusion model caused by retraining.
[0043] In the architecture 300, if v is used to represent the frame-level visual label 304 of the image frame to be generated, h is used text Represents a sequence of text tokens 324, and uses h box To represent the sequence of control tags 316, the enhanced spatial attention block can be described by the following equations (1)-(3):
[0044] v = v + SelfAttn(v) (1)
[0045] v = v + TS(SelfAttn([v, h box ])) (2)
[0046] v= v + CrossAttn(v, h text ) (3)
[0047] Where TS(·) represents a tag selection operation that specifically considers visual tags, SelfAttn represents the spatial self-attention layer 302, and CrossAttn represents the spatial cross-attention layer 322.
[0048] In the architecture 300, the number of control tags 316 depends on the number of bounding boxes that the video generation model supports to exist simultaneously in an image frame, and the control tags 316 correspond to the bounding boxes one by one. For example, if the video generation model only supports the inclusion of a bounding box for one object in an image frame, the number of control tags 316 is 1; if the video generation model supports the inclusion of 5 bounding boxes for 5 objects in an image frame at the same time, the number of control tags 316 is 5. If the video generation model supports providing 5 bounding boxes simultaneously in an image frame, but only the movement of two objects needs to be controlled in the video to be generated (i.e., only 2 bounding boxes are provided), then the vacant 3 control tags can be filled with learnable specific tags. In the architecture 300, the text tag 324 is not necessary, that is, if the user does not provide a text description of the video to be generated, the missing text tags can be filled with learnable tags.
[0049] like Figure 3 As shown, to generate a control mark 316, the coordinates 310 of the bounding box 328, the unique object identifier 312 for identifying the bounding box 328 (or the object corresponding to the bounding box 328), and the bounding box type 314 can be determined. Each control mark 316 can be defined by the following equation (4):
[0050] t b =MLP(Fourier([b loc , b id , b flag ])) (4)
[0051] where b loc represents the 4-dimensional vector containing the coordinates of the upper left corner and the lower right corner of the bounding box (i.e., coordinate 310), and is normalized to between 0 and 1. id 312, which is used to identify and link bounding boxes between various image frames. flag Indicates the bounding box type 314, for example, 1 indicates a hard bounding box, and 0 indicates a soft bounding box. In addition, Fourier indicates a Fourier embedding operation, and MLP indicates a multi-layer perceptron. In this way, a multi-layer perceptron can be used to make the control mark include a higher level and more abstract semantic feature, so as to improve the performance of the motion control module 318 and improve the motion effect of the generated image frame.
[0052] In some embodiments, b may be represented in the color RGB space.id , where each object corresponds to a bounding box with a unique color such that b id is a vector of 3D RGB values normalized to between 0 and 1. loc , b id and b flag The RGB values are concatenated into a vector and a corresponding embedding is generated via a Fourier embedding operation. The embedding is then input into a multi-layer perceptron 308 to generate a control tag 316. By using RGB values to generate object identifiers, corresponding bounding boxes can be generated in image frames based on the object identifiers during the training phase, thereby facilitating alignment between the generated bounding boxes and the ground truth bounding boxes, thereby improving the effect of model training.
[0053] Using Fourier embedding to encode b loc , b id and b flag , it is ensured that all input dimensions are scaled to between 0 and 1. For any given input x in this range, its Fourier embedding is defined by the following equation (5):
[0054] Fourier(x)=[cos(x·100 0 / 8 ), ..., cos(x·100 7 / 8 ), sin(x·100 0 / 8 ), ..., sin(x·100 7 / 8 )] (5)
[0055] In some embodiments, the Fourier embeddings of each input may be combined to generate an overall embedding, which has a dimension of 128. These embeddings may then be processed through a multilayer perceptron. The multilayer perceptron may have three hidden layers, each with a dimension of 512. The output control mark may then be adjusted to match the dimension of the visual mark (i.e., 1024).
[0056] It should be understood that while control tag 316 is shown in architecture 300 as being generated based on coordinates 310 of bounding box 328, object identifier 312, and bounding box type 314, in some embodiments object identifier 312 and bounding box type 314 are not necessary. For example, in some embodiments, if only one specific type of bounding box (e.g., a hard bounding box) is supported, control tag 316 may be generated based only on coordinates 310. In some embodiments, if only multiple specific types of bounding boxes are supported, control tag 316 may be generated based only on coordinates 310 and object identifier 312.
[0057] In this way, the motion control module 318 can provide accurate motion control information for the original video diffusion model, thereby improving the effect of the generated image frame, so that the object can move according to the movement mode expected by the user. In addition, since the inserted motion control module 318 does not change the structure and parameters of the original video diffusion model, the architecture 300 can reuse the capabilities of the trained video diffusion model, thereby improving the motion control of the object in the video while ensuring the picture quality of the generated video.
[0058] In the training phase, in order to obtain a training data set, training data that meets the conditions can be obtained from an existing publicly accessible video data set. For example, each video in the existing video data set can be evaluated and the embeddings of its start frame and end frame can be compared. If the cosine similarity between the embeddings of the start frame and the end frame is lower than a predetermined threshold, it means that the video shows obvious object motion or camera motion, so that the data can be put into the data set to form a selected data set.
[0059] For a video in a selected dataset, the starting frame of the video can be obtained and an existing model can be used to generate a description of the content of the starting frame. Noun phrases (e.g., young people, white shirts, etc.) can then be extracted from these descriptions for use as object cues. These object cues can then be used to identify a rectangular bounding box that encloses the object in the starting frame. These bounding boxes can then be tracked and propagated across all image frames of the video, so that a large number of objects enclosed by the bounding boxes can be obtained.
[0060] During training, the video can be randomly cropped according to a specific aspect ratio, and then all bounding boxes are projected into the cropped area. If the bounding box is completely outside the cropped area, such a bounding box can be projected as a line segment along the boundary of the cropped area (or a bounding box approximating a line segment). Figure 4 FIG. 4 is a schematic diagram showing an example 400 of generating training data from an existing video dataset according to some embodiments of the present disclosure. Figure 4 As shown, example 400 includes image 402, which includes identified bounding boxes 404, 406, and 408. Since the size of image 402 is wider than the size required by the video generation model, image 402 can be cropped according to the size required by the video generation model to obtain a cropped area 412. Figure 4As shown, bounding box 404 is completely within cropping area 412, so no additional operation is required. A portion of bounding box 406 is outside cropping area 412, so bounding box 406 can be cropped, and only the portion within cropping area 412, i.e., bounding box 416, is retained. In addition, bounding box 408 is completely outside cropping area 412, so it can be projected to bounding box 418 located at the boundary of cropping area 412, and bounding box 418 can be regarded as a line segment or a rectangular box with a very small width, and the height of bounding box 418 is related to the height of bounding box 408. In the training data set, bounding box 418 can indicate that an object enters from outside the image frame or indicates that an object moves from inside the image frame to outside the image frame.
[0061] In this way, training samples that can be used to train the motion control module of the present disclosure can be generated from the existing training data set, solving the problem of lack of training data corresponding to the method for generating videos provided by the embodiment of the present disclosure. In addition, the training data generated in this way has good diversity, thereby improving the training effect.
[0062] In some embodiments, objects in a video can be annotated in three steps. In the first step, dynamic video clips can be selected by comparing the start and end frames of each 4-second video clip in the dataset. In some embodiments, these image frames can be processed and their cosine similarity of feature embeddings at the average pooling layer can be calculated. Video clips with similarity scores below 0.65 will be retained for further processing. In the second step, for each selected video clip, a three-sentence description of the video content can be created, and then noun phrases in these descriptions can be identified. Since most of these phrases are abstract nouns rather than specific object names, these noun phrases can be filtered and only phrases representing specific object names can be retained. Subsequently, these filtered noun phrases can be processed to identify the initial bounding box in the starting frame of the video clip. These bounding boxes can then be tracked in subsequent frames. For each detected object, some bounding boxes can be provided, one for each image frame in the video clip. Then, objects in some image frames where bounding boxes cannot be detected or where the detection confidence is below a threshold can be excluded, so that the successfully tracked exclusive use can be used as ground truth data for training.
[0063] During training, in some embodiments, the motion control module can be trained by adjusting the parameters of the motion control module while fixing the parameters of the base model. In some embodiments, the motion control module can be trained by applying a self-alignment operation, the self-alignment operation comprising: generating an identification image frame based on a target bounding box in a training data set, the identification image frame including an identification bounding box that identifies an object constrained by the target bounding box, and training the motion control module by aligning the identification bounding box with the target bounding box. In some embodiments, a loss between the identification bounding box and the target bounding box can be determined, and the motion control module can be trained by making the loss satisfy a predetermined condition.
[0064] Figure 5 A schematic diagram of an example process 500 of a self-alignment operation according to some embodiments of the present disclosure is shown. Process 500 can train a model to generate a bounding box of a different color for each encoded object in each image frame, and specify the color in the control mark of the object. This method can decompose the problem of associating bounding boxes with objects and maintaining temporal consistency between multiple image frames into two more manageable tasks: generating a bounding box with the correct color for each object, and aligning these boxes with the bounding boxes in each image frame that provide motion control information. In this way, it is ensured that bounding boxes of the same color always surround the same object in different image frames. For hard bounding boxes, the model only needs to generate bounding boxes at specified coordinates, while for soft bounding boxes, bounding boxes can be generated within a specified area. The self-aligned bounding boxes can be used as an intermediate representation, and the model can follow the constraints provided by the target bounding box to guide the generation of these self-aligned bounding boxes, which can then guide the generation of visual objects. After completing the training phase of performing the self-alignment operation, the same dataset can be used to further train the model to eliminate bounding boxes in the generated image frames.
[0065] like Figure 5 As shown, when the process 500 generates image frames 502, 512, 522, 532 and 542, it can predict the bounding boxes 504, 506, 514, 516, 524, 526, 534 and 536 around each controlled object, and draw these bounding boxes into the corresponding image frames as part of the image frames. The bounding boxes identifying the same object are represented by the same color, and the bounding boxes identifying different objects are represented by different colors. For example, the bounding boxes 504, 514, 524 and 534 identifying the same object are all represented by white, the bounding boxes 506, 516, 526 and 536 identifying the same object are all represented by black, and the bounding boxes 504 and 506 identifying different objects are represented by different colors.
[0066] In this way, the self-alignment operation can effectively associate bounding boxes with objects and maintain temporal consistency across multiple frames. In addition, the model can quickly learn to stop generating visible bounding boxes, but the alignment of these bounding boxes still exists. In this way, the self-alignment operation can help the model build a proper internal representation.
[0067] In some embodiments, a first training data set may be obtained, the first training data set including a hard bounding box, and based on the first training data set, a motion control module may be trained by applying a self-alignment operation. In some embodiments, a second training data set may be generated by converting a portion of the bounding box of the first training data set into a soft bounding box, and based on the second training data set, the motion control module may be trained by applying a self-alignment operation. In some embodiments, the motion control module may be trained based on the second training data set by utilizing a loss for a base model without applying a self-alignment operation.
[0068] Figure 6 600 is a flow chart showing an example process of a multi-stage training process according to some embodiments of the present disclosure. Figure 6 As shown, in the first stage described in block 602, process 600 can use all the provided ground truth bounding boxes as hard bounding boxes to train the model. Since the motion control of hard bounding boxes is easier to learn than soft bounding boxes, this stage can be used as a preliminary stage to establish the model's initial understanding of coordinates and object identifiers.
[0069] In the second stage described in block 604, process 600 may replace a portion of the hard bounding box with a soft bounding box to train the model. For example, 80% of the hard bounding box may be replaced with a soft bounding box. Process 600 may perform independent random extensions of the hard bounding box in four directions (not exceeding the boundaries of the image frame) up, down, left, and right, and use the extended bounding box as the soft bounding box. In the first and second stages of blocks 602 and 604, a self-alignment operation may be applied for training.
[0070] In the third stage described in block 606, the process 600 can train the model using the training data in the second stage, but without performing the self-alignment operation. In this way, the first and second stages can effectively enable the model to handle hard bounding boxes and soft bounding boxes, and in the third stage, image frames that do not include self-aligned bounding boxes can be generated. This can not only improve the training efficiency of the model, but also make full use of the self-alignment operation to improve the training effect of the model.
[0071] Figure 7A schematic diagram of an example process 700 for generating a video by generating a hard bounding box and expanding it into a soft bounding box when a hard bounding box in an end frame is provided according to some embodiments of the present disclosure. In the inference stage, a user may identify a bounding box in only a few image frames (e.g., in the start frame and the end frame). Figure 7 As shown, the example process 700 can identify a bounding box 704 in the starting frame 702 and a bounding box 734 in the ending frame 732, indicating that the object identified by the bounding box 704 moves from the location of the bounding box 704 to the location specified by the bounding box 734, and the size of the object in the ending frame 732 should correspond to the size of the bounding box 734. In order to provide more stable motion control, a soft bounding box can be inserted into the intermediate frames 712 and 722 by the model. In some embodiments, a linear interpolation between the bounding boxes 704 and 734 specified by the user can be applied to the intermediate frames 712 and 722 to generate hard bounding boxes 714 and 724. Then, the process 700 can appropriately expand the hard bounding boxes 714 and 724 to generate soft bounding boxes 716 and 726. In this way, the process 700 can generate an intermediate frame 712 with a soft bounding box 716 and an intermediate frame 722 with a soft bounding box 726.
[0072] In this way, we ensure that the object roughly follows the expected trajectory during movement, while the soft bounding box can also introduce various variations to the model, thereby improving the diversity of the generated videos.
[0073] Figure 8 A schematic diagram of an example process of generating a video by generating a hard bounding box and expanding it into a soft bounding box when a motion trajectory of an object is provided according to some embodiments of the present disclosure. Figure 8 As shown, the user provides a bounding box 804 and a motion trajectory 808 in the starting frame 802. The process 800 can generate hard bounding boxes 814, 824, and 834 in subsequent image frames 812, 822, and 832 by interpolation and along the motion trajectory 808. For example, the hard bounding boxes 814, 824, and 834 can be generated by generating hard bounding boxes whose center positions are on the motion trajectory. Then, the process 800 can expand the hard bounding boxes 814, 824, and 834 to generate soft bounding boxes 816, 826, and 836. In this way, the object identified by the bounding box 804 can move along the motion trajectory 808 to the range defined by the soft bounding box 836 in the image frame 832, and the object can remain within the range defined by the soft bounding boxes 816 and 826 in the image frames 812 and 822.
[0074] In this way, users can perform more precise motion control by specifying the motion trajectory of the object, and can achieve effects that cannot be expressed through text description.
[0075] Fig. 9 FIG. 9 is a schematic diagram showing an example 900 of providing a plurality of bounding boxes at a start frame and providing a bounding box close to a right boundary at an end frame according to some embodiments of the present disclosure. Fig. 9 As shown, in example 900, a start frame 902 and a text description 908 (i.e., "a puppy is running toward the camera") are provided, wherein the start frame 902 includes a white puppy and a black puppy. In example 900, the start frame 902 includes a black bounding box 904 identifying the white puppy, and a white bounding box 906 identifying the black puppy. In addition, the end frame 912 includes a black hard bounding box 914 and a white hard bounding box 916, wherein the hard bounding box 916 is close to the right border of the end frame 912 and has a very narrow width (e.g., less than a threshold width), which indicates that the black puppy needs to run out of the screen from the right before the end frame.
[0076] In example 900, the model generates a sequence of image frames 920 in which two puppies gradually run toward the camera, and the white puppy runs to the position specified by the hard bounding box 914 in the end frame, and the size of the white puppy corresponds to the hard bounding box 914. In addition, the black puppy runs out of the right boundary in the end frame, and it can be seen that in the image frame where the black puppy runs out of the right boundary, its size corresponds to the hard bounding box 916.
[0077] Fig.10 A schematic diagram of an example 1000 of providing an object bounding box and a motion trajectory and a bounding box of another object at a start frame and providing the bounding box of the other object at an end frame according to some embodiments of the present disclosure is shown. Fig.10 As shown, in example 1000, a starting frame 1002 and a text description 1010 (i.e., "a person is throwing a Frisbee") are provided, wherein the starting frame 1002 includes a person and a Frisbee. In example 1000, the starting frame 1002 includes a white hard bounding box 1004 identifying the person, a black hard bounding box 1006 identifying the Frisbee, and a motion track 1008 identifying the motion path of the Frisbee.
[0078] In example 1000, the model generates a sequence of image frames 1020 in which the person is located at the position specified by the hard bounding box 1014 in the end frame, and the size of the person corresponds to the hard bounding box 1014. In addition, the Frisbee is thrown by the person in the sequence of image frames 1020, flies along the motion trajectory 1008, and finally flies back to the position of the end point indicated by the motion trajectory 1008.
[0079] It can be seen that the motion control module is able to achieve precise motion control based on bounding boxes, which can manage the motion of foreground and background objects and modify their postures by adjusting smaller components of larger objects. In addition, in scenarios where videos are generated based on images, users can easily select objects by drawing hard bounding boxes around them. This vision-based approach is easier to operate than text- or language-based control. In addition, for intermediate frames that lack user-provided bounding boxes, the motion control module can algorithmically generate soft bounding boxes to approximate motion trajectories. These soft bounding boxes can be constructed based on bounding boxes in user-specified start and end frames, or can be constructed based on user-specified bounding boxes and motion trajectories, thereby enabling precise motion control and improving user experience.
[0080] Fig.11 1 shows a block diagram of an apparatus 1100 for generating a video according to some embodiments of the present disclosure. Fig.11 As shown, the apparatus 1100 includes a visual marker acquisition module 1102, which is configured to acquire visual markers for generating image frames in a video. The apparatus 1100 also includes a control marker acquisition module 1104, which is configured to acquire control markers for constraining position information of objects in the image frames. In addition, the apparatus 1100 also includes a video image generation module 1106, which is configured to generate image frames in the video based on the visual markers and the control markers, wherein the objects in the image frames satisfy the position information.
[0081] It can be understood that by using the device 1100 of the present disclosure, at least one of the many advantages that can be achieved by the method or process described above can be achieved. For example, the device 1100 can use the control mark generated based on the bounding box to understand the position to which the user expects the target object to move, so that the control mark can be used to constrain the content of the generated image frame. In this way, the user can simply and accurately express the desired object movement mode, and compared with only providing a text description or a reference image, this method can improve the motion effect of the generated video and improve the matching degree between the generated video and the user's requirements, thereby improving the user experience.
[0082] Fig.12 1 shows a block diagram of a device 1200 capable of implementing multiple embodiments of the present disclosure. The device 1200 may be a device or apparatus described in the embodiments of the present disclosure. Fig.12As shown, the device 1200 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 1201, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1202 or loaded from a storage unit 1208 to a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The CPU / GPU 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204. Although not shown in FIG. Fig.12 As shown in FIG. 1 , device 1200 may also include a coprocessor.
[0083] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0084] The various methods or processes described above may be executed by the CPU / GPU 1201. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the CPU / GPU 1201, one or more steps or actions in the methods or processes described above may be performed.
[0085] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.
[0086] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0087] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0088] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.
[0089] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0091] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0092] Some example implementations of the present disclosure are listed below.
[0093] Example 1. A method for generating a video, comprising:
[0094] Obtaining visual markers for generating image frames in the video;
[0095] Acquiring a control mark for constraining position information of an object in the image frame; and
[0096] The image frame in the video is generated based on the visual mark and the control mark, wherein the object in the image frame satisfies the position information.
[0097] Example 2. The method of Example 1, wherein the position information is a bounding box, and obtaining the control mark for constraining the position information of the object in the image frame comprises:
[0098] determining coordinates of the bounding box in the image frame; and
[0099] The control mark is generated based on the coordinates.
[0100] Example 3. The method of examples 1-2, wherein generating the control mark based on the coordinates comprises:
[0101] generating an object identifier for the bounding box based on a color of the bounding box; and
[0102] The control mark is generated based on the coordinates and the object identifier.
[0103] Example 4. The method of Examples 1-3, wherein generating the control mark based on the coordinates and the object identifier comprises:
[0104] determining a type of the bounding box, wherein the type includes a first type that constrains a specific position and a specific size of the object to be generated and a second type that constrains a position range and a size range of the object to be generated; and
[0105] The control tag is generated based on the coordinates, the object identifier, and the type.
[0106] Example 5. The method of Examples 1-4, wherein generating the image frame in the video based on the visual marker and the control marker comprises:
[0107] In response to the type of the bounding box being the first type, a center position of the object coincides with a center position of the bounding box, and a size of the object corresponds to a size of the bounding box; and
[0108] In response to the type of the bounding box being the second type, a center position of the object is within the bounding box, and a size of the object does not exceed the bounding box.
[0109] Example 6. The method of Examples 1-5, wherein generating the control tag based on the coordinates, the object identifier, and the type comprises:
[0110] generating a plurality of embeddings based on the coordinates, the object identifier, and the type; and
[0111] The control mark is generated based on the plurality of embeddings by utilizing a multi-layer perceptron.
[0112] Example 7. The method of Examples 1-6, wherein the visual markers are a first set of visual markers, and generating the image frame in the video based on the visual markers and the control markers comprises:
[0113] A second set of visual indicia is generated based on the first set of visual indicia and the control indicia, wherein the number of visual indicia in the first set of visual indicia is the same as the number of visual indicia in the second set of visual indicia.
[0114] Example 8. The method of Examples 1-7, wherein the first set of visual markers and the image frame are generated by a base model, the second set of visual markers are generated by a motion control module, and the method further comprises:
[0115] The motion control module is trained by fixing the parameters of the base model while adjusting the parameters of the motion control module.
[0116] Example 9. The method of Examples 1-8, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation comprises:
[0117] Based on the target bounding box in the training data set, generating an identification image frame, wherein the identification image frame includes an identification bounding box that identifies an object constrained by the target bounding box; and
[0118] The motion control module is trained by aligning the identification bounding box with the target bounding box.
[0119] Example 10. The method of Examples 1-9, wherein training the motion control module by aligning the identification bounding box with the target bounding box comprises:
[0120] determining a loss between the identification bounding box and the target bounding box; and
[0121] The motion control module is trained by making the loss satisfy a predetermined condition.
[0122] Example 11. The method of Examples 1-10, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0123] Obtaining a first training data set, the first training data set including bounding boxes of the first type; and
[0124] The motion control module is trained by applying the self-alignment operation based on the first training data set.
[0125] Example 12. The method of Examples 1-11, wherein training the motion control module by adjusting the parameters of the motion control module while fixing the parameters of the base model further comprises:
[0126] generating a second training data set by converting a portion of a bounding box of the first training data set into a bounding box of the second type; and
[0127] Based on the second training data set, the motion control module is trained by applying the self-alignment operation.
[0128] Example 13. The method of Examples 1-12, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0129] Based on the second training dataset, the motion control module is trained by utilizing a loss for the basis model without applying the self-alignment operation.
[0130] Example 14. An apparatus for generating a video, comprising:
[0131] A visual marker acquisition module, configured to acquire visual markers for generating image frames in the video;
[0132] a control mark acquisition module, configured to acquire a control mark for constraining position information of an object in the image frame; and
[0133] The video image generation module is configured to generate the image frame in the video based on the visual mark and the control mark, wherein the object in the image frame satisfies the position information.
[0134] Example 15. The apparatus of Example 14, wherein the position information is a bounding box, and obtaining the control mark for constraining the position information of the object in the image frame comprises:
[0135] a coordinate determination module configured to determine the coordinates of the bounding box in the image frame; and
[0136] The coordinate using module is configured to generate the control mark based on the coordinates.
[0137] Example 16. The apparatus of Examples 14-15, wherein generating the control mark based on the coordinates comprises:
[0138] an identifier generation module configured to generate an object identifier for the bounding box based on the color of the bounding box; and
[0139] An identifier using module is configured to generate the control mark based on the coordinates and the object identifier.
[0140] Example 17. The apparatus of Examples 14-16, wherein generating the control mark based on the coordinates and the object identifier comprises:
[0141] a type determination module configured to determine a type of the bounding box, wherein the type includes a first type that constrains a specific position and a specific size of the object to be generated and a second type that constrains a position range and a size range of the object to be generated; and
[0142] A type usage module is configured to generate the control tag based on the coordinates, the object identifier, and the type.
[0143] Example 18. The apparatus of Examples 14-17, wherein generating the image frame in the video based on the visual marker and the control marker comprises:
[0144] In response to the type of the bounding box being the first type, a center position of the object coincides with a center position of the bounding box, and a size of the object corresponds to a size of the bounding box; and
[0145] In response to the type of the bounding box being the second type, a center position of the object is within the bounding box, and a size of the object does not exceed the bounding box.
[0146] Example 19. The apparatus of Examples 14-18, wherein generating the control tag based on the coordinates, the object identifier, and the type comprises:
[0147] an embedding generation module configured to generate a plurality of embeddings based on the coordinates, the object identifier, and the type; and
[0148] The embedding using module is configured to generate the control mark by utilizing a multi-layer perceptron based on the plurality of embeddings.
[0149] Example 20. The apparatus of Examples 14-19, wherein the visual markers are a first set of visual markers, and generating the image frame in the video based on the visual markers and the control markers comprises:
[0150] A visual mark generation module is configured to generate a second set of visual marks based on the first set of visual marks and the control marks, wherein the number of visual marks in the first set of visual marks is the same as the number of visual marks in the second set of visual marks.
[0151] Example 21. The apparatus of Examples 14-20, wherein the first set of visual markers and the image frame are generated by a base model, the second set of visual markers are generated by a motion control module, and the apparatus further comprises:
[0152] The motion control training module is configured to train the motion control module by fixing the parameters of the base model and adjusting the parameters of the motion control module.
[0153] Example 22. The apparatus of Examples 14-21, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation comprises:
[0154] a marking frame generating module configured to generate a marking image frame based on a target bounding box in a training data set, wherein the marking image frame includes a marking bounding box identifying an object constrained by the target bounding box; and
[0155] A bounding box alignment module is configured to train the motion control module by aligning the identification bounding box with the target bounding box.
[0156] Example 23. The apparatus of Examples 14-22, wherein training the motion control module by aligning the identification bounding box with the target bounding box comprises:
[0157] a loss determination module configured to determine a loss between the identification bounding box and the target bounding box; and
[0158] The loss using module is configured to train the motion control module by making the loss satisfy a predetermined condition.
[0159] Example 24. The apparatus of Examples 14-23, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0160] A first training set acquisition module is configured to acquire a first training data set, where the first training data set includes a bounding box of the first type; and
[0161] The first training set using module is configured to train the motion control module by applying the self-alignment operation based on the first training data set.
[0162] Example 25. The apparatus of Examples 14-24, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0163] A second training set acquisition module is configured to generate a second training data set by converting a portion of a bounding box of the first training data set into a bounding box of the second type; and
[0164] The second training set using module is configured to train the motion control module by applying the self-alignment operation based on the second training data set.
[0165] Example 26. The apparatus of Examples 14-25, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0166] A third training set using module is configured to train the motion control module based on the second training data set by utilizing the loss for the base model without applying the self-alignment operation.
[0167] Example 27. An electronic device comprising:
[0168] Processor; and
[0169] A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the electronic device to perform actions when executed by the processor, the actions comprising:
[0170] Obtaining visual markers for generating image frames in the video;
[0171] Acquiring a control mark for constraining position information of an object in the image frame; and
[0172] The image frame in the video is generated based on the visual mark and the control mark, wherein the object in the image frame satisfies the position information.
[0173] Example 28. The apparatus of Example 27, wherein obtaining the control mark for constraining a bounding box of an object in the image frame comprises:
[0174] determining coordinates of the bounding box in the image frame; and
[0175] The control mark is generated based on the coordinates.
[0176] Example 29. The apparatus of Examples 27-28, wherein generating the control mark based on the coordinates comprises:
[0177] generating an object identifier for the bounding box based on a color of the bounding box; and
[0178] The control mark is generated based on the coordinates and the object identifier.
[0179] Example 30. The apparatus of Examples 27-29, wherein generating the control mark based on the coordinates and the object identifier comprises:
[0180] determining a type of the bounding box, wherein the type includes a first type that constrains a specific position and a specific size of the object to be generated and a second type that constrains a position range and a size range of the object to be generated; and
[0181] The control tag is generated based on the coordinates, the object identifier, and the type.
[0182] Example 31. The apparatus of Examples 27-30, wherein generating the image frame in the video based on the visual marker and the control marker comprises:
[0183] In response to the type of the bounding box being the first type, a center position of the object coincides with a center position of the bounding box, and a size of the object corresponds to a size of the bounding box; and
[0184] In response to the type of the bounding box being the second type, the center position of the object is within the bounding box, and the size of the object does not exceed the bounding box.
[0185] Example 32. The apparatus of Examples 27-31, wherein generating the control tag based on the coordinates, the object identifier, and the type comprises:
[0186] generating a plurality of embeddings based on the coordinates, the object identifier, and the type; and
[0187] The control mark is generated based on the plurality of embeddings by utilizing a multi-layer perceptron.
[0188] Example 33. The apparatus of Examples 27-32, wherein the visual markers are a first set of visual markers, and generating the image frame in the video based on the visual markers and the control markers comprises:
[0189] A second set of visual indicia is generated based on the first set of visual indicia and the control indicia, wherein the number of visual indicia in the first set of visual indicia is the same as the number of visual indicia in the second set of visual indicia.
[0190] Example 34. The apparatus of Examples 27-33, wherein the first set of visual markers and the image frame are generated by a base model, the second set of visual markers are generated by a motion control module, and the apparatus further comprises:
[0191] The motion control module is trained by fixing the parameters of the base model while adjusting the parameters of the motion control module.
[0192] Example 35. The apparatus of Examples 27-34, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation comprises:
[0193] Based on the target bounding box in the training data set, generating an identification image frame, wherein the identification image frame includes an identification bounding box that identifies an object constrained by the target bounding box; and
[0194] The motion control module is trained by aligning the identification bounding box with the target bounding box.
[0195] Example 36. The apparatus of Examples 27-35, wherein training the motion control module by aligning the identification bounding box with the target bounding box comprises:
[0196] determining a loss between the identification bounding box and the target bounding box; and
[0197] The motion control module is trained by making the loss satisfy a predetermined condition.
[0198] Example 37. The apparatus of Examples 27-36, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0199] Obtaining a first training data set, the first training data set including bounding boxes of the first type; and
[0200] The motion control module is trained by applying the self-alignment operation based on the first training data set.
[0201] Example 38. The apparatus of Examples 27-37, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0202] generating a second training data set by converting a portion of a bounding box of the first training data set into a bounding box of the second type; and
[0203] Based on the second training data set, the motion control module is trained by applying the self-alignment operation.
[0204] Example 39. The apparatus of Examples 27-38, wherein training the motion control module by adjusting parameters of the motion control module while fixing parameters of the base model further comprises:
[0205] Based on the second training dataset, the motion control module is trained by utilizing a loss for the basis model without applying the self-alignment operation.
[0206] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for generating a video, comprising: Acquire a visual mark for generating an image frame in the video, wherein the visual mark indicates visual information of the image frame to be generated; Acquire a control mark for constraining position information of an object in the image frame, wherein the control mark is generated based on the position information and indicates motion control information for the object; as well as The image frame in the video is generated based on the visual marker and the control marker by a video generation model having a motion control module, wherein the object in the image frame satisfies the position information.
2. The method of claim 1 , wherein the position information is a bounding box, and acquiring the control mark for constraining the position information of the object in the image frame comprises: determining coordinates of the bounding box in the image frame; as well as The control mark is generated based on the coordinates.
3. The method of claim 2, wherein generating the control mark based on the coordinates comprises: generating an object identifier for the bounding box based on a color of the bounding box; as well as The control mark is generated based on the coordinates and the object identifier.
4. The method of claim 3, wherein generating the control mark based on the coordinates and the object identifier comprises: Determining a type of the bounding box, wherein the type includes a first type that constrains a specific position and a specific size of the object to be generated and a second type that constrains a position range and a size range of the object to be generated; as well as The control tag is generated based on the coordinates, the object identifier, and the type.
5. The method of claim 4, wherein generating the image frame in the video based on the visual marker and the control marker comprises: In response to the type of the bounding box being the first type, a center position of the object coincides with a center position of the bounding box, and a size of the object corresponds to a size of the bounding box; as well as In response to the type of the bounding box being the second type, a center position of the object is within the bounding box, and a size of the object does not exceed the bounding box.
6. The method of claim 5, wherein generating the control mark based on the coordinates, the object identifier, and the type comprises: generating a plurality of embeddings based on the coordinates, the object identifier, and the type; as well as The control mark is generated based on the plurality of embeddings by utilizing a multi-layer perceptron.
7. The method of claim 5, wherein the visual markers are a first set of visual markers, and generating the image frame in the video based on the visual markers and the control markers comprises: A second set of visual indicia is generated based on the first set of visual indicia and the control indicia, wherein the number of visual indicia in the first set of visual indicia is the same as the number of visual indicia in the second set of visual indicia.
8. The method of claim 7, wherein the first set of visual markers and the image frame are generated by a base model, the second set of visual markers are generated by the motion control module, and the method further comprises: The motion control module is trained by fixing the parameters of the base model while adjusting the parameters of the motion control module.
9. The method of claim 8, wherein the motion control module is trained by applying a self-alignment operation, and the self-alignment operation comprises: Based on the target bounding box in the training data set, generating an identification image frame, wherein the identification image frame includes an identification bounding box that identifies the object constrained by the target bounding box; as well as The motion control module is trained by aligning the identification bounding box with the target bounding box.
10. The method of claim 9, wherein training the motion control module by aligning the identification bounding box with the target bounding box comprises: determining a loss between the identification bounding box and the target bounding box; as well as The motion control module is trained by making the loss satisfy a predetermined condition.
11. The method according to claim 9, wherein training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module further comprises: Acquire a first training data set, wherein the first training data set includes bounding boxes of the first type; as well as The motion control module is trained by applying the self-alignment operation based on the first training data set.
12. The method according to claim 11, wherein training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module further comprises: generating a second training data set by converting a portion of a bounding box of the first training data set into a bounding box of the second type; as well as Based on the second training data set, the motion control module is trained by applying the self-alignment operation.
13. The method according to claim 12, wherein training the motion control module by fixing the parameters of the base model while adjusting the parameters of the motion control module further comprises: Based on the second training dataset, the motion control module is trained by utilizing a loss for the basis model without applying the self-alignment operation.
14. An apparatus for generating a video, comprising: A visual mark acquisition module, configured to acquire a visual mark for generating an image frame in the video, wherein the visual mark indicates visual information of the image frame to be generated; a control mark acquisition module, configured to acquire a control mark for constraining position information of an object in the image frame, wherein the control mark is generated based on the position information and indicates motion control information for the object; as well as The video image generation module is configured to generate the image frame in the video based on the visual marker and the control marker by using a video generation model with a motion control module, wherein the object in the image frame satisfies the position information.
15. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 13.
16. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions which, when executed, cause a machine to implement the method according to any one of claims 1 to 13.
Citation Information
Patent Citations
Video generation method and server
CN116233491A
Video generation method and device, model training method and device, equipment and medium
CN116320216A
Video acquisition method and device, storage medium and electronic equipment
CN116664726A
Video generation method, and method and device for training video generation model
CN116863003A
Computerized system and method for fine-grained event detection and content hosting therefrom
US20220309279A1
