Method, apparatus, device and computer program product for generating video

Abstract

The present disclosure relates to a method, apparatus, device and computer program product for generating a video. The method includes obtaining content information related to the content of the video to be generated, and the content information includes at least one of text or an image. The method also includes obtaining position information indicating the position of an object in the video in a starting frame. The method also includes obtaining control information that constrains the position of the object in an ending frame. In addition, the method also includes generating the video based on the content information, the position information, and the control information. In this way, the user can use the position information in the starting frame to accurately identify the object to be controlled. In addition, the user can accurately express the position of the identified object moving from the starting frame to the ending frame, so that the application can receive accurate motion control requirements, thereby generating a video that meets the user's expectations.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence, and more specifically, to methods, apparatuses, electronic devices, and computer program products for generating videos. Background Art

[0002] Text-guided video generation is a technology that uses natural language text to guide video content generation. Through deep learning and artificial intelligence technology, the system can understand the input text description and convert it into specific visual content to generate the corresponding video. This method can be applied to film production, virtual reality, animation production and other fields, providing creators with a more intuitive and efficient way to express their creativity.

[0003] Motion control refers to the precise control of the movement of objects or cameras to achieve various special effects and dynamic effects in videos. This technology can be achieved through programming or automated systems, making video production more creative and dynamic. Motion control is widely used in the fields of film, virtual reality, etc., providing viewers with a more immersive and fascinating visual experience.

[0004] Text-guided video generation combined with motion control enables more intelligent and personalized video creation. With text guidance, creators can describe the scenes and effects they want in natural language, while motion control ensures that these ideas can be presented in the video in an accurate and smooth way, providing greater flexibility and creativity in the creation process. Summary of the invention

[0005] In a first aspect of an embodiment of the present disclosure, a method for generating a video is provided. The method includes obtaining content information related to the content of the video to be generated, and the content information includes at least one of text or an image. The method also includes obtaining position information indicating the position of an object in the video in a starting frame. The method also includes obtaining control information that constrains the position of the object in an ending frame. In addition, the method also includes generating the video based on the content information, the position information, and the control information.

[0006] In a second aspect of an embodiment of the present disclosure, a device for generating a video is provided. The device includes a content information acquisition module configured to acquire content information related to the content of the video to be generated, the content information including at least one of text or image. The device also includes a position information acquisition module configured to acquire position information indicating the position of an object in the video in a starting frame. The device also includes a control information acquisition module configured to acquire control information constraining the position of the object in an ending frame. In addition, the device also includes a video generation module configured to generate the video based on the content information, the position information, and the control information.

[0007] In a third aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for generating a video. The method includes obtaining content information related to the content of the video to be generated, and the content information includes at least one of text or an image. The method also includes obtaining position information indicating the position of an object in the video in the starting frame. The method also includes obtaining control information that constrains the position of the object in the ending frame. In addition, the method also includes generating the video based on the content information, the position information, and the control information.

[0008] In a fourth aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine executable instructions, which, when executed, cause a machine to implement a method for generating a video. The method includes obtaining content information related to the content of the video to be generated, the content information including at least one of text or an image. The method also includes obtaining position information indicating the position of an object in the video in a starting frame. The method also includes obtaining control information constraining the position of the object in an ending frame. In addition, the method also includes generating the video based on the content information, the position information, and the control information.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0012] Figure 2 A flowchart of a method for generating a video according to some embodiments of the present disclosure is shown;

[0013] Figure 3A-3B A schematic diagram showing an example in which a user inputs multiple bounding boxes in a start frame and an end frame respectively according to some embodiments of the present disclosure;

[0014] Figure 4A-4B A schematic diagram showing an example of a bounding box and a motion trajectory of an object input by a user in a starting frame according to some embodiments of the present disclosure;

[0015] Figure 5A-5B A schematic diagram showing an example in which a user inputs multiple soft bounding boxes and does not input a text description in an end frame according to some embodiments of the present disclosure;

[0016] Figure 6A-6B A schematic diagram showing an example of a user inputting multiple bounding boxes on a starting frame where no image is provided according to some embodiments of the present disclosure;

[0017] Figure 7 A schematic diagram showing an example in which a user creates an intermediate frame and inputs multiple soft bounding boxes in the intermediate frame according to some embodiments of the present disclosure;

[0018] Figure 8 A block diagram showing an apparatus for generating a video according to some embodiments of the present disclosure; and

[0019] Fig. 9 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] It is understood that all user-related data involved in this technical solution should be obtained and used after the user's authorization. This means that in this technical solution, if the user's personal information needs to be used, the user's explicit consent and authorization are required before obtaining this data, otherwise the relevant data will not be collected and used. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly observed during the collection, use and storage of data, and necessary technologies and measures should be taken to protect the user's data security and ensure the safe use of data.

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0023] In some text or reference image guided video generation scenarios, users want to provide information about the movement of objects in the generated video by entering text descriptions. For example, a user can provide a reference image of a building, and then enter the text description "tilt the camera upwards and reveal the top of the building". At this point, the user hopes that in the generated video, the camera gradually raises the lens upward from the perspective of shooting the ground and finally shoots the top of the building. However, in the related art, although it is possible to generate videos with high picture quality and slow lens movement based on the reference images and text descriptions provided by the user, the model cannot well understand the user's requirements for the movement of objects in the video, and thus cannot accurately reveal the top of the building in the generated video.

[0024] In addition, in some usage scenarios, when the user's requirements for the movement method are very precise, it is difficult to accurately describe the desired picture in words. For example, when the user wants two puppies to run towards the camera in the video, one of the white puppies is getting closer and closer to the camera and runs to the center of the screen. At this time, the white puppy fills one-third of the screen. At the same time, the other black puppy is also getting closer and closer to the camera, but it is running towards the toy next to the camera, so it deviates more and more from the center of the screen and eventually disappears from the right side of the screen. For ordinary users, it is very difficult to accurately describe such movement requirements, and thus it is impossible to generate the desired video.

[0025] To this end, an embodiment of the present disclosure provides a scheme for generating a video. In this scheme, a user can input content information related to the content of the video to be generated in a user interface provided by an application. Such content information can be a text description, a key frame of a video, or both. Then, the user can use a bounding box to identify a motion-controlled object in a starting frame, and input control information of how the object is expected to move in the user interface, and the control information at least includes the position of the object in the ending frame. Then, the application can generate a video between the content information, the bounding box identifying the object, and the control information.

[0026] In this way, the user can use the bounding box in the start frame to accurately identify the object they want to control. In addition, the user can accurately express the position of the identified object from the start frame to the end frame, so that the application can receive precise motion control requirements and generate a video that meets the user's expectations.

[0027] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1 As shown, environment 100 includes user 102 and computing device 104. Computing device 104 may be a user terminal, a mobile device, a computer, or a computing system, a single server, a distributed server, or a cloud-based server. Computing device 104 includes application 106, which may provide a function of generating a video. Application 106 may be any application such as a local application, a remote application, an application in a browser / server architecture, an application in a client / server architecture, or the like.

[0028] like Figure 1 As shown, in the environment 100, the user 102 can interact with the application 106 via the user interface 108. The user 102 can input content information 110 related to the content of the video to be generated on the user interface 108. The content information 110 can be a text description 112, or a reference image 114, or both can be provided at the same time. For example, the text description 112 can be "two puppies running towards the camera", and the reference image 114 can be an image of two puppies running on the grass. It should be noted that in some implementations, the text description 112 can be provided indirectly via user interaction methods such as voice, so the text description 112 also includes text provided indirectly via methods such as voice. The user 102 can identify the object whose movement he wants to control by drawing bounding boxes 118-1, 118-2, ..., 118-N (collectively referred to as bounding boxes 118) in the area where the start frame 116 is displayed. It should be understood that although in Figure 1, but the user can also draw only one bounding box 118 to control the motion of only one object. If the content information 110 includes a reference image 114, the reference image 114 can be displayed in the area of ​​the starting frame 116 so that the user can use the bounding box 118 to identify the object whose motion is to be controlled.

[0029] like Figure 1 As shown, in the environment 100, the user 102 may input control information 120 via the user interface 108 to at least describe the position of the controlled object in the end frame of the video. In some embodiments, the control information 120 may be a bounding box drawn by the user 102 in the end frame area to indicate the position to which the controlled object is expected to move. In some embodiments, the control information 120 may be a motion track of the controlled object drawn by the user 102 in the start frame 116 area.

[0030] In the environment 100, after the user 102 inputs the content information 110, the bounding box 118 in the starting frame 116, and the control information 120, the application 106 may generate a video 122 based on the user inputs, and provide the video 122 to the user 102 via the user interface 108. For example, the computing device 104 may send the user inputs to a server and receive the address of the video 122 from the server, or the computing device 104 may generate the video 122 locally. In the user interface 108, the video 122 may be displayed to the user 102 via a video playback control, or may be provided to the user 102 in the form of a download control. The content of the video 122 will be related to the content information 110, and the controlled object identified by the bounding box 118 in the video 122 will move from the position in the starting frame 116 to the specified position in the ending frame according to the constraints of the control information 120.

[0031] It should be understood that although the content information 110, the start frame 116, and the control information 120 are included in one user interface 108 in the environment 100, in some embodiments, the user 102 may input these information in different user interfaces. In addition, the video 122 may also be provided to the user 102 in a separate user interface.

[0032] In this way, the user 102 can accurately identify the object that he wants to control using the bounding box 118 in the starting frame 116. In addition, the user 102 can accurately express the position of the identified object moving from the starting frame 116 to the ending frame, so that the application 106 can receive accurate motion control requirements, thereby generating a video that meets the user's expectations.

[0033] Figure 2 FIG. 2 is a flow chart of a method 200 for generating a video according to some embodiments of the present disclosure. Figure 2 As shown, at block 202, method 200 may obtain content information related to the content of the video to be generated, the content information including at least one of text or image. Figure 1 In the illustrated environment 100 , a computing device 104 may obtain content information 110 related to the content of a video to be generated, which is input by a user 102 . The content information 110 includes at least one of a text description 112 and a reference image 114 .

[0034] At block 204, the method 200 may obtain position information indicating the position of the object in the video in the starting frame. The position information may be information associated with the position, such as a bounding box, an outline, a coordinate value, a coordinate range, etc. For example, in Figure 1 In the illustrated environment 100, the computing device 104 may obtain a bounding box 118 input by the user 102 in the starting frame 116, and the bounding box 118 is used to identify the object under motion control in the starting frame 116. When the user 102 provides a reference image 114, the content of the starting frame 116 may be the reference image 114, so that the user can directly identify the object under motion control on the reference image 114. When the user 102 does not provide a reference image 114, the content of the starting frame 116 may be blank, and the user can use the bounding box 118 to identify the position and size of the object under motion control in the blank area.

[0035] At block 206, method 200 may obtain control information for constraining the position of the object in the ending frame. Figure 1 In the environment 100 shown, the computing device 104 can obtain control information 120 input by the user 102, and the control information 120 can constrain the position of the controlled object identified by the bounding box 118 in the end frame. In some embodiments, the user 102 can provide the control information 120 by drawing a bounding box in the end frame. In some embodiments, the user can provide the control information 120 by drawing a motion trajectory in the start frame 116.

[0036] At block 208, method 200 may generate a video based on the content information, the location information, and the control information. Figure 1 In the illustrated environment 100, the computing device 104 may generate a video 122 based on the content information 110, the bounding box 118, and the control information 120. The content of the video 122 will be related to the content information 110, and the controlled object identified by the bounding box 118 in the video 122 will move from the position in the starting frame 116 to the specified position according to the constraints of the control information 120.

[0037] In this way, the user can use the position information in the start frame to accurately identify the object they want to control. In addition, the user can accurately express the position of the identified object moving from the start frame to the end frame, so that the application can receive accurate motion control requirements and generate a video that meets the user's expectations.

[0038] In some embodiments, the object is a first object, the position information is a first bounding box having a first color, the control information is first control information having a first color, and generating a video may include obtaining a second bounding box indicating a position of a second object in the video in a start frame, the second bounding box having a second color different from the first color. In addition, second control information constraining a position of the second object in an end frame may also be obtained, the second control information having a second color. Then, a video may be generated based on the content information, the first bounding box, the first control information, the second bounding box, and the second control information.

[0039] In some embodiments, the position information is a starting bounding box, the control information is an ending bounding box of the object in the ending frame, and the starting bounding box and the ending bounding box are rectangular boxes. In some embodiments, in response to a user selecting the first type as the type of the ending bounding box, a video may be generated by moving the object from a position indicated by the starting bounding box to a specific position indicated by the ending bounding box, wherein the size of the object in the ending frame corresponds to the ending bounding box. In some embodiments, the image content identified by the bounding box in the starting frame may be determined as the object, and a video may be generated based on the image, the object, and the control information of the starting frame, wherein the content of the video is associated with the image.

[0040] In some embodiments, in response to the end bounding box being close to the left or right boundary of the end frame and the width of the end bounding box being less than a threshold width, the video can be generated by moving the object from the position in the start frame to outside the left or right boundary of the end frame, and the size of the object when it moves outside the left or right boundary is associated with the height of the end bounding box. In some embodiments, in response to the end bounding box being close to the upper or lower boundary of the end frame and the height of the end bounding box being less than a threshold height, the video is generated by moving the object from the position in the start frame to outside the upper or lower boundary of the end frame, and the size of the object when it moves outside the upper or lower boundary is associated with the width of the end bounding box, wherein the content of the video is associated with the content information.

[0041] In some embodiments, in response to the start bounding box being close to the left or right boundary of the start frame and the width of the start bounding box being less than a threshold width, a video is generated by moving an object from outside the left or right boundary to a position constrained by control information, and the size of the object when entering the left or right boundary is associated with the height of the start bounding box. In some embodiments, in response to the start bounding box being close to the upper or lower boundary of the start frame and the height of the start bounding box being less than a threshold height, a video is generated by moving an object from outside the upper or lower boundary of the start frame to a position constrained by control information, and the size of the object when entering the upper or lower boundary is associated with the width of the end bounding box, wherein the content of the video is associated with the content information.

[0042] In some embodiments, the object gradually moves from a first position to a second position in the generated video. In some embodiments, the generated video causes the object to move relative to the camera by changing the camera viewing angle.

[0043] Figure 3A-3B FIG. 3 is a schematic diagram showing an example 300 in which a user inputs multiple bounding boxes at a start frame and an end frame, respectively, according to some embodiments of the present disclosure. Figure 3A As shown, example 300 includes a user interface 302, which provides a control 304 for inputting a text description. A user can enter a text description for describing the content of the video to be generated in the text input control 304. For example, in example 300, the user enters the text description "two puppies running towards the camera", and the content of the generated video should include two puppies running towards the camera. User interface 302 also provides a selection picture control 306 for inputting a reference image, and the user can select a reference image by interacting with control 306. When the user selects a reference image, the reference image can guide the content of the generated video. For example, in example 300, the user selects an image of two puppies (including a white puppy and a black puppy) running on the grass, and the generated video should include the two puppies running on the grass as the background.

[0044] like Figure 3AAs shown, the user interface 302 also provides a start frame control 308. When the user selects a reference image, the content of the start frame control 308 can display the reference image. When the user does not select a reference image, the content of the start frame control 308 can be blank (similar to the control 318 used to indicate the end frame in FIG. 3). The user interface 302 also provides a color display control 310 corresponding to object 1 and a color display control 312 corresponding to object 2 associated with the start frame control 308, wherein the color corresponding to each object is different. When the user interacts with the color display control 310 or 312, it can be selected which object is currently to be identified on the start frame control 308. For example, in example 300, object 1 is selected, indicating that the next operation to be performed in the start frame control 308 is an operation for object 1. The user interface 302 also provides an operation type selection control 314, and the operation type includes a bounding box and a motion track. When the value of the operation type selection control 314 is a bounding box, the start bounding box for identifying object 1 can be input in the start frame control 308 by dragging. When the value of the operation type selection control 314 is a motion track, the motion track for the object 1 can be input in the start frame control 308 in a brush manner.

[0045] like Figure 3A As shown, the user interface 302 also provides an end frame control 318, the content of the end frame control 318 can be blank, and the user can draw an end bounding box in the end frame control 318, and the end bounding box constrains the position and size of the object to which it moves. The user interface 302 also provides a color display control 320 corresponding to object 1 and a color display control 312 corresponding to object 2 associated with the end frame control 318, wherein the value of the color display control 320 is the same as the value of the color display control 310, and the value of the color display control 322 is the same as the value of the color display control 312. In other words, the same object is represented by the same color in the start frame control 308 and the end frame control 318. It should be understood that although specific colors such as black and white are assigned to each object in the embodiment of the present disclosure, it is not intended to limit the color corresponding to each object, and the color may also be other colors such as yellow, purple, green, etc.

[0046] like Figure 3AAs shown, the user interface 302 also provides a bounding box type selection control 324. There are two types of bounding boxes, hard bounding boxes and soft bounding boxes. The hard bounding box is used to specify a specific position and a specific size of an object, which means that in the generated image frame, the object is generated at the coordinates specified by the hard bounding box (for example, the center coordinates of the bounding box), and the size of the object corresponds to the size of the hard bounding box. The soft bounding box is used to specify the position range and size range of the object, which means that in the generated image frame, the object is generated within the range framed by the soft bounding box, and the size of the object does not exceed the range. When the user selects the object 1 to be operated in the end frame and the value of the bounding box type control is a hard bounding box, then in the end frame control 318, for example, a hard bounding box that constrains the specific position of the object 1 after the movement can be drawn in a dragging manner. When the user selects the object 1 to be operated in the end frame and the value of the bounding box type control is a soft bounding box, then in the end frame control 318, a soft bounding box that constrains the position range of the object 1 after the movement can be drawn.

[0047] like Figure 3A As shown, in example 300, the user identifies the white puppy as object 1 with a black bounding box 330 in the start frame control 308, and identifies the specific position and specific size to which object 1 should move with a black hard bounding box 334 in the end frame control 318. In addition, the user identifies the black puppy as object 2 with a white bounding box 332 in the start frame control 308, and identifies the position range and size range to which object 2 should move with a white hard bounding box 336 in the end frame control 318.

[0048] like Figure 3A As shown, in the end frame control 318 of the example 300, the hard bounding box 336 of the object 2 is close to the right border of the end frame, and its width is narrow (e.g., less than the threshold width), which means that the black puppy corresponding to the object 2 will run out of the right border of the video, and when the black puppy runs out of the right border, its size is associated with the height of the hard bounding box 336. In other words, the height of the hard bounding box 336 is the projection of its height onto the right border when the object 2 moves out of the right border.

[0049] Then, the user can click the video generation control 328 to generate a video, and the generated video is as follows: Figure 3B As shown. Figure 3B In the video 340 shown, the white puppy (object 1) runs from the position in the starting frame to the position specified by the hard bounding box 334 in the ending frame, and the size of the white puppy in the ending frame corresponds to the hard bounding box 334. At the same time, the black puppy (object 2) runs out of the screen from the position in the starting frame, and when the black puppy starts to run out of the screen in the third frame, its height is associated with the hard bounding box 324.

[0050] In this way, users can use different colors to identify multiple controlled objects, so that the content of the generated video can be made more dynamic and rich. In addition, by identifying the bounding box in the end frame, the position and size of the object can be accurately controlled, so that users can accurately express and control the movement of the object, improve the motion effect of the generated video, and enhance the user experience. In addition, it is also possible to achieve the effect of making the object move out of the picture, and to specify the position and size when moving out of the picture, so as to provide a wealth of motion control instructions to meet the needs of users in different scenarios.

[0051] In some embodiments, the position information is a bounding box, and the control information is a motion track drawn on the starting frame. In some embodiments, a video can be generated by moving the object along the motion track starting from the position indicated by the bounding box, wherein the content of the video is associated with the content information.

[0052] Figure 4A-4B A schematic diagram of an example 400 of a user inputting a bounding box and a motion trajectory of an object in a starting frame according to some embodiments of the present disclosure is shown. Figure 4A As shown, in example 400, the user enters a text description "a person throwing a Frisbee" in the text input control 404 of the user interface 402, and as shown in the starting frame control 408, the user selects a reference image, which includes a person holding a Frisbee. In example 400, the user identifies object 1 (i.e., Frisbee) with a black bounding box 432 in the starting frame control 408. Then, the user selects the value of the operation type selection control 414 as motion trajectory, and depicts the motion trajectory 434 of the Frisbee in the starting frame control 408. Figure 4A As shown, the motion trajectory 434 of the flying disc is similar to a "U" shape, and the starting position and the ending position are close, which indicates that the flying disc will fly out from the starting position and eventually fly back to the position specified by the arrow of the trajectory. In addition, the user identifies the person as object 2 with a white bounding box 430 in the starting frame control 408, and identifies a hard bounding box 436 of object 2 in the ending frame control 418, which indicates that object 2 will move from the position in the starting frame to the position specified by the hard bounding box 436, and the size after the movement corresponds to the size of the hard bounding box 436.

[0053] Figure 4B The generated video 440 is shown, as shown in the figure, the flying disc (object 1) flies from the starting position along the motion trajectory 434, and finally flies to the end position of the motion trajectory 434. At the same time, after a series of movements, the person (object 2) is located at the position specified by the hard bounding box 436, and the size of the person corresponds to the hard bounding box 436.

[0054] In this way, users can control the movement of objects by describing the movement trajectory of the object, and because the movement trajectory contains richer information about the movement process, the movement of the object can be controlled more accurately. In addition, for slightly complex movement trajectories, it is difficult for users to express them through text descriptions, so describing the movement trajectory allows users to easily express how the object moves, thereby improving the user experience.

[0055] In some embodiments, in response to a user selecting the second type as the type of the ending bounding box, a video is generated by moving the object from a position indicated by the starting bounding box to a range of positions indicated by the ending bounding box, wherein the size of the object in the ending frame does not exceed the ending bounding box, and the content of the video is associated with the content information.

[0056] Figure 5A-5B A schematic diagram of an example 500 in which a user enters multiple soft bounding boxes and does not enter a text description at the end frame according to some embodiments of the present disclosure is shown. Figure 5A As shown, in example 500, the user does not enter anything in the text input control 504 of the user interface 502, but selects a reference image. As shown in the start frame control 508, there are multiple eggs placed in a basket in the reference image. In example 500, the user marks object 1 (i.e., the egg located on the right side of the basket) with a black bounding box 528 in the start frame control 508, marks object 2 (i.e., the egg located behind the basket) with a white bounding box 530, and marks object 3 (i.e., the egg located on the left side of the basket) with a gray bounding box 532. In the end frame, the user selects the values ​​of the bounding box type selection controls 522, 524, and 526 of object 1, object 2, and object 3 as soft bounding boxes, and specifies the position range and size range to which the egg on the right side of the basket moves with a black soft bounding box 534 in the end frame control 518, that is, the position of the egg on the right side of the basket in the end frame does not exceed the range specified by the soft bounding box 534 and the size does not exceed the soft bounding box 534. In addition, the user also specifies the position range and size range to which the eggs on the left side of the basket move using white soft bounding box 536 in the end frame control 518, and specifies the position range and size range to which the eggs on the back side of the basket move using gray soft bounding box 538.

[0057] Figure 5BThe generated video 540 is shown, and as shown in the figure, the egg (object 1) on the right side of the basket moves from the starting position to the area defined by the soft bounding box 534, and the size does not exceed the range defined by the soft bounding box 534. The egg (object 2) behind the basket moves from the starting position to the area defined by the soft bounding box 536, and the size does not exceed the range defined by the soft bounding box 536. The egg (object 3) on the left side of the basket moves from the starting position to the area defined by the soft bounding box 538, and the size does not exceed the range defined by the soft bounding box 538.

[0058] In this way, users can use the soft bounding box to expand the constraints on the motion of the controlled object, so that the generated video can increase diversity while satisfying the constraints. In addition, it can also reduce the requirements on users, that is, only a certain range needs to be constrained, which can make the operation easier and improve the user experience.

[0059] In some embodiments, noun phrases in the text may be identified and the noun phrases may be determined as objects. Then, a video may be generated based on the text, the objects, and the control information, wherein the content of the video is associated with the text.

[0060] Figure 6A-6B A schematic diagram of an example 600 in which a user enters multiple bounding boxes on a starting frame without providing an image according to some embodiments of the present disclosure is shown. As shown in FIG6 , in example 600, a user enters a text description "Four pigs running on the snow" describing the content of a video in a text input control 604 of a user interface 602. However, the user does not select a reference image, so the background of the starting frame control 608 is blank. In example 600, the user uses a black bounding box 630, a white bounding box 632, a gray bounding box 634, and a yellow bounding box 636 in the starting frame control 608 to mark the starting position and size of the four pigs. In the ending frame control 618, the user uses a black hard bounding box 640, a white hard bounding box 642, a gray hard bounding box 640, and a yellow hard bounding box 646 to identify the position and size of the four pigs in the ending frame.

[0061] When generating videos, the video generation model can identify noun phrases in these descriptions. Since most of these phrases are abstract nouns rather than specific object names, these noun phrases can be filtered and only phrases representing specific object names can be retained. These filtered noun phrases can then be processed to identify the controlled objects and associate these objects with bounding boxes.

[0062] Figure 6BThe generated video 650 is shown, as shown in the figure, the video generation model generates four pigs with specified sizes at the positions specified by the bounding boxes 630, 632, 634 and 636 based on the text description, and the generated four pigs are associated with the four bounding boxes. The four pigs each run from the initial position to the position specified by the hard bounding boxes 640, 642, 644 and 646, and the sizes of the four pigs correspond to these hard bounding boxes.

[0063] In this way, objects in text descriptions can be identified without providing reference images, allowing users to generate desired videos even when reference images cannot be provided. This can reduce the prerequisites for users to use the application to generate videos, thereby enabling more users to use the application to generate videos.

[0064] In some embodiments, third control information constraining the position of the object in the intermediate frame may be obtained, and a video may be generated based on the content information, the bounding box, the first control information, and the third control information.

[0065] Figure 7 A schematic diagram of an example 700 of a user creating an intermediate frame and inputting multiple soft bounding boxes in the intermediate frame according to some embodiments of the present disclosure is shown. Figure 7 As shown, in example 700, the user enters a text description describing the video content "Two puppies running toward the camera" in the text input control 704 of the user interface 702, and selects a reference image (as shown in the start frame control 708). The user identifies the white puppy as object 1 using a black bounding box 730 in the start frame control 708, identifies the black puppy as object 2 using a white bounding box 732, and specifies the position and size to which the white puppy moves using a black hard bounding box 740 in the end frame control 718, and specifies the position and size to which the black puppy moves using a white hard bounding box 742. However, the bounding box in the end frame cannot directly intervene in the movement process of the white puppy from the position of the bounding box 730 to the position of the bounding box 740. Therefore, in example 700, the user inserts an intermediate frame between the start frame and the end frame, and accordingly, an intermediate frame control 728 and a group of controls associated with the intermediate frame control 728 (such as color display controls for each object, bounding box type selection controls, etc.) are displayed in the user interface 702. The functions of these controls are the same as those of the group of controls associated with the end frame control 718.

[0066] like Figure 7As shown, in example 700, the user uses a black soft bounding box 750 in the intermediate frame control 728 to constrain the position and size of the white puppy's movement, and uses a white soft bounding box 752 to constrain the position and size of the black puppy's movement. In this way, in the generated video, when the white puppy runs from the position of the bounding box 730 to the position specified by the hard bounding box 740, it will first pass through a position in the area specified by the soft bounding box 750, and then go to the position specified by the hard bounding box 740. When the black puppy runs from the position of the bounding box 732 to the position specified by the hard bounding box 742, it will first pass through a position in the area specified by the soft bounding box 752. In addition, in the user interface 702, the user can also insert more intermediate frames by interacting with the image frame insertion controls 760 and 762, so as to achieve more precise control over the movement process of the two puppies.

[0067] In this way, the motion process of the controlled object can be finely controlled by inserting an intermediate frame between the start frame and the end frame. Compared with the motion trajectory, the method of inserting the intermediate frame can also control the size of the object during the motion process, thereby making the motion control function of the application more complete.

[0068] In order to realize bounding box guided video generation, a motion control module can be inserted into the existing video generation model. The motion control module can process the bounding box into a control token and use a self-attention layer to fuse the control token with the visual token used to generate the image frame, thereby generating a fused visual token. These fused visual tokens contain the motion control information provided by the bounding box.

[0069] In an example architecture, a spatial self-attention layer, a multi-layer perceptron, a motion control module, and a spatial cross-attention layer are included. The spatial self-attention layer and the spatial cross-attention layer can be, for example, modules in a video diffusion model based on a three-dimensional U-shaped network (3D U-Net) architecture. The video diffusion model can iteratively predict the noise vector in the noisy video input, thereby gradually converting the pure Gaussian noise into high-quality video frames. The 3D U-Net consists of alternating convolution blocks and attention blocks. Each block contains two components, namely, a spatial component that processes each image frame as a separate image, and a time component that facilitates information exchange between image frames. In each attention block, the spatial component typically includes a self-attention layer, followed by a cross-attention layer, which is used to adjust the generation of the video based on text prompts. The motion control module is inserted between the two attention layers so that the model can manage motion control in video generation.

[0070] This example architecture inserts a motion control module between the spatial self-attention layer and the spatial cross-attention layer of the original video diffusion model. The spatial self-attention layer receives frame-level visual tags and generates visual tags based on the frame-level visual tags. The motion control module receives visual tags and control tags as input and outputs fused visual tags. Among them, each control tag in the control tag corresponds to a corresponding object (or bounding box). Since the control tag includes motion control information provided by the bounding box, the fused visual tag will also contain the motion control information provided by the bounding box. The visual tag is then input to the spatial cross-attention layer, which can generate an updated frame-level visual tag based on the visual tag and the text tag. The video diffusion model can then generate an image frame based on the updated frame-level visual tag. In order not to change the original structure of the spatial cross-attention layer, the number of visual tags and visual tags can be kept the same. In this way, the parameters of the original video diffusion model (including the spatial self-attention layer and the spatial cross-attention layer) can be fixed during the training phase, and only the parameters of the motion control module can be adjusted, thereby avoiding retraining caused by modifying the structure of the video diffusion model, thereby saving costs and avoiding a decrease in the accuracy of the original video diffusion model caused by retraining.

[0071] In this example architecture, the number of control tags depends on the number of bounding boxes that the video generation model supports to exist simultaneously in an image frame, and the control tags correspond to the bounding boxes one by one. For example, if the video generation model only supports the inclusion of a bounding box for one object in an image frame, the number of control tags is 1; if the video generation model supports the inclusion of 5 bounding boxes for 5 objects in an image frame at the same time, the number of control tags is 5. If the video generation model supports providing 5 bounding boxes simultaneously in an image frame, but only the movement of two objects needs to be controlled in the video to be generated (i.e., only 2 bounding boxes are provided), then the missing 3 control tags can be filled with learnable specific tags. In this example architecture, text tags are not necessary, i.e., if the user does not provide a text description of the video to be generated, learnable tags can be used to fill the missing text tags.

[0072] In order to generate a control mark, the coordinates of the bounding box, the unique object identifier for identifying the bounding box, and the bounding box type can be determined. Then, the control mark is generated based on the coordinates, the object identifier and the bounding box type. For example, the object identifier can be represented in the color RGB space, where each object corresponds to a bounding box with a unique color, so that the object identifier is a vector with a 3-dimensional RGB value standardized between 0 and 1. The coordinates, object identifier and bounding box type are spliced ​​into a vector and a corresponding embedding is generated via a Fourier embedding operation. Then, the embedding is input into a multi-layer perceptron to generate a control mark. By using RGB values ​​to generate object identifiers, corresponding bounding boxes can be generated in image frames based on object identifiers during the training phase, thereby promoting the alignment between the generated bounding boxes and the ground truth bounding boxes, and improving the effect of model training.

[0073] It should be understood that although the control mark is shown to be generated based on the coordinates of the bounding box, the object identifier, and the bounding box type in this example architecture, in some embodiments the object identifier and the bounding box type are not necessary. For example, in some embodiments, if only one specific type of bounding box (e.g., a hard bounding box) is supported, the control mark can be generated based only on the coordinates. In some embodiments, if only multiple specific types of bounding boxes are supported, the control mark can be generated based only on the coordinates and the object identifier.

[0074] In this way, the motion control module can provide accurate motion control information for the original video diffusion model, thereby improving the effect of the generated image frame and enabling the object to move in the desired motion mode. In addition, since the inserted motion control module does not change the structure and parameters of the original video diffusion model, the example architecture can reuse the capabilities of the trained video diffusion model, thereby improving the motion control of the object in the video while ensuring the picture quality of the generated video.

[0075] Figure 8 FIG. 8 is a block diagram of an apparatus 800 for generating a video according to some embodiments of the present disclosure. Figure 8 As shown, the device 800 includes a content information acquisition module 802, which is configured to acquire content information related to the content of the video to be generated, and the content information includes at least one of text or image. The device 800 also includes a position information acquisition module 804, which is configured to acquire position information indicating the position of the object in the start frame of the video. The device 800 also includes a control information acquisition module 806, which is configured to acquire control information that constrains the position of the object in the end frame. In addition, the device 800 also includes a video generation module 808, which is configured to generate the video based on the content information, the position information, and the control information.

[0076] It can be understood that, by using the device 800 of the present disclosure, at least one of the advantages that can be achieved by the method or process described above can be achieved. For example, the device 800 can enable the user to accurately identify the object to be controlled using the bounding box in the start frame. In addition, the user can accurately express the position of the identified object moving from the start frame to the end frame, so that the application can receive accurate motion control requirements, thereby generating a video that meets the user's expectations.

[0077] Fig. 9 1 shows a block diagram of a device 900 capable of implementing multiple embodiments of the present disclosure. The device 900 may be a device or apparatus described in the embodiments of the present disclosure. Fig. 9 As shown, the device 900 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The CPU / GPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904. Although not shown in FIG. Fig. 9 As shown in FIG. 9 , device 900 may further include a co-processor.

[0078] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0079] The various methods or processes described above may be performed by the CPU / GPU 901. For example, in some embodiments, the methods may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the CPU / GPU 901, one or more steps or actions in the methods or processes described above may be performed.

[0080] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0081] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0082] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0083] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0084] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0085] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0086] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] Some example implementations of the present disclosure are listed below.

[0088] Example 1. A method for generating a video, comprising:

[0089] Acquire content information related to the content of the video to be generated, wherein the content information includes at least one of text or image;

[0090] Obtaining position information indicating a position of an object in the video in a starting frame;

[0091] Obtaining control information constraining the position of the object in the end frame; and

[0092] The video is generated based on the content information, the position information, and the control information.

[0093] Example 2. The method of Example 1, wherein the object is a first object, the position information is a first bounding box having a first color, the control information is first control information having the first color, and generating the video comprises:

[0094] Obtaining a second bounding box indicating a position of a second object in the video in the starting frame, the second bounding box having a second color different from the first color;

[0095] acquiring second control information constraining a position of the second object in the end frame, the second control information having the second color; and

[0096] The video is generated based on the content information, the first bounding box, the first control information, the second bounding box, and the second control information.

[0097] Example 3. The method according to Examples 1-2, wherein the position information is a starting bounding box, the control information is an ending bounding box of the object in the ending frame, and the starting bounding box and the ending bounding box are rectangular boxes.

[0098] Example 4. The method of Examples 1-3, wherein generating the video based on the content information, the location information, and the control information comprises:

[0099] In response to a user selecting a first type as the type of the ending bounding box, the video is generated by moving the object from a position indicated by the starting bounding box to a specific position indicated by the ending bounding box, wherein a size of the object in the ending frame corresponds to the ending bounding box.

[0100] Example 5. The method of Examples 1-4, wherein generating the video based on the content information, the location information, and the control information comprises:

[0101] In response to a user selecting a second type as the type of the ending bounding box, the video is generated by moving the object from a position indicated by the starting bounding box to a position range indicated by the ending bounding box, wherein a size of the object in the ending frame does not exceed the ending bounding box, and content of the video is associated with the content information.

[0102] Example 6. The method of Examples 1-5, wherein generating the video based on the content information, the location information, and the control information comprises:

[0103] In response to the end bounding box being close to the left boundary or the right boundary of the end frame and the width of the end bounding box being less than a threshold width, generating the video by moving the object from a position in the start frame to outside the left boundary or the right boundary of the end frame, wherein the size of the object when moving outside the left boundary or the right boundary is associated with the height of the end bounding box; or

[0104] In response to the end bounding box being close to an upper boundary or a lower boundary of the end frame and a height of the end bounding box being less than a threshold height, generating the video by moving the object from a position in the start frame to outside the upper boundary or the lower boundary of the end frame, wherein a size of the object when moving outside the upper boundary or the lower boundary is associated with a width of the end bounding box,

[0105] The content of the video is associated with the content information.

[0106] Example 7. The method of Examples 1-6, wherein generating the video based on the content information, the location information, and the control information comprises:

[0107] In response to the starting bounding box being close to a left boundary or a right boundary of a starting frame and a width of the starting bounding box being less than a threshold width, generating the video by moving the object from outside the left boundary or the right boundary to a position constrained by the control information, wherein a size of the object when entering the left boundary or the right boundary is associated with a height of the starting bounding box; or

[0108] In response to the start bounding box being close to an upper boundary or a lower boundary of the start frame and a height of the start bounding box being less than a threshold height, generating the video by moving the object from outside the upper boundary or the lower boundary of the start frame to a position constrained by the control information, wherein a size of the object when entering the upper boundary or the lower boundary is associated with a width of the end bounding box,

[0109] The content of the video is associated with the content information.

[0110] Example 8. The method of Examples 1-7, wherein the position information is a bounding box and the control information is a motion trajectory drawn on the starting frame.

[0111] Example 9. The method of Examples 1-8, wherein generating the video based on the content information, the location information, and the control information comprises:

[0112] The video is generated by causing the object to move along the motion trajectory starting from a position indicated by the bounding box, wherein content of the video is associated with the content information.

[0113] Example 10. The method of Examples 1-9, wherein the position information is a bounding box, and the content information includes an image selected by a user for the starting frame, and generating the video based on the content information, the position information, and the control information includes:

[0114] determining the image content identified by the bounding box in the starting frame as the object; and

[0115] The video is generated based on the image of the starting frame, the object, and the control information, wherein content of the video is associated with the image.

[0116] Example 11. The method of Examples 1-10, wherein the content information includes user-entered text describing the content of the video, and the example further includes:

[0117] identifying noun phrases in the text;

[0118] identifying the noun phrase as the object; and

[0119] The video is generated based on the text, the object, and the control information, wherein content of the video is associated with the text.

[0120] Example 12. The method of Examples 1-11, wherein the control information is first control information, further comprising:

[0121] acquiring third control information constraining the position of the object in the intermediate frame; and

[0122] The video is generated based on the content information, the position information, the first control information, and the third control information.

[0123] Example 13. A method according to Example 1, wherein the object gradually moves from a first position to a second position in the video.

[0124] Example 14. The method of Example 1, wherein the video causes the object to move relative to the camera by changing the camera's viewing angle.

[0125] Example 15. An apparatus for generating a video, comprising:

[0126] A content information acquisition module, configured to acquire content information related to the content of the video to be generated, wherein the content information includes at least one of text or image;

[0127] a position information acquisition module, configured to acquire position information indicating a position of an object in the video in a starting frame;

[0128] a control information acquisition module, configured to acquire control information constraining the position of the object in the end frame; and

[0129] The video generation module is configured to generate the video based on the content information, the location information, and the control information.

[0130] Example 16. The apparatus of Example 15, wherein the object is a first object, the position information is a first bounding box having a first color, the control information is first control information having the first color, and generating the video comprises:

[0131] A second bounding box acquisition module is configured to acquire a second bounding box indicating a position of a second object in the video in the start frame, the second bounding box having a second color different from the first color;

[0132] A second control information acquisition module is configured to acquire second control information constraining a position of the second object in the end frame, the second control information having the second color; and

[0133] The second bounding box using module is configured to generate the video based on the content information, the first bounding box, the first control information, the second bounding box, and the second control information.

[0134] Example 17. An apparatus according to Examples 15-16, wherein the position information is a starting bounding box, the control information is an ending bounding box of the object in the ending frame, and the starting bounding box and the ending bounding box are rectangular boxes.

[0135] Example 18. The apparatus of Examples 15-17, wherein generating the video based on the content information, the location information, and the control information comprises:

[0136] A first type video generation module is configured to generate the video in response to a user selecting the first type as the type of the ending bounding box by moving the object from a position indicated by the starting bounding box to a specific position indicated by the ending bounding box, wherein the size of the object in the ending frame corresponds to the ending bounding box.

[0137] Example 19. The apparatus of Examples 15-18, wherein generating the video based on the content information, the location information, and the control information comprises:

[0138] The second type video generation module generates the video in response to the user selecting the second type as the type of the ending bounding box by moving the object from the position indicated by the starting bounding box to the position range indicated by the ending bounding box, wherein the size of the object in the ending frame does not exceed the ending bounding box, and the content of the video is associated with the content information.

[0139] Example 20. The apparatus of Examples 15-19, wherein generating the video based on the content information, the location information, and the control information comprises:

[0140] a first boundary judgment module, configured to generate the video by moving the object from a position in the starting frame to outside the left boundary or the right boundary of the ending frame in response to the ending boundary box being close to the left boundary or the right boundary of the ending frame and the width of the ending boundary box being less than a threshold width, wherein the size of the object when moving outside the left boundary or the right boundary is associated with the height of the ending boundary box; or

[0141] a second boundary judgment module, configured to generate the video by moving the object from a position in the starting frame to outside the upper boundary or the lower boundary of the ending frame in response to the ending boundary box being close to the upper boundary or the lower boundary of the ending frame and the height of the ending boundary box being less than a threshold height, wherein the size of the object when moving outside the upper boundary or the lower boundary is associated with the width of the ending boundary box,

[0142] The content of the video is associated with the content information.

[0143] Example 21. The apparatus of Examples 15-20, wherein generating the video based on the content information, the location information, and the control information comprises:

[0144] a third boundary judgment module, configured to generate the video by moving the object from outside the left boundary or the right boundary to a position constrained by the control information in response to the starting boundary box being close to the left boundary or the right boundary of the starting frame and the width of the starting boundary box being less than a threshold width, wherein the size of the object when entering the left boundary or the right boundary is associated with the height of the starting boundary box; or

[0145] a fourth boundary judgment module, configured to generate the video by moving the object from outside the upper boundary or the lower boundary of the starting frame to a position constrained by the control information in response to the starting boundary box being close to the upper boundary or the lower boundary of the starting frame and the height of the starting boundary box being less than a threshold height, wherein the size of the object when entering the upper boundary or the lower boundary is associated with the width of the ending boundary box,

[0146] The content of the video is associated with the content information.

[0147] Example 22. An apparatus according to examples 15-21, wherein the position information is a bounding box and the control information is a motion track drawn on the starting frame.

[0148] Example 23. The apparatus of Examples 15-22, wherein generating the video based on the content information, the location information, and the control information comprises:

[0149] The motion trajectory using module is configured to generate the video by causing the object to move along the motion trajectory starting from a position indicated by the bounding box, wherein content of the video is associated with the content information.

[0150] Example 24. The apparatus of Examples 15-23, wherein the content information includes an image selected by a user for the starting frame, and generating the video based on the content information, the position information, and the control information includes:

[0151] an object determination module, configured to determine the image content identified by the bounding box in the starting frame as the object; and

[0152] The video is generated based on the image of the starting frame, the object, and the control information, wherein content of the video is associated with the image.

[0153] Example 25. The apparatus of Examples 15-24, wherein the content information comprises user-entered text describing the content of the video, and the examples further comprising:

[0154] a noun recognition module configured to recognize noun phrases in the text;

[0155] a noun usage module configured to determine the noun phrase as the object; and

[0156] The object using module is configured to generate the video based on the text, the object, and the control information, wherein the content of the video is associated with the text.

[0157] Example 26. The apparatus of Examples 15-25, wherein the control information is first control information, the examples further comprising:

[0158] A third control information acquisition module, configured to acquire third control information constraining the position of the object in the intermediate frame; and

[0159] The third control information using module is configured to generate the video based on the content information, the location information, the first control information, and the third control information.

[0160] Example 27. An apparatus according to Examples 15-26, wherein the object gradually moves from a first position to a second position in the video.

[0161] Example 28. An apparatus according to Examples 15-27, wherein the video causes the object to move relative to the camera by changing the camera perspective.

[0162] Example 29. An electronic device comprising:

[0163] Processor; and

[0164] A memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, causing the electronic device to implement a method for generating a video, the method comprising:

[0165] Acquire content information related to the content of the video to be generated, wherein the content information includes at least one of text or image;

[0166] Obtaining position information indicating a position of an object in the video in a starting frame;

[0167] Obtaining control information constraining the position of the object in the end frame; and

[0168] The video is generated based on the content information, the position information, and the control information.

[0169] Example 30. The apparatus of Example 29, wherein the object is a first object, the position information is a first bounding box having a first color, the control information is first control information having the first color, and generating the video comprises:

[0170] Obtaining a second bounding box indicating a position of a second object in the video in the starting frame, the second bounding box having a second color different from the first color;

[0171] acquiring second control information constraining a position of the second object in the end frame, the second control information having the second color; and

[0172] The video is generated based on the content information, the first bounding box, the first control information, the second bounding box, and the second control information.

[0173] Example 31. An apparatus according to Examples 29-30, wherein the position information is a starting bounding box, the control information is an ending bounding box of the object in the ending frame, and the starting bounding box and the ending bounding box are rectangular boxes.

[0174] Example 32. The apparatus of Examples 29-31, wherein generating the video based on the content information, the location information, and the control information comprises:

[0175] In response to a user selecting a first type as the type of the ending bounding box, the video is generated by moving the object from a position indicated by the starting bounding box to a specific position indicated by the ending bounding box, wherein a size of the object in the ending frame corresponds to the ending bounding box.

[0176] Example 33. The apparatus of Examples 29-32, wherein generating the video based on the content information, the location information, and the control information comprises:

[0177] In response to a user selecting a second type as the type of the ending bounding box, the video is generated by moving the object from a position indicated by the starting bounding box to a position range indicated by the ending bounding box, wherein a size of the object in the ending frame does not exceed the ending bounding box, and content of the video is associated with the content information.

[0178] Example 34. The apparatus of Examples 29-33, wherein generating the video based on the content information, the location information, and the control information comprises:

[0179] In response to the end bounding box being close to the left boundary or the right boundary of the end frame and the width of the end bounding box being less than a threshold width, generating the video by moving the object from a position in the start frame to outside the left boundary or the right boundary of the end frame, wherein the size of the object when moving outside the left boundary or the right boundary is associated with the height of the end bounding box; or

[0180] In response to the end bounding box being close to an upper boundary or a lower boundary of the end frame and a height of the end bounding box being less than a threshold height, generating the video by moving the object from a position in the start frame to outside the upper boundary or the lower boundary of the end frame, wherein a size of the object when moving outside the upper boundary or the lower boundary is associated with a width of the end bounding box,

[0181] The content of the video is associated with the content information.

[0182] Example 35. The apparatus of Examples 29-34, wherein generating the video based on the content information, the location information, and the control information comprises:

[0183] In response to the starting bounding box being close to a left boundary or a right boundary of a starting frame and a width of the starting bounding box being less than a threshold width, generating the video by moving the object from outside the left boundary or the right boundary to a position constrained by the control information, wherein a size of the object when entering the left boundary or the right boundary is associated with a height of the starting bounding box; or

[0184] In response to the start bounding box being close to an upper boundary or a lower boundary of the start frame and a height of the start bounding box being less than a threshold height, generating the video by moving the object from outside the upper boundary or the lower boundary of the start frame to a position constrained by the control information, wherein a size of the object when entering the upper boundary or the lower boundary is associated with a width of the end bounding box,

[0185] The content of the video is associated with the content information.

[0186] Example 36. An apparatus according to Examples 29-35, wherein the position information is a bounding box and the control information is a motion trajectory drawn on the starting frame.

[0187] Example 37. The apparatus of Examples 29-36, wherein generating the video based on the content information, the location information, and the control information comprises:

[0188] The video is generated by causing the object to move along the motion trajectory starting from a position indicated by the bounding box, wherein content of the video is associated with the content information.

[0189] Example 38. The apparatus of Examples 29-37, wherein the content information includes an image selected by a user for the starting frame, and generating the video based on the content information, the position information, and the control information includes:

[0190] determining the image content identified by the bounding box in the starting frame as the object; and

[0191] The video is generated based on the image of the starting frame, the object, and the control information, wherein content of the video is associated with the image.

[0192] Example 39. The apparatus of Examples 29-38, wherein the content information comprises user-entered text describing the content of the video, and the examples further comprising:

[0193] identifying noun phrases in the text;

[0194] identifying the noun phrase as the object; and

[0195] The video is generated based on the text, the object, and the control information, wherein content of the video is associated with the text.

[0196] Example 40. The apparatus of Examples 29-39, wherein the control information is first control information, the examples further comprising:

[0197] acquiring third control information constraining the position of the object in the intermediate frame; and

[0198] The video is generated based on the content information, the position information, the first control information, and the third control information.

[0199] Example 41. An apparatus according to Examples 29-40, wherein the object gradually moves from a first position to a second position in the video.

[0200] Example 42. The method of Examples 29-41, wherein the video causes the object to move relative to the camera by changing the camera's viewing angle.

[0201] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a video, comprising: Acquire content information related to the content of the video to be generated, the content information including an image used to generate the video; Acquiring, via a user interface, position information input by a user for identifying a position of a motion-controlled object in the image in a start frame, wherein the image includes the object, and the object is identified in the image by the user via the user interface; Acquiring, via a user interface, control information constraining a position of the object in an end frame, the control information comprising at least one of a drawn bounding box or a drawn motion trajectory; as well as The video is generated based on the content information, the position information, and the control information.

2. The method of claim 1, wherein the object is a first object, the position information is a first bounding box having a first color, the control information is first control information having the first color, and generating the video comprises: Obtaining a second bounding box indicating a position of a second object in the video in the starting frame, the second bounding box having a second color different from the first color; Acquire second control information constraining a position of the second object in the end frame, the second control information having the second color; as well as The video is generated based on the content information, the first bounding box, the first control information, the second bounding box, and the second control information. 3 . The method according to claim 1 , wherein the position information is a start bounding box, the control information is an end bounding box of the object in the end frame, and the start bounding box and the end bounding box are rectangular boxes.

4. The method of claim 3, wherein generating the video based on the content information, the location information, and the control information comprises: In response to a user selecting a first type as the type of the ending bounding box, the video is generated by moving the object from a position indicated by the starting bounding box to a specific position indicated by the ending bounding box, wherein a size of the object in the ending frame corresponds to the ending bounding box.

5. The method according to claim 3 or 4, wherein generating the video based on the content information, the location information, and the control information comprises: In response to a user selecting a second type as the type of the ending bounding box, the video is generated by moving the object from a position indicated by the starting bounding box to a position range indicated by the ending bounding box, wherein a size of the object in the ending frame does not exceed the ending bounding box, and content of the video is associated with the content information.

6. The method of claim 3, wherein generating the video based on the content information, the location information, and the control information comprises: In response to the end bounding box being close to a left boundary or a right boundary of the end frame and a width of the end bounding box being less than a threshold width, generating the video by moving the object from a position in the start frame to outside the left boundary or the right boundary of the end frame, wherein a size of the object when moving outside the left boundary or the right boundary is associated with a height of the end bounding box; or In response to the end bounding box being close to an upper boundary or a lower boundary of the end frame and a height of the end bounding box being less than a threshold height, generating the video by moving the object from a position in the start frame to outside the upper boundary or the lower boundary of the end frame, wherein a size of the object when moving outside the upper boundary or the lower boundary is associated with a width of the end bounding box, The content of the video is associated with the content information.

7. The method of claim 3, wherein generating the video based on the content information, the location information, and the control information comprises: In response to the starting bounding box being close to a left boundary or a right boundary of a starting frame and a width of the starting bounding box being less than a threshold width, generating the video by moving the object from outside the left boundary or the right boundary to a position constrained by the control information, wherein a size of the object when entering the left boundary or the right boundary is associated with a height of the starting bounding box; or In response to the start bounding box being close to an upper boundary or a lower boundary of the start frame and a height of the start bounding box being less than a threshold height, generating the video by moving the object from outside the upper boundary or the lower boundary of the start frame to a position constrained by the control information, wherein a size of the object when entering the upper boundary or the lower boundary is associated with a width of the end bounding box, The content of the video is associated with the content information.

8. The method of claim 1, wherein the position information is a bounding box, and the control information is a motion trajectory drawn on the starting frame.

9. The method of claim 8, wherein generating the video based on the content information, the location information, and the control information comprises: The video is generated by causing the object to move along the motion trajectory starting from a position indicated by the bounding box, wherein content of the video is associated with the content information.

10. The method of claim 1, wherein the position information is a bounding box, and generating the video based on the content information, the position information, and the control information comprises: Determine the image content identified by the bounding box in the starting frame as the object; as well as The video is generated based on the image of the starting frame, the object, and the control information, wherein content of the video is associated with the image.

11. The method of claim 10, wherein the content information comprises user-entered text describing the content of the video, and the method further comprises: identifying noun phrases in the text; identifying the noun phrase as the object; as well as The video is generated based on the text, the object, and the control information, wherein content of the video is associated with the text.

12. The method according to claim 1, wherein the control information is first control information, the method further comprising: Acquire third control information constraining the position of the object in the intermediate frame; as well as The video is generated based on the content information, the position information, the first control information, and the third control information.

13. The method of claim 1, wherein the object gradually moves from a first position to a second position in the video. The method of claim 1 , wherein the video causes the object to move relative to the camera by changing the camera viewing angle.

15. An apparatus for generating a video, comprising: A content information acquisition module, configured to acquire content information related to the content of the video to be generated, wherein the content information includes an image used to generate the video; a position information acquisition module, configured to acquire, via a user interface, position information input by a user for identifying a position of a motion-controlled object in the image in a start frame, wherein the image includes the object, and the object is identified in the image by the user via the user interface; a control information acquisition module configured to acquire, via a user interface, control information constraining a position of the object in an end frame, the control information comprising at least one of a drawn bounding box or a drawn motion trajectory; as well as The video generation module is configured to generate the video based on the content information, the location information, and the control information.

16. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 14.

17. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions which, when executed, cause a machine to implement the method according to any one of claims 1 to 14.

Images