Video generation method and device, equipment and medium

Abstract

The embodiment of the invention relates to a video generation method and device, equipment and a medium, and the method comprises the steps: obtaining the information of a motion track corresponding to a track drawing operation in response to the detection of the track drawing operation for a target object in a target reference image; generating sparse optical flow information based on the information of the motion track; generating a target video by using a preset target network model based on the sparse optical flow information and the target reference image; wherein the target video is a video in which the target object moves based on the movement track. According to the embodiment of the invention, on the basis of the target reference image, the target network model can be simply and efficiently guided to generate the video in which the target object moves according to the track required by the user by utilizing the sparse optical flow information, the user requirements can be well met, and the image-generated video effect is enriched.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video generation method, apparatus, device, and medium. Background Technology

[0002] In image-processing scenarios such as computer vision and multimedia editing, image-to-video generation technology has gradually become a research hotspot. For example, by inputting an image, a video showing changes in a specific object within that image can be generated. However, the inventors have found that the image-to-video effects achievable by existing technologies are limited and cannot meet the diverse needs of users. Therefore, new image-to-video technologies are urgently needed. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a video generation method, apparatus, device, and medium.

[0004] This disclosure provides a video generation method, the method comprising: in response to detecting a trajectory drawing operation for a target object in a target reference image, acquiring information of a motion trajectory corresponding to the trajectory drawing operation; generating sparse optical flow information based on the motion trajectory information; and generating a target video using a preset target network model based on the sparse optical flow information and the target reference image; wherein the target video is a video of the target object moving based on the motion trajectory.

[0005] Optionally, the method further includes: in response to an object selection operation on a target reference image, taking the object in the target reference image corresponding to the object selection operation as the target object, and detecting a trajectory drawing operation on the target object.

[0006] Optionally, the trajectory drawing operation is determined based on the control information of a preset trajectory drawing control; or, the trajectory drawing operation is determined based on touch operations and / or sliding operations within the area of ​​the target reference image.

[0007] Optionally, generating sparse optical flow information based on the motion trajectory information includes: obtaining the motion direction and motion distance corresponding to the target object based on the motion trajectory information; and generating sparse optical flow information based on the motion direction and motion distance corresponding to the target object.

[0008] Optionally, the target network model is obtained through the following steps: acquiring sample videos and corresponding reference images; acquiring dense optical flow information corresponding to the sample videos; training a preset initial generation model based on the dense optical flow information, the corresponding reference image, and noisy sample videos, to obtain a first generation model based on the trained initial generation model; acquiring sparse optical flow information corresponding to the sample videos based on the dense optical flow information; training the first generation model based on the sparse optical flow information, the corresponding reference image, and noisy sample videos, to obtain the target network model based on the trained first generation model.

[0009] Optionally, obtaining the dense optical flow information corresponding to the sample video includes: using a preset optical flow estimation algorithm to obtain the dense optical flow information between the first frame and the last frame of the sample video, and using the obtained dense optical flow information as the dense optical flow information corresponding to the sample video.

[0010] Optionally, obtaining the sparse optical flow information corresponding to the sample video based on the dense optical flow information corresponding to the sample video includes: extracting the sparse optical flow information corresponding to the sample video from the dense optical flow information corresponding to the sample video based on the watershed algorithm.

[0011] Optionally, the reference image corresponding to the sample video is the first frame of the sample video.

[0012] This disclosure also provides a video generation apparatus, comprising: a trajectory information acquisition module, configured to acquire motion trajectory information corresponding to the trajectory drawing operation in response to detecting a trajectory drawing operation for a target object in a target reference image; a sparse optical flow generation module, configured to generate sparse optical flow information based on the motion trajectory information; and a video generation module, configured to generate a target video using a preset target network model based on the sparse optical flow information and the target reference image; wherein the target video is a video of the target object moving based on the motion trajectory.

[0013] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the video generation method provided in this disclosure.

[0014] This disclosure also provides a computer-readable storage medium storing a computer program for performing the video generation method provided in this disclosure.

[0015] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the video generation method provided in this disclosure.

[0016] The technical solution provided in this disclosure can respond to the detection of a trajectory drawing operation for a target object in a target reference image, obtain information on the motion trajectory corresponding to the trajectory drawing operation, and generate sparse optical flow information based on the motion trajectory information. Based on the target reference image, the sparse optical flow information can be used to simply and efficiently guide the target network model to generate a video of the target object moving according to the trajectory required by the user, which can better meet the user's needs and enrich the image-generated video effect.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a video generation method provided in this embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of trajectory control provided in an embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram of a video generation process provided in an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram of dense optical flow acquisition provided in an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of model training provided in an embodiment of this disclosure;

[0025] Figure 6 A schematic diagram of model training provided in an embodiment of this disclosure;

[0026] Figure 7This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present disclosure;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of the present disclosure. The method can be executed by a video generation device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0031] Step S102: In response to detecting a trajectory drawing operation for a target object in the target reference image, information about the motion trajectory corresponding to the trajectory drawing operation is obtained. This motion trajectory information may include one or more of the following: the starting point, ending point, motion path, motion direction, and motion distance.

[0032] In some embodiments, the method provided by this disclosure further includes: responding to an object selection operation on a target reference image, identifying the object in the target reference image corresponding to the object selection operation as the target object, and detecting a trajectory drawing operation on the target object. For example, a touch operation on an object region in the detected target reference image can be determined as an object selection operation, and the object acted upon by the object selection operation (such as an object touched by a user on the target reference image) is the target object. For example, an object selection operation can also be determined using an object selection control. For instance, the object selection control can be a draggable control, and the target object can be determined based on the area where the drag position of the object selection control is located. Alternatively, the object selection control can directly display all movable objects in the target reference image, and the user can select the target object from the displayed movable objects based on the object selection control. This disclosure does not limit the form of the object selection control or the method of selecting the target object; the above are merely illustrative examples and should not be considered as limitations.

[0033] In some implementations, the trajectory drawing operation is determined based on the control information of a preset trajectory drawing control; or, the trajectory drawing operation is determined based on touch and / or swipe operations within the area of ​​the target reference image. For example, a trajectory drawing control can be provided on the display interface of the target reference image. This control could be a draggable control, a pen control, or a control providing multiple selectable trajectory paths. This disclosure does not limit the form of the trajectory drawing control; the trajectory drawing control and the aforementioned object selection control can be the same control or different controls, and can be flexibly configured. The user's trajectory drawing operation on the target object can be determined based on the control information of the trajectory drawing control. Alternatively, a trajectory drawing control may not be provided, and the user can directly perform touch or swipe operations on the target object. For example, after detecting that the user has selected the target object, the position where the user performs another touch operation on the target reference image is taken as the motion termination point of the target object, and the line connecting the current position of the target object and the motion termination point is taken as the trajectory drawn by the user; or, the trajectory corresponding to the user's swipe operation on the target reference image is taken as the trajectory drawn by the user.

[0034] For easier understanding, please refer to Figure 2 The diagram illustrates a trajectory control method where the user can designate both the vehicle and the lantern as target objects. The user can then instruct the vehicle to move forward and the lantern to move upward using trajectory drawing operations such as tapping / swiping on the interface or manipulating trajectory drawing controls. It should be noted that... Figure 2 This is just an example; the movement trajectories of the vehicle and the lantern are simply marked with arrows. In practical applications, you can select only some objects in the image to move, such as selecting only one of the vehicle or the lantern to move, and the movement trajectories of each object can also be drawn flexibly.

[0035] Step S104: Generate sparse optical flow information based on motion trajectory information.

[0036] For example, the motion direction and distance of the target object can be obtained based on the motion trajectory information; sparse optical flow information can be generated based on the motion direction and distance of the target object. The sparse optical flow information can be presented in the form of a sparse optical flow graph. The method for generating the sparse optical flow graph based on the motion direction and distance can be found in relevant technologies and will not be elaborated here.

[0037] Step S106: Based on sparse optical flow information and target reference image, generate target video using a preset target network model; wherein, target video is video of target object moving based on motion trajectory.

[0038] The target network model can be implemented using a generative model, or in some specific implementation examples, a diffusion model or a network model containing a denoising network can be used; there are no limitations on this. For example, a motion video of the target object can be generated using a preset target network model based on sparse optical flow information, a target reference image, and randomly generated noise data.

[0039] Since sparse optical flow information is generated based on motion trajectory information, that is, it carries motion trajectory information, the above-mentioned method provided in this disclosure can, based on the target reference image, use sparse optical flow information to simply and efficiently guide the target network model to generate a video of the target object moving according to the trajectory required by the user. It can better meet the user's needs and enrich the image-generated video effects. For example, for the same image, the user can flexibly specify the motion of different objects in the image according to the needs, and even if the same object is specified, the motion trajectory can be flexibly set. Therefore, it is possible to generate rich and diverse motion videos that meet the user's needs based on the image.

[0040] Furthermore, this disclosure provides some implementation examples of a target network model. Exemplarily, the target network model includes a first encoder, a second encoder, a denoising network, and a decoder. For ease of understanding, please refer to... Figure 3The diagram illustrates a video generation process, demonstrating how a target video is generated using a target network model. The focus is on showcasing the structure of this model. For example, the process begins by detecting the trajectory drawing operation of the target object in the target reference image, obtaining information about the motion trajectory corresponding to the trajectory drawing operation. Then, sparse optical flow information is generated based on this motion trajectory information. Noise data, the target reference image, and the sparse optical flow information are then input to a first encoder for encoding. This first encoder, such as a VAE (Variational Auto-Encoder) network, encodes the input data and inputs the encoded result to a denoising network. The second encoder takes the target reference image as input and can be implemented using an image encoder. It extracts features from the target reference image and injects the extracted features into the denoising network. The denoising network, such as a U-net network, performs layer-by-layer denoising based on the outputs of the first and second encoders. Finally, a decoding network decodes the denoised result to obtain the desired target video, i.e., a video of the target object moving based on its motion trajectory. The decoding network, such as a VAE decoding network, can be used for this purpose. The above are examples. In practical applications, the target network model can be implemented in other ways, which are not limited here. It should be emphasized that the above target network model does not require an additional trajectory encoder. Instead, the sparse optical flow information and the target reference image can be directly used as the input of the first encoder. Compared with adding a trajectory encoder, this can effectively avoid adding more model parameters and avoid increasing the training difficulty.

[0041] For ease of understanding, this disclosure further provides a method for obtaining the target network model. Exemplarily, the target network model is obtained through the following steps A to E:

[0042] Step A: Obtain the sample video and its corresponding reference image. This embodiment of the disclosure does not limit the content of the sample video; generally, the sample video contains one or more moving objects. For ease of processing, exemplarily, the reference image corresponding to the sample video is the first frame of the sample video. Furthermore, it should be noted that the number of frames in the sample video determines the final number of video frames generated by the model. For example, if the sample video contains N frames, the trained target network model will ultimately generate a video containing N frames.

[0043] Step B: Obtain the dense optical flow information corresponding to the sample video.

[0044] For example, a preset optical flow estimation algorithm can be used to obtain dense optical flow information between the first and last frames of a sample video, and this obtained dense optical flow information can be used as the dense optical flow information corresponding to the sample video. The dense optical flow information can be represented by a dense optical flow graph. For ease of understanding, refer to... Figure 4 The diagram illustrates a dense optical flow acquisition method. From the first and last frames of a video, it's clear that a moving car exists in the video, while the rest are considered stationary. The resulting dense optical flow map only shows the moving vehicle; pixels outside the vehicle area are considered stationary and therefore appear blank. It should be noted that in practical applications, the dense optical flow map can also use pixel color to represent information such as the direction and / or magnitude of pixel displacement. For example, different colors correspond to different directions or velocities. This method allows for a more visual representation of dense optical flow information, facilitating analysis and processing.

[0045] Step C: Based on the dense optical flow information corresponding to the sample video, the reference image corresponding to the sample video, and the sample video with added noise, train a preset initial generation model to obtain the first generation model based on the trained initial generation model.

[0046] In practical applications, the structure of the initial generative model is the same as that of the first generative model and the aforementioned target network model; the main difference lies in the parameters. When training the initial generative model, the parameters can be adjusted based on the differences between the output video of the initial generative model and the sample video to obtain an initial generative model that meets the requirements (i.e., the trained initial generative model). Then, the trained initial generative model can be directly used as the first generative model.

[0047] In practical applications, see Figure 5 The diagram illustrates a model training process where dense optical flow information corresponding to a sample video and a noisy sample video are concatenated along the channel dimension and input into the first encoder of the initial generation model. A reference image (such as the first frame of the sample video) corresponding to the sample video is input into the second encoder of the initial generation model. The denoising network in the initial generation model performs layer-by-layer denoising based on the outputs of the first and second encoders. Finally, the denoised result output by the denoising network is decoded by the decoding network in the initial generation model to obtain the model output video. The model parameters are adjusted based on the differences between the model output video and the sample video to train the initial model, thus obtaining the first generation model.

[0048] The first generative model trained in the above manner outputs a video of object motion that meets expectations based on the dense optical flow information of the input.

[0049] Step D: Based on the dense optical flow information corresponding to the sample video, obtain the sparse optical flow information corresponding to the sample video.

[0050] It is understandable that the motion trajectory information given by the user is usually just simple information such as direction, position or distance, which is very sparse. The first generative model obtained through the aforementioned step C only has a strong processing capability for dense optical flow information, but still cannot handle sparse information well. Therefore, this embodiment of the present disclosure will continue to extract the sparse optical flow information corresponding to the sample video from the dense optical flow information corresponding to the sample video in order to further train the first generative model.

[0051] For example, sparse optical flow information corresponding to a sample video can be extracted from dense optical flow information based on the watershed algorithm. The watershed algorithm can conveniently and accurately determine object boundaries, and then sparse optical flow points can be extracted for each object appearing in the sample video. The generated sparse optical flow information can represent the optical flow information of moving objects in the sample video. Sparse optical flow information can be represented by a sparse optical flow graph. In practical applications, a single sparse optical flow graph can represent the sparse optical flow information of all moving objects in the sample video. Since the dense optical flow information corresponding to the sample video is essentially the dense optical flow information between the first and last frames of the sample video, the sparse optical flow information corresponding to the sample video obtained based on the dense optical flow information is also essentially the sparse optical flow information between the first and last frames of the sample video.

[0052] Step E involves training a first generative model based on the sparse optical flow information corresponding to the sample video and the reference image corresponding to the sample video, and then obtaining the target network model based on the trained first generative model. In practical applications, the trained first generative model can be directly used as the target network model.

[0053] The training method for the first generative model can refer to the training method for the initial generative model described above, and will not be repeated here. See [link to relevant documentation]. Figure 6 The diagram shown illustrates the training process of the first generative model, which is related to... Figure 5 The main difference is that in the first stage of training, the input of the first encoder of the initial generative model is the dense optical flow information corresponding to the sample video, while in the second stage of training, the input of the first encoder of the first generative model is the sparse optical flow information corresponding to the sample video.

[0054] Through steps A through D above, the model can be trained in stages. First, in the first stage of training, the model can learn and process dense optical flow information with greater precision. For example, the model can learn relevant information from the optical flow map more clearly and reliably, such as understanding the direction and distance of pixel movement represented by colors in the optical flow map. Building on this, a second stage of training is performed using sparse optical flow information. This effectively improves the convergence speed and enhances the model's ability to process sparse optical flow information. The resulting target network model can then learn matching motion direction information based on the sparse optical flow information, thereby generating a video of the object's motion as expected.

[0055] Understandably, directly training the initial generative model using sparse optical flow information often results in poor convergence and extremely difficult training. However, the aforementioned two-stage training method, which first trains the model based on dense optical flow information and then further trains it based on sparse optical flow information, more efficiently and reliably trains a target network model with strong understanding and processing capabilities for sparse optical flow information. This model can accurately understand the required motion trajectory based on sparse optical flow information and control the target object to move according to that trajectory, producing a video of the object's motion that meets expectations and significantly enriching the image-generated video effect. Furthermore, this two-stage training method effectively improves model convergence speed, simplifies model training, and eliminates the need for an additional trajectory encoder. Consequently, it avoids training an additional trajectory encoder and increasing the number of model parameters, effectively preventing model structural complexity.

[0056] Corresponding to the aforementioned video generation method, this disclosure further provides a video generation apparatus. Figure 7 This is a schematic diagram of a video generation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 7 As shown, the video generation device includes:

[0057] The trajectory information acquisition module 702 is used to acquire information about the motion trajectory corresponding to the trajectory drawing operation in response to the detection of a trajectory drawing operation for a target object in the target reference image.

[0058] The sparse optical flow generation module 704 is used to generate sparse optical flow information based on motion trajectory information;

[0059] The video generation module 706 is used to generate a target video based on sparse optical flow information and a target reference image using a preset target network model; wherein, the target video is a video of the target object moving based on a motion trajectory.

[0060] The apparatus provided in this disclosure can, based on a target reference image, use sparse optical flow information to simply and efficiently guide the target network model to generate a video of the target object moving along a trajectory required by the user, which can better meet the user's needs and enrich the image-generated video effect.

[0061] In some embodiments, the apparatus further includes a detection module for identifying, in response to an object selection operation on a target reference image, an object in the target reference image corresponding to the object selection operation as a target object, and detecting a trajectory drawing operation on the target object.

[0062] In some implementations, the trajectory drawing operation is determined based on the control information of a preset trajectory drawing control; or, the trajectory drawing operation is determined based on touch and / or swipe operations within the area of ​​the target reference image.

[0063] In some embodiments, the sparse optical flow generation module 704 is specifically used to: obtain the motion direction and motion distance corresponding to the target object based on the motion trajectory information; and generate sparse optical flow information based on the motion direction and motion distance corresponding to the target object.

[0064] In some embodiments, the apparatus further includes a model acquisition module, configured to obtain the target network model through the following steps: acquiring a sample video and a reference image corresponding to the sample video; acquiring dense optical flow information corresponding to the sample video; training a preset initial generation model based on the dense optical flow information corresponding to the sample video, the reference image corresponding to the sample video, and the sample video with added noise, to obtain a first generation model based on the trained initial generation model; acquiring sparse optical flow information corresponding to the sample video based on the dense optical flow information corresponding to the sample video; training the first generation model based on the sparse optical flow information corresponding to the sample video, the reference image corresponding to the sample video, and the sample video with added noise, to obtain the target network model based on the trained first generation model.

[0065] In some implementations, the model acquisition module is specifically used to: acquire dense optical flow information between the first frame and the last frame of the sample video using a preset optical flow estimation algorithm, and use the acquired dense optical flow information as the dense optical flow information corresponding to the sample video.

[0066] In some implementations, the model acquisition module is specifically used to: extract the sparse optical flow information corresponding to the sample video from the dense optical flow information corresponding to the sample video based on the watershed algorithm.

[0067] In some implementations, the reference image corresponding to the sample video is the first frame of the sample video.

[0068] The video generation apparatus provided in this disclosure can execute the video generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0069] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0070] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0071] The following is for reference. Figure 8 This illustration shows a structural schematic of an electronic device 800 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0072] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0073] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0074] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0075] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0076] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the video generation method provided in embodiments of this disclosure.

[0077] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0078] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the video generation method of this disclosure.

[0079] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0080] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0081] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0082] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that, include: In response to detecting a trajectory drawing operation for a target object in a target reference image, information about the motion trajectory corresponding to the trajectory drawing operation is obtained; Sparse optical flow information is generated based on the information of the motion trajectory; Based on the sparse optical flow information and the target reference image, a target video is generated using a preset target network model; wherein, the target video is a video of the target object moving based on the motion trajectory.

2. The method according to claim 1, characterized in that, The method further includes: In response to an object selection operation on a target reference image, the object in the target reference image corresponding to the object selection operation is taken as the target object, and a trajectory drawing operation is detected for the target object.

3. The method according to claim 1, characterized in that, The trajectory drawing operation is determined based on the preset trajectory drawing control's operation information; Alternatively, the trajectory drawing operation is determined based on touch and / or swipe operations within the area of ​​the target reference image.

4. The method according to claim 1, characterized in that, The generation of sparse optical flow information based on the motion trajectory information includes: Based on the information of the motion trajectory, the motion direction and motion distance of the target object are obtained; Based on the motion direction and distance of the target object, sparse optical flow information is generated.

5. The method according to any one of claims 1 to 4, characterized in that, The target network model is obtained through the following steps: Acquire sample videos and corresponding reference images; Obtain the dense optical flow information corresponding to the sample video; Based on the dense optical flow information corresponding to the sample video, the reference image corresponding to the sample video, and the sample video with added noise, a preset initial generation model is trained, and a first generation model is obtained based on the trained initial generation model. Based on the dense optical flow information corresponding to the sample video, obtain the sparse optical flow information corresponding to the sample video; Based on the sparse optical flow information corresponding to the sample video, the reference image corresponding to the sample video, and the sample video with added noise, the first generative model is trained to obtain the target network model based on the trained first generative model.

6. The method according to claim 5, characterized in that, The step of obtaining the dense optical flow information corresponding to the sample video includes: Using a preset optical flow estimation algorithm, dense optical flow information between the first and last frames of the sample video is obtained, and the obtained dense optical flow information is used as the dense optical flow information corresponding to the sample video.

7. The method according to claim 5, characterized in that, The step of obtaining sparse optical flow information corresponding to the sample video based on the dense optical flow information includes: Based on the watershed algorithm, the sparse optical flow information corresponding to the sample video is extracted from the dense optical flow information corresponding to the sample video.

8. The method according to claim 5, characterized in that, The reference image corresponding to the sample video is the first frame of the sample video.

9. A video generation apparatus, characterized in that, include: The trajectory information acquisition module is used to acquire information about the motion trajectory corresponding to the trajectory drawing operation in response to the detection of a trajectory drawing operation for a target object in a target reference image. A sparse optical flow generation module is used to generate sparse optical flow information based on the information of the motion trajectory. The video generation module is used to generate a target video based on the sparse optical flow information and the target reference image using a preset target network model; wherein, the target video is a video of the target object moving based on the motion trajectory.

10. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the video generation method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the video generation method according to any one of claims 1-8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1-8.

Images