Video generation method and device, equipment and medium

Abstract

The embodiment of the invention relates to a video generation method and device, equipment and a medium, and the method comprises the steps: responding to a region selection operation for a target reference image, and determining a target region based on the region selection operation; acquiring target track information of the target area moving in the target reference image; wherein the target track information is track information represented by three-dimensional space information; based on the target area, the target reference image and the target trajectory information, generating a motion video of a target object by using a preset target network model; wherein the target object is the target area or an object contained in the target area. According to the embodiment of the invention, the user demand can be well met, the image video effect is enriched, and the video infection is enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generation method, device, equipment and medium. Background Art

[0002] In scenarios where image processing is required, such as computer vision and multimedia editing, the technology of generating videos based on images has gradually become a research hotspot. For example, by inputting an image, a video showing changes in an object in the image can be generated. However, the inventors have found through research that the image-generated videos that can be achieved by related technologies have poor effects and are difficult to meet the diverse needs of users. Therefore, new image-generated video technologies are urgently needed. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a video generation method, device, equipment and medium.

[0004] An embodiment of the present disclosure provides a video generation method, the method comprising: in response to a region selection operation on a target reference image, determining a target region based on the region selection operation; obtaining target trajectory information of the target region moving in the target reference image; wherein the target trajectory information is trajectory information represented by three-dimensional space information; based on the target region, the target reference image and the target trajectory information, generating a motion video of a target object using a preset target network model; wherein the target object is the target region or an object contained in the target region.

[0005] Optionally, the three-dimensional spatial information includes three-dimensional optical flow information, and obtaining target trajectory information of the target area moving in the target reference image includes: upon receiving expected trajectory information set for the target area, generating three-dimensional optical flow information corresponding to the expected trajectory information, and obtaining target trajectory information of the target area moving in the target reference image based on the three-dimensional optical flow information corresponding to the expected trajectory information; when not receiving the expected trajectory information set for the target area, using preset three-dimensional optical flow information as target trajectory information of the target area moving in the target reference image; wherein, the preset three-dimensional optical flow information is represented by a specific numerical value.

[0006] Optionally, the target network model is obtained through the following steps: obtaining a target video sample; wherein the target video sample is a video with a moving object in the foreground and a still background; determining a moving area in the target video sample, and obtaining three-dimensional optical flow information corresponding to the moving area; wherein the moving area is an area containing a moving object; based on the target video sample, the moving area and the three-dimensional optical flow information corresponding to the moving area, adjusting the parameters of a preset generation model to obtain the target network model based on the preset generation model with adjusted parameters.

[0007] Optionally, obtaining a target video sample includes: obtaining a first video sample; wherein the number of the first video samples is multiple; obtaining optical flow information corresponding to the first video sample, and based on the optical flow information corresponding to the first video sample, filtering out a second video sample from the multiple first video samples; wherein the second video sample is a non-static video; based on the optical flow information corresponding to the second video sample, determining target mapping information corresponding to a video frame group in the second video sample; wherein the video frame group includes two video frame images with a preset interval; based on the target mapping information, filtering out a third video sample from the second video sample, and obtaining a target video sample based on the third video sample; wherein the target video sample is the third video sample, or the target video sample is a sample obtained by segmenting the third video sample.

[0008] Optionally, the optical flow information corresponding to the second video sample includes dense optical flow information corresponding to the video frame group in the second video sample; and determining the target mapping information corresponding to the video frame group in the second video sample based on the optical flow information corresponding to the second video sample includes: obtaining a plurality of sparse optical flow points using a preset uniform sampling strategy based on the dense optical flow information corresponding to the video frame group in the second video sample; determining the successfully matched key point pairs corresponding to the video frame group in the second video sample based on the plurality of sparse optical flow points; filtering out the key point pairs corresponding to the video frame background in the video frame group from the successfully matched key point pairs, and obtaining the target mapping information corresponding to the video frame group in the second video sample based on the mapping information of the key point pairs corresponding to the video frame background.

[0009] Optionally, determining the motion area in the target video sample includes: performing object segmentation processing on the video frame image of the target video sample to obtain an object area; determining two-dimensional optical flow information corresponding to the video frame image of the target video sample based on the video frame image of the target video sample and a subsequent frame image adjacent to the video frame image of the target video sample; and determining the motion area in the video frame image of the target video sample based on the object area and the two-dimensional optical flow information.

[0010] Optionally, determining the motion area in the video frame image of the target video sample based on the object area and the two-dimensional optical flow information includes: screening out a first area from the object area based on the two-dimensional optical flow information; wherein an optical flow value of the two-dimensional optical flow information corresponding to the first area is greater than a preset threshold; determining a second area from the video frame image of the target video sample based on the two-dimensional optical flow information; wherein the second area is an area other than the first area, and an optical flow value of the two-dimensional optical flow information corresponding to the second area is greater than the preset threshold; and obtaining the motion area in the video frame image of the target video sample based on the first area and the second area.

[0011] Optionally, obtaining the three-dimensional optical flow information corresponding to the moving area includes: obtaining the distance gradient field information corresponding to the moving area; obtaining the two-dimensional optical flow information corresponding to the moving area, and obtaining the three-dimensional optical flow information corresponding to the moving area based on the two-dimensional optical flow information and the distance gradient field information.

[0012] Optionally, obtaining the distance gradient field information corresponding to the motion area includes: obtaining the distance between the pixels of the motion area and the edge of the motion area; and determining the distance gradient field information corresponding to the motion area based on the distance corresponding to the pixels of the motion area and the distance corresponding to adjacent pixels of the pixels.

[0013] Optionally, the preset generation model includes a denoising network; wherein, the first frame image information of the target video sample, the mask image corresponding to the motion area, and the three-dimensional optical flow information corresponding to the motion area are introduced into the denoising network in the form of input channel splicing.

[0014] Optionally, the method also includes: when initializing the preset generation model, setting the weights corresponding to the first newly added channel and the second newly added channel of the denoising network to zero; wherein the first newly added channel is used to input the mask image corresponding to the motion area, and the second newly added channel is used to input the three-dimensional optical flow information corresponding to the motion area.

[0015] The disclosed embodiment also provides a video generating device, including: a region determining module, for responding to a region selection operation on a target reference image, and determining a target region based on the region selection operation; a trajectory acquiring module, for acquiring target trajectory information of the target region moving in the target reference image; wherein the target trajectory information is trajectory information represented by three-dimensional space information; a video generating module, for generating a motion video of a target object based on the target region, the target reference image, and the target trajectory information, using a preset target network model; wherein the target object is the target region or an object contained in the target region.

[0016] An embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the video generation method provided in the embodiment of the present disclosure.

[0017] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the video generation method provided by the embodiment of the present disclosure.

[0018] The above technical solution provided by the embodiment of the present disclosure can respond to the region selection operation for the target reference image, determine the target region based on the region selection operation, and further obtain the target trajectory information represented by the three-dimensional space information, so as to generate the motion video of the target object based on the target region, the target reference image and the target trajectory information using the preset target network model, and the target object is the target region or the object contained in the target region. Through the above method, the user can select any target region of the required motion in the image according to the needs, and on this basis, guide the target network model to generate the motion video of the target object through the target trajectory information, which can achieve the effect of controllable motion object and controllable motion trajectory; further, since the target trajectory information is represented by the three-dimensional space information, it helps to present the motion effect of the target object in the three-dimensional space, better meet the needs of users, and enrich the image-generated video effect and enhance the appeal of the video.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A schematic diagram of a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of image interaction provided by an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of determining a motion area provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a model training provided in an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of the structure of a preset generation model provided in an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the structure of a video generating device provided by an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] The inventor has found through research that the image-generated video method used in the related art is not good. For example, it is necessary to globally control the motion of the image content through information such as text or images, and it is difficult to accurately control the motion of the local area in the image, and it is also impossible to directionally control the motion trajectory of the local area. In addition, the generated video does not give people a good dynamic perception and poor realism. In order to improve the above problems, the embodiments of the present disclosure provide a video generation method, device, equipment and medium, which are described in detail below:

[0032] Figure 1The flowchart of a video generation method provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be executed by a video generation device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0033] Step S102 , in response to a region selection operation on a target reference image, determining a target region based on the region selection operation.

[0034] The target reference image may be, for example, an image input by a user, and the content contained in the target reference image is not limited herein. Exemplarily, the region selection operation includes a smearing operation or a box selection operation. In actual applications, a region selection control such as a brush may be provided to the user, so that the user can perform a region selection operation through the region selection control, such as the user performing a smearing operation or a box selection operation on the target reference image through a brush, or the user can also perform a smearing operation or a box selection operation by touching the screen with a finger, drawing lines, and the like, thereby determining the target area. In some embodiments, the smearing area corresponding to the smearing operation is the target area, and the box selection area corresponding to the box selection operation is the target area. It should be noted that the target area may be any area, such as the entire object, a part of an object, or a non-object area, and is not limited here.

[0035] Step S104, obtaining target trajectory information of the target area moving in the target reference image; wherein the target trajectory information is trajectory information represented by three-dimensional space information.

[0036] In practical applications, the target trajectory information may be information determined based on a user-specified trajectory. If the user does not specify a trajectory, the target trajectory information may be default information or information automatically generated based on a target area in a target reference image, which is not limited here. The target trajectory information represented by three-dimensional spatial information is more helpful in controlling the model to generate a motion trajectory with a three-dimensional visual effect, effectively improving the fidelity of the final generated video, and making it easier to present a good dynamic viewing experience to the user.

[0037] In some implementation examples, the three-dimensional spatial information includes three-dimensional optical flow information, that is, the target trajectory information can be represented by the three-dimensional optical flow information. On this basis, the target trajectory information of the target area moving in the target reference image can be obtained by referring to the following (1) and (2):

[0038] (1) When receiving expected trajectory information set for a target area, generate three-dimensional optical flow information corresponding to the expected trajectory information, and obtain target trajectory information of the target area moving in a target reference image based on the three-dimensional optical flow information corresponding to the expected trajectory information. For example, the three-dimensional optical flow information corresponding to the expected trajectory information is directly used as the target trajectory information.

[0039] Exemplarily, the user can draw the expected trajectory in the form of a line for the target area through a trajectory drawing control or finger touch drawing, and obtain two-dimensional expected trajectory information. And / or, the user can be provided with trajectory input controls such as X direction, Y direction or Z direction, so that the user can input information in each direction in digital form to obtain three-dimensional expected trajectory information. In practical applications, any of the above methods can be selected, or the above two methods can be combined, such as the user first draws the expected trajectory in two dimensions, and then inputs the depth information in the Z direction in digital form to obtain the expected trajectory in three dimensions. The specific method of obtaining the expected trajectory information can be flexibly set and is not limited here.

[0040] For easier understanding, please refer to Figure 2 A schematic diagram of image interaction is shown in FIG. Figure 2 The left side (a) corresponds to the target reference image. Figure 2 The right side of (b) corresponds to the smeared area and the expected trajectory corresponding to the smeared area, that is, Figure 2 The arm area of ​​the cartoon object in the image is the smearing area, that is, the target area. There are upward arrow tracks drawn on both sides of the two arms. The track can also be drawn by the user, indicating that the arms of the cartoon object are expected to be raised upward. It can be understood that the expected track drawn by the user on the image is usually a two-dimensional track, on this basis, the information of the X dimension and the Y dimension can be obtained. For example, the information about the Z dimension can be realized by automatically filling in the preset value or directly obtaining the information input by the user for the Z dimension. There is no limitation here. Through the above method, three-dimensional expected track information can be obtained, and thus the three-dimensional optical flow information corresponding to the expected track information can be obtained. The method of converting the track information into optical flow information can refer to the relevant technology, such as calculating the corresponding optical flow information based on the key pixel points in the track information. For example, the starting point, the end point and the track of the movement of a key pixel point can be determined based on the track information, so as to obtain the optical flow information corresponding to the key pixel point. The above is only an exemplary description and is not limited here.

[0041] (2) When the expected trajectory information set for the target area is not received, the preset three-dimensional optical flow information is used as the target trajectory information of the target area moving in the target reference image; wherein the preset three-dimensional optical flow information can be represented by specific numerical values, such as setting preset optical flow values ​​based on the X, Y, and Z dimensions respectively, which can be flexibly set and is not limited here.

[0042] Step S106, based on the target area, the target reference image and the target trajectory information, a motion video of the target object is generated using a preset target network model; wherein the target object is the target area or an object contained in the target area.

[0043] The target network model can be implemented by a generative model, and in some specific implementation examples, a diffusion model or a network model including a denoising network can be used, which is not limited here. For example, a motion video of a target object can be generated based on a target area, a target reference image, target trajectory information, and randomly generated noise data using a preset target network model. In practical applications, the target object can be a complete target area or an object body contained in the target area, which can be flexibly set.

[0044] Through the above method, the user can select any target area of ​​required movement in the image according to the needs, and on this basis, guide the target network model to generate a motion video of the target object through the target trajectory information, so as to achieve the effect of controllable moving object and controllable motion trajectory; further, since the target trajectory information is represented by three-dimensional spatial information, it is helpful to present the motion effect of the target object in three-dimensional space, better meet the needs of users, enrich the image-generated video effect, and enhance the appeal of the video.

[0045] For ease of understanding, the present disclosure provides a method for training a target network model. Exemplarily, the target network model is obtained through the following steps A to C:

[0046] Step A, obtaining a target video sample; wherein the target video sample is a video with a moving object in the foreground and a still background, that is, a video with a stable lens.

[0047] In some implementation examples, step A may be performed with reference to the following steps A1 to A4:

[0048] Step A1, obtaining a first video sample, wherein the number of the first video samples is multiple. The first video sample is the initially obtained video.

[0049] Step A2, obtaining optical flow information corresponding to the first video sample, and based on the optical flow information corresponding to the first video sample, selecting a second video sample from multiple first video samples; wherein the second video sample is a non-static video.

[0050] In order to improve processing efficiency and reduce the amount of resource calculation occupied, in some specific implementation examples, multiple video frame groups can be first extracted from the first video sample; wherein the video frame group includes a first video frame image and a second video frame image, and the first video frame image and the second video frame image are two frames of images separated by a preset number of frames; then the optical flow information corresponding to each of the multiple video frame groups is obtained, and the optical flow information corresponding to each of the multiple video frame groups is used as the optical flow information corresponding to the first video sample. In practical applications, the number of the above-mentioned video frame groups and the number of preset number of frames between the first video frame image and the second video frame image can be flexibly set according to demand, such as setting the number of video frame groups to 4, and the first video frame image and the second video frame image are separated by 4 frames. For example, 4 video frame groups can be uniformly extracted from the first video sample, and the two frames of images contained in each video frame group are separated by 4 frames, that is, a total of 8 frames of images can be extracted from the first video sample for analysis to obtain the optical flow information corresponding to each video frame group. Afterwards, the still video can be excluded based on the optical flow information of the multiple video frame groups corresponding to the first video sample, so as to screen out the second video sample. Exemplarily, for each first video sample, it is determined whether the maximum value of the absolute value of the optical flow of multiple video frame groups corresponding to the first video sample is greater than a preset value, if yes, the first video sample is used as the filtered second video sample, if no, the first video sample is filtered out. In the above manner, non-static videos can be filtered out efficiently and reliably.

[0051] Step A3, based on the optical flow information corresponding to the second video sample, determine the target mapping information corresponding to the video frame group in the second video sample; wherein the video frame group includes two video frame images with a preset interval, and exemplarily, the target mapping information can be represented by a homography matrix.

[0052] In some specific implementation examples, the optical flow information corresponding to the second video sample includes dense optical flow information corresponding to the video frame group in the second video sample; it should be noted that the video frame group in the first video sample (which may be referred to as the first video frame group for short) and the video frame group in the second video sample (which may be referred to as the second video frame group for short) may be different. Specifically, the number of the first video frame group and the second video frame group and / or the interval between the two video frame images contained therein may be different. For example, each two adjacent frame images in the second video sample may be used as the second video frame group. Assuming that the second video sample has a total of N frames, the number of the second video frame groups is N-1, and the preset interval between the two video frame images in the second video frame group is zero. The above is an implementation example and should not be regarded as a limitation. It can also be flexibly set in actual applications and is not limited here. On the basis of the above, step A3 can be performed with reference to the following steps A3.1 to A3.3:

[0053] Step A3.1, based on the dense optical flow information corresponding to the video frame group in the second video sample, a plurality of sparse optical flow points are obtained using a preset uniform sampling strategy. In the above manner, the dense optical flow can be converted into a sparse optical flow on the grid points. For example, the uniform sampling strategy can be equidistant sampling, and uniform sampling of a total of a specified number of points is performed on both the long side and the short side, such as uniform sampling of a total of 30 points on both the long side and the short side, and a total of 900 sparse optical flow points are sampled.

[0054] Step A3.2: Based on the multiple sparse optical flow points, determine the key point pairs that are successfully matched corresponding to the video frame group in the second video sample.

[0055] The disclosed embodiment can convert the sparse optical flow into the form of key points that are successfully matched. Specifically, for each sparse optical flow point obtained from the above steps, its current coordinates are recorded as (x, y), and the corresponding optical flow value is (u, v), then the point pair in the form of {(x, y), (x+u, y+v)} is defined as the key point pair that is successfully matched. In other words, assuming that the pixel coordinates of the previous frame image in the video frame group are (x, y), then the pixel coordinates that are successfully matched with it in the subsequent frame image in the video frame group are (x+u, y+v).

[0056] Step A3.3, select the key point pairs corresponding to the video frame background in the video frame group from the successfully matched key point pairs, and obtain the target mapping information corresponding to the video frame group in the second video sample based on the mapping information of the key point pairs corresponding to the video frame background. For example, the mapping information of the key point pairs corresponding to the video frame background can be directly used as the target mapping information.

[0057] In some specific implementation examples, based on the successfully matched key point pairs, the RANSAC algorithm can be used to implement the above step A3.3, that is, the key point pairs in the successfully matched key point pairs that are located within the foreground object containing movement are filtered out through the RANSAC algorithm, and the mapping information of the key point pairs corresponding to the background of the video frame is obtained, which is represented in the form of a homography matrix. In this way, it can be effectively ensured that the obtained target mapping information can accurately present the background movement of the second video sample.

[0058] Through the above manner, the target mapping information (such as homography matrix) of each of the multiple video frame groups corresponding to the second video sample can be obtained and stored for subsequent further processing.

[0059] Step A4, based on the target mapping information, screen the third video sample from the second video sample, and obtain the target video sample based on the third video sample; wherein the target video sample is the third video sample, or the target video sample is a sample obtained by segmenting the third video sample.

[0060] The second video sample is a non-static video. Based on the target mapping information of each of the multiple video frame groups corresponding to the second video sample, the affine transformation principle and the preset threshold value can be used to screen out the video whose background remains static. The foreground of the screened third video sample has a moving object and the background remains static. Exemplarily, the target mapping information can be represented by a homography matrix, which is referred to as the target homography matrix here, assuming that the size is M*M, such as a 3*3 matrix representation. By comparing the target homography matrix with a preset standard homography matrix, for example, the standard homography matrix is ​​a matrix in which the values ​​of the diagonal elements are all 1 and the values ​​of the remaining elements are 0. By comparing the numerical difference between each element of the target homography matrix and the elements at the same position of the standard homography matrix, based on the difference corresponding to each of the 9 elements of the homography matrix and the target threshold value, the third video sample can be screened out from the second video sample, such as, if the fusion value (such as the sum, average value, etc.) of the absolute value of the difference corresponding to each of the 9 elements is less than the target threshold value, it means that the target homography matrix indicates that the background is roughly static. Through the above method, it is possible to reasonably and reliably filter out a video with a still background from a non-still video, that is, to obtain a third video sample.

[0061] Afterwards, the third video sample can be directly used as the target video sample, or the third video sample can be segmented as needed. For example, if the third video sample is M frames, but the model only needs to output N frames of video, the third video sample can be segmented to obtain the target video sample.

[0062] Step B, determining the motion region in the target video sample, and obtaining the three-dimensional optical flow information corresponding to the motion region; wherein the motion region is a region containing a moving object. In some specific implementation examples, the motion region in the target video sample can be determined with reference to the following steps 1 to 3:

[0063] Step 1, perform object segmentation processing on the video frame image of the target video sample to obtain the object area. The specific method of object segmentation processing can refer to the relevant technology and will not be repeated here. In practical applications, the object area can be represented in the form of an object mask. In practical applications, object segmentation processing can be performed on each video frame image in the target video sample. For ease of processing, the object area of ​​the subsequent frame can also be quickly determined by object tracking and other methods based on the object segmentation results of the previous frame image.

[0064] Step 2, based on the video frame image of the target video sample and the subsequent frame image adjacent to the video frame image of the target video sample, determine the two-dimensional optical flow information corresponding to the video frame image of the target video sample. In other words, for each video frame image (not the last frame image) in the target video sample, the two-dimensional optical flow information between two adjacent frame images can be determined based on the video frame image and the subsequent frame image adjacent to the video frame image.

[0065] Step 3, based on the object area and the two-dimensional optical flow information, determine the motion area in the video frame image of the target video sample. It can be understood that the object segmentation method is mainly to segment the salient objects, and then combined with the two-dimensional optical flow information, on the one hand, the moving objects can be screened out from the salient objects, and on the other hand, the area with an optical flow value greater than a threshold can be further determined based on the two-dimensional optical flow information, and it can be supplemented as the motion area.

[0066] In some specific implementation examples, step 3 may be performed with reference to the following steps 3.1 to 3.3:

[0067] Step 3.1, based on the two-dimensional optical flow information, a first region is selected from the object region, wherein the optical flow value of the two-dimensional optical flow information corresponding to the first region is greater than a preset threshold. In this way, the moving first region can be selected from the object region.

[0068] Step 3.2, based on the two-dimensional optical flow information, determine a second area from the video frame image of the target video sample; wherein the second area is an area other than the first area, and the optical flow value of the two-dimensional optical flow information corresponding to the second area is greater than a preset threshold. The second area is a motion area supplemented on the basis of the moving object. Although the area does not belong to the object, it also has motion.

[0069] Step 3.3, based on the first area and the second area, obtain the motion area in the video frame image of the target video sample. In practical applications, the first area and the second area can be directly used as motion areas. In addition, the first area and the second area can also be post-processed by corrosion operations, and the post-processed first area and second area can be used as motion areas. In specific implementation, the first area and the second area correspond to a label value respectively, so as to distinguish different areas based on the label value, and then the first area and the second area are corroded. Specifically, the processing can be directly based on the mask corresponding to each area. In the above manner, it can be effectively ensured that different areas are not connected.

[0070] For ease of understanding, please refer to Figure 3The schematic diagram of determining a motion area shown in the figure, the areas numbered 1, 2, 3, and 4 are all object areas obtained through object segmentation, and the area numbered 5 is an area where the optical flow value determined based on the two-dimensional optical flow information is greater than the optical flow threshold, wherein the area corresponding to the number 6 is also indicated in the number 5, which is actually also an object area, but is an area missed by the object segmentation, that is, the second area obtained based on the two-dimensional optical flow information will also cover the object area where the object segmentation is inaccurate. Based on the above steps, it can be known that the target area is the area corresponding to the numbers 2, 4, 5, and 6, wherein the area corresponding to the number 5 can be regarded as an area other than the areas 2, 4, and 6. That is, even in the case of inaccurate object segmentation, reasonable supplementation can be achieved based on the two-dimensional optical flow information. Alternatively, the target area can also be referred to as the area corresponding to the numbers 2, 4, and 5, wherein the area corresponding to the number 5 includes the unsegmented area 6. Both of the above division methods are acceptable and can be flexibly selected. In addition, it should be noted that the area where the optical flow value determined based on the two-dimensional optical flow information is greater than the optical flow threshold does not necessarily have to include the area where the object is detected, such as Figure 3 The small independent area corresponding to the number 5 shown in the figure does not include the object detection area, but it will also be used as the motion area. Through the above method, the motion area can be determined more comprehensively and accurately.

[0071] In some implementations after the motion region is determined, the three-dimensional optical flow information corresponding to the motion region may be obtained by referring to the following steps 1) and 2):

[0072] Step 1), obtain the distance gradient field information corresponding to the motion area. Specifically, the corresponding distance field can be calculated for each motion area to obtain the distance between the pixel of the motion area and the edge of the motion area; based on the distance corresponding to the pixel of the motion area and the distance corresponding to the adjacent pixel of the pixel, the distance gradient field information corresponding to the motion area is determined, and the direction pointing to the center of the object can be determined based on the distance gradient field information.

[0073] Step 2), obtain the two-dimensional optical flow information corresponding to the moving area, and obtain the three-dimensional optical flow information corresponding to the moving area based on the two-dimensional optical flow information and the distance gradient field information. The information in the X direction and the Y direction can be determined based on the two-dimensional optical flow information, and then the information in the Z direction can be determined based on the distance gradient field information, and the Z direction is the direction pointing to the center of the object. For the X direction and the Y direction, the average optical flow of the entire moving area can be directly calculated as the component of the plane motion. Specifically, the average value of the components of the optical flow values ​​of all pixels in the moving area in the X direction can be used as the X motion component value corresponding to the moving area, and the average value of the components of the optical flow values ​​of all pixels in the moving area in the Y direction can be used as the Y motion component value corresponding to the moving area. When an object is far away from the camera, the pixels in the area occupied by the object will shrink toward the center of the object. Similarly, when the object approaches the camera, the pixels in the area occupied by the object will diffuse toward the edge of the object. Based on this principle, for the Z direction (that is, the distance gradient field direction), the average value of the components of the optical flow values ​​of all pixels in the moving area in the Z direction can be used as the Z motion component value corresponding to the moving area.

[0074] In practical applications, the three-dimensional optical flow information of each moving area in the target video sample can be obtained, and the three-dimensional optical flow information can accurately present the moving direction and intensity of the moving area in the three-dimensional space.

[0075] Step C, based on the target video sample, the motion area and the three-dimensional optical flow information corresponding to the motion area, adjust the parameters of the preset generation model to obtain the target network model based on the preset generation model after the parameters are adjusted.

[0076] In practical applications, the structure of the preset generation model is the same as that of the target network model, and the main difference is in the parameters. When training the preset generation model, the parameters of the preset generation model can be adjusted based on the difference between the video output by the preset generation model and the target video sample, so as to obtain a preset generation model that meets the requirements (that is, the preset generation model after parameter adjustment), and then the preset generation model after parameter adjustment can be directly used as the target network model.

[0077] For ease of understanding, for example, reference may be made to Figure 4 A model training schematic diagram is shown, which focuses on illustrating the method of obtaining target video samples, motion areas, and 3D optical flow information. Finally, by training a preset generation model, the parameters of the preset generation model are adjusted based on the difference between the model's output video and the target video sample, and finally a target network model whose output results meet the expectations can be obtained. The specific process of obtaining the target video sample, motion area, and 3D optical flow information can refer to the above-mentioned related content and will not be repeated here.

[0078] Furthermore, the preset generation model provided by the embodiment of the present disclosure includes a denoising network; wherein the first frame image information of the target video sample, the mask image corresponding to the motion area, and the three-dimensional optical flow information corresponding to the motion area are introduced into the denoising network in the form of input channel splicing. The first frame image information of the target video sample can be the encoding result of the first frame image of the target video sample. For ease of understanding, please refer to Figure 5 A structural diagram of a preset generation model shown in the figure illustrates an encoder, a denoising network and a decoder, wherein the denoising network can be implemented by a Unet network, for example, and is not limited here. The target video sample can be input into the encoder to obtain the encoding result of the encoder for the target video sample, wherein the encoding result of the first frame image of the target video sample, the mask image corresponding to the motion area after scale processing, and the three-dimensional optical flow information corresponding to the motion area can be input into the denoising network in the form of input channel splicing to obtain the denoising result of the denoising network, and then decoded based on the decoder to obtain the output video, and the model parameters can be adjusted based on the information difference between the output video and the target video sample to make the output video of the model as close as possible to the target video sample, so that the final model can output the expected video result. It should be noted that Figure 5 The key point is to illustrate the difference between the input of the denoising network and the existing network. In practical applications, the denoising network will also have noisy data input, and the generative model can also have other network modules. Figure 5 No longer indicate one by one.

[0079] Through the above-mentioned input channel splicing method, the newly added control information such as the mask map corresponding to the moving area and the three-dimensional optical flow information corresponding to the moving area can be introduced into the original model framework. It can be understood that even if the existing image-generated video model may also have the input of a mask map (such as one used to indicate the frames that need to be generated), but considering the subsequent zero-impact initialization, the embodiment of the present disclosure will not input the mask map corresponding to the moving area on the original basis, but will add a new input channel for the mask map corresponding to the moving area.

[0080] Exemplarily, when initializing the preset generation model, the weights corresponding to the first newly added channel and the second newly added channel of the denoising network are set to zero; wherein the first newly added channel is used to input the mask image corresponding to the motion area, and the second newly added channel is used to input the three-dimensional optical flow information corresponding to the motion area.

[0081] That is, the weights corresponding to the first newly added channel and the second newly added channel are not randomly initialized. By setting the weights to zero during the initialization process, such as multiplying the control information such as the mask map corresponding to the moving area and the three-dimensional optical flow information by zero, zero influence is achieved, that is, maximizing the original image-generated video capability of the model, and then gradually adjusting the weights during the subsequent model training process. Through the above method, the ability of regional interactive control can be added without affecting the image-to-video capability of the original model as much as possible. After zero-impact initialization, for the same input and arbitrary control information (the mask map corresponding to the moving area and the three-dimensional optical flow information corresponding to the moving area), the above method provided by the embodiment of the present disclosure can be used to achieve the same output form as the original image-to-video model. In summary, the embodiment of the present disclosure can achieve a new image-generated video effect without major adjustments and changes on the basis of the existing image-generated video model, and it is easy to implement.

[0082] To sum up, the target network model adopted in the video generation method provided by the embodiment of the present disclosure is easy to implement, and the user can select any target area of ​​required movement in the image according to needs, and on this basis, guide the target network model to generate a motion video of the target object through the target trajectory information, which can achieve the effect of controllable moving objects and controllable motion trajectories; further, since the target trajectory information is represented by three-dimensional spatial information, it helps to present the motion effect of the target object in three-dimensional space, better meet user needs, enrich the image-generated video effect, and enhance the appeal of the video.

[0083] Corresponding to the aforementioned video generation method, the embodiment of the present disclosure further provides a video generation device, Figure 6 FIG. 1 is a schematic diagram of a video generation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 6 As shown, the video generating device includes:

[0084] A region determination module 602, configured to respond to a region selection operation on a target reference image and determine a target region based on the region selection operation;

[0085] The trajectory acquisition module 604 is used to acquire target trajectory information of the target area moving in the target reference image; wherein the target trajectory information is trajectory information represented by three-dimensional space information;

[0086] The video generation module 606 is used to generate a motion video of the target object based on the target area, the target reference image and the target trajectory information using a preset target network model; wherein the target object is the target area or an object contained in the target area.

[0087] Through the above device, the user can select any target area of ​​required movement in the image according to needs, and on this basis, guide the target network model to generate a motion video of the target object through the target trajectory information, so as to achieve the effect of controllable moving object and controllable motion trajectory; further, since the target trajectory information is represented by three-dimensional spatial information, it is helpful to present the motion effect of the target object in three-dimensional space, better meet user needs, enrich the image-generated video effect, and enhance the appeal of the video.

[0088] In some embodiments, the three-dimensional spatial information includes three-dimensional optical flow information, and the trajectory acquisition module 604 is specifically used to: upon receiving expected trajectory information set for the target area, generate three-dimensional spatial information corresponding to the expected trajectory information, and obtain target trajectory information of the target area moving in the target reference image based on the three-dimensional optical flow information corresponding to the expected trajectory information; upon not receiving the expected trajectory information set for the target area, use the preset three-dimensional optical flow information as the target trajectory information of the target area moving in the target reference image; wherein the preset three-dimensional optical flow information is represented by a specific numerical value.

[0089] In some embodiments, the device also includes a model acquisition module, which is used to obtain the target network model through the following steps: obtaining a target video sample; wherein the target video sample is a video with a moving object in the foreground and a stationary background; determining the moving area in the target video sample, and obtaining three-dimensional optical flow information corresponding to the moving area; wherein the moving area is an area containing a moving object; based on the target video sample, the moving area and the three-dimensional optical flow information corresponding to the moving area, adjusting the parameters of the preset generation model to obtain the target network model based on the preset generation model after the parameters are adjusted.

[0090] In some embodiments, the model acquisition module is specifically used to: obtain a first video sample; wherein the number of the first video samples is multiple; obtain optical flow information corresponding to the first video sample, and based on the optical flow information corresponding to the first video sample, screen out a second video sample from the multiple first video samples; wherein the second video sample is a non-static video; based on the optical flow information corresponding to the second video sample, determine the target mapping information corresponding to the video frame group in the second video sample; wherein the video frame group includes two video frame images with a preset interval; based on the target mapping information, screen out a third video sample from the second video sample, and obtain a target video sample based on the third video sample; wherein the target video sample is the third video sample, or the target video sample is a sample obtained by segmenting the third video sample.

[0091] In some embodiments, the optical flow information corresponding to the second video sample includes dense optical flow information corresponding to the video frame group in the second video sample; the model acquisition module is specifically used to: based on the dense optical flow information corresponding to the video frame group in the second video sample, obtain a plurality of sparse optical flow points using a preset uniform sampling strategy; based on the plurality of sparse optical flow points, determine the successfully matched key point pairs corresponding to the video frame group in the second video sample; filter out the key point pairs corresponding to the video frame background in the video frame group from the successfully matched key point pairs, and based on the mapping information of the key point pairs corresponding to the video frame background, obtain the target mapping information corresponding to the video frame group in the second video sample.

[0092] In some embodiments, the model acquisition module is specifically used to: perform object segmentation processing on the video frame image of the target video sample to obtain an object area; determine the two-dimensional optical flow information corresponding to the video frame image of the target video sample based on the video frame image of the target video sample and the subsequent frame image adjacent to the video frame image of the target video sample; determine the motion area in the video frame image of the target video sample based on the object area and the two-dimensional optical flow information.

[0093] In some embodiments, the model acquisition module is specifically used to: screen out a first area from the object area based on the two-dimensional optical flow information; wherein the optical flow value of the two-dimensional optical flow information corresponding to the first area is greater than a preset threshold; based on the two-dimensional optical flow information, determine a second area from the video frame image of the target video sample; wherein the second area is an area other than the first area, and the optical flow value of the two-dimensional optical flow information corresponding to the second area is greater than the preset threshold; based on the first area and the second area, obtain a motion area in the video frame image of the target video sample.

[0094] In some embodiments, the model acquisition module is specifically used to: obtain the distance gradient field information corresponding to the motion area; obtain the two-dimensional optical flow information corresponding to the motion area, and based on the two-dimensional optical flow information and the distance gradient field information, obtain the three-dimensional optical flow information corresponding to the motion area.

[0095] In some embodiments, the model acquisition module is specifically used to: obtain the distance between the pixels of the motion area and the edge of the motion area; based on the distance corresponding to the pixels of the motion area and the distance corresponding to the adjacent pixels of the pixels, determine the distance gradient field information corresponding to the motion area.

[0096] In some embodiments, the preset generation model includes a denoising network; wherein, the first frame image information of the target video sample, the mask image corresponding to the motion area, and the three-dimensional optical flow information corresponding to the motion area are introduced into the denoising network in the form of input channel splicing.

[0097] In some embodiments, the device also includes a weight setting module, which is used to set the weights corresponding to the first newly added channel and the second newly added channel of the denoising network to zero when initializing the preset generation model; wherein the first newly added channel is used to input the mask image corresponding to the motion area, and the second newly added channel is used to input the three-dimensional optical flow information corresponding to the motion area.

[0098] The video generating device provided in the embodiments of the present disclosure can execute the video generating method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0100] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0101] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0102] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0103] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0104] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0105] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0106] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the video generation method provided by the embodiment of the present disclosure.

[0107] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0108] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the video generation method in the embodiment of the present disclosure when the computer program / instruction is executed by a processor.

[0109] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0110] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0111] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0112] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0113] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0114] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that: include: In response to a region selection operation on a target reference image, determining a target region based on the region selection operation; Acquire target trajectory information of the target area moving in the target reference image; wherein the target trajectory information is trajectory information represented by three-dimensional space information; Based on the target area, the target reference image and the target trajectory information, a motion video of the target object is generated using a preset target network model; wherein the target object is the target area or an object contained in the target area.

2. The method according to claim 1, characterized in that: The three-dimensional space information includes three-dimensional optical flow information, and the step of obtaining target trajectory information of the target area moving in the target reference image includes: In the case of receiving expected trajectory information set for the target area, generating three-dimensional optical flow information corresponding to the expected trajectory information, and obtaining target trajectory information of the target area moving in the target reference image based on the three-dimensional optical flow information corresponding to the expected trajectory information; In the case where the expected trajectory information set for the target area is not received, preset three-dimensional optical flow information is used as target trajectory information of the target area moving in the target reference image; wherein the preset three-dimensional optical flow information is represented by a specific numerical value.

3. The method according to claim 1, characterized in that The target network model is obtained through the following steps: Obtain a target video sample; wherein the target video sample is a video with a moving object in the foreground and a stationary background; Determine a motion region in the target video sample, and obtain three-dimensional optical flow information corresponding to the motion region; wherein the motion region is a region containing a moving object; Based on the target video sample, the motion area and the three-dimensional optical flow information corresponding to the motion area, the parameters of the preset generation model are adjusted to obtain the target network model based on the preset generation model after the parameters are adjusted.

4. The method according to claim 3, characterized in that The step of obtaining a target video sample comprises: Acquire a first video sample; wherein the number of the first video samples is multiple; Obtaining optical flow information corresponding to the first video sample, and based on the optical flow information corresponding to the first video sample, screening a second video sample from a plurality of the first video samples to obtain a second video sample; wherein the second video sample is a non-static video; Based on the optical flow information corresponding to the second video sample, determine the target mapping information corresponding to the video frame group in the second video sample; wherein the video frame group includes two video frame images with a preset interval; Based on the target mapping information, a third video sample is screened from the second video sample, and a target video sample is obtained based on the third video sample; wherein the target video sample is the third video sample, or the target video sample is a sample obtained by segmenting the third video sample.

5. The method according to claim 4, characterized in that The optical flow information corresponding to the second video sample includes dense optical flow information corresponding to the video frame group in the second video sample; and determining the target mapping information corresponding to the video frame group in the second video sample based on the optical flow information corresponding to the second video sample includes: Based on the dense optical flow information corresponding to the video frame group in the second video sample, a plurality of sparse optical flow points are obtained by using a preset uniform sampling strategy; Determine, based on the multiple sparse optical flow points, a pair of key points that are successfully matched and correspond to a group of video frames in the second video sample; The key point pairs corresponding to the video frame background in the video frame group are screened out from the successfully matched key point pairs, and based on the mapping information of the key point pairs corresponding to the video frame background, the target mapping information corresponding to the video frame group in the second video sample is obtained.

6. The method according to claim 3, characterized in that Determining a motion region in the target video sample includes: Performing object segmentation processing on the video frame image of the target video sample to obtain an object area; Determine two-dimensional optical flow information corresponding to the video frame image of the target video sample based on the video frame image of the target video sample and a subsequent frame image adjacent to the video frame image of the target video sample; Based on the object area and the two-dimensional optical flow information, a motion area in the video frame image of the target video sample is determined.

7. The method according to claim 6, characterized in that The determining, based on the object area and the two-dimensional optical flow information, a motion area in the video frame image of the target video sample comprises: Based on the two-dimensional optical flow information, a first area is screened out from the object area; wherein an optical flow value of the two-dimensional optical flow information corresponding to the first area is greater than a preset threshold; Based on the two-dimensional optical flow information, determining a second area from the video frame image of the target video sample; wherein the second area is an area other than the first area, and the optical flow value of the two-dimensional optical flow information corresponding to the second area is greater than the preset threshold; Based on the first area and the second area, a motion area in the video frame image of the target video sample is obtained.

8. The method according to claim 3, characterized in that The obtaining of the three-dimensional optical flow information corresponding to the motion area includes: Acquiring distance gradient field information corresponding to the motion area; The two-dimensional optical flow information corresponding to the moving area is acquired, and based on the two-dimensional optical flow information and the distance gradient field information, the three-dimensional optical flow information corresponding to the moving area is obtained.

9. The method according to claim 8, characterized in that The obtaining of distance gradient field information corresponding to the motion region includes: Acquire the distance between the pixel of the motion area and the edge of the motion area; Based on the distance corresponding to the pixel of the motion area and the distance corresponding to the adjacent pixels of the pixel, the distance gradient field information corresponding to the motion area is determined.

10. The method according to claim 3, characterized in that: The preset generation model includes a denoising network; wherein, the first frame image information of the target video sample, the mask image corresponding to the motion area and the three-dimensional optical flow information corresponding to the motion area are introduced into the denoising network in the form of input channel splicing.

11. The method according to claim 10, characterized in that The method further comprises: When initializing the preset generation model, the weights corresponding to the first newly added channel and the second newly added channel of the denoising network are set to zero; wherein the first newly added channel is used to input the mask image corresponding to the motion area, and the second newly added channel is used to input the three-dimensional optical flow information corresponding to the motion area.

12. A video generating device, characterized in that: include: A region determination module, configured to respond to a region selection operation on a target reference image and determine a target region based on the region selection operation; A trajectory acquisition module, used to acquire target trajectory information of the target area moving in the target reference image; wherein the target trajectory information is trajectory information represented by three-dimensional space information; A video generation module is used to generate a motion video of a target object based on the target area, the target reference image and the target trajectory information using a preset target network model; wherein the target object is the target area or an object contained in the target area.

13. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the video generation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the video generation method according to any one of claims 1 to 11.

Images