Image-to-video method and device

Abstract

The embodiment of the invention discloses a picture video generation method and device. According to the embodiment of the invention, a source image containing a target object is input into a first video generation model to obtain a material video, an inter-frame transformation matrix sequence is determined according to the material video, and then an object mask image corresponding to the target object is obtained from the source image; the inter-frame transformation matrix sequence is applied to an object mask image to obtain a plurality of mask images so as to form a mask image sequence, and the inter-frame transformation matrix sequence is applied to a source image to obtain a plurality of target object images so as to form a target object image sequence. And determining target input data according to the source image, the mask image sequence and the target object image sequence, and inputting the target input data into a second video generation model supporting local redrawing to obtain a corresponding target video. The video is generated through the two models, intelligent end-to-end video generation is achieved, and the diversity of motion tracks can be achieved while the target object is kept not to be diffused without introducing preset motion parameters.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for generating video from an image. Background Art

[0002] With the rapid development of artificial intelligence, video generation technology has received extensive attention and research. The commonly used diffusion model-based image-generated video model may cause deformation and distortion of objects in the video when generating corresponding videos from images, resulting in the video being unable to accurately reflect the true shape and details of the objects. In addition, the image-generated video model usually requires users to preset the motion trajectory, which is complicated for users to operate. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an image-generated video method and device to realize intelligent end-to-end image-generated video, which can achieve the diversity of motion trajectories while keeping the target object area from spreading without introducing preset motion parameters.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating video from an image, the method comprising:

[0005] Acquire a source image, where the source image is an image containing a target object;

[0006] Inputting the source image into a first video generation model to obtain a corresponding material video, wherein the first video generation model is a pre-trained image-generated video model;

[0007] Determine an inter-frame transformation matrix sequence corresponding to the material video;

[0008] Determining an object mask image corresponding to the target object in the source image;

[0009] Determine a target object image sequence according to the inter-frame transformation matrix sequence and the source image;

[0010] Determining a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image;

[0011] Determining target input data according to the source image, the target object image sequence and the mask image sequence;

[0012] The target input data is input into a second video generation model to obtain a corresponding target video, wherein the second video generation model is an image-generated video model or a video-generated video model with a local redrawing function.

[0013] In a second aspect, an embodiment of the present invention provides a device for generating video from an image, the device comprising:

[0014] An acquisition module, used for acquiring a source image, wherein the source image is an image containing a target object;

[0015] A first input module, used for inputting the source image into a first video generation model to obtain a corresponding material video, wherein the first video generation model is a pre-trained image-generated video model;

[0016] A first determination module is used to determine an inter-frame transformation matrix sequence corresponding to the material video;

[0017] A second determination module, configured to determine an object mask image corresponding to the target object in the source image;

[0018] A third determination module is used to determine a target object image sequence according to the inter-frame transformation matrix sequence and the source image;

[0019] A fourth determination module, configured to determine a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image;

[0020] a fifth determination module, configured to determine target input data according to the source image, the target object image sequence and the mask image sequence;

[0021] The second input module is used to input the target input data into a second video generation model to obtain a corresponding target video. The second video generation model is an image-generated video model or a video-generated video model with a local redrawing function.

[0022] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, wherein the computer program instructions implement the method described in the first aspect when executed by a processor.

[0023] In a fourth aspect, an embodiment of the present invention provides an electronic device, the device comprising:

[0024] a memory for storing one or more computer program instructions;

[0025] A processor, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0026] In a fifth aspect, an embodiment of the present invention provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method as described in the first aspect.

[0027] In the embodiment of the present invention, a source image containing a target object is input into a first video generation model to obtain a material video, an inter-frame transformation matrix sequence is determined according to the material video, and then an object mask image corresponding to the target object is obtained from the source image. The inter-frame transformation matrix sequence is applied to the object mask image to obtain multiple mask images to form a mask image sequence, and the inter-frame transformation matrix sequence is applied to the source image to obtain multiple target object images to form a target object image sequence. The source image, the mask image sequence and the target object image sequence are converted into target input data that meet the input requirements of the second video generation model, and the target input data is input into a second video generation model that supports local redrawing to obtain the corresponding target video. Thus, when the model graph is used to generate video for the first time, the inter-frame transformation matrix can be determined according to the obtained video, and the motion trajectory of the target object can be described by using the inter-frame transformation matrix. When the model graph is used to generate video for the second time, the local redrawing of the video frame can be controlled according to the inter-frame transformation matrix, so that the target object in the generated target video is clear and does not diffuse. By generating videos from the model twice, intelligent end-to-end graph generation video is realized, and the diversity of motion trajectories can be achieved while keeping the target object area from diffusing without introducing preset motion parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0029] Figure 1 is a flow chart of a method for generating video from an image according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a first video generation model workflow according to an embodiment of the present invention;

[0031] Figure 3 A flowchart of a method for determining an inter-frame transformation matrix sequence according to an embodiment of the present invention;

[0032] Figure 4 A flowchart of a method for determining a target object image sequence according to an embodiment of the present invention;

[0033] Figure 5 A flowchart of a method for determining a target object image sequence according to an embodiment of the present invention;

[0034] Figure 6 is a flowchart of a method for determining a mask image sequence according to an embodiment of the present invention;

[0035] Figure 7 A flowchart of a method for determining a target object image sequence according to an embodiment of the present invention;

[0036] Figure 8 A flowchart of a method for determining target input data according to an embodiment of the present invention;

[0037] Fig. 9 is a flow chart of a target video generation method according to an embodiment of the present invention;

[0038] Fig.10 A schematic diagram of a second video generation model workflow according to an embodiment of the present invention;

[0039] Fig.11 is a flow chart of a target video generation method according to an embodiment of the present invention;

[0040] Fig.12 A schematic diagram of a second video generation model workflow according to an embodiment of the present invention;

[0041] Fig.13 A flowchart of a method for generating video from an image according to an embodiment of the present invention;

[0042] Fig.14 A schematic diagram of a device for generating video from an image according to an embodiment of the present invention;

[0043] Fig.15 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0045] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0046] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0047] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0048] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0049] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0050] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0051] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0052] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0053] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0054] Figure 1 FIG. 4 is a flow chart of a method for generating video from an image according to an embodiment of the present invention. Figure 1 As shown, the image-to-video method comprises the following steps:

[0055] Step S100, obtaining a source image.

[0056] The source image is an image containing a target object, and the target object is a subject highlighted in the source image, such as a commodity, an animal, a plant, a person or an animation character.

[0057] Step S200: input the source image into a first video generation model to obtain a corresponding material video.

[0058] The first video generation model is a pre-trained Image-to-Video (I2V) model, which is used to convert static images into dynamic videos and is widely used in video production, advertising creativity, virtual reality and other fields. The Image-to-Video model can be a generative model based on a diffusion model, such as stable diffusion, sora, etc.

[0059] It is worth noting that different types of target objects have different motion trajectories and motion features. For example, if the target object is a person's head, the motion of the person in the corresponding generated video may be changes in facial expressions such as blinking, smiling, and crying. If the target object is the whole body of the person, the motion of the person in the corresponding generated video may be body movements such as walking, turning around, and swinging arms. If the target object is a commodity, the motion of the commodity in the corresponding video may be changes in light and shadow, perspective, or color projected on the commodity. Therefore, the image-generated video model of this embodiment needs to be trained according to the type of target object before use. The following content will be explained using the target object as a commodity as an example.

[0060] The pre-training process of the first video generation model is to collect product display videos with smooth, stable pictures and clear picture quality, and select appropriate video frames from the product display videos as source images, wherein the appropriate video frames refer to video frames containing clear images of products. A sample data set is constructed based on the product display videos and source images, and the sample data set is used to train and fine-tune the image-generated video model to obtain the first video generation model with vertical characteristics. The vertical characteristics refer to the attributes and requirements that are unique to a specific industry or field. In the scenario of generating product display videos, the vertical characteristics include light and shadow settings, detail display, product usage scenario display, product performance parameters and other characteristics.

[0061] In a possible implementation, the length of the material video output by the first video generation model may be preset, for example, the length of the video output by the first video generation model may be preset to 2 seconds, 4 seconds, 8 seconds, etc.

[0062] After the first video generation model is trained, it can be put into use. After receiving the source image, the first video generation model can generate multiple corresponding material video frames, and generate corresponding material videos based on the multiple material video frames, that is, the material video contains multiple material video frames.

[0063] Figure 2 FIG. 1 is a schematic diagram of the workflow of the first video generation model according to an embodiment of the present invention. Figure 2As shown, after the source image is input into the first video generation model, the encoding module in the first video generation model encodes it to obtain the corresponding encoding result, such as using a convolutional neural network to extract high-level features of the input source image. These features can capture important information in the image, such as edges, textures, and objects. Then the encoding result is copied i to obtain i copies of the encoding result, that is, an intermediate frame is generated, such as using a generative adversarial network or a variational autoencoder to generate an intermediate frame, and by learning the time evolution law of the image, a series of intermediate frames are generated to gradually transition the static image to a dynamic video. Then, each intermediate frame is diffused to gradually add and remove noise to generate data to improve the video quality, and finally the diffused video frame is decoded to obtain the material video.

[0064] Step S300, determining an inter-frame transformation matrix sequence corresponding to the material video.

[0065] Among them, the inter-frame transformation matrix sequence is a series of transformation matrices calculated for the change of objects or scenes between consecutive video frames. These transformations may include affine transformations, projective transformations, etc., which are used to capture the relative motion or change between two adjacent video frames. Affine transformation is a linear transformation combined with a translation operation. It retains the "straightness" and "collinearity" of points, lines, and line segments, but does not retain distances and angles. Common affine transformations include scaling, rotation, translation, and shearing. Projective transformation (also called perspective transformation) is a more extensive transformation that allows images to be manipulated with perspective effects. It retains the properties of straight lines, but does not retain parallelism and proportion. Applications of projective transformations include image correction and 3D projection.

[0066] Figure 3 FIG. 4 is a flow chart of a method for determining an inter-frame transformation matrix sequence according to an embodiment of the present invention. Figure 3 As shown, the method for determining the inter-frame transformation matrix sequence includes the following steps:

[0067] Step S301, performing image segmentation on each of the material video frames to obtain a corresponding foreground image.

[0068] The material video frame includes a target object and a background other than the target object, and the foreground image is an image of the target object. Object sub-images (ie foreground images) and background sub-images can be obtained by cutting out each material video frame.

[0069] Step S302: extracting image features of the foreground image.

[0070] Among them, the calculation of various types of transformations, such as affine transformation or projective transformation, usually requires the use of image features, including scale-invariant feature transform (SIFT), accelerated robust features (SURF), corner detection, Canny edge features, FAST (Features from Accelerated Segment Test) features, feature matching, etc.

[0071] Step S303: determining an inter-frame transformation matrix between adjacent material video frames according to the image features.

[0072] Specifically, after determining the image features of the foreground image, it is necessary to perform feature matching between adjacent source video frames to find the corresponding relationship. Common feature matching methods include Brute-Force Matcher (BFMatcher), Fast Library for Approximate Nearest Neighbors (FLANN), etc.

[0073] After the matching is completed, the transformation matrix between frames is calculated through the matched feature point pairs. Common transformation matrix calculation methods include: homography matrix (Homography), affine transformation matrix (Affine Transformation) and rigid transformation matrix (Rigid Transformation). The homography matrix is ​​applicable to the transformation of planar scenes, the affine transformation matrix is ​​applicable to linear transformation, and the rigid transformation matrix is ​​applicable to rigid body motion, including translation and rotation.

[0074] In a possible implementation manner, the mask image in the background motion video frame may also be smoothed.

[0075] In a possible implementation manner, the inter-frame transformation matrix may also be smoothed.

[0076] Step S304: determining the inter-frame transformation matrix sequence according to the inter-frame transformation matrices between adjacent material video frames.

[0077] Steps S200-S300 use the first video generation model to determine the material video, and then determine the inter-frame transformation matrix sequence based on the material video. The inter-frame transformation matrix sequence can replace the motion trajectory parameters that need to be preset by the user in the prior art, thereby reducing the difficulty of use and realizing intelligent generation of motion parameters.

[0078] Step S400: determining an object mask image corresponding to the target object in the source image.

[0079] The object mask image is a binary image with the same size or a matching size as the object sub-image (ie, the foreground image).

[0080] Step S500: determining a target object image sequence according to the inter-frame transformation matrix sequence and the source image.

[0081] Specifically, the first inter-frame transformation matrix in the inter-frame transformation matrix sequence is applied to the source image to determine the first target object image of the target object image sequence, and the i-th inter-frame transformation matrix in the inter-frame transformation matrix sequence is applied to the last target object image of the current target object image sequence in an iterative manner to determine the i-th target object image of the target object image sequence until all inter-frame transformation matrices in the inter-frame transformation matrix sequence are applied, and i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0082] Figure 4 FIG. 4 is a flow chart of a method for determining a target object image sequence according to an embodiment of the present invention. Figure 4 As shown, the target object image sequence determination method includes the following steps:

[0083] Step S511: Apply the first inter-frame transformation matrix in the inter-frame transformation matrix sequence to the source image to determine the first target object image in the target object image sequence.

[0084] Step S512, sequentially taking the i-th inter-frame transformation matrix in the inter-frame transformation matrix sequence.

[0085] The i-th inter-frame transformation matrix is ​​an unapplied inter-frame transformation matrix.

[0086] Step S513: Apply the i-th inter-frame transformation matrix to the last target object image in the current target object image sequence to determine the corresponding target object image.

[0087] Step S514: adding the target object image to the end of the current target object image.

[0088] Step S515: determine whether there is an unapplied inter-frame transformation matrix in the inter-frame transformation matrix sequence.

[0089] If it exists, go to step S512; if it does not exist, go to step S516.

[0090] Step S516, determining the target object image sequence.

[0091] Above Figure 4 The target object image sequence determination method shown is to apply each inter-frame transformation matrix in the inter-frame transformation matrix sequence to the last target object image of the current target object image sequence in sequence, and then add the obtained target object image to the end of the current target object image sequence in sequence until all inter-frame transformation matrices are applied.

[0092] Figure 5 FIG. 4 is a flow chart of a method for determining a target object image sequence according to an embodiment of the present invention. Figure 5 As shown, the target object image sequence determination method includes the following steps:

[0093] Step S521, determining the i-th target transformation matrix according to the first i inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0094] Step S522: Apply the i-th target transformation matrix to the source image to obtain the i-th target object image in the target object image sequence, where i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0095] Above Figure 5 The target object image sequence determination method shown is to determine the inter-frame transformation matrix corresponding to each target object image in the target object image sequence according to the inter-frame transformation matrix sequence, and calculate the target transformation matrix required from the source image directly to the i-th target object image according to the 1st to ith inter-frame transformation matrices. For example, the inter-frame transformation matrix corresponding to the second target object image needs to be determined according to the first inter-frame transformation matrix and the second inter-frame transformation matrix in the inter-frame transformation matrix sequence, and the inter-frame transformation matrix corresponding to the fifth target object image needs to be determined according to the first, second, third, fourth and fifth inter-frame transformation matrices in the inter-frame transformation matrix sequence. After determining the target transformation matrix, directly applying it to the source image can directly obtain the corresponding target object image. This method obtains a target object image with higher definition.

[0096] Step S600: determining a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image.

[0097] Specifically, the first inter-frame transformation matrix in the inter-frame transformation matrix sequence is applied to the object mask image to determine the first mask image of the mask image sequence, and the i-th inter-frame transformation matrix in the inter-frame transformation matrix sequence is applied to the last mask image of the current mask image sequence in an iterative manner to determine the i-th mask image of the mask image sequence until all inter-frame transformation matrices in the inter-frame transformation matrix sequence are applied, and i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0098] Figure 6 FIG. 4 is a flowchart of a method for determining a mask image sequence according to an embodiment of the present invention. Figure 6 As shown, the mask image sequence determination method comprises the following steps:

[0099] Step S611: Apply the first inter-frame transformation matrix in the inter-frame transformation matrix sequence to the object mask image to determine the first mask image in the mask image sequence.

[0100] Step S612, sequentially taking the i-th inter-frame transformation matrix in the inter-frame transformation matrix sequence.

[0101] The i-th inter-frame transformation matrix is ​​an unapplied inter-frame transformation matrix.

[0102] Step S613: Apply the i-th inter-frame transformation matrix to the last mask image in the current mask image sequence to determine the corresponding mask image.

[0103] Step S614: adding the mask image to the end of the current mask image.

[0104] Step S615: determine whether there is an unapplied inter-frame transformation matrix in the inter-frame transformation matrix sequence.

[0105] If it exists, go to step S612; if it does not exist, go to step S616.

[0106] Step S616, determining a mask image sequence.

[0107] Above Figure 6 The mask image sequence determination method shown is to apply each inter-frame transformation matrix in the inter-frame transformation matrix sequence to the last mask image of the current mask image sequence in sequence, and then add the newly obtained mask image to the end of the current mask image sequence in sequence until all inter-frame transformation matrices are applied.

[0108] Figure 7 FIG. 4 is a flowchart of a method for determining a mask image sequence according to an embodiment of the present invention. Figure 7 As shown, the mask image sequence determination method comprises the following steps:

[0109] Step S621, determining the i-th target transformation matrix according to the first i inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0110] Step S622: Apply the i-th target transformation matrix to the object mask image to obtain the i-th mask image in the mask image sequence, where i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0111] Above Figure 7 The mask image sequence determination method shown is to determine the inter-frame transformation matrix corresponding to each mask image in the mask image sequence according to the inter-frame transformation matrix sequence, and calculate the target transformation matrix required from the object mask image directly to the i-th mask image according to the 1st to ith inter-frame transformation matrices. For example, the inter-frame transformation matrix corresponding to the second mask image needs to be determined according to the first inter-frame transformation matrix and the second inter-frame transformation matrix in the inter-frame transformation matrix sequence, and the inter-frame transformation matrix corresponding to the fifth mask image needs to be determined according to the first, second, third, fourth and fifth inter-frame transformation matrices in the inter-frame transformation matrix sequence. After determining the target transformation matrix, directly applying it to the object mask image can directly obtain the corresponding mask image. The mask image obtained by this method has higher clarity.

[0112] Step S700: determining target input data according to the source image, the target object image sequence and the mask image sequence.

[0113] The target input data is input data of a second video generation model, and the second video generation model is an image-generated video model or a video-generated video model with a local redrawing function.

[0114] It is worth noting that compared with the first video generation model, in addition to the input data format, the second video generation model also adds a local redrawing module to achieve the redrawing of the corresponding area of ​​the target object in the video.

[0115] In a possible implementation, the second video generation model is a picture-generated video model with a local redrawing function, and the corresponding input data should be in image format, so the source image, the target object image sequence and the mask image sequence can be directly determined as the target input data.

[0116] In a possible implementation, the second video generation model is a video generation model with a local redrawing function, and the corresponding input data should be in video format. Therefore, the source image, the target object image sequence and the mask image sequence need to be converted into corresponding video formats.

[0117] Figure 8 Flow chart of the target input data determination method according to an embodiment of the present invention. Figure 8 As shown, the target input data determination method includes the following steps:

[0118] Step S701: Generate multiple background motion video frames according to the source image and the mask image sequence.

[0119] Specifically, the source image is firstly copied and diffused, and then each mask image in the mask image sequence is used to process the source image, and pixels in corresponding areas of the source image are modified to obtain multiple background motion video frames.

[0120] Step S702: determining a corresponding background motion video according to the plurality of background motion video frames.

[0121] Step S703: determining a corresponding target object motion video according to the target object image sequence.

[0122] Step S704: determine the background motion video and the target object motion video as target input data.

[0123] Step S800: input the target input data into a second video generation model to obtain a corresponding target video.

[0124] In a possible implementation, the second video generation model is a picture-generated video model with a local redrawing function, and the processing process of the second video generation model is as follows: Figure 7 and Figure 8 shown.

[0125] Fig. 9 The figure is a flow chart of a target video generating method according to an embodiment of the present invention. Fig.10 FIG. 1 is a schematic diagram of the workflow of the second video generation model according to an embodiment of the present invention. Fig.10 The content shown is Fig. 9 The method is explained as follows: Fig. 9 As shown, the target video generation method includes the following steps:

[0126] Step S811, copying and diffusing the source image to obtain a plurality of background motion video frames.

[0127] For details on the implementation process, see Figure 2 The corresponding contents are shown and will not be repeated here.

[0128] Step S812: According to the mask image sequence, each target object image in the target object image sequence is used to replace the mask image in the corresponding background motion video frame to obtain a plurality of target video frames.

[0129] Specifically, the mask image sequence, multiple background motion video frames and the target object image sequence are matched. After the matching is completed, multiple groups of replacement data consisting of mask images, background motion video frames and target object images are obtained. For each group of replacement data, according to the mask image, the corresponding local area in the background motion video frame is replaced with the target object image to obtain the corresponding target video frame.

[0130] Step S813: Generate a corresponding target video according to the multiple target video frames.

[0131] In a possible implementation manner, the second video generation model is a video generation model with a local redrawing function, and the processing process of the second video generation model is as follows: Fig. 9 and Fig.10 shown.

[0132] Fig.11 The figure is a flow chart of a target video generating method according to an embodiment of the present invention. Fig.12 FIG. 1 is a schematic diagram of the workflow of the second video generation model according to an embodiment of the present invention. Fig.12 The content shown is Fig.11 The method is explained as follows: Fig.11 As shown, the target video generation method includes the following steps:

[0133] Step S821, determining a plurality of background motion video frames corresponding to the background motion video.

[0134] This step is to convert the input video into corresponding multiple video frames.

[0135] Step S822: determining a plurality of target object images corresponding to the target object motion video.

[0136] That is, the target object image sequence corresponding to the target object motion video is determined.

[0137] Step S823: using each target object image in the target object image sequence to replace the mask image in the corresponding background motion video frame to obtain a plurality of target video frames.

[0138] Step S824: Generate a corresponding target video according to the multiple target video frames.

[0139] Fig.11 The method shown is simply to parse the image into image frames, and the processing method after obtaining the image frames is the same as the above Fig. 9 The methods shown are similar and will not be repeated here.

[0140] The method of the embodiment of the present invention includes inputting a source image containing a target object into a first video generation model to obtain a material video, determining an inter-frame transformation matrix sequence according to the material video, and then obtaining an object mask image corresponding to the target object from the source image, applying the inter-frame transformation matrix sequence to the object mask image to obtain multiple mask images to form a mask image sequence, applying the inter-frame transformation matrix sequence to the source image to obtain multiple target object images to form a target object image sequence, converting the source image, the mask image sequence and the target object image sequence into target input data that meets the input requirements of the second video generation model, and inputting the target input data into a second video generation model that supports local redrawing to obtain the corresponding target video. Thus, the method of the embodiment of the present invention directly calculates the inter-frame transformation matrix sequence from the material video generated by the first video generation model to describe the motion trajectory of the target object, fully follows the motion trajectory generated by intelligent generation, and solves the problem that the preset motion trajectory of the target object is relatively single or the motion trajectory remains stationary. The method of the embodiment of the present method also uses the calculated inter-frame transformation matrix sequence to add a local drawing function in the video generation process of the second video generation model to keep the foreground area corresponding to the target object from spreading within a controllable range.

[0141] Fig.13 FIG. 1 is a flow chart of a method for generating video from an image according to an embodiment of the present invention. Fig.13 As shown, the image-to-video method comprises the following steps:

[0142] Step S1301, obtaining a source image.

[0143] Step S1302: Generate a material video using a first video generation model.

[0144] The material video includes a plurality of material video frames.

[0145] Step S1303, cut out each material video frame.

[0146] Step S1304, obtaining a foreground image.

[0147] Step S1305: determining, according to the image features of the foreground image, a sequence of inter-frame transformation matrices to be calculated according to the foreground features.

[0148] Step S1306, performing matting on the source image to obtain a mask image.

[0149] Step S1307: Apply the inter-frame transformation matrix to the object mask image to obtain a mask image sequence.

[0150] Step S1308: Apply the inter-frame transformation matrix to the source image to obtain a target object image sequence.

[0151] Step S1309: input the source image, the target object image sequence and the mask image sequence into a second video generation model to obtain a corresponding target video.

[0152] The detailed implementation process of the above steps S1301 to S1309 is similar to the corresponding process of the above embodiment, and will not be repeated here.

[0153] The method of the embodiment of the present invention includes inputting a source image containing a target object into a first video generation model to obtain a material video, determining an inter-frame transformation matrix sequence according to the material video, then obtaining an object mask image corresponding to the target object from the source image, applying the inter-frame transformation matrix sequence to the object mask image to obtain multiple mask images to form a mask image sequence, applying the inter-frame transformation matrix sequence to the source image to obtain multiple target object images to form a target object image sequence, converting the source image, the mask image sequence and the target object image sequence into target input data that meets the input requirements of the second video generation model, and inputting the target input data into a second video generation model that supports local redrawing to obtain the corresponding target video. Thus, the method of the embodiment of the present invention directly calculates the inter-frame transformation matrix sequence from the material video generated by the first video generation model to describe the motion trajectory of the target object, fully follows the motion trajectory generated by intelligent generation, and solves the problem that the preset motion trajectory of the target object is relatively single or the motion trajectory remains stationary. The method of the embodiment of the present method also uses the calculated inter-frame transformation matrix sequence to add a local drawing function in the video generation process of the second video generation model to keep the foreground area corresponding to the target object from spreading within a controllable range.

[0154] Fig.14 FIG. 1 is a schematic diagram of a device for generating video from an image according to an embodiment of the present invention. Fig.14 As shown, the image-generated video device comprises:

[0155] The acquisition module 1401 is used to acquire a source image, where the source image is an image containing a target object.

[0156] The first input module 1402 is used to input the source image into a first video generation model to obtain a corresponding material video, and the first video generation model is a pre-trained image-generated video model.

[0157] The first determination module 1403 is used to determine an inter-frame transformation matrix sequence corresponding to the material video.

[0158] The second determining module 1404 is configured to determine an object mask image corresponding to the target object in the source image.

[0159] The third determination module 1405 is used to determine a target object image sequence according to the inter-frame transformation matrix sequence and the source image.

[0160] The fourth determining module 1406 is configured to determine a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image.

[0161] The fifth determination module 1407 is configured to determine target input data according to the source image, the target object image sequence and the mask image sequence.

[0162] The second input module 1408 is used to input the target input data into a second video generation model to obtain a corresponding target video. The second video generation model is an image-generated video model or a video-generated video model with a local redrawing function.

[0163] The device of the embodiment of the present invention inputs the source image containing the target object into the first video generation model to obtain the material video, determines the inter-frame transformation matrix sequence according to the material video, and then obtains the object mask image corresponding to the target object from the source image, applies the inter-frame transformation matrix sequence to the object mask image to obtain multiple mask images to form a mask image sequence, applies the inter-frame transformation matrix sequence to the source image to obtain multiple target object images to form a target object image sequence, converts the source image, the mask image sequence and the target object image sequence into target input data that meets the input requirements of the second video generation model, and inputs the target input data into the second video generation model that supports local redrawing to obtain the corresponding target video. Thus, when the model graph is used to generate the video for the first time, the inter-frame transformation matrix can be determined according to the obtained video, and the motion trajectory of the target object can be described by using the inter-frame transformation matrix. When the model graph is used to generate the video for the second time, the local redrawing of the video frame can be controlled according to the inter-frame transformation matrix, so that the target object in the generated target video is clear and does not diffuse. By generating the video twice by the model, intelligent end-to-end graph generation video is realized, and the diversity of motion trajectories can be achieved while keeping the target object area from diffusing without introducing preset motion parameters.

[0164] Fig.15 Schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 150 includes a server, a terminal, etc. Fig.15 As shown, the electronic device 150: includes at least one processor 1501; and a memory 1502 communicatively connected to at least one processor 1501; and a communication component 1503 communicatively connected to a scanning device, the communication component 1503 receiving and sending data under the control of the processor 1501; wherein the memory 1502 stores instructions executable by at least one processor 1501, and the instructions are executed by at least one processor 1501 to implement the above-mentioned image-to-video method.

[0165] Specifically, the electronic device includes: one or more processors 1501 and a memory 1502, Fig.15A processor 1501 is taken as an example. The processor 1501 and the memory 1502 may be connected via a bus or other means. Fig.15 In the example, the bus connection is used. The memory 1502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1501 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1502, that is, realizing the above-mentioned image-to-video method.

[0166] The memory 1502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 1502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1502 may optionally include a memory remotely arranged relative to the processor 1501, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] One or more modules are stored in the memory 1502, and when executed by one or more processors 1501, the image-to-video method in any of the above method embodiments is executed.

[0168] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0169] The embodiment of the present invention obtains a material video by inputting a source image containing a target object into a first video generation model, determines an inter-frame transformation matrix sequence according to the material video, and then obtains an object mask image corresponding to the target object from the source image, applies the inter-frame transformation matrix sequence to the object mask image to obtain multiple mask images to form a mask image sequence, applies the inter-frame transformation matrix sequence to the source image to obtain multiple target object images to form a target object image sequence, converts the source image, the mask image sequence and the target object image sequence into target input data that meets the input requirements of the second video generation model, and inputs the target input data into a second video generation model that supports local redrawing to obtain the corresponding target video. Thus, the method of the embodiment of the present invention directly calculates the inter-frame transformation matrix sequence from the material video generated by the first video generation model to describe the motion trajectory of the target object, fully follows the motion trajectory generated by intelligent generation, and solves the problem that the preset target object motion trajectory is relatively single or the motion trajectory remains stationary. The method of the embodiment of the present method also uses the calculated inter-frame transformation matrix sequence to add a local drawing function in the video generation process of the second video generation model to keep the foreground area corresponding to the target object from spreading within a controllable range.

[0170] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0171] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0172] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating video from an image, characterized in that: The method comprises: Acquire a source image, where the source image is an image containing a target object; Inputting the source image into a first video generation model to obtain a corresponding material video, wherein the first video generation model is a pre-trained image-generated video model; Determine an inter-frame transformation matrix sequence corresponding to the material video; Determining an object mask image corresponding to the target object in the source image; Determine a target object image sequence according to the inter-frame transformation matrix sequence and the source image; Determining a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image; Determining target input data according to the source image, the target object image sequence and the mask image sequence; The target input data is input into a second video generation model to obtain a corresponding target video, wherein the second video generation model is an image-generated video model or a video-generated video model with a local redrawing function.

2. The method according to claim 1, characterized in that The material video includes a plurality of material video frames; The determining of the inter-frame transformation matrix sequence corresponding to the material video includes: Performing image segmentation on each of the material video frames to obtain a corresponding foreground image, wherein the foreground image is an image of the target object; Extracting image features of the foreground image; Determining an inter-frame transformation matrix between adjacent material video frames according to the image features; The inter-frame transformation matrix sequence is determined according to the inter-frame transformation matrices between adjacent material video frames.

3. The method according to claim 1, characterized in that Determining a target object image sequence according to the inter-frame transformation matrix sequence and the source image comprises: Applying a first inter-frame transformation matrix in the inter-frame transformation matrix sequence to the source image to determine a first target object image in the target object image sequence; Iteratively apply the i-th inter-frame transformation matrix in the inter-frame transformation matrix sequence to the last target object image of the current target object image sequence to determine the i-th target object image of the target object image sequence until all inter-frame transformation matrices in the inter-frame transformation matrix sequence are applied, and i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

4. The method according to claim 1, characterized in that: Determining a target object image sequence according to the inter-frame transformation matrix sequence and the source image comprises: Determine an i-th target transformation matrix according to the first i inter-frame transformation matrices in the inter-frame transformation matrix sequence; The i-th target transformation matrix is ​​applied to the source image to obtain the i-th target object image in the target object image sequence, where i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

5. The method according to claim 1, characterized in that Determining a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image comprises: Applying a first inter-frame transformation matrix in the inter-frame transformation matrix sequence to the object mask image to determine a first mask image in the mask image sequence; Iteratively applying the i-th inter-frame transformation matrix in the inter-frame transformation matrix sequence to the last mask image of the current mask image sequence, determining the i-th mask image of the mask image sequence, until all inter-frame transformation matrices in the inter-frame transformation matrix sequence are applied, i being a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

6. The method according to claim 1, characterized in that Determining a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image comprises: Determine an i-th target transformation matrix according to the first i inter-frame transformation matrices in the inter-frame transformation matrix sequence; The i-th target transformation matrix is ​​applied to the object mask image to obtain the i-th mask image in the mask image sequence, where i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

7. The method according to claim 1, characterized in that The second video generation model is a picture-generated video model with a local redrawing function; The determining target input data according to the source image, the target object image sequence and the mask image sequence comprises: The source image, the target object image sequence and the mask image sequence are determined as the target input data.

8. The method according to claim 1, characterized in that The second video generation model is a video generation model with a local redrawing function; The determining target input data according to the source image, the target object image sequence and the mask image sequence comprises: generating a plurality of background motion video frames according to the source image and the mask image sequence; Determining a corresponding background motion video according to the multiple background motion video frames; Determining a corresponding target object motion video according to the target object image sequence; The background motion video and the target object motion video are determined as target input data.

9. The method according to claim 8, characterized in that The step of generating a plurality of background motion video frames according to the source image and the mask image sequence comprises: Copying and diffusing the source image; A plurality of background motion video frames are generated according to the source image after the replication and diffusion processing and the mask image sequence.

10. The method according to claim 7, characterized in that After the target input data is input into the second video generation model, the second video generation model outputs the target video through the following steps: Copying and diffusing the source image to obtain a plurality of background motion video frames; According to the mask image sequence, using each target object image in the target object image sequence to replace the mask image in the corresponding background motion video frame to obtain a plurality of target video frames; A corresponding target video is generated according to the multiple target video frames.

11. The method according to claim 8, characterized in that After the target input data is input into the second video generation model, the second video generation model outputs the target video through the following steps: Determining a plurality of background motion video frames corresponding to the background motion video; Determine a plurality of target object images corresponding to the target object motion video; Using each target object image in the target object image sequence to replace the mask image in the corresponding background motion video frame to obtain a plurality of target video frames; A corresponding target video is generated according to the multiple target video frames.

12. A device for generating video from an image, characterized in that: The device comprises: An acquisition module, used for acquiring a source image, wherein the source image is an image containing a target object; A first input module, used for inputting the source image into a first video generation model to obtain a corresponding material video, wherein the first video generation model is a pre-trained image-generated video model; A first determination module is used to determine an inter-frame transformation matrix sequence corresponding to the material video; A second determination module, configured to determine an object mask image corresponding to the target object in the source image; A third determination module is used to determine a target object image sequence according to the inter-frame transformation matrix sequence and the source image; A fourth determination module, configured to determine a mask image sequence according to the inter-frame transformation matrix sequence and the object mask image; a fifth determination module, configured to determine target input data according to the source image, the target object image sequence and the mask image sequence; The second input module is used to input the target input data into a second video generation model to obtain a corresponding target video. The second video generation model is an image-generated video model or a video-generated video model with a local redrawing function.

13. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images