A graph-based video method and apparatus

Abstract

This invention discloses a method and apparatus for generating video from images. In this embodiment, a source image containing a target object is input into a first video generation model to obtain a source video. An inter-frame transform matrix sequence is determined based on the source video. Then, an object mask image corresponding to the target object is obtained from the source image. Applying the inter-frame transform matrix sequence to the object mask image yields multiple mask images, forming a mask image sequence. Applying the inter-frame transform matrix sequence to the source image yields multiple target object images, forming a target object image sequence. Target input data is determined based on the source image, the mask image sequence, and the target object image sequence. This target input data is then input into a second video generation model supporting local redrawing to obtain the corresponding target video. By generating video through two models, intelligent end-to-end image-to-video generation is achieved. It can achieve diverse motion trajectories while maintaining the target object's non-diffusing state without introducing preset motion parameters.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and apparatus for generating video from images. Background Technology

[0002] With the rapid development of artificial intelligence, video generation technology has received widespread attention and research. Currently used image-to-video models based on diffusion models may suffer from distortion of objects in the video when generating corresponding videos from images, resulting in the video failing to accurately reflect the true shape and details of the objects. Furthermore, image-to-video models typically require users to preset motion trajectories, making the user operation cumbersome. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for generating video from images, so as to realize intelligent end-to-end video from images, without the need to introduce preset motion parameters, and can achieve diversity of motion trajectories while keeping the target object area from spreading.

[0004] In a first aspect, embodiments of the present invention provide a method for generating video from images, the method comprising:

[0005] Obtain a source image, which is an image containing the target object;

[0006] The source image is input into the first video generation model to obtain the corresponding source video. The first video generation model is a pre-trained image-generated video model.

[0007] Determine the sequence of inter-frame transform matrices corresponding to the source video;

[0008] Determine the object mask image corresponding to the target object in the source image;

[0009] Based on the inter-frame transform matrix sequence and the source image, determine the target object image sequence;

[0010] Determine the mask image sequence based on the inter-frame transform matrix sequence and the object mask image;

[0011] The target input data is determined based on the source image, the target object image sequence, and the mask image sequence.

[0012] The target input data is input into the second video generation model to obtain the corresponding target video. The second video generation model is a graph-generated video model or a video-generated video model with local redrawing function.

[0013] Secondly, embodiments of the present invention provide an image-generated video device, the device comprising:

[0014] The acquisition module is used to acquire a source image, wherein the source image is an image containing the target object;

[0015] The first input module is used to input the source image into the first video generation model to obtain the corresponding source video. The first video generation model is a pre-trained image-generated video model.

[0016] The first determining module is used to determine the inter-frame transformation matrix sequence corresponding to the source video;

[0017] The second determining module is used to determine the object mask image corresponding to the target object in the source image;

[0018] The third determining module is used to determine the target object image sequence based on the inter-frame transformation matrix sequence and the source image;

[0019] The fourth determining module is used to determine a mask image sequence based on the inter-frame transform matrix sequence and the object mask image;

[0020] The fifth determining module is used to determine target input data based on the source image, the target object image sequence, and the mask image sequence;

[0021] The second input module is used to input the target input data into the second video generation model to obtain the corresponding target video. The second video generation model is a graph-generated video model or a video-generated video model with local redrawing function.

[0022] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0023] Fourthly, embodiments of the present invention provide an electronic device, the device comprising:

[0024] Memory is used to store one or more computer program instructions;

[0025] A processor, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0026] Fifthly, embodiments of the present invention provide a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0027] In this embodiment of the invention, a source image containing the target object is input into a first video generation model to obtain a source video. An inter-frame transformation matrix sequence is determined based on the source video. Then, an object mask image corresponding to the target object is obtained from the source image. Applying the inter-frame transformation matrix sequence to the object mask image yields multiple mask images, forming a mask image sequence. Applying the inter-frame transformation matrix sequence to the source image yields multiple target object images, forming a target object image sequence. The source image, mask image sequence, and target object image sequence are transformed into target input data that meets the input requirements of a second video generation model. This target input data is then input into a second video generation model that supports local redrawing to obtain the corresponding target video. Thus, the first use of the model to generate video allows for the determination of the inter-frame transformation matrix, which can be used to describe the motion trajectory of the target object. The second use of the model to generate video allows for control of local redrawing of video frames based on the inter-frame transformation matrix, ensuring that the target object in the generated target video is clear and does not diffuse. By generating video through two model generation processes, intelligent end-to-end image-to-video generation is achieved. Without introducing preset motion parameters, it can maintain the target object region from diffusion while achieving diverse motion trajectories. Attached Figure Description

[0028] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0029] Figure 1 This is a flowchart of the image-to-video method according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the workflow of the first video generation model according to an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of the method for determining the inter-frame transform matrix sequence according to an embodiment of the present invention;

[0032] Figure 4 This is a flowchart of the target object image sequence determination method according to an embodiment of the present invention;

[0033] Figure 5 This is a flowchart of the target object image sequence determination method according to an embodiment of the present invention;

[0034] Figure 6 This is a flowchart of the mask image sequence determination method according to an embodiment of the present invention;

[0035] Figure 7 This is a flowchart of the target object image sequence determination method according to an embodiment of the present invention;

[0036] Figure 8 This is a flowchart of the target input data determination method according to an embodiment of the present invention;

[0037] Figure 9 This is a flowchart of the target video generation method according to an embodiment of the present invention;

[0038] Figure 10 This is a schematic diagram of the workflow of the second video generation model according to an embodiment of the present invention;

[0039] Figure 11 This is a flowchart of the target video generation method according to an embodiment of the present invention;

[0040] Figure 12 This is a schematic diagram of the workflow of the second video generation model according to an embodiment of the present invention;

[0041] Figure 13 This is a flowchart of the image-to-video method according to an embodiment of the present invention;

[0042] Figure 14 This is a schematic diagram of the image-generated video device according to an embodiment of the present invention;

[0043] Figure 15 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0044] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0045] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0046] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0047] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0048] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0049] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0050] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0051] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0052] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0053] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0054] Figure 1 This is a flowchart of the image-to-video method according to an embodiment of the present invention. Figure 1 As shown, the image-to-video method includes the following steps:

[0055] Step S100: Obtain the source image.

[0056] The source image is an image containing the target object, which is the main subject highlighted in the source image, such as a product, animal, plant, person, or animated character.

[0057] Step S200: Input the source image into the first video generation model to obtain the corresponding source video.

[0058] The first video generation model is a pre-trained image-to-video (I2V) model. I2V models are used to convert static images into dynamic videos and are widely used in video production, advertising design, virtual reality, and other fields. The I2V model can be a generative model based on a diffusion model, such as stablediffusion or sora.

[0059] It is worth noting that different types of target objects have different motion trajectories and motion characteristics. For example, if the target object is a person's head, the motion of the person in the generated video may include facial expressions such as blinking, smiling, or crying. If the target object is a person's whole body, the motion of the person in the generated video may include body movements such as walking, spinning, or swinging arms. If the target object is a product, the motion of the product in the generated video may include changes in light and shadow projected onto the product, changes in perspective, or changes in color. Therefore, the image-generated video model in this embodiment needs to be trained according to the type of target object before use. The following explanation will use a product as an example.

[0060] The pre-training process of the first video generation model involves acquiring smooth, stable, and clear product display videos. Suitable video frames are selected from these videos as source images; a suitable video frame is defined as a video frame containing a clear image of the product. A sample dataset is constructed based on the product display videos and source images. This dataset is then used to train and fine-tune the image-generated video model, resulting in a first video generation model with vertical-specific characteristics. Vertical-specific characteristics refer to attributes and requirements unique to a specific industry or field. In the context of generating product display videos, these characteristics include lighting settings, detail display, product usage scenario display, and product performance parameters.

[0061] In one possible implementation, the length of the source video output by the first video generation model can be preset, for example, the length of the video output by the first video generation model can be preset to 2 seconds, 4 seconds, 8 seconds, etc.

[0062] Once the first video generation model is trained, it can be put into use. After receiving the source image, the first video generation model can generate multiple corresponding source video frames, and generate the corresponding source video based on the multiple source video frames. That is, the source video contains multiple source video frames.

[0063] Figure 2 This is a schematic diagram of the workflow of the first video generation model according to an embodiment of the present invention. Figure 2As shown, after the source image is input into the first video generation model, the encoding module in the first video generation model encodes it to obtain the corresponding encoding result, such as using a convolutional neural network to extract high-level features of the input source image. These features can capture important information in the image, such as edges, textures, and objects. Then, the encoding result is copied i times to obtain i copies of the encoding result, which is to generate intermediate frames. For example, a generative adversarial network or variational autoencoder can be used to generate intermediate frames. By learning the temporal evolution of the image, a series of intermediate frames are generated, so that the static image gradually transitions to dynamic video. Then, each intermediate frame is diffused to gradually add and remove noise to generate data, thereby improving video quality. Finally, the diffused video frames are decoded to obtain the source video.

[0064] Step S300: Determine the sequence of inter-frame transformation matrices corresponding to the source video.

[0065] The inter-frame transformation matrix sequence is a series of transformation matrices calculated between consecutive video frames to account for changes in objects or scenes. These transformations can include affine transformations, projective transformations, etc., used to capture relative motion or changes between two adjacent video frames. An affine transformation is a linear transformation that incorporates translation operations. It preserves the straightness and collinearity of points, lines, and line segments, but not distances and angles. Common affine transformations include scaling, rotation, translation, and shearing. A projective transformation (also known as perspective transformation) is a broader type of transformation that allows for perspective manipulation of images. It preserves the properties of straight lines, but not parallelism or scale. Applications of projective transformations include image correction and 3D projection.

[0066] Figure 3 This is a flowchart of the method for determining the inter-frame transform matrix sequence according to an embodiment of the present invention. Figure 3 As shown, the method for determining the inter-frame transform matrix sequence includes the following steps:

[0067] Step S301: Perform image segmentation on each of the aforementioned video frames to obtain the corresponding foreground image.

[0068] The source video frames contain a target object and a background other than the target object, and the foreground image is an image of the target object. By performing image matting on each source video frame, the object sub-image (i.e., the foreground image) and the background sub-image can be obtained.

[0069] Step S302: Extract the image features of the foreground image.

[0070] Calculating various types of transformations, such as affine or projective transformations, typically requires the use of image features, including Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Corner detection, Canny edge features, FAST (Features from Accelerated Segment Test) features, and feature matching pairs.

[0071] Step S303: Determine the inter-frame transformation matrix between adjacent video frames based on the image features.

[0072] Specifically, after determining the image features of the foreground image, feature matching needs to be performed between adjacent video frames to find the corresponding relationships. Commonly used feature matching methods include Brute-Force Matcher (BFMatcher) and Fast Library for Approximate Nearest Neighbors (FLANN).

[0073] After matching is complete, the transformation matrix between frames is calculated using the matched feature points. Commonly used methods for calculating the transformation matrix include homography, affine transformation, and rigid transformation. Homography is suitable for transformations in planar scenes, affine transformation is suitable for linear transformations, and rigid transformation is suitable for rigid body motion, including translation and rotation.

[0074] In one possible implementation, the mask image in the background motion video frame can also be smoothed.

[0075] In one possible implementation, the inter-frame transform matrix can also be smoothed.

[0076] Step S304: Determine the sequence of inter-frame transformation matrices based on the inter-frame transformation matrices between adjacent video frames.

[0077] Steps S200-S300 use the first video generation model to determine the source video, and then determine the inter-frame transformation matrix sequence based on the source video. The inter-frame transformation matrix sequence can replace the motion trajectory parameters that need to be preset by the user in the prior art, thereby reducing the difficulty of use and realizing intelligent generation of motion parameters.

[0078] Step S400: Determine the object mask image corresponding to the target object in the source image.

[0079] The object mask image is a binary image with the same or matching size as the object sub-image (i.e., the foreground image).

[0080] Step S500: Determine the target object image sequence based on the inter-frame transformation matrix sequence and the source image.

[0081] Specifically, the first inter-frame transform matrix in the inter-frame transform matrix sequence is applied to the source image to determine the first target object image of the target object image sequence. The i-th inter-frame transform matrix in the inter-frame transform matrix sequence is applied to the last target object image of the current target object image sequence in an iterative manner to determine the i-th target object image of the target object image sequence, until all inter-frame transform matrices in the inter-frame transform matrix sequence have been applied, where i is a positive integer not greater than the number of inter-frame transform matrices in the inter-frame transform matrix sequence.

[0082] Figure 4 This is a flowchart illustrating the method for determining a target object image sequence according to an embodiment of the present invention. Figure 4 As shown, the method for determining the target object image sequence includes the following steps:

[0083] Step S511: Apply the first inter-frame transform matrix in the inter-frame transform matrix sequence to the source image to determine the first target object image in the target object image sequence.

[0084] Step S512: Take the i-th inter-frame transform matrix from the inter-frame transform matrix sequence in order.

[0085] Among them, the i-th inter-frame transform matrix is ​​the inter-frame transform matrix that has not been applied.

[0086] Step S513: Apply the i-th inter-frame transformation matrix to the last target object image in the current target object image sequence to determine the corresponding target object image.

[0087] Step S514: Add the target object image to the end of the current target object image.

[0088] Step S515: Determine whether there are any unapplied inter-frame transform matrices in the inter-frame transform matrix sequence.

[0089] If it exists, proceed to step S512; if it does not exist, proceed to step S516.

[0090] Step S516: Determine the target object image sequence.

[0091] The above Figure 4 The method for determining the target object image sequence shown is to apply each inter-frame transformation matrix in the inter-frame transformation matrix sequence to the last target object image in the current target object image sequence in sequence, and then add the obtained target object images to the end of the current target object image sequence in sequence until all inter-frame transformation matrices have been applied.

[0092] Figure 5 This is a flowchart illustrating the method for determining a target object image sequence according to an embodiment of the present invention. Figure 5 As shown, the method for determining the target object image sequence includes the following steps:

[0093] Step S521: Determine the i-th target transformation matrix based on the first i inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0094] Step S522: Apply the i-th target transformation matrix to the source image to obtain the i-th target object image in the target object image sequence, where i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0095] The above Figure 5 The method for determining the target object image sequence shown involves determining the inter-frame transformation matrix corresponding to each target object image in the sequence based on the inter-frame transformation matrix sequence. Based on the first to the i-th inter-frame transformation matrices, the target transformation matrix required to directly transform the source image into the i-th target object image is calculated. For example, the inter-frame transformation matrix corresponding to the second target object image needs to be determined based on the first and second inter-frame transformation matrices in the sequence, and the inter-frame transformation matrix corresponding to the fifth target object image needs to be determined based on the first, second, third, fourth, and fifth inter-frame transformation matrices in the sequence. After determining the target transformation matrix, it can be directly applied to the source image to obtain the corresponding target object image. This method yields target object images with higher clarity.

[0096] Step S600: Determine the mask image sequence based on the inter-frame transform matrix sequence and the object mask image.

[0097] Specifically, the first inter-frame transform matrix in the inter-frame transform matrix sequence is applied to the object mask image to determine the first mask image of the mask image sequence. The i-th inter-frame transform matrix in the inter-frame transform matrix sequence is applied to the last mask image of the current mask image sequence in an iterative manner to determine the i-th mask image of the mask image sequence, until all inter-frame transform matrices in the inter-frame transform matrix sequence have been applied, where i is a positive integer not greater than the number of inter-frame transform matrices in the inter-frame transform matrix sequence.

[0098] Figure 6 This is a flowchart illustrating the mask image sequence determination method according to an embodiment of the present invention. Figure 6 As shown, the mask image sequence determination method includes the following steps:

[0099] Step S611: Apply the first inter-frame transform matrix in the inter-frame transform matrix sequence to the object mask image to determine the first mask image in the mask image sequence.

[0100] Step S612: Take the i-th inter-frame transform matrix from the inter-frame transform matrix sequence in order.

[0101] Among them, the i-th inter-frame transform matrix is ​​the inter-frame transform matrix that has not been applied.

[0102] Step S613: Apply the i-th inter-frame transform matrix to the last mask image of the current mask image sequence to determine the corresponding mask image.

[0103] Step S614: Add the mask image to the end of the current mask image.

[0104] Step S615: Determine whether there are any unapplied inter-frame transform matrices in the inter-frame transform matrix sequence.

[0105] If it exists, proceed to step S612; if it does not exist, proceed to step S616.

[0106] Step S616: Determine the mask image sequence.

[0107] The above Figure 6 The method for determining the mask image sequence shown is to apply each inter-frame transform matrix in the inter-frame transform matrix sequence to the last mask image of the current mask image sequence in turn, and then add the newly obtained mask images to the end of the current mask image sequence in order, until all inter-frame transform matrices have been applied.

[0108] Figure 7 This is a flowchart illustrating the mask image sequence determination method according to an embodiment of the present invention. Figure 7 As shown, the mask image sequence determination method includes the following steps:

[0109] Step S621: Determine the i-th target transformation matrix based on the first i inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0110] Step S622: Apply the i-th target transformation matrix to the object mask image to obtain the i-th mask image in the mask image sequence, where i is a positive integer not greater than the number of inter-frame transformation matrices in the inter-frame transformation matrix sequence.

[0111] The above Figure 7 The method for determining the mask image sequence shown involves determining the inter-frame transformation matrix corresponding to each mask image in the sequence based on the inter-frame transformation matrix sequence. Based on the first to the ith inter-frame transformation matrices, the target transformation matrix required to directly transform the object mask image into the ith mask image is calculated. For example, the inter-frame transformation matrix corresponding to the second mask image needs to be determined based on the first and second inter-frame transformation matrices in the sequence, and the inter-frame transformation matrix corresponding to the fifth mask image needs to be determined based on the first, second, third, fourth, and fifth inter-frame transformation matrices in the sequence. After determining the target transformation matrix, it can be directly applied to the object mask image to obtain the corresponding mask image. This method yields mask images with higher clarity.

[0112] Step S700: Determine target input data based on the source image, the target object image sequence, and the mask image sequence.

[0113] The target input data is the input data of the second video generation model, which is a graph-generated video model or a video-generated video model with local redrawing function.

[0114] It is worth noting that, compared with the first video generation model, in addition to the input data format, the second video generation model also adds a local redrawing module to redraw the corresponding area of ​​the target object in the video.

[0115] In one possible implementation, the second video generation model is a graph-generated video model with local redrawing capabilities. The corresponding input data should be in image format. Therefore, the source image, the target object image sequence, and the mask image sequence can be directly determined as the target input data.

[0116] In one possible implementation, the second video generation model is a video generation model with local redrawing capabilities, in which case the corresponding input data should be in video format. Therefore, it is necessary to convert the source image, the target object image sequence, and the mask image sequence into the corresponding video format.

[0117] Figure 8 This is a flowchart of a method for determining target input data according to an embodiment of the present invention. Figure 8 As shown, the target input data determination method includes the following steps:

[0118] Step S701: Generate multiple background motion video frames based on the source image and the mask image sequence.

[0119] Specifically, the source image is first copied and diffused, and then each mask image in the mask image sequence is used to process the source image. By modifying the pixels in the corresponding areas of the source image, multiple background motion video frames can be obtained.

[0120] Step S702: Determine the corresponding background motion video based on the plurality of background motion video frames.

[0121] Step S703: Determine the corresponding target object motion video based on the target object image sequence.

[0122] Step S704: The background motion video and the target object motion video are determined as target input data.

[0123] Step S800: Input the target input data into the second video generation model to obtain the corresponding target video.

[0124] In one possible implementation, the second video generation model is a graph-based video model with local redrawing capabilities. The processing procedure of the second video generation model is as follows: Figure 7 and Figure 8 As shown.

[0125] Figure 9 This is a flowchart of the target video generation method according to an embodiment of the present invention. Figure 10 This is a schematic diagram of the workflow of the second video generation model according to an embodiment of the present invention. The following is in conjunction with... Figure 10 The content shown is for Figure 9 The method will be explained and illustrated as follows: Figure 9 As shown, the target video generation method includes the following steps:

[0126] Step S811: The source image is copied and diffused to obtain multiple background motion video frames.

[0127] For details on the implementation process, please refer to [link / document / etc.]. Figure 2 The corresponding content is shown below, and will not be repeated here.

[0128] Step S812: Based on the mask image sequence, replace the mask image in the corresponding background motion video frame with each target object image in the target object image sequence to obtain multiple target video frames.

[0129] Specifically, the mask image sequence, multiple background motion video frames, and target object image sequence are matched. After the matching is completed, multiple sets of replacement data consisting of mask images, background motion video frames, and target object images are obtained. For each set of replacement data, the corresponding local area in the background motion video frame is replaced with the target object image according to the mask image to obtain the corresponding target video frame.

[0130] Step S813: Generate a corresponding target video based on the plurality of target video frames.

[0131] In one possible implementation, the second video generation model is a video generation model with local redrawing capabilities. The processing procedure of the second video generation model is as follows: Figure 9 and Figure 10 As shown.

[0132] Figure 11 This is a flowchart of the target video generation method according to an embodiment of the present invention. Figure 12 This is a schematic diagram of the workflow of the second video generation model according to an embodiment of the present invention. The following is in conjunction with... Figure 12 The content shown is for Figure 11 The method will be explained and illustrated as follows: Figure 11 As shown, the target video generation method includes the following steps:

[0133] Step S821: Determine multiple background motion video frames corresponding to the background motion video.

[0134] This step converts the input video into multiple corresponding video frames.

[0135] Step S822: Determine multiple target object images corresponding to the target object motion video.

[0136] That is, determining the target object image sequence corresponding to the target object motion video.

[0137] Step S823: Use each target object image in the target object image sequence to replace the mask image in the corresponding background motion video frame to obtain multiple target video frames.

[0138] Step S824: Generate a corresponding target video based on the plurality of target video frames.

[0139] Figure 11 The method shown can be simply described as parsing an image into image frames, and the processing methods after obtaining the image frames are the same as those described above. Figure 9 The methods shown are similar and will not be repeated here.

[0140] The method of this invention includes inputting a source image containing a target object into a first video generation model to obtain a source video; determining an inter-frame transformation matrix sequence based on the source video; obtaining an object mask image corresponding to the target object from the source image; applying the inter-frame transformation matrix sequence to the object mask image to obtain multiple mask images to form a mask image sequence; applying the inter-frame transformation matrix sequence to the source image to obtain multiple target object images to form a target object image sequence; converting the source image, mask image sequence, and target object image sequence into target input data that meets the input requirements of a second video generation model; and inputting the target input data into a second video generation model that supports local redrawing to obtain the corresponding target video. Therefore, the method of this invention directly calculates the inter-frame transformation matrix sequence from the source video generated by the first video generation model to describe the motion trajectory of the target object, fully following the intelligently generated motion trajectory, solving the problem that the preset target object motion trajectory is relatively simple or remains stationary. This method also utilizes the calculated inter-frame transformation matrix sequence to add a local drawing function during the video generation process of the second video generation model, keeping the foreground area corresponding to the target object within a controllable range and preventing diffusion.

[0141] Figure 13 This is a flowchart of the image-to-video method according to an embodiment of the present invention. Figure 13 As shown, the image-to-video method includes the following steps:

[0142] Step S1301: Obtain the source image.

[0143] Step S1302: Generate source video using the first video generation model.

[0144] The source video includes multiple source video frames.

[0145] Step S1303: Perform image cutout on each video frame.

[0146] Step S1304: Obtain the foreground image.

[0147] Step S1305: Determine the sequence of inter-frame transformation matrices based on the image features of the foreground image.

[0148] Step S1306: Perform image masking on the source image to obtain a mask image.

[0149] Step S1307: Apply the inter-frame transformation matrix to the object mask image to obtain a mask image sequence.

[0150] Step S1308: Apply the inter-frame transformation matrix to the source image to obtain the target object image sequence.

[0151] Step S1309: Input the source image, the target object image sequence, and the mask image sequence into the second video generation model to obtain the corresponding target video.

[0152] The detailed implementation process of steps S1301-S1309 is similar to the corresponding process in the above embodiments, and will not be repeated here.

[0153] The method of this invention includes inputting a source image containing a target object into a first video generation model to obtain a source video; determining an inter-frame transformation matrix sequence based on the source video; obtaining an object mask image corresponding to the target object from the source image; applying the inter-frame transformation matrix sequence to the object mask image to obtain multiple mask images to form a mask image sequence; applying the inter-frame transformation matrix sequence to the source image to obtain multiple target object images to form a target object image sequence; converting the source image, mask image sequence, and target object image sequence into target input data that meets the input requirements of a second video generation model; and inputting the target input data into a second video generation model that supports local redrawing to obtain the corresponding target video. Therefore, the method of this invention directly calculates the inter-frame transformation matrix sequence from the source video generated by the first video generation model to describe the motion trajectory of the target object, fully following the intelligently generated motion trajectory, solving the problem that the preset target object motion trajectory is relatively simple or remains stationary. This method also utilizes the calculated inter-frame transformation matrix sequence to add a local drawing function during the video generation process of the second video generation model, keeping the foreground area corresponding to the target object within a controllable range and preventing diffusion.

[0154] Figure 14 This is a schematic diagram of a video generation device according to an embodiment of the present invention. Figure 14 As shown, the image-generating video device includes:

[0155] The acquisition module 1401 is used to acquire a source image, wherein the source image is an image containing the target object.

[0156] The first input module 1402 is used to input the source image into the first video generation model to obtain the corresponding source video. The first video generation model is a pre-trained image-generated video model.

[0157] The first determining module 1403 is used to determine the inter-frame transformation matrix sequence corresponding to the source video.

[0158] The second determining module 1404 is used to determine the object mask image corresponding to the target object in the source image.

[0159] The third determining module 1405 is used to determine the target object image sequence based on the inter-frame transformation matrix sequence and the source image.

[0160] The fourth determining module 1406 is used to determine a mask image sequence based on the inter-frame transformation matrix sequence and the object mask image.

[0161] The fifth determining module 1407 is used to determine target input data based on the source image, the target object image sequence, and the mask image sequence.

[0162] The second input module 1408 is used to input the target input data into the second video generation model to obtain the corresponding target video. The second video generation model is a graph-generated video model or a video-generated video model with local redrawing function.

[0163] The apparatus of this invention inputs a source image containing a target object into a first video generation model to obtain a source video. Based on the source video, it determines an inter-frame transformation matrix sequence. Then, it obtains an object mask image corresponding to the target object from the source image. Applying the inter-frame transformation matrix sequence to the object mask image yields multiple mask images, forming a mask image sequence. Applying the inter-frame transformation matrix sequence to the source image yields multiple target object images, forming a target object image sequence. The source image, mask image sequence, and target object image sequence are transformed into target input data that meets the input requirements of a second video generation model. This target input data is then input into a second video generation model that supports local redrawing to obtain the corresponding target video. Thus, the first use of the model to generate video allows for the determination of the inter-frame transformation matrix, which can be used to describe the motion trajectory of the target object. The second use of the model to generate video allows for the control of local redrawing of video frames based on the inter-frame transformation matrix, ensuring that the target object in the generated target video is clear and does not diffuse. By generating video through two model generation processes, intelligent end-to-end image-to-video generation is achieved. Without introducing preset motion parameters, it can maintain the target object area from diffusion while achieving diverse motion trajectories.

[0164] Figure 15 This is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 150 includes a server, a terminal, etc. Figure 15 As shown, the electronic device 150 includes at least one processor 1501; a memory 1502 communicatively connected to at least one processor 1501; and a communication component 1503 communicatively connected to a scanning device, the communication component 1503 receiving and transmitting data under the control of the processor 1501; wherein the memory 1502 stores instructions executable by at least one processor 1501, the instructions being executed by at least one processor 1501 to implement the above-described image-to-video method.

[0165] Specifically, the electronic device includes: one or more processors 1501 and a memory 1502. Figure 15Taking a processor 1501 as an example, the processor 1501 and the memory 1502 can be connected via a bus or other means. Figure 15 Taking a bus connection as an example, memory 1502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 1501 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 1502, thereby realizing the above-mentioned image-to-video method.

[0166] Memory 1502 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 1502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 1502 may optionally include memory remotely located relative to processor 1501, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] One or more modules are stored in memory 1502 and, when executed by one or more processors 1501, perform the image-generated video method in any of the above method embodiments.

[0168] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0169] This invention embodiment obtains source video by inputting a source image containing the target object into a first video generation model. Based on the source video, an inter-frame transformation matrix sequence is determined. Then, an object mask image corresponding to the target object is obtained from the source image. Applying the inter-frame transformation matrix sequence to the object mask image yields multiple mask images, forming a mask image sequence. Applying the inter-frame transformation matrix sequence to the source image yields multiple target object images, forming a target object image sequence. The source image, mask image sequence, and target object image sequence are transformed into target input data that meets the input requirements of a second video generation model. This target input data is then input into a second video generation model that supports local redrawing to obtain the corresponding target video. Therefore, the method of this invention embodiment directly calculates the inter-frame transformation matrix sequence from the source video generated by the first video generation model to describe the motion trajectory of the target object, fully adhering to the intelligently generated motion trajectory. This solves the problem of the preset target object motion trajectory being too simple or remaining stationary. Furthermore, this method embodiment utilizes the calculated inter-frame transformation matrix sequence to add a local drawing function during the video generation process of the second video generation model, ensuring that the foreground area corresponding to the target object remains within a controllable range and does not spread.

[0170] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0171] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0172] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A graph-based video method, characterized by, The method comprises: acquiring a source image, the source image being an image containing a target object; inputting the source image into a first video generation model to obtain a corresponding material video, the first video generation model being a pre-trained graph video model; determining a sequence of inter-frame transformation matrices corresponding to the material video; determining an object mask image corresponding to the target object in the source image; determining a target object image sequence according to the sequence of inter-frame transformation matrices and the source image; determining a mask image sequence according to the sequence of inter-frame transformation matrices and the object mask image; determining target input data according to the source image, the target object image sequence and the mask image sequence; inputting the target input data into a second video generation model to obtain a corresponding target video, the second video generation model being a graph video model with a local redrawing function or a video video model.

2. The method of claim 1, wherein, The material video comprises a plurality of material video frames; The determination of the sequence of inter-frame transformation matrices corresponding to the material video comprises: performing image segmentation on each of the material video frames to obtain a corresponding foreground image, the foreground image being an image of the target object; extracting image features of the foreground image; determining inter-frame transformation matrices between adjacent material video frames according to the image features; determining the sequence of inter-frame transformation matrices according to the inter-frame transformation matrices between adjacent material video frames.

3. The method of claim 1, wherein, The determination of the target object image sequence according to the sequence of inter-frame transformation matrices and the source image comprises: applying a first inter-frame transformation matrix in the sequence of inter-frame transformation matrices to the source image to determine a first target object image in the target object image sequence; applying an i-th inter-frame transformation matrix in the sequence of inter-frame transformation matrices to a last target object image in a current target object image sequence in an iterative manner to determine an i-th target object image in the target object image sequence until all inter-frame transformation matrices in the sequence of inter-frame transformation matrices are applied, i being a positive integer not greater than the number of inter-frame transformation matrices in the sequence of inter-frame transformation matrices.

4. The method of claim 1, wherein, The determination of the target object image sequence according to the sequence of inter-frame transformation matrices and the source image comprises: determining an i-th target transformation matrix according to the first i inter-frame transformation matrices in the sequence of inter-frame transformation matrices; applying the i-th target transformation matrix to the source image to obtain an i-th target object image in the target object image sequence, i being a positive integer not greater than the number of inter-frame transformation matrices in the sequence of inter-frame transformation matrices.

5. The method of claim 1, wherein, The determination of the mask image sequence according to the sequence of inter-frame transformation matrices and the object mask image comprises: applying a first inter-frame transformation matrix in the sequence of inter-frame transformation matrices to the object mask image to determine a first mask image in the mask image sequence; apply the i-th inter-frame transformation matrix in the sequence of inter-frame transformation matrices to a last mask image in the sequence of current mask images in an iterative manner to determine an i-th mask image in the sequence of mask images until all the inter-frame transformation matrices in the sequence of inter-frame transformation matrices are applied, i being a positive integer not greater than a number of inter-frame transformation matrices in the sequence of inter-frame transformation matrices.

6. The method of claim 1, wherein, The determining the sequence of mask images according to the sequence of inter-frame transformation matrices and the object mask image comprises: determining an i-th target transformation matrix according to the first i inter-frame transformation matrices in the sequence of inter-frame transformation matrices; applying the i-th target transformation matrix to the object mask image to obtain an i-th mask image in the sequence of mask images, i being a positive integer not greater than a number of inter-frame transformation matrices in the sequence of inter-frame transformation matrices.

7. The method of claim 1, wherein, The second video generation model is a graph-to-video model with a local redraw function. The determining the target input data according to the source image, the sequence of target object images and the sequence of mask images comprises: determining the source image, the sequence of target object images and the sequence of mask images as the target input data.

8. The method of claim 1, wherein, The second video generation model is a video-to-video model with a local redraw function. The determining the target input data according to the source image, the sequence of target object images and the sequence of mask images comprises: generating a plurality of background motion video frames according to the source image and the sequence of mask images; determining a corresponding background motion video according to the plurality of background motion video frames; determining a corresponding target object motion video according to the sequence of target object images; determining the background motion video and the target object motion video as the target input data.

9. The method of claim 8, wherein, The generating a plurality of background motion video frames according to the source image and the sequence of mask images comprises: performing copy and diffusion processing on the source image; generating a plurality of background motion video frames according to the source image after being processed by copying and diffusion and the sequence of mask images.

10. The method of claim 7, wherein, After the target input data is input into the second video generation model, the second video generation model outputs the target video by the following steps: performing copy and diffusion processing on the source image to obtain a plurality of background motion video frames; replacing mask images in corresponding background motion video frames with each target object image in the sequence of target object images according to the sequence of mask images to obtain a plurality of target video frames; generating a corresponding target video according to the plurality of target video frames.

11. The method of claim 8, wherein, After the target input data is input into the second video generation model, the second video generation model outputs the target video by the following steps: determining a plurality of background motion video frames corresponding to the background motion video; determining a plurality of target object images corresponding to the target object motion video; replacing mask images in corresponding background motion video frames with each target object image in the sequence of target object images to obtain a plurality of target video frames; generating a corresponding target video according to the plurality of target video frames.

12. A graph-based video device, comprising: The apparatus comprises: an acquisition module configured to acquire a source image, the source image being an image containing a target object; The first input module is configured to input the source image into a first video generation model to obtain a corresponding material video, the first video generation model being a pre-trained graph video model. The first determination module is configured to determine a sequence of inter-frame transformation matrices corresponding to the material video. The second determination module is configured to determine an object mask image corresponding to the target object in the source image. The third determination module is configured to determine a sequence of target object images according to the sequence of inter-frame transformation matrices and the source image. The fourth determination module is configured to determine a sequence of mask images according to the sequence of inter-frame transformation matrices and the object mask image. The fifth determination module is configured to determine target input data according to the source image, the sequence of target object images, and the sequence of mask images. The second input module is configured to input the target input data into a second video generation model to obtain a corresponding target video, the second video generation model being a graph video model with a local redrawing function or a video video model.

13. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method according to any one of claims 1-11.

15. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the method according to any one of claims 1-11.

Images