Custom area dynamic video generation method and device, equipment, medium and product

Abstract

The invention provides a user-defined area dynamic video generation method and device, equipment, a medium and a product, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a text used for video generation content description and a picture used as a video background; generating a mask picture in response to a user-defined area selected by a user in the picture; inputting a text, a picture and a mask picture to the trained video generation model to obtain a target video; the video generation model is obtained by embedding a time sequence alignment module and / or an image control module in a preset diffusion model, the time sequence alignment module is used for determining the time sequence of each frame of image in the target video, and the condition control module is at least used for determining a dynamic region of the target video according to the mask picture. The video generation quality and content are optimized by adding the time sequence alignment module and the condition control module, dynamic generation of the designated area is achieved through the mask operation, and the problems that in the prior art, dynamic area selection cannot be carried out, and the video content is difficult to control are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and product for generating dynamic video of a custom area. Background Art

[0002] As the application of artificial intelligence in life increases, it provides users with more entertaining, intelligent and convenient options. Creative functions such as text generation and image generation have emerged, driving people's exploration of video generation. With the emergence of diffusion models, pre-training generation and other methods, the demand for high-quality and personalized video generation is increasing.

[0003] Current video generation methods rely on traditional text-to-image models, which generate several images from the same text and then splice them together. However, due to the lack of action continuity in the generation process of each image, all images are generated by random denoising of the model, which leads to problems such as the inability to select dynamic areas and difficulty in controlling video content. Summary of the invention

[0004] The present application provides a method, device, equipment, medium and product for generating dynamic video of a custom area, so as to solve the defects in the prior art that dynamic area selection cannot be performed and video content is difficult to control.

[0005] The present application provides a method for generating dynamic video of a custom area, comprising: Get the text used to describe the video content and the picture used as the video background; generating a mask image in response to a user-defined area selected in the image; Inputting the text, the image and the mask image into a pre-trained video generation model to obtain a target video output by the video generation model; Among them, the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame image in the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask image.

[0006] As an embodiment, the video generation model includes a 3D encoder, a noise adding module, a condition control module, a UNet module, a temporal alignment model and a 3D decoder; The 3D encoder is used to output a latent vector according to the picture; The noise adding module is used to add preset noise or user input noise to the latent vector to obtain a noisy latent vector; The condition control module is used to splice the noise latent vector and the mask image, and encode the text based on a preset text encoder; The UNet module is used to obtain a latent vector for completing text control and image control according to the spliced ​​noise latent vector and the mask image and the encoded text; The temporal alignment model is used to perform temporal alignment on latent vectors that complete text control and image control based on a self-attention mechanism to obtain a target latent vector; The 3D decoder is used to generate the target video according to the target latent vector.

[0007] As an embodiment, the training step of the video generation model includes: Obtain a public video data set, and perform a mask operation on the first frame image of the video in the public video data set to obtain a training set; Freezing parameters of the 3D encoder, the denoising module, the UNet module, and the 3D decoder, and determining a loss function of the video generation model based on minimizing negative log-likelihood; The video generation model is trained based on the training set, and the parameters of the conditional control module and / or the parameters of the timing alignment model are updated. The training is completed when a preset number of iterations is reached or the loss function reaches a preset threshold.

[0008] As an embodiment, the reasoning step of the video generation model includes: Get inference text and inference pictures; In response to a user-defined area selected in the inference image by a user, generating an inference mask image; The inference text, the inference picture, the inference mask picture and preset random noise are input into the trained video generation model to obtain the inference video output by the video generation model.

[0009] As an embodiment, generating a mask image in response to a user-defined area selected in the image by a user includes: Providing a brush tool or an image segmentation tool to the user based on a preset user interaction interface; In response to a smearing instruction issued by the user based on the brush tool, the area corresponding to the smearing instruction is used as a custom area to generate a mask image; or,

[0010] In response to a click instruction issued by a user, a mask image is generated based on the image segmentation tool using an area corresponding to the click instruction as a custom area.

[0011] As an embodiment, after generating the mask image in response to the user-defined area selected in the image by the user, the method further includes: Acquiring a custom parameter input by a user based on the user interaction interface; Correspondingly, the inputting the text, the image, and the mask image into a pre-trained video generation model to obtain a target video output by the video generation model includes: The text, the image, the mask image and the custom parameters are input into a pre-trained video generation model to obtain a target video output by the video generation model.

[0012] The present application also provides a device for generating dynamic video of a custom area, comprising: An acquisition module, used to acquire text used to describe the video content and a picture used as a video background; A first generating module, configured to generate a mask image in response to a user-defined area selected in the image by a user; A second generation module is used to input the text, the picture and the mask picture into a pre-trained video generation model to obtain a target video output by the video generation model; Among them, the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame image in the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask image.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-described methods for generating dynamic video in a custom area is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for generating dynamic video of a custom area as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method for generating dynamic video of a custom area as described above is implemented.

[0016] The custom area dynamic video generation method, device, equipment, medium and product provided in the present application optimize the video generation quality and content by adding a timing alignment module and a conditional control module, and realize the dynamic generation of the specified area through mask operation, thereby solving the problem that the traditional video generation model cannot perform dynamic area selection and the video content is difficult to control. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the method for generating dynamic video in a custom area provided in this application.

[0019] Figure 2 It is a structural diagram of the video generation model provided in this application.

[0020] Figure 3 It is a structural diagram of the conditional control module, UNet module, and timing alignment model provided in this application.

[0021] Figure 4 It is a schematic diagram of the reasoning process of the video generation model provided in this application.

[0022] Figure 5 This is the second flow chart of the method for generating dynamic video in a custom area provided in this application.

[0023] Figure 6a This is a schematic diagram of selecting a custom area based on a brush tool provided by this application. Figure 6b This is a schematic diagram of selecting a custom area based on an image segmentation tool provided by this application.

[0024] Figure 7 It is a structural schematic diagram of the custom area dynamic video generation device provided in this application.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0028] At present, the video generation method relies on the traditional text-image model, which generates several images from the same text and splices them together. Due to the lack of action continuity in the generation process of each image, all images are randomly denoised by the model, so they cannot have the authenticity and watchability of real videos. In addition, there are also additional auxiliary videos to achieve high quality of generated videos, but the input conditions required by such methods are harsh. The model needs to process video files and image files at the same time, which consumes a lot of resources and occupies a lot of video memory, which is not conducive to service deployment. With the development of large language models, some scholars have also used large language models to achieve video generation, but this method requires pre-training of models with a huge number of parameters in advance, and its inference parameters are large and the generation speed is slow.

[0029] Existing video generation methods have the following problems: dynamic area selection cannot be performed in traditional video generation models, the generated video quality is poor, and the video content is difficult to control; the large number of component modules and complex interfaces result in slow calling speed and low efficiency; the video generation process has poor flexibility and interactivity, and its application is easily restricted.

[0030] In order to solve at least one of the above technical problems, the present application provides a method, device, equipment, medium and product for generating dynamic video of a custom area. The present application is described in detail below in conjunction with the accompanying drawings.

[0031] Figure 1 This is one of the flow charts of the method for generating dynamic video in a custom area provided by this application, such as Figure 1 As shown, the present application provides a method for generating dynamic video of a custom area, including steps S100 to S300.

[0032] Step S100, obtain text for describing the video content generation and a picture used as the video background; if the picture is an image containing a basketball and a backboard, the text may be "basketball touches the backboard"; if the picture is an image containing a person, an animal, or a cartoon character, the text may be "shaking head" or "shaking head left and right", etc., that is, the text is used to specify a target in the picture to perform a corresponding action.

[0033] Step S200, in response to the user's selected custom area in the image, a mask image is generated; the present application adds image condition control to the traditional text condition control module, and the user can customize the area selection based on the uploaded image, and the selected area is input into the video generation model as a mask image to realize dynamic video generation of the specified area, while other areas are different from the original area. Figure 1 It increases the controllability of the model and is applicable to multiple scenarios.

[0034] Step S300, input the text, the image and the mask image into a pre-trained video generation model to obtain a target video output by the video generation model.

[0035] Among them, the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame image in the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask image.

[0036] The diffusion model can be a traditional Stable Diffusion model or Imagen model or Latent Diffusion Models (LDM) model or Craiyon model, etc. The embodiment of the present application takes the Stable Diffusion model as an example for explanation. The present application takes the Stable Diffusion model as the main model, and embeds the timing alignment module in the Stable Diffusion model, so that the video generation model can generate a video with coherent actions, and realizes dynamic video generation of a specified area through the image control module, thereby increasing the controllability of the model and being applicable to a variety of scenarios. It should be noted that the timing alignment module can be freely combined with a variety of Vincent graph models (i.e., diffusion models) to increase flexibility and applicability.

[0037] It can be understood that the present application optimizes the video generation quality and content by adding a timing alignment module and a conditional control module, and realizes the dynamic generation of specified areas through mask operations, thereby solving the problems of the inability to perform dynamic area selection and the difficulty in controlling video content in traditional video generation models.

[0038] Figure 2 is a structural diagram of the video generation model provided by this application, Figure 3 It is a structural diagram of the conditional control module, UNet module, and timing alignment model provided by this application, such as Figure 2 and Figure 3 As shown, based on the above embodiment, as an optional embodiment, the video generation model includes a 3D encoder, a noise addition module, a condition control module, a UNet module, a timing alignment model and a 3D decoder.

[0039] The 3D encoder is used to output a latent vector according to the picture; the picture can be one or more pictures, and when inputting the picture into the video generation model or before inputting the picture into the video generation model, the video parameters need to be input into the video generation model, and the video parameters are identified as ,in, is the BatchSize size, is the video height, is the video width, is the number of video frames, is the number of video channels. The 3D encoder generates latent vectors based on video parameters and pictures. , In the embodiment of the present application, Both are set to 4.

[0040] Among them, the 3D encoder is pre-trained on the UCF101 dataset, which is a standard dataset widely used for video classification and action recognition.

[0041] The noise adding module is used to add preset noise or user input noise to the latent vector to obtain a noisy latent vector; the preset noise refers to the random noise generated by the video generation model according to a preset distribution, and the user input noise refers to the random noise generated according to the user input instruction.

[0042] Adding preset noise or user input noise to the latent vector means continuously diffusing the latent vector. The principle is to use the Markov chain to generate a constant number of noises at each time step. Gradually add noise to the data To obtain the posterior probability ,in … The data representing the input is also the latent vector. T is the number of time steps and is also the number of times noise is added.

[0043] For a given latent vector , the diffusion process passes through Add Gaussian noise to it cumulatively and get the total Noise latent vectors, each noise addition process is composed of a series of hyperparameters with Gaussian distribution variances Control, the latent vector generated after adding noise is only related to the input at the previous moment, so it can be regarded as a Markov process. The specific formula is as follows: .

[0044] in, Indicates the state at a given previous moment Under the condition of The probability distribution of is a Gaussian distribution. represents a Gaussian distribution (normal distribution), where: is the state variable at the current moment, is the mean of this Gaussian distribution, which depends on the state of the previous moment , is the covariance matrix of this Gaussian distribution, where is the identity matrix, indicating that the variables are independent of each other and their variance is It can be seen that the diffusion process of the latent vector is as increase, The closer to pure noise, the It becomes a complete Gaussian noise. After the noise addition process is completed, the latent vector approximate to pure noise can be obtained. .

[0045] The conditional control module is used to splice the noise latent vector and the mask image, and encode the text based on a preset text encoder; in order to achieve high-quality video generation, the present application adds a timing alignment module based on the Stable Diffusion model of the traditional text image. In addition, in order to customize the generated content, the present application also adds conditional control, including text control and image control.

[0046] The UNet module is used to obtain a latent vector that completes text control and image control based on the spliced ​​noise latent vector and the mask image and the encoded text.

[0047] After obtaining the latent vector that is approximately pure noise, the reverse process, namely the denoising process, is required. The reverse denoising process is predicted with the help of a neural network model (UNet module). In this application, the UNet and cross-attention combined model is used to predict the reverse probability distribution. , the specific formula is as follows: ; .

[0048] In this application, the super parameter As a conditional probability The variance of , so the UNet module only needs to predict its mean. Although the inverse process and the diffusion process are both Markov processes, the difference is that the mean in the Gaussian distribution needs to be determined by network learning rather than hyperparameters.

[0049] The UNet module includes a trainable convolution module, a residual convolution module, and a cross attention module. The noise latent vector is Time is ,in, b for t The BatchSize size at the moment is collaged with the input image and then input into the trainable convolution module. The output result of the trainable convolution module is dimensionally transformed to generate The original intention is to integrate the video length dimension into the batch dimension, which can then be input into the two-dimensional residual convolution module in the UNet module, and the cross-attention mechanism is used to enhance the global capture ability of the network and learn the information between different frames. At this time, the prompt word corresponding to the text is also used as the input of the cross-attention module. After being converted into a token, it is embedded in the cross-attention module to control the generation of the target video. After the residual convolution module and the cross-attention module, Perform dimension changes to generate .

[0050] The temporal alignment model is used to perform temporal alignment on latent vectors that complete text control and image control based on a self-attention mechanism to obtain a target latent vector; the temporal alignment module includes at least one mapping module. Input to the first mapping module in the timing alignment module and generate The mapping vector is then used as the self-attention mechanism Q, K, V Perform self-attention operation, the specific operation is shown in the following formula: ； in, Q Query is the query vector ，K is the key, V is the value, c is the channel size (channel) 。

[0051] The above steps generated The output is mapped and input into the next mapping module. The vector output by the last mapping module is the target latent vector. The target latent vector is the vector obtained by denoising the noisy latent vector with text control and image control added.

[0052] It should be noted that since the structure of the residual convolution module and the cross-attention module has not been changed, the structure of the original Stable Diffusion model is retained. Therefore, the temporal alignment module can be embedded in other Wensheng graph model frameworks with residual convolution modules and cross-attention modules to increase flexibility and applicability. It can also be embedded in different styles of LoRA to generate videos of specified style types. LoRA (Low-Rank Adaptation) is a technology used for model fine-tuning, especially a method for adaptive training on large-scale pre-trained models (such as language models and generative models).

[0053] The 3D decoder is used to generate the target video according to the target latent vector. The 3D decoder in this application is also pre-trained on the UCF101 dataset, and can generate a video according to the denoised target latent vector, and use the generated video as the target video. Among them, the 3D encoder and the 3D decoder are both 3D CNN network structures.

[0054] It is understandable that the present application provides a high-quality and efficient end-to-end video generation model. The video generation model includes an encoder and a decoder, and adopts an end-to-end network framework to improve training efficiency and resource utilization, facilitate subsequent function deployment and application, and achieve efficient video generation. In addition, by inputting a masked picture and controlling the dynamic video generation area during the denoising process of the video generation model, the user's dynamic generation requirements for specified areas are met. In addition, the various modules obtained by training in this application can be inserted into a Wensheng graph model with a similar architecture, and support integration with the style LoRA to achieve specific style video generation and broaden the scope of application.

[0055] Based on the above embodiment, as an optional embodiment, the training steps of the video generation model include step S011 to step S013, and the order of step S011 and step S012 is adjustable.

[0056] Step S011, obtain a public video data set, perform a mask operation on the first frame image of the video in the public video data set, and obtain a training set; specifically, the public video data set is the MSR-VTT public data set, which consists of a total of 100,000 video clips and their manual annotations. Part or all of the videos can be selected from the data set and preprocessed to obtain a training set.

[0057] Masking the first frame image of the video in the public video dataset may include performing a masking operation on the first frame of each video, randomly selecting a preset percentage (e.g., 50%-85%, preferably 75%) of the area for masking, thereby enhancing the network's ability to recognize the subject and improving the quality of the network-generated video.

[0058] Optionally, the user can customize the output video size. If not set, the input image size is used as the output video resolution by default. In order to achieve dynamic generation in the selected area while other areas remain unchanged, the pixel value of the background part except the selected area in each denoising process of this application is consistent with the denoised image of the first frame, that is, the dynamic generation of the background area of ​​other frames except the first frame is suppressed.

[0059] Step S012, freezing the parameters of the 3D encoder, the denoising module, the UNet module and the 3D decoder, and determining the loss function of the video generation model based on minimizing the negative log-likelihood.

[0060] This application uses maximum likelihood estimation to find the probability distribution of Markov chain transitions in the inverse diffusion process, that is, to maximize the logarithmic likelihood of the model prediction. Therefore, the loss function of the training process is to minimize the negative logarithmic likelihood. The specific loss function expression is as follows: ; ; in, represents the expected value; Represents the features of each frame of the input; represents the noise sampled from a normal distribution with mean 0 and covariance as the identity matrix; represents the time step; Indicates additional conditional information, such as custom dynamic areas, etc. Represents the square of the L2 norm. is an output of the model, representing the estimated noise; represents a sequence of time steps containing features; represents the masked original input; is the current time step; A transformation that represents additional conditional information.

[0061] Step S013, training the video generation model based on the training set, updating the parameters of the condition control module and / or the parameters of the timing alignment model, reaching a preset number of iterations or the loss function reaching a preset threshold, and completing the training.

[0062] During the training process, the parameters of the original Stable Diffusion model are frozen and do not participate in the training. Only the parameters of the timing alignment module and the condition control module are updated.

[0063] It can be understood that the present application can enhance the network's ability to recognize subjects and improve the quality of network-generated videos by performing mask operations on the first frame image of the video in the public video dataset; only the parameters of the conditional control module and / or the parameters of the timing alignment model are updated to reduce the amount of video memory required for training and improve the training rate.

[0064] Figure 4 This is a schematic diagram of the reasoning process of the video generation model provided by this application, such as Figure 4 As shown, based on the above embodiment, as an optional embodiment, the reasoning steps of the video generation model include step S021-step S023, and the order of step S021 and step S022 can be adjusted.

[0065] Step S021, obtaining inference text and inference pictures; the inference text and inference pictures are input by the user or extracted from an existing text set and an existing picture set.

[0066] Step S022, generating an inference mask image in response to a user-defined area selected in the inference image; this step is the same as step S200.

[0067] Step S023, input the inference text, the inference picture, the inference mask picture and the preset random noise into the trained video generation model to obtain the inference video output by the video generation model. There is no noise adding step in the inference process, and a user-defined noise with the same size as the video and conforming to the Gaussian distribution is directly generated as one of the inputs, and is input into the trained UNet module and the time alignment module together with the prompt word prompt information of the inference text and the inference picture, and then the inference video is generated through the 3D decoder.

[0068] Optionally, the user can customize the output video size. If not set, the input image size is used as the output video resolution by default. In order to achieve dynamic generation in the selected area while other areas remain unchanged, the pixel value of the background part except the selected area in each denoising process of this application is consistent with the denoised image of the first frame, that is, the dynamic generation of the background area of ​​other frames except the first frame is suppressed.

[0069] It can be understood that the present application provides an inference process for the video generation model, which can be used to evaluate the actual application performance of the model, including the clarity, coherence and creativity of the generated video, and make adjustments based on user feedback to achieve a personalized and customized content generation experience.

[0070] Figure 5 This is the second flow chart of the method for generating dynamic video in a custom area provided by this application. Figure 5 As shown, based on the above embodiment, as an optional embodiment, generating a mask image in response to a custom area selected by a user in the image includes the following steps.

[0071] Step S210: providing a brush tool or an image segmentation tool to the user based on a preset user interaction interface.

[0072] Step S220, in response to a smearing instruction issued by the user based on the brush tool, the area corresponding to the smearing instruction is used as a custom area to generate a mask image; or, in response to a click instruction issued by the user, the area corresponding to the click instruction is used as a custom area based on the image segmentation tool to generate a mask image.

[0073] Figure 6a This is a schematic diagram of selecting a custom area based on a brush tool provided by this application. Figure 6b is a schematic diagram of selecting a custom area based on an image segmentation tool provided by this application, such as Figure 6a and Figure 6bAs shown, in order to facilitate users to select dynamic areas, the present application provides a user interaction interface, which can support users to use the brush tool to paint the area by themselves, or use the SAM tool to click any point in the area and use the open source SAM model to select the area. SAM (Segment Anything Model) is an open source image segmentation model.

[0074] Optionally, after generating the mask image in response to the user-defined area selected in the image by the user, the method further includes: obtaining user-defined parameters input by the user based on the user interaction interface. The user-defined parameters include parameters such as the generated video size, the number of times the diffusion process is adopted, the video length, and fps.

[0075] Correspondingly, the inputting of the text, the picture and the mask picture into a pre-trained video generation model to obtain the target video output by the video generation model includes: inputting the text, the picture, the mask picture and the custom parameters into a pre-trained video generation model to obtain the target video output by the video generation model.

[0076] After the user selects a custom area, he or she can also customize the parameters such as the generated video size, number of times the diffusion process is used, video length, fps, etc. Based on the above parameters and conditions, a high-quality dynamic area video can be generated.

[0077] It is understandable that this application provides two interactive methods for selecting regions. Users can use the brush tool to customize the smear area as the mask area; or use the SAM tool to automatically obtain the entire region range of the subject by selecting any point in the subject. Through this interactive method, users can quickly and flexibly select the required range to generate a video that better meets user needs.

[0078] In summary, the present application can realize the generation of high-quality videos, and can realize accurate content production based on the text and pictures provided by the user. The present application supports video content as input by embedding the timing alignment module and the conditional control module in the diffusion model, and uses the self-attention operation to enhance the global modeling ability of the model, so as to make the actions between each frame coherent, while not affecting the existing diffusion model structure and weights, and the timing alignment module can be inserted into any Wensheng graph model of the same architecture, so as to facilitate the loading of the specified LoRA, generate video content of a specific style, increase flexibility and application scope, and provide users with a variety of functional options and optimizations. It solves the problems that the existing video generation scheme is based on the traditional Wensheng graph model, the generated video action is incoherent, and its content style or background cannot be specified.

[0079] This application greatly reduces the training and transmission costs by constructing an end-to-end model framework. Multiple modules and structures are built in one framework, avoiding the efficiency loss caused by multi-model scheduling. At the same time, during the reasoning process, the automatic latent vector generation and the dynamic combination of the input image are input into the model together with the text prompt word prompt, so that high-quality videos can be generated quickly and efficiently. In addition, during the training process, the UNet module parameters in the model are in a frozen state, that is, the weights in the text graph model are retained, the amount of video memory required for training is reduced, and the training rate is increased. It solves the problems of low efficiency of existing video generation solutions, the need for multi-service scheduling or multiple training times, and huge storage and transmission overheads.

[0080] This application can enable users to customize the selection of any area in the image, and keep the area in a dynamic state, while the unselected area remains static. The selected area is input into the video generation model as a mask image, and the diffusion area is controlled during the denoising process, the pixel value of the unselected area is maintained, and the target video is directly generated to prevent the inefficiency caused by post-processing. This solution can be flexibly applied to multiple scenarios, including dynamic greeting cards, dynamic backgrounds, etc., and can also increase fun and flexibility, which is conducive to use by different groups. In addition, this application also provides two area selection methods, namely, the user uses a brush to smear arbitrarily, and the SAM method of quickly selecting an area, which further improves the user's convenience. It solves the problem that the existing video generation solution cannot customize the selection of dynamic areas, but can only perform dynamic generation of all areas in the image.

[0081] The following is a description of the custom area dynamic video generation device provided in the present application. The custom area dynamic video generation device described below and the custom area dynamic video generation method described above can be referenced to each other.

[0082] Figure 7 It is a timing diagram of the custom area dynamic video generation device provided by this application, such as Figure 7 As shown, the present application also provides a device for generating dynamic video of a custom area, including the following modules.

[0083] An acquisition module 710 is used to acquire text used for video generation content description and a picture used as a video background; A first generating module 720, configured to generate a mask image in response to a user-defined area selected in the image by a user; A second generation module 730 is used to input the text, the picture and the mask picture into a pre-trained video generation model to obtain a target video output by the video generation model; Among them, the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame image in the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask image.

[0084] As an embodiment, the video generation model includes a 3D encoder, a noise adding module, a condition control module, a UNet module, a temporal alignment model and a 3D decoder; The 3D encoder is used to output a latent vector according to the picture; The noise adding module is used to add preset noise or user input noise to the latent vector to obtain a noisy latent vector; The condition control module is used to splice the noise latent vector and the mask image, and encode the text based on a preset text encoder; The UNet module is used to obtain a latent vector for completing text control and image control according to the spliced ​​noise latent vector and the mask image and the encoded text; The temporal alignment model is used to perform temporal alignment on latent vectors that complete text control and image control based on a self-attention mechanism to obtain a target latent vector; The 3D decoder is used to generate the target video according to the target latent vector.

[0085] As an embodiment, it also includes: A training module is used to obtain a public video data set, perform a mask operation on the first frame image of the video in the public video data set to obtain a training set; freeze the parameters of the 3D encoder, the denoising module, the UNet module and the 3D decoder, and determine the loss function of the video generation model based on minimizing the negative log-likelihood; train the video generation model based on the training set, update the parameters of the conditional control module and / or the parameters of the timing alignment model, reach a preset number of iterations or the loss function reaches a preset threshold, and complete the training.

[0086] As an embodiment, it also includes: An inference module is used to obtain inference text and inference pictures; generate an inference mask picture in response to a custom area selected by a user in the inference picture; input the inference text, the inference picture, the inference mask picture and preset random noise into the trained video generation model to obtain the inference video output by the video generation model.

[0087] As an embodiment, the first generating module 720 is further configured to: Providing a brush tool or an image segmentation tool to the user based on a preset user interaction interface; In response to a smearing instruction issued by the user based on the brush tool, the area corresponding to the smearing instruction is used as a custom area to generate a mask image; or,

[0088] In response to a click instruction issued by a user, a mask image is generated based on the image segmentation tool using an area corresponding to the click instruction as a custom area.

[0089] As an embodiment, the first generating module 720 is further configured to: Acquiring a custom parameter input by a user based on the user interaction interface; Correspondingly, the second generating module 730 is further used for: The text, the image, the mask image and the custom parameters are input into a pre-trained video generation model to obtain a target video output by the video generation model.

[0090] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the custom region dynamic video generation method, the method comprising: obtaining text for describing the video generation content and a picture used as a video background; generating a mask picture in response to a custom region selected by a user in the picture; inputting the text, the picture and the mask picture into a pre-trained video generation model to obtain a target video output by the video generation model; wherein the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame of the target video, and the condition control module is at least used to determine the dynamic region of the target video according to the mask picture.

[0091] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0092] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the custom area dynamic video generation method provided by the above methods, the method including: obtaining text for describing the video generation content and a picture used as a video background; generating a mask picture in response to a custom area selected by a user in the picture; inputting the text, the picture and the mask picture into a pre-trained video generation model to obtain a target video output by the video generation model; wherein the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame of the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask picture.

[0093] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the custom area dynamic video generation method provided by the above-mentioned methods, the method comprising: obtaining text for describing the video generation content and a picture for serving as a video background; generating a mask picture in response to a custom area selected by a user in the picture; inputting the text, the picture and the mask picture into a pre-trained video generation model to obtain a target video output by the video generation model; wherein the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame of the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask picture.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating dynamic video in a custom area, characterized in that: include: Get the text used to describe the video content and the picture used as the video background; generating a mask image in response to a user-defined area selected in the image; Inputting the text, the image and the mask image into a pre-trained video generation model to obtain a target video output by the video generation model; Among them, the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame image in the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask image.

2. The method for generating dynamic video of a custom area according to claim 1, characterized in that: The video generation model includes a 3D encoder, a noise adding module, a condition control module, a UNet module, a temporal alignment model and a 3D decoder; The 3D encoder is used to output a latent vector according to the picture; The noise adding module is used to add preset noise or user input noise to the latent vector to obtain a noisy latent vector; The condition control module is used to splice the noise latent vector and the mask image, and encode the text based on a preset text encoder; The UNet module is used to obtain a latent vector for completing text control and image control according to the spliced ​​noise latent vector and the mask image and the encoded text; The temporal alignment model is used to perform temporal alignment on latent vectors that complete text control and image control based on a self-attention mechanism to obtain a target latent vector; The 3D decoder is used to generate the target video according to the target latent vector.

3. The method for generating dynamic video of a custom area according to claim 2, characterized in that: The training steps of the video generation model include: Obtain a public video data set, and perform a mask operation on the first frame image of the video in the public video data set to obtain a training set; Freezing parameters of the 3D encoder, the denoising module, the UNet module, and the 3D decoder, and determining a loss function of the video generation model based on minimizing negative log-likelihood; The video generation model is trained based on the training set, and the parameters of the conditional control module and / or the parameters of the timing alignment model are updated. The training is completed when a preset number of iterations is reached or the loss function reaches a preset threshold.

4. The method for generating dynamic video of a custom area according to claim 2, characterized in that: The reasoning steps of the video generation model include: Get inference text and inference pictures; In response to a user-defined area selected in the inference image by a user, generating an inference mask image; The inference text, the inference picture, the inference mask picture and preset random noise are input into the trained video generation model to obtain the inference video output by the video generation model.

5. The method for generating dynamic video of a custom area according to any one of claims 1 to 4, characterized in that: The step of generating a mask image in response to a user-defined area selected in the image by a user comprises: Providing a brush tool or an image segmentation tool to the user based on a preset user interaction interface; In response to a smearing instruction issued by the user based on the brush tool, the area corresponding to the smearing instruction is used as a custom area to generate a mask image; or, In response to a click instruction issued by a user, a mask image is generated based on the image segmentation tool using an area corresponding to the click instruction as a custom area.

6. The method for generating dynamic video of a user-defined area according to claim 5, characterized in that: After generating the mask image in response to the user-defined area selected in the image by the user, the method further includes: Acquiring a custom parameter input by a user based on the user interaction interface; Correspondingly, the inputting the text, the image, and the mask image into a pre-trained video generation model to obtain a target video output by the video generation model includes: The text, the image, the mask image and the custom parameters are input into a pre-trained video generation model to obtain a target video output by the video generation model.

7. A device for generating dynamic video in a custom area, characterized in that: include: An acquisition module, used to acquire text used to describe the video content and a picture used as a video background; A first generating module, configured to generate a mask image in response to a user-defined area selected in the image by a user; A second generation module is used to input the text, the picture and the mask picture into a pre-trained video generation model to obtain a target video output by the video generation model; Among them, the video generation model is obtained by embedding a timing alignment module and / or an image control module in a preset diffusion model, the timing alignment model is used to determine the timing of each frame image in the target video, and the condition control module is at least used to determine the dynamic area of ​​the target video based on the mask image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating dynamic video in a custom area as claimed in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating dynamic video in a custom area as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating dynamic video in a custom area as claimed in any one of claims 1 to 6 is implemented.

Images