Method and apparatus for generating dynamic video, electronic device, and storage medium

Abstract

The present application discloses a method and apparatus for generating a dynamic video, an electronic device, and a storage medium. The method comprises: acquiring a textual script of a storyboard; performing semantic expansion on the textual script by means of a large language model to obtain a plurality of scene descriptions, and generating storyboard still images corresponding to the scene descriptions; using a fusion inpainting model to perform inpainting processing on a target storyboard still image selected from the plurality of storyboard still images; generating a mask image of a dynamic area on the target storyboard still image selected by a user; by means of an image-to-video model, using feature information of the target storyboard still image and the mask image to generate a dynamic video of the storyboard according to image generation prompt information; performing video post-processing on the dynamic video; generating audio generation prompt information on the basis of a theme and style of the dynamic video, and generating a current video-based audio according to the audio generation prompt information by means of an audio generation model; and adding the current video-based audio to the dynamic video of the storyboard to obtain a final dynamic video of the storyboard.

Description

A dynamic video generation method and device, electronic equipment, and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202411081878.2, filed on August 8, 2024, and entitled "A dynamic video generation method and device, electronic equipment, and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of video production, and in particular to a dynamic video generation method and device, electronic equipment, and storage medium. BACKGROUND

[0003] Currently, in the process of program video production, whether it is real scene shooting or dynamic cartoon drawing, a long time is needed, that is, the efficiency is relatively low, and a high cost is also needed. Therefore, a way that can efficiently and at a low cost produce program videos that meet the needs is needed.

[0004] Currently, artificial intelligence technology has made some progress in video generation, that is, some companies have provided text-to-video models and image-to-video models. Therefore, users can input the corresponding description text according to the video generation needs, and completely generate the corresponding video through the text-to-video model according to the semantic analysis of the input description text. Or according to the video to be generated, first draw an image, and then input it into the image-to-video model, and then expand the input image through the image-to-video model, so as to form a short video by using the expanded image.

[0005] However, since the current text-to-video model and image-to-video model are in the initial development stage, when generating a video, the model completely generates the video according to the analysis, and cannot control the fine details of the video content, only an overall relatively required video can be generated, so it cannot meet the fine detail requirements of the program video. In addition, the quality of the video generated by the current model is also limited, and it cannot meet the high quality requirements of the program video. SUMMARY

[0006] Based on the above-mentioned deficiencies of the prior art, the present application provides a dynamic video generation method and device, electronic equipment, and storage medium to solve the problem that the prior art cannot control the fine details of the video content and cannot guarantee the quality requirements.

[0007] In order to achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0008] The first aspect of the present application provides a dynamic video generation method, comprising:

[0009] acquire a text script of a shot;

[0010] perform semantic expansion on the text script of the shot by a large language model to obtain expanded scene descriptions of the shot, and generate a shot static image corresponding to each of the scene descriptions;

[0011] perform redrawing processing on a target shot static image by a fusion redrawing model to obtain the redrawing target shot static image; wherein the target shot static image is a selected one of the shot static images;

[0012] generate a mask image of a dynamic region on the target shot static image selected by a user;

[0013] input the target shot static image, the mask image, and set image generation prompt information into a pre-constructed image-to-video model, use the image generation prompt information to gradually predict a next frame of video by the image-to-video model according to feature information of the target shot static image and the mask image, and use each of the predicted frames of video to compose a dynamic video of the shot;

[0014] perform video post-processing on the dynamic video of the shot;

[0015] generate audio generation prompt information according to a theme style of the dynamic video of the shot, and generate a current video audio by a pre-constructed audio generation model according to the audio generation prompt information;

[0016] add the current video audio to the dynamic video of the shot to obtain a final dynamic video of the shot.

[0017] Optionally, in the above dynamic video generation method, the processing of the target shot static image by the fusion redrawing model to obtain the redrawing target shot static image comprises:

[0018] input the target shot static image into the fusion redrawing model, and perform quality and / or size adjustment on the target shot static image by a drawing model in the fusion redrawing model to obtain an adjusted target shot static image;

[0019] use image edge information to redraw a defect region in the adjusted target shot static image by an image redrawing plug-in in the fusion redrawing model to obtain the redrawing target shot static image; wherein the image edge information is extracted from the adjusted target shot static image by an edge detection algorithm.

[0020] Optionally, in the dynamic video generation method described above, the target static shot, the mask image, and the set image generation prompt information are input into a pre-constructed image-to-video model, the image-to-video model uses feature information of the target static shot and the mask image to gradually predict a next frame of video according to the image generation prompt information, and each frame of the video predicted is used to compose a dynamic video of the shot.

[0021] The target static shot, the mask image, and the set image generation prompt information are input into the pre-constructed image-to-video model, and the target static shot and the mask image are encoded to obtain image feature information.

[0022] The internal parameters and the image feature information are used to compose guidance condition information.

[0023] Based on the gradually diffused noise introduced by the target static shot at each time step, the next frame of video is gradually predicted according to the guidance condition information and the image generation prompt information.

[0024] The frames of the video are arranged in time to obtain the dynamic video of the shot and output.

[0025] Optionally, in the dynamic video generation method described above, the dynamic video of the shot is video post-processed, including:

[0026] The first frame of the dynamic video of the shot is extracted.

[0027] The first frame of the dynamic video of the shot is displayed on a drawing interface, and a mask region drawn by a user on the first frame of the dynamic video of the shot is determined in response to a drawing operation of the user.

[0028] The image of the region on the target static shot that is at the same position as the mask region is used to replace the image of the region on each frame of the dynamic video of the shot that is at the same position as the mask region.

[0029] The resolution and consistency of the dynamic video of the shot are improved by a video enhancement model.

[0030] Optionally, in the dynamic video generation method described above, the current video audio is generated by the pre-constructed audio generation model according to the audio generation prompt information, including:

[0031] The audio data and the audio generation prompt information are input into a variational autoencoder in the audio generation model, the audio data is encoded by the variational autoencoder to obtain audio encoding data, and the audio generation prompt information is encoded by a text encoder in the audio generation model to obtain a high-dimensional feature vector;

[0032] The audio encoding data, the high-dimensional feature vector, and time embedding are input into an image segmentation network in the audio generation model, and after feature fusion and noise reduction processing are performed by using self-attention and cross-attention layers, the audio encoding data, the high-dimensional feature vector, and the time embedding are collected to obtain current video audio.

[0033] The second aspect of the application provides a dynamic video generation device, comprising:

[0034] An acquisition unit is configured to acquire a text script of a split shot.

[0035] An image generation unit is configured to perform semantic extension on the text script of the split shot by using a large language model to obtain extended scene descriptions of the split shot, and generate a split shot static image corresponding to each of the scene descriptions.

[0036] A redrawing unit is configured to perform redrawing processing on a target split shot static image by using a fusion redrawing model to obtain a redrawing result of the target split shot static image, wherein the target split shot static image is selected from the split shot static images.

[0037] A mask image generation unit is configured to generate a mask image of a dynamic region on the target split shot static image selected by a user.

[0038] A video generation unit is configured to input the target split shot static image, the mask image, and set image generation prompt information into a pre-constructed image-to-video model, use feature information of the target split shot static image and the mask image, and gradually predict a next frame of video according to the image generation prompt information by using the image-to-video model, and use the predicted frames of video to compose a dynamic video of the split shot.

[0039] A post-processing unit is configured to perform video post-processing on the dynamic video of the split shot.

[0040] An audio generation unit is configured to generate audio generation prompt information according to a theme style of the dynamic video of the split shot, and generate current video audio according to the audio generation prompt information by using a pre-constructed audio generation model.

[0041] An adding unit is configured to add the current video audio to the dynamic video of the split shot to obtain a final dynamic video of the split shot.

[0042] Optionally, in the dynamic video generation apparatus, the redrawing unit comprises:

[0043] An image adjusting unit is configured to input the target split shot static image into the fusion redrawing model, perform quality and / or size adjustment on the target split shot static image by using a drawing model in the fusion redrawing model, and obtain an adjusted target split shot static image.

[0044] A region redrawing unit is configured to redraw a defect region in the adjusted target split shot static image by using image edge information obtained by using an edge detection algorithm on the adjusted target split shot static image by using an image redrawing plug-in in the fusion redrawing model, and obtain a redrew target split shot static image.

[0045] Optionally, in the dynamic video generation apparatus, the video generation unit comprises:

[0046] A feature obtaining unit is configured to input the target split shot static image, the mask image, and the set image generation prompt information into a pre-constructed image generation model, encode the target split shot static image and the mask image, and obtain image feature information.

[0047] A feature combination unit is configured to combine internal parameters and the image feature information to obtain guidance condition information.

[0048] A prediction unit is configured to gradually predict a next frame of video based on gradually diffused noise introduced by the target split shot static image at each time step, the guidance condition information, and the image generation prompt information.

[0049] An image combination unit is configured to arrange each frame of the video according to time to obtain the dynamic video of the split shot and output the dynamic video of the split shot.

[0050] Optionally, in the dynamic video generation apparatus, the post-processing unit comprises:

[0051] An extraction unit is configured to extract a first frame image of the dynamic video of the split shot.

[0052] A drawing unit is configured to display the first frame image on a drawing interface, and determine a mask region drawn by a user on the first frame image in response to a drawing operation of the user.

[0053] a replacing unit configured to replace an image of an area on each frame of the dynamic video of the split shot that is in the same position as the mask area with an image of an area on the target split shot static image that is in the same position as the mask area;

[0054] an enhancing unit configured to improve resolution and consistency of the dynamic video of the split shot by a video enhancement model.

[0055] Optionally, in the dynamic video generation apparatus, the audio generation unit comprises:

[0056] an encoding unit configured to encode the audio data by a variational autoencoder in the audio generation model to obtain audio encoding data, and encode the audio generation prompt information by a text encoder in the audio generation model to obtain a high-dimensional feature vector;

[0057] a feature processing unit configured to input the audio encoding data, the high-dimensional feature vector and time embedding into an image segmentation network in the audio generation model, and perform feature fusion and noise reduction processing by self-attention and cross-attention layers in the image segmentation network, and then collect the audio encoding data, the high-dimensional feature vector and the time embedding to obtain a current video audio.

[0058] The third aspect of the present application provides an electronic device, comprising:

[0059] a memory and a processor;

[0060] wherein the memory is configured to store a program;

[0061] the processor is configured to execute the program, and the program, when executed, is specifically configured to implement the dynamic video generation method according to any one of the preceding aspects.

[0062] The fourth aspect of the present application provides a computer storage medium for storing a computer program, wherein the computer program, when executed, is configured to implement the dynamic video generation method according to any one of the preceding aspects.

[0063] The embodiment of the application provides a dynamic video generation method, acquires a text script of a shot, then performs semantic extension on the text script of the shot through a large language model, obtains a plurality of scene descriptions of the extended shot, and generates a shot static picture corresponding to each scene description, so that the shot picture meeting the user demand can be generated through the extension mode, the user can select, and it is guaranteed that the generated dynamic video is more in line with the actual demand of the user. Then, a fusion redrawing model is used for redrawing processing on a target shot static picture selected by the user from the shot static pictures, to obtain a redrawing target shot static picture, so that the shot picture is finely controlled by redrawing the area needing adjustment in the static picture, the quality of the shot picture is guaranteed, and the quality of the generated video is improved. Then, a mask picture of a dynamic area on the target shot static picture selected by the user is generated, so that the user can select and control the area of dynamic change. After that, the target shot static picture, the mask picture and set image generation prompt information are input into a pre-constructed picture generation video model, the picture generation video model uses the feature information of the target shot static picture and the mask picture, and gradually predicts a next frame of video according to the image generation prompt information, and uses the predicted frames of video to form a dynamic video of the shot, so as to generate a dynamic video of dynamic change according to the dynamic area selected by the user. Then, the dynamic video of the shot is subjected to video post-processing, so as to improve the video of each frame of the generated dynamic video and guarantee whether the dynamic video meets the requirement. Then, audio generation prompt information is generated according to the theme style of the dynamic video of the shot, and a pre-constructed audio generation model is used to generate a current video audio according to the audio generation prompt information, so as to generate audio data meeting the theme style of the dynamic video. Finally, the current video audio is added to the dynamic video of the shot, to obtain a final dynamic video of the shot. Thus, a video generation method capable of fine control is realized, the demand of the user can be well met, and the quality of the video can be effectively guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0065] FIG. 1 is a flowchart of a dynamic video generation method provided by the embodiment of the present application;

[0066] FIG. 2 is a flowchart of a method for redrawing processing on a target shot static picture provided by the embodiment of the present application;

[0067] FIG. 3 is a schematic diagram of an example of controlling a dynamic region of an image according to an embodiment of the present application;

[0068] FIG. 4 is a flowchart of a method of generating a dynamic video by a graph-based video model according to an embodiment of the present application;

[0069] FIG. 5 is a flowchart of a method of performing video post-processing on a dynamic video according to an embodiment of the present application;

[0070] FIG. 6 is a flowchart of a method of generating audio of a current video according to an embodiment of the present application;

[0071] FIG. 7 is a schematic diagram of an architecture of a dynamic video generation apparatus according to an embodiment of the present application;

[0072] FIG. 8 is a schematic diagram of an architecture of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0074] In the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement “including a…” does not exclude the presence of another identical element in the process, method, article or device including the element.

[0075] The embodiments of the present application provide a dynamic video generation method, as shown in FIG. 1, including the following steps:

[0076] S101, acquiring a text script of a split shot.

[0077] The split shot refers to any one shot in a program.

[0078] Specifically, the user can edit a corresponding text script according to the generated video of the split shot, that is, a text description of the split shot picture, so that the video of the split shot can be generated based on the obtained text script of the split shot, without the user drawing the image of the split shot. Wherein, the text script of the split shot can be Prompt.

[0079] S102, the text script of the split shot is expanded by the large language model, and a plurality of scene descriptions of the expanded split shot are obtained, and a split static picture corresponding to each scene description is generated.

[0080] Because the granularity of the content described by the text script of the split shot edited by the user is limited, the video generated directly according to the text script of the split shot edited by the user may only meet the user's demand on the whole theme, and cannot well meet the user's real demand, so in the present application, in order to generate a video that meets the user's real demand, the text script of the split shot is expanded by the large language model, so that a plurality of scene descriptions of a plurality of split shots are further expanded on the basis of the original scene description of the split shot, that is, a plurality of scripts Prompt of the split shot are generated. For example, the user starts to edit "I want a picture of a person building bricks", and accordingly, a plurality of scene descriptions such as "the person builds bricks in mountains and rivers", "the person builds bricks in bustling streets", and "the person builds bricks in the universe" can be expanded.

[0081] Then, each scene description of the split shot is used to generate a split static picture corresponding to the scene description for the user to select, so that the user selects a split shot picture that meets the user's real demand for subsequent video generation, and thus a video that meets the demand is obtained, that is, the split shot picture is generated by expansion, so that an image that meets the user's demand can be generated and selected by the user.

[0082] Therefore, the split static picture corresponding to each scene description is generated, which can be displayed to the user for selection.

[0083] Optionally, the large language model used can be developed and constructed based on GPT-4.

[0084] S103, the target split static picture is redrawn by using a fusion redrawing model to obtain a redrawn target split static picture.

[0085] Among them, the target split static picture is a split static picture selected from the split static pictures.

[0086] It should be noted that the target split shot static picture selected by the user is a static picture that meets the user's demand as a whole, but the picture quality, size, etc. of the target split shot static picture, and some details in the target split shot static picture, such as character costumes, hair ornaments, and part of the articles, etc. do not meet the demand. Therefore, in order to realize fine control of the target split shot static picture and realize fine control of the generated video, the target split shot static picture is further redrawn in the embodiment of the present application. Specifically, a fusion redrawing model is constructed by combining a plurality of models, and the target split shot static picture is redrawn according to the user's demand through the fusion redrawing model, that is, part of the area in the target split shot static picture is redrawn according to the user's demand, so as to realize fine control of the target split shot static picture, so that the target split shot static picture also meets the user's demand in details.

[0087] Optionally, in another embodiment of the present application, a specific implementation of step S103 includes the following steps as shown in FIG. 2:

[0088] S201, input the target split shot static picture into the fusion redrawing model, and adjust the picture quality and / or size of the target split shot static picture through the drawing model in the fusion redrawing model to obtain an adjusted target split shot static picture.

[0089] It should be noted that in the embodiment of the present application, the drawing model and the image redrawing plug-in are fused to obtain the fusion redrawing model in combination with the mask technology. The drawing model is mainly used to adjust the target split shot static picture as a whole, and the mask technology can be used to select the area in the target split shot static picture that needs to be redrawn, and the image redrawing plug-in is used to redraw the selected area. Optionally, the drawing model can specifically adopt Stable diffusion, and the image redrawing plug-in is specifically Adetailer.

[0090] Therefore, after the target split shot static picture is input into the fusion redrawing model, the drawing model in the fusion redrawing model first adjusts the picture quality and / or size of the target split shot static picture to obtain an adjusted target split shot static picture.

[0091] S202, redraw the defect area in the adjusted target split shot static picture by using image edge information through the image redrawing plug-in in the fusion redrawing model to obtain a redrawn target split shot static picture.

[0092] The image edge information is obtained by using an edge detection algorithm to extract the adjusted target split shot static picture. The defect area refers to an area in the target split shot static picture that does not meet the user's expectation.

[0093] Specifically, the user can draw the mask area by using a mouse or other interactive operation means, and then the image redrawing plug-in redraws the defect area. In order to make the redrawing area perfectly blend with other areas, in the embodiment of the present application, an edge detection algorithm is combined in the redrawing process. The edge information in the original image is extracted by the edge detection algorithm, and the redrawing is performed based on the edge information, so that the redrawing content and other content are more naturally blended.

[0094] S104, generating a mask map of the dynamic area on the target split shot static image selected by the user.

[0095] It should be noted that, unlike the prior art in which the model autonomously generates dynamic changes in multiple objects in a static image to generate a dynamic video. In the embodiment of the present application, the dynamic changes in the dynamic video are finely controlled to meet the user's demand for the program video. Specifically, in the embodiment of the present application, the user selects a dynamic area on a target split shot static image, and a mask map of the dynamic area is generated accordingly, so as to facilitate subsequent dynamic processing of the dynamic area, so that the objects in the dynamic area realize dynamic changes, and other areas are not dynamically processed.

[0096] Optionally, the "pre-processing motion brush function" developed can be used for selection, that is, a mouse-interactive drawing mask method is made, and the user can directly draw the mask area that wants to be dynamically changed on the target split shot static image, so as to achieve pixel-level controllability. For example, as shown in FIG. 3, the user only wants to shake the candle flame in the picture shown in FIG. 3, and can select the candle flame by using a mouse interaction or other drawing method, and a mask map of the dynamic area shown in (a) of FIG. 3, that is, a mask control map of the dynamic area, is generated accordingly. In the subsequent generated video, only the candle flame in each predicted image is dynamically changed. For example, as shown in (b), (c), (d) and (e) of FIG. 3, only the candle flame is different, so that these images can constitute a video of the candle flame change.

[0097] S105, inputting the target split shot static image, the mask map and the set image generation prompt information into a pre-constructed image video generation model, using the feature information of the target split shot static image and the mask map, and gradually predicting the next frame of video according to the image generation prompt information by using the image video generation model, and using the predicted frames of video to compose a dynamic video of a split shot.

[0098] It should be noted that the mask map is mainly to determine the area that needs to be dynamically changed, and in order to control the change of the dynamic area according to the user's needs, in the embodiment of the present application, the user can also input the set image generation prompt information for indicating image generation.

[0099] Optionally, the image-to-video model can be an extensible diffusion model improved based on stable video diffusion, and the video generation effect is optimized by adding multiple time sequence layers containing 3D convolution in the model U-Net part, and then using the self-recurrent generation method and the pre-trained VAE model to encode and decode the input image.

[0100] Specifically, the image-to-video model can encode the target shot script static image and the mask map to obtain feature information of the target shot script static image and the mask map, and then based on the target shot script static image and the mask map and the feature information, according to the image generation prompt information, predict the next frame of video frame of the target shot script static image, and then on the basis of a frame of video frame, predict the next frame of video frame, so as to obtain multiple video frames through multiple predictions, and sort the video frames according to time to form a dynamic video of a shot.

[0101] Optionally, in another embodiment of the present application, a specific embodiment of step S105 includes the following steps as shown in FIG. 4:

[0102] S401, input the target shot script static image, the mask map and the set image generation prompt information into the pre-constructed image-to-video model, encode the target shot script static image and the mask map to obtain image feature information.

[0103] S402, combine the internal parameters and the image feature information to form the guidance condition information.

[0104] Among them, the internal parameter refers to the parameter in the image-to-video model. Because the prediction of the image needs to be based on the internal of the image-to-video model, therefore the internal parameter and the image feature information are combined.

[0105] S403, based on the step-by-step diffusion noise introduced by the target shot script static image at each time step, gradually predict the next frame of video frame according to the guidance condition information and the image generation prompt information.

[0106] S404, arrange the video frames according to time to obtain a dynamic video of a shot and output.

[0107] S106, video post-processing is performed on the dynamic video of a shot.

[0108] It should be noted that the quality of the video frame generated by the video generation model considering the program video may not be good enough to meet the quality requirements, and therefore in the embodiment of the present application, the generated dynamic video of the shot also needs to be post-processed. Specifically, the quality of each frame of the dynamic video of the shot can be optimized.

[0109] Optionally, in another embodiment of the present application, a specific implementation of step S106 includes the following steps as shown in FIG. 5.

[0110] S501, extract the first frame image of the dynamic video of the shot.

[0111] It should be noted that if there is a defect in a certain area of the first frame image of the dynamic video of the shot, for example, there is a case of deformation of a person or object, then the subsequent video frames will also have the same situation in the same area, so the first frame image of the dynamic video of the shot is extracted to determine the area with defects.

[0112] S502, display the first frame image on the drawing interface, and determine the mask area drawn by the user on the first frame image in response to the user's drawing operation.

[0113] In order to facilitate processing according to user needs, the first frame image is displayed on the drawing interface to select the area to be processed by the user. Specifically, the user can also draw the area to be processed on the interface through the "post-processing motion brush function", which will generate a mask area, i.e. mask area.

[0114] S503, replace the image of the area in the same position as the mask area on each frame image of the dynamic video of the shot with the image of the area in the same position as the mask area on the target shot static image.

[0115] It should be noted that since the generated target shot static image is processed by the fusion redrawing model, it has high quality, and the size of the target shot static image and each frame image of the dynamic video is consistent, so in the embodiment of the present application, the image of the same area in the target shot static image is selected to replace the mask area of each frame image of the dynamic video, so that the image of the area with defects is replaced.

[0116] S504, improve the resolution and consistency of the dynamic video of the shot by the video enhancement model.

[0117] Specifically, the video enhancement model combining the U-Net and VAE-Decoder algorithms can be used to improve the resolution of the video picture to a specified resolution, for example, to 4k, and also perform dynamic frame interpolation to supplement the frame rate to a set frame rate, for example, to 60fps. Moreover, the model divides the dynamic video into multiple segments and processes them using the U-Net with a time layer to achieve consistency within the segments.

[0118] S107, generating audio generation prompt information according to the theme style of the dynamic video by shot, and generating the current video audio according to the audio generation prompt information through the pre-constructed audio generation model.

[0119] It should be noted that since the generated dynamic video only contains the video part, it is necessary to generate corresponding audio data for the dynamic video to form a complete video together with the dynamic video. In order to generate audio that is in harmony with the dynamic video, the audio generation prompt information is generated according to the theme style of the dynamic video by shot, that is, a plurality of words are outlined as input prompts according to the theme style of the dynamic video, which are input into the audio generation model, so that the audio generation model generates audio in accordance with the audio generation prompt information, which is in accordance with the theme style of the dynamic video.

[0120] Optionally, in another embodiment of the present application, a specific implementation of step S107 is shown in FIG. 6, which includes the following steps:

[0121] S601, the audio data and the audio generation prompt information are pre-constructed in the audio generation model, the audio data is encoded by the variational autoencoder in the audio generation model to obtain audio encoding data, and the audio generation prompt information is encoded by the text encoder in the audio generation model to obtain a high-dimensional feature vector.

[0122] Optionally, the audio generation model can be based on an optimized Transformer model. The Audio audio data that has obtained the copyright is input into the model, and the convolutional variational autoencoder (VAE) in the model is used to compress the representation and retain enough feature information, so as to generate audio data in accordance with the dynamic video on the basis of the audio data. Then the text encoder CLAP in the model is used to encode the audio generation prompt information into a high-dimensional feature vector, and these vectors establish a meaningful connection between words and corresponding sounds, so that audio generation can be based on them.

[0123] S602, input the audio encoding data, high-dimensional feature vector and time embedding into the image segmentation network in the audio generation model, and after feature fusion and noise reduction processing by the self-attention and cross-attention layers of the image segmentation network, the audio encoding data, high-dimensional feature vector and time embedding are collected to obtain the current video audio.

[0124] Specifically, the audio encoding data, high-dimensional feature vector and time embedding are input into the image segmentation network (U-Net architecture) in the audio generation model to perform feature fusion and noise reduction processing by the self-attention and cross-attention layers. Finally, the high-dimensional feature vector, time embedding and VAE encoded audio encoding data are used to generate a high-quality audio segment of a user-specified length and text prompt by the U-Net.

[0125] S108, add the current video audio to the dynamic video of the split shot to obtain the final dynamic video of the split shot.

[0126] Specifically, the current video audio and the dynamic video of the split shot can be combined according to time to obtain the final dynamic video of the split shot.

[0127] The embodiment of the application provides a dynamic video generation method, a text script of a split shot is acquired, then a large language model is used for semantic extension of the text script of the split shot, a plurality of scene descriptions of the extended split shot are obtained, and a split shot static picture corresponding to each scene description is generated, so that the split shot picture meeting the user demand can be generated in an extended manner for the user to select, so that the generated dynamic video is more in line with the actual demand of the user. Then, a fusion redrawing model is used for redrawing processing of a target split shot static picture selected by the user from the split shot static pictures, to obtain a redrawing target split shot static picture, so that the split shot picture is finely controlled by redrawing the area needing adjustment in the static picture, the quality of the split shot picture is ensured, and the quality of the generated video is improved. Then, a mask picture of a dynamic area on the target split shot static picture selected by the user is generated, so that the user selects and controls the area that dynamically changes. Finally, the target split shot static picture, the mask picture and set image generation prompt information are input into a pre-constructed picture generation video model, the picture generation video model uses the feature information of the target split shot static picture and the mask picture, and gradually predicts a next frame of video according to the image generation prompt information, and uses the predicted frames of video to form a dynamic video of the split shot, so that the dynamic video dynamically changes according to the dynamic area selected by the user. Then, the dynamic video of the split shot is subjected to video post-processing, so that the quality of each frame of video of the generated dynamic video is improved, and whether the dynamic video meets the requirement is ensured. Then, audio generation prompt information is generated according to the theme style of the dynamic video of the split shot, and a pre-constructed audio generation model is used to generate current video audio according to the audio generation prompt information, so that audio data meeting the theme style of the dynamic video is generated. Finally, the current video audio is added to the dynamic video of the split shot, to obtain a final dynamic video of the split shot. Thus, a video generation method capable of fine control is realized, the demand of the user can be well met, and the quality of the video can be effectively ensured.

[0128] Another embodiment of the application provides a dynamic video generation device, as shown in FIG. 7, comprising:

[0129] The acquisition unit 701 is configured to acquire a text script of a split shot.

[0130] The image generation unit 702 is configured to use a large language model to perform semantic extension on the text script of the split shot, obtain a plurality of scene descriptions of the extended split shot, and generate a split shot static picture corresponding to each scene description.

[0131] The redrawing unit 703 is configured to use a fusion redrawing model to perform redrawing processing on a target split shot static picture, to obtain a redrawing target split shot static picture.

[0132] The target split shot static picture is a split shot static picture selected from the split shot static pictures.

[0133] The mask image generation unit 704 is configured to generate a mask image of a dynamic region on the target split shot static image selected by the user.

[0134] The video generation unit 705 is configured to input the target split shot static image, the mask image, and the set image generation prompt information into a pre-constructed video generation model, and use the video generation model to gradually predict a next frame of video according to the image generation prompt information based on feature information of the target split shot static image and the mask image, and use the predicted frames of video to compose a dynamic video of a split shot.

[0135] The post-processing unit 706 is configured to perform video post-processing on the dynamic video of the split shot.

[0136] The audio generation unit 707 is configured to generate audio generation prompt information according to a theme style of the dynamic video of the split shot, and generate a current video audio according to the audio generation prompt information by using a pre-constructed audio generation model.

[0137] The adding unit 708 is configured to add the current video audio to the dynamic video of the split shot to obtain a final dynamic video of the split shot.

[0138] Optionally, in the dynamic video generation apparatus provided in another embodiment of the present application, the redrawing unit comprises:

[0139] The image adjustment unit is configured to input the target split shot static image into a fusion redrawing model, and perform quality and / or size adjustment on the target split shot static image by using a drawing model in the fusion redrawing model to obtain an adjusted target split shot static image.

[0140] The region redrawing unit is configured to perform redrawing on a defect region in the adjusted target split shot static image by using image edge information to obtain a redrawing target split shot static image, where the image edge information is obtained by using an edge detection algorithm to extract the adjusted target split shot static image.

[0141] Optionally, in the dynamic video generation apparatus provided in another embodiment of the present application, the video generation unit comprises:

[0142] The feature acquisition unit is configured to input the target split shot static image, the mask image, and the set image generation prompt information into a pre-constructed video generation model, and encode the target split shot static image and the mask image to obtain image feature information.

[0143] The feature combination unit is configured to combine the internal parameters and the image feature information to obtain guidance condition information.

[0144] The prediction unit is configured to predict a next frame of video based on a step-by-step diffusion noise introduced by the target split shot shot sheet at each time step, the guidance condition information, and the image generation prompt information.

[0145] The image combination unit is configured to arrange the frames of video in time to obtain the split shot dynamic video and output the split shot dynamic video.

[0146] Optionally, in another embodiment of the dynamic video generation apparatus provided in the present application, the post-processing unit comprises:

[0147] The extraction unit is configured to extract a first frame of image of the split shot dynamic video.

[0148] The drawing unit is configured to display the first frame of image on a drawing interface and determine a mask region drawn by a user on the first frame of image in response to a drawing operation of the user.

[0149] The replacement unit is configured to replace an image of a region at a same position as the mask region on each frame of image of the split shot dynamic video with an image of a region at a same position as the mask region on the target split shot static image.

[0150] The enhancement unit is configured to improve resolution and consistency of the split shot dynamic video by a video enhancement model.

[0151] Optionally, in another embodiment of the dynamic video generation apparatus provided in the present application, the audio generation unit comprises:

[0152] The encoding unit is configured to encode the audio data by a variational autoencoder in the audio generation model, to obtain audio encoding data, and encode the audio generation prompt information by a text encoder in the audio generation model, to obtain a high-dimensional feature vector.

[0153] The feature processing unit is configured to input the audio encoding data, the high-dimensional feature vector, and the time embedding into an image segmentation network in the audio generation model, and perform feature fusion and noise reduction processing by self-attention and cross-attention layers in the image segmentation network, and then combine the audio encoding data, the high-dimensional feature vector, and the time embedding to obtain a current video audio.

[0154] It should be noted that the specific working processes of the various units provided in the above embodiments of the present application can correspondingly refer to the implementation processes of the corresponding steps in the method embodiments provided in the above method embodiments, which will not be described here.

[0155] Another embodiment of the present application provides an electronic device, as shown in FIG. 8, comprising:

[0156] a memory 801 and a processor 802.

[0157] The memory 801 is configured to store a program.

[0158] The processor 802 is configured to execute the program stored in the memory 801, and when the program is executed, specifically configured to implement the dynamic video generation method provided by any one of the above embodiments.

[0159] Another embodiment of the present application provides a computer storage medium for storing a computer program, and when the computer program is executed, the computer program is configured to implement the dynamic video generation method provided by any one of the above embodiments.

[0160] The computer storage medium includes permanent and non-permanent, removable and non-removable media, and can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program models or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium such as modulated data signal and carrier wave.

[0161] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0162] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of dynamic video generation, the method comprising: The method comprises the following steps: acquiring a text script of a shot; performing semantic extension on the text script of the shot by a large language model to obtain extended scene descriptions of the shot, and generating a shot static image corresponding to each scene description; performing redraw processing on a target shot static image by a fusion redraw model to obtain a redrawn target shot static image, wherein the target shot static image is a selected shot static image from the shot static images; generating a mask image of a dynamic region on the target shot static image selected by a user; inputting the target shot static image, the mask image, and set image generation prompt information into a pre-constructed image-to-video model, using feature information of the target shot static image and the mask image, and gradually predicting a next frame of video according to the image generation prompt information, and using each frame of the predicted video to form a dynamic video of the shot; performing video post-processing on the dynamic video of the shot; generating audio generation prompt information according to the theme style of the dynamic video of the shot, and generating a current video audio by a pre-constructed audio generation model according to the audio generation prompt information; adding the current video audio to the dynamic video of the shot to obtain a final dynamic video of the shot.

2. The method of claim 1, wherein, The method comprises the following steps: inputting the target shot static image into the fusion redraw model, adjusting the quality and / or size of the target shot static image by a drawing model in the fusion redraw model to obtain an adjusted target shot static image; using image edge information to redraw a defect area in the adjusted target shot static image by an image redraw plug-in in the fusion redraw model to obtain a redrawn target shot static image, wherein the image edge information is extracted from the adjusted target shot static image by an edge detection algorithm.

3. The method of claim 1, wherein, The method comprises the following steps: inputting the target shot static image, the mask image, and the set image generation prompt information into the pre-constructed image-to-video model, using the feature information of the target shot static image and the mask image, and gradually predicting a next frame of video according to the image generation prompt information, and using each frame of the predicted video to form a dynamic video of the shot; inputting the target shot static image, the mask image, and the set image generation prompt information into the pre-constructed image-to-video model, encoding the target shot static image and the mask image to obtain image feature information; combining internal parameters and the image feature information to form guidance condition information; based on the gradually diffused noise introduced by the target shot image at each time step, gradually predicting a next frame of video according to the guidance condition information and the image generation prompt information; arranging each frame of the video according to time to obtain and output the dynamic video of the shot.

4. The method of claim 1, wherein, The video post-processing on the dynamic video of the sub-shot comprises: extracting a first frame image of the dynamic video of the sub-shot; displaying the first frame image on a drawing interface and determining a mask area drawn by a user on the first frame image in response to a drawing operation of the user; replacing an image of a region at the same position as the mask area on each frame image of the dynamic video of the sub-shot with an image of a region at the same position as the mask area on the target sub-shot static image; improving resolution and consistency of the dynamic video of the sub-shot by a video enhancement model.

5. The method of claim 1, wherein, The generating of the current video audio according to the audio generation prompt information by the pre-constructed audio generation model comprises: encoding the audio data by a variational autoencoder in the audio generation model to obtain audio encoding data, and encoding the audio generation prompt information by a text encoder in the audio generation model to obtain a high-dimensional feature vector; inputting the audio encoding data, the high-dimensional feature vector and time embedding into an image segmentation network in the audio generation model, performing feature fusion and noise reduction processing by the image segmentation network using self-attention and cross-attention layers, and then collecting the audio encoding data, the high-dimensional feature vector and the time embedding to obtain the current video audio.

6. A dynamic video generation apparatus, characterized by comprising: comprise: an acquisition unit configured to acquire a text script of a sub-shot; an image generation unit configured to perform semantic extension on the text script of the sub-shot by a large language model to obtain a plurality of scene descriptions of the extended sub-shot, and generate a sub-shot static image corresponding to each scene description; a redrawing unit configured to perform redrawing processing on a target sub-shot static image by a fusion redrawing model to obtain the redrawing target sub-shot static image, wherein the target sub-shot static image is a selected sub-shot static image from the plurality of sub-shot static images; a mask image generation unit configured to generate a mask image of a dynamic region on the target sub-shot static image selected by a user; a video generation unit configured to input the target sub-shot static image, the mask image and a set image generation prompt information into a pre-constructed image video generation model, use feature information of the target sub-shot static image and the mask image to gradually predict a next frame video frame according to the image generation prompt information by the image video generation model, and use the predicted video frames to compose a dynamic video of the sub-shot; a post-processing unit configured to perform video post-processing on the dynamic video of the sub-shot; an audio generation unit configured to generate audio generation prompt information according to a theme style of the dynamic video of the sub-shot, and generate a current video audio according to the audio generation prompt information by a pre-constructed audio generation model; an adding unit configured to add the current video audio to the dynamic video of the sub-shot to obtain a final dynamic video of the sub-shot.

7. The apparatus of claim 6, wherein, The redrawing unit comprises: An image adjusting unit is configured to input the target split shot static image into the fusion redrawing model, and perform quality and / or size adjustment on the target split shot static image by using a drawing model in the fusion redrawing model, to obtain an adjusted target split shot static image. A region redrawing unit is configured to redraw a defect region in the adjusted target split shot static image by using image edge information obtained by using an edge detection algorithm on the adjusted target split shot static image, by using an image redrawing plug-in in the fusion redrawing model, to obtain a redrew target split shot static image.

8. The apparatus of claim 6, wherein, The video generation unit comprises: A feature acquisition unit is configured to input the target split shot static image, the mask image, and set image generation prompt information into a pre-constructed image video generation model, encode the target split shot static image and the mask image, and obtain image feature information. A feature combination unit is configured to combine internal parameters and the image feature information to obtain guidance condition information. A prediction unit is configured to gradually predict a next frame of video based on gradually diffused noise introduced by the target split shot image at each time step, the guidance condition information, and the image generation prompt information. An image combination unit is configured to arrange the video frames in time sequence, obtain a dynamic video of the split shot, and output the dynamic video.

9. An electronic device, comprising: comprise: a memory and a processor; wherein the memory is configured to store a program; the processor is configured to execute the program, and when the program is executed, specifically used to implement the dynamic video generation method in any one of claims 1 to 5.

10. A computer storage medium, characterized in that, a computer program is stored, and when the computer program is executed, used to implement the dynamic video generation method in any one of claims 1 to 5.

Images