Video generation method and device and storage medium

Abstract

The embodiment of the invention provides a video generation method and device and a storage medium, and the method comprises the steps: obtaining the input information of a user, and the input information comprises an image and a user editing instruction; performing intention recognition on the image and the user editing instruction to obtain image main body information, image static information description and image dynamic information description; editing the image main body information according to the image static information description to obtain a result image; acquiring the confusion degree of the image dynamic information description, judging the reasonability of the statement generated by the image dynamic information description according to the confusion degree, and if the statement is reasonable, generating a dynamic video for the text described by the image dynamic information; if not, determining a dynamic video according to user prompt text semantics in the image dynamic information description; and generating a target video according to the result image and the dynamic video. According to the method, automatic processing of variable address space information tasks in different application scenes is realized, and the security of the system is improved.

Description

Technical Field

[0001] The present application relates to the field of graphic, text and video, and in particular to a method, device and storage medium for generating a video. Background Art

[0002] Existing enterprise internal application systems require users to register information, and the registration information will be integrated into dynamic information, such as generating a video information as material to be integrated into the user registration information. For example, when registering, new employees of the enterprise upload one or more personal photos, or generate a video dynamic information based on the pictures provided by the user. These help to improve the diversity of system information and enhance user experience.

[0003] However, in the prior art, dynamic videos generated based on portrait images usually use fixed control modes to achieve relatively fixed scenes. Both the control of the portrait and the content of the video background are relatively simple, and the controllable factors are limited, which affects the security of the system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device and storage medium for generating a video, which realizes the automated processing of variable address space information tasks in different application scenarios and improves the security of the system.

[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides a method for generating a video, the method comprising:

[0006] Acquiring user input information, wherein the input information includes an image and a user editing instruction;

[0007] Performing intention recognition on the image and the user editing instruction to obtain processing requirement information, wherein the processing requirement information includes image main body information, image static information description and image dynamic information description;

[0008] Editing the main information of the image according to the image static information description to obtain a result image;

[0009] Obtaining the perplexity of the image dynamic information description, judging the rationality of the image dynamic information description generation sentence according to the perplexity, and if it is reasonable, generating a dynamic video for the text of the image dynamic information description; if it is unreasonable, determining the dynamic video according to the semantics of the user prompt text in the image dynamic information description;

[0010] A target video is generated according to the result image and the dynamic video.

[0011] Optionally, the performing intention recognition on the image and the user editing instruction to obtain processing requirement information includes:

[0012] Performing image-text conversion on the image to obtain semantic information of the image;

[0013] According to the user editing instruction, the semantic information is subjected to intention recognition to obtain image subject information, image static information description and image dynamic information description.

[0014] Optionally, the editing the image main body information according to the image static information description to obtain a result image includes:

[0015] Obtaining static attribute characteristics of the image subject information;

[0016] The image main body information is edited according to the static attribute characteristics and the image static information description to obtain a result image.

[0017] Optionally, the obtaining the perplexity of the image dynamic information description and judging the rationality of the generated sentence of the image dynamic information description according to the perplexity includes:

[0018] Obtaining the conditional probability of each word in the image dynamic information description;

[0019] Determining the perplexity of each word in the image dynamic information description according to the conditional probability;

[0020] When the perplexity meets a certain threshold, the generated sentence describing the image dynamic information is reasonable, otherwise it is unreasonable.

[0021] Optionally, generating a dynamic video from the text describing the dynamic information of the image includes:

[0022] Constructing a first text video model based on the text describing the dynamic information of the image by the neural network;

[0023] Generate a dynamic video according to the first text video model.

[0024] Optionally, determining the dynamic video according to the semantics of the user prompt text in the image dynamic information description includes:

[0025] Set up video list;

[0026] Combined with the semantic similarity, a video with semantic similarity to the user prompt text in the image dynamic information description is obtained from the video list as a dynamic video.

[0027] Optionally, generating a target video according to the result image and the dynamic video includes:

[0028] Performing DDIM inversion processing on the result image and dynamic video to obtain latent space vectors, spatial features, spatial attention vectors, and temporal attention vectors of each frame of the image;

[0029] Constructing a second text video model according to the latent space vector, the spatial feature, the spatial attention vector and the temporal attention vector;

[0030] The spatial features and the time domain features are injected into the text video decoding data of the second text video model to generate the target video.

[0031] On the other hand, the present invention also provides a device for generating a video, the device comprising:

[0032] An acquisition module, used to acquire user input information, wherein the input information includes an image and a user editing instruction;

[0033] A first processing module, configured to perform intention recognition on the image and the user editing instruction to obtain processing requirement information, wherein the processing requirement information includes image main body information, image static information description and image dynamic information description;

[0034] A second processing module, configured to edit the main image information according to the image static information description to obtain a result image;

[0035] The third processing module is used to obtain the perplexity of the image dynamic information description, and judge the rationality of the image dynamic information description generation sentence according to the perplexity. If it is reasonable, a dynamic video is generated for the text of the image dynamic information description; if it is unreasonable, a dynamic video is determined according to the semantics of the user prompt text in the image dynamic information description;

[0036] The fourth processing module is used to generate a target video according to the result image and the dynamic video.

[0037] Optionally, the obtaining the perplexity of the image dynamic information description and judging the rationality of the generated sentence of the image dynamic information description according to the perplexity includes:

[0038] Obtaining the conditional probability of each word in the image dynamic information description;

[0039] Determining the perplexity of each word in the image dynamic information description according to the conditional probability;

[0040] When the perplexity meets a certain threshold, the generated sentence describing the image dynamic information is reasonable, otherwise it is unreasonable.

[0041] On the other hand, the present invention further provides a processor configured to execute the method for generating a video as described above.

[0042] On the other hand, the present invention further proposes a machine-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the method for generating a video as described above.

[0043] On the other hand, the present invention further proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for generating a video as described above.

[0044] A method for generating a video in the present application includes: obtaining user input information, the input information includes an image and a user editing instruction; performing intent recognition on the image and the user editing instruction to obtain processing requirement information, the processing requirement information includes image main body information, image static information description and image dynamic information description; editing the image main body information according to the image static information description to obtain a result image; obtaining the perplexity of the image dynamic information description, judging the rationality of the image dynamic information description generation statement according to the perplexity, if it is reasonable, generating a dynamic video for the text of the image dynamic information description; if it is unreasonable, determining the dynamic video according to the semantics of the user prompt text in the image dynamic information description; generating a target video according to the result image and the dynamic video. The method obtains the target video by performing intent recognition, editing and perplexity analysis on the image and the user editing instruction, thereby realizing the controllable generation of image information and video information, realizing the automated processing of variable address space information tasks in different application scenarios, and improving the security of the system.

[0045] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0047] Figure 1 A schematic diagram of a process of generating a video according to the present application is shown schematically;

[0048] Figure 2 A schematic diagram of a process according to an embodiment of the present application is shown;

[0049] Figure 3 The structure block diagram of a device for generating a video according to the present application is schematically shown;

[0050] Figure 4 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown.

[0051] Description of Reference Numerals

[0052] 300- means for generating video;

[0053] 301-Get module;

[0054] 302-first processing module;

[0055] 303-second processing module;

[0056] 304-third processing module;

[0057] 305-the fourth processing module. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0060] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0061] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In addition, it should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0062] This application proposes a method for generating a video, such as Figure 1 As shown, the method includes: step S101 is to obtain user input information, wherein the input information includes an image and a user editing instruction, wherein the user editing instruction includes specific requirements and goals for editing the image.

[0063] Step S102 is to perform intention recognition on the image and the user editing instruction to obtain processing requirement information, wherein the processing requirement information includes image main body information, image static information description and image dynamic information description.

[0064] According to a specific implementation, the intention recognition of the image and the user editing instruction to obtain the processing requirement information includes: converting the image into text to obtain the semantic information of the image; and according to the user editing instruction, the intention recognition of the semantic information to obtain the image subject information, the image static information description and the image dynamic information description. For example, if the image input by the user is a portrait, and the user editing instruction is "process the person in the photo to be younger", then the semantic information of the image is the person.

[0065] Specifically, the intent recognition refers to the technology of automatically identifying the core needs and intentions of users by analyzing the text or voice input by the user. In the search field, intent recognition can improve the accuracy of search and user experience. For example, when a user enters "Legend of Sword and Fairy", the search engine can recognize that the user's intention is about the TV series rather than games or news. By returning the TV series as a result to the user, the user's search time and clicks can be saved.

[0066] The semantic information of the edited image description, the edited description of the static content of the image and the description of the dynamic content generated from the static content of the image (i.e., the image main information, the image static information description and the image dynamic information description) are generated according to the input image and the user editing instructions.

[0067] The present application can be implemented by combining local fine-tuning of the GLMV3 model with calling GLMV4. The main solution is based on the prompt method (providing input to the model to guide it to generate text or instructions for specific output) to guide the large model to give results.

[0068] Step S103 is to edit the main image information according to the image static information description to obtain a result image.

[0069] According to a specific implementation, the editing of the image main information according to the image static information description to obtain a result image includes: obtaining static attribute characteristics of the image main information; and editing the image main information according to the static attribute characteristics and the image static information description to obtain a result image.

[0070] According to a specific implementation, the main information of the image is obtained through the GLMV4 model (an AI model), the dynamic content of the image is prompted to learn to obtain the image dynamic information description, the static content of the image is extracted according to the user editing instruction, and the image static information description is generated, and then the main information of the image, the image static information description and the image dynamic information description are input based on the prompt learning by using the GLM4 language model to obtain the modified result image. For example, if the main information of the image is a middle-aged male with curly hair, the image static information description is that the hair is straight, and the image dynamic information description is blinking, then the result image is a middle-aged male with straight hair, blinking.

[0071] When editing the main image information according to the image static information description, a processing method based on a diffusion model is mainly used, which requires ensuring that the main image information of the edited image (ie, the result after editing by a person) is as consistent as possible with the input.

[0072] The editing method based on the diffusion model mainly includes diffusion reconstruction of the original image, processing based on the DDIMInversion method, portrait editing based on prompts, LORA combination mechanism and prompt2prompt mechanism.

[0073] According to a specific implementation, the diffusion reconstruction of the original image includes: performing diffusion reconstruction on the original image, that is, obtaining a predicted noise vector that satisfies the accurate reconstruction of the image based on the visual encoding of the original image and the text encoding of the semantic text description of the original image. To realize controllable editing based on text based on the DDPM method, it is first necessary to establish a controllable reconstruction generation process based on the original description of noise. The input is the original ID photo image and the image semantic text information (image subject information), and the noise vector information that can accurately restore the original image is obtained. The image passes through VAE as an encoder to obtain an embedding vector, and the multimodal Clip model is used as a text encoder. The semantic text information of the image is encoded to obtain a text vector encoding.

[0074] In order to reduce the impact of cumulative errors that may be encountered in multiple noise predictions, the DDIM Inversion method introduces a null-text embedding optimization processing mechanism. The obtained predicted noise vector and optimized null-text embedding can be used for subsequent prompt-based image editing.

[0075] The prompt-based image editing is used to complete the static editing operation of the image, that is, based on the predicted noise vector and the corresponding picture description, it is realized by the combination of two strategies based on the prompt2prompt mechanism and the LORA combination mechanism. The overall strategy is to divide the static attribute editing prompt of the image into a combination of single puppets, and then map it to the existing LORA image editing operation category, and process it through the LOAR combination mechanism to realize editing; if it cannot be mapped to the existing LORA image editing operation category, the prompt2prompt mechanism is used.

[0076] Low-Rank Adaptation (LoRA) is a method for fine-tuning large pre-trained language models. The core idea is to add small, low-rank matrices in the key layers of the model to adjust the model's behavior without changing the entire model structure. The present invention uses decoding-centric multi-LoRA combination technology (i.e., LoRASwitch and LoRA Composite) to enable image editing to manipulate more LoRA transformations and achieve better results.

[0077] The promt-to-promt mechanism is based on the semantic coding vectors of the text description before and after the modification, and calls a preset diffusion model to perform a reverse diffusion process, so as to edit the noise vector based on the cross-attention mechanism, iteratively generate the corresponding target latent vector, and obtain the result image by decoding the target latent vector.

[0078] Step S104 is to obtain the perplexity of the image dynamic information description, and judge the rationality of the image dynamic information description generation sentence according to the perplexity. If it is reasonable, a dynamic video is generated for the text of the image dynamic information description; if it is unreasonable, a dynamic video is determined according to the semantics of the user prompt text in the image dynamic information description. The perplexity is a measurement indicator for judging a probability model or a probability distribution prediction, and can be used to evaluate the quality of the model.

[0079] According to a specific implementation, the obtaining of the perplexity of the image dynamic information description and judging the rationality of the sentence generated by the image dynamic information description according to the perplexity include: obtaining the conditional probability of each word in the image dynamic information description; confirming the perplexity of each word in the image dynamic information description according to the conditional probability; when the perplexity meets a certain threshold, the sentence generated by the image dynamic information description is reasonable, otherwise it is unreasonable. Generally speaking, a lower perplexity value indicates that the sentence is more fluent.

[0080] The method of generating a dynamic video from the text describing the dynamic information of the image includes: constructing a first text video model from the text describing the dynamic information of the image according to a neural network; and generating a dynamic video according to the first text video model.

[0081] Determining the dynamic video according to the semantics of the user prompt text in the image dynamic information description includes: setting a video list; and obtaining a video with similar semantics to the user prompt text in the image dynamic information description from the video list as a dynamic video in combination with semantic similarity.

[0082] When the prompt provided by the user does not have a description that can support the generation of dynamic videos, you can combine semantic similarity and select a video with the closest semantics from the optional video list as reference information to generate dynamic feature information. That is, the target dynamic feature information is generated based on the picture and reference video as input. For the selected video, the latent space vector of each frame is obtained through DDIM inversion processing, and its spatial features, spatial attention vectors, and temporal attention vectors can be obtained. By introducing the Image2Video model and utilizing the feature injection mechanism of its decoding part, the injection mechanism of spatial features and temporal features is realized to enhance the background consistency and overall structural consistency of the edited video. When the prompt description provided by the user can support the construction of dynamic videos, the Vincent video model architecture established by the diffusion model based on the Transformer backbone is used.

[0083] Step S105 is to generate a target video according to the result image and the dynamic video.

[0084] According to a specific implementation manner, generating a target video based on the result image and the dynamic video includes: performing DDIM inversion processing on the result image and the dynamic video to obtain a latent space vector, spatial features, spatial attention vector and temporal attention vector for each frame of the image; constructing a second text video model based on the latent space vector, spatial features, spatial attention vector and temporal attention vector; and injecting spatial features and time domain features into the text video decoding data of the second text video model to generate a target video.

[0085] The application of DDIM inversion technology in image generation and editing includes: first, mapping the input to the hidden space through the VAE variable encoder, and then using sampling technology to gradually denoise and generate the target image. DDIM inversion starts from a clear image and gradually adds noise to achieve the reverse process from the hidden space to the input image. Through forward sampling and inversion functions, the image can be edited and modified in different steps to achieve closeness or changes to the original image.

[0086] Figure 2 A schematic diagram of a process flow according to an embodiment of the present application is shown schematically. Figure 2 As shown, first, the user inputs the request information, and the user's processing requirement information is determined through intention understanding. The processing requirement information includes the image main body information, the image static information description and the image dynamic information description. If the above information cannot be effectively determined, the judgment fails. Otherwise, subsequent processing is performed; then, the image main body information is edited according to the image static information description to obtain the edited static picture (i.e., the result image); secondly, it is determined whether the image dynamic information description can generate a video. If it can be generated, the text is processed to generate the video. If not, according to the semantics of the user's prompt text information, a reference video with similar semantics is selected for subsequent processing; finally, the target video is generated based on the result image and the dynamic video.

[0087] On the other hand, the present invention also provides a device for generating a video, such as Figure 3 As shown, the device 300 for generating a video includes: an acquisition module 301, which is used to acquire user input information, and the input information includes an image and a user editing instruction; a first processing module 302, which is used to perform intent recognition on the image and the user editing instruction to obtain processing requirement information, and the processing requirement information includes image main information, image static information description and image dynamic information description; a second processing module 303, which is used to edit the image main information according to the image static information description to obtain a result image; a third processing module 304, which is used to acquire the perplexity of the image dynamic information description, and judge the rationality of the image dynamic information description generation statement according to the perplexity, if it is reasonable, generate a dynamic video for the text of the image dynamic information description; if it is unreasonable, determine the dynamic video according to the semantics of the user prompt text in the image dynamic information description; a fourth processing module 305, which is used to generate a target video according to the result image and the dynamic video.

[0088] Optionally, the performing of intent recognition on the image and the user editing instruction to obtain the processing requirement information includes: performing image-text conversion on the image to obtain the semantic information of the image; performing intent recognition on the semantic information according to the user editing instruction to obtain the image subject information, the image static information description and the image dynamic information description. The performing of editing the image subject information according to the image static information description to obtain the result image includes: obtaining the static attribute characteristics of the image subject information; and editing the image subject information according to the static attribute characteristics and the image static information description to obtain the result image.

[0089] The obtaining of the perplexity of the image dynamic information description and judging the rationality of the sentence generated by the image dynamic information description according to the perplexity include: obtaining the conditional probability of each word in the image dynamic information description; confirming the perplexity of each word in the image dynamic information description according to the conditional probability; when the perplexity meets a certain threshold, the sentence generated by the image dynamic information description is reasonable, otherwise it is unreasonable.

[0090] Determining the dynamic video according to the semantics of the user prompt text in the dynamic information description of the image includes: setting a video list; combining semantic similarity, obtaining a video with similar semantics to the user prompt text in the dynamic information description of the image from the video list as a dynamic video. Generating the target video according to the result image and the dynamic video includes: performing DDIM inversion processing on the result image and the dynamic video to obtain the latent space vector, spatial features, spatial attention vector and time attention vector of each frame image; constructing a second text video model according to the latent space vector, spatial features, spatial attention vector and time attention vector; injecting spatial features and time domain features into the text video decoding data of the second text video model to generate the target video.

[0091] The device provides an effective auxiliary mechanism from the perspective of dynamic and automatic generation of information, which can support relevant application scenarios of various applications within the enterprise.

[0092] A method for generating a video in the present application includes: obtaining user input information, the input information includes an image and a user editing instruction; performing intent recognition on the image and the user editing instruction to obtain processing requirement information, the processing requirement information includes image main body information, image static information description and image dynamic information description; editing the image main body information according to the image static information description to obtain a result image; obtaining the perplexity of the image dynamic information description, judging the rationality of the image dynamic information description generation statement according to the perplexity, if it is reasonable, generating a dynamic video for the text of the image dynamic information description; if it is unreasonable, determining the dynamic video according to the semantics of the user prompt text in the image dynamic information description; generating a target video according to the result image and the dynamic video. The method obtains the target video by performing intent recognition, editing and perplexity analysis on the image and the user editing instruction, thereby realizing the controllable generation of image information and video information, realizing the automated processing of variable address space information tasks in different application scenarios, and improving the security of the system.

[0093] An embodiment of the present application provides a storage medium on which a program is stored. When the program is executed by a processor, the method for generating a video is implemented.

[0094] An embodiment of the present application provides a processor, which is used to run a program, wherein the program executes the above-mentioned method for generating a video when running.

[0095] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a method for data processing is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0096] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method for generating a video according to any embodiment of the present application is implemented.

[0097] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the method for generating a video according to any embodiment of the present invention.

[0098] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0103] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0104] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0105] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0106] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0107] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating a video, characterized in that: The method includes: Acquiring user input information, wherein the input information includes an image and a user editing instruction; Performing intention recognition on the image and the user editing instruction to obtain processing requirement information, wherein the processing requirement information includes image main body information, image static information description and image dynamic information description; Editing the main information of the image according to the image static information description to obtain a result image; Obtaining the perplexity of the image dynamic information description, judging the rationality of the image dynamic information description generation sentence according to the perplexity, and if it is reasonable, generating a dynamic video for the text of the image dynamic information description; if it is unreasonable, determining the dynamic video according to the semantics of the user prompt text in the image dynamic information description; A target video is generated according to the result image and the dynamic video.

2. The method according to claim 1, characterized in that The performing intention recognition on the image and the user editing instruction to obtain processing requirement information includes: Performing image-text conversion on the image to obtain semantic information of the image; According to the user editing instruction, the semantic information is subjected to intention recognition to obtain image subject information, image static information description and image dynamic information description.

3. The method according to claim 1, characterized in that The step of editing the main image information according to the image static information description to obtain a result image includes: Obtaining static attribute characteristics of the image subject information; The image main body information is edited according to the static attribute characteristics and the image static information description to obtain a result image.

4. The method according to claim 1, characterized in that The obtaining of the perplexity of the image dynamic information description and judging the rationality of the generated sentence of the image dynamic information description according to the perplexity include: Obtaining the conditional probability of each word in the image dynamic information description; Determining the perplexity of each word in the image dynamic information description according to the conditional probability; When the perplexity meets a certain threshold, the generated sentence describing the image dynamic information is reasonable, otherwise it is unreasonable.

5. The method according to claim 1, characterized in that The generating a dynamic video from the text describing the dynamic information of the image comprises: Constructing a first text video model based on the text describing the dynamic information of the image by the neural network; Generate a dynamic video according to the first text video model.

6. The method according to claim 1, characterized in that The determining of the dynamic video according to the semantics of the user prompt text in the image dynamic information description includes: Set up video list; Combined with the semantic similarity, a video with semantic similarity to the user prompt text in the image dynamic information description is obtained from the video list as a dynamic video.

7. The method according to claim 1, characterized in that The step of generating a target video according to the result image and the dynamic video comprises: Performing DDIM inversion processing on the result image and dynamic video to obtain latent space vectors, spatial features, spatial attention vectors, and temporal attention vectors of each frame of the image; Constructing a second text video model according to the latent space vector, the spatial feature, the spatial attention vector and the temporal attention vector; The spatial features and the time domain features are injected into the text video decoding data of the second text video model to generate the target video.

8. A device for generating a video, characterized in that: The device includes: An acquisition module, used to acquire user input information, wherein the input information includes an image and a user editing instruction; A first processing module, configured to perform intention recognition on the image and the user editing instruction to obtain processing requirement information, wherein the processing requirement information includes image main body information, image static information description and image dynamic information description; A second processing module, configured to edit the main image information according to the image static information description to obtain a result image; The third processing module is used to obtain the perplexity of the image dynamic information description, and judge the rationality of the image dynamic information description generation sentence according to the perplexity. If it is reasonable, a dynamic video is generated for the text of the image dynamic information description; if it is unreasonable, a dynamic video is determined according to the semantics of the user prompt text in the image dynamic information description; The fourth processing module is used to generate a target video according to the result image and the dynamic video.

9. The device according to claim 8, characterized in that The obtaining of the perplexity of the image dynamic information description and judging the rationality of the generated sentence of the image dynamic information description according to the perplexity include: Obtaining the conditional probability of each word in the image dynamic information description; Determining the perplexity of each word in the image dynamic information description according to the conditional probability; When the perplexity meets a certain threshold, the generated sentence describing the image dynamic information is reasonable, otherwise it is unreasonable.

10. A processor, characterized in that: The method is configured to execute the method for generating a video according to any one of claims 1 to 7.

11. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to perform the method for generating a video according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method for generating a video according to any one of claims 1 to 7.

Images