Video generation method and device, equipment and storage medium

Abstract

The invention provides a video generation method and device, equipment and a storage medium, and belongs to the technical field of computers. According to the method, multiple pieces of reference information are coded respectively, the feature of the reference object instance corresponding to each piece of reference information and the feature of the description text can be bound, and the features of different pieces of reference information are unbound, so that the confusion of different pieces of reference information is avoided; therefore, the attribute of each reference object instance can be accurately restored in the generated video. Moreover, the scheme does not depend on regional constraints, so that the generated video has diversity and motion smoothness.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generation method, device, equipment and storage medium. Background Art

[0002] With the development of computer technology, video generation schemes have emerged that generate video content of specific object instances through image guidance. In practical applications, users may want to generate video content containing multiple specific object instances. Therefore, how to implement the above scheme to meet the needs of users is a research direction.

[0003] Currently, there are image generation schemes based on multiple object instances in an image. The scheme usually imposes region constraints on each object instance when generating an image to avoid confusion of the attributes of the object instance.

[0004] However, since the content of the video is dynamic, the position of the object instance will change over time, and the regional constraints will affect the diversity and motion smoothness of the generated video content, resulting in the image generation scheme based on multiple instances of the image being unable to be directly transferred to the video generation scenario. Therefore, a new video generation scheme is urgently needed. Summary of the invention

[0005] The present disclosure provides a video generation method, apparatus, device and storage medium. The solution unbinds the features of different reference information to avoid confusion between different reference information, so that the attributes of each reference object instance can be accurately restored in the generated video. In addition, since the above solution does not rely on regional constraints, the generated video has diversity and motion smoothness.

[0006] According to one aspect of an embodiment of the present disclosure, a video generation method is provided, the method comprising:

[0007] Encode the text prompt information to obtain text prompt features;

[0008] Encoding the plurality of reference information respectively to obtain a plurality of semantic visual features, each reference information comprising a reference image and a description text, the reference image comprising a reference object instance, and the description text being used to describe the corresponding reference object instance;

[0009] A target video is generated based on the text prompt feature and the plurality of semantic visual features.

[0010] According to another aspect of an embodiment of the present disclosure, a video generating device is provided, the device comprising:

[0011] A first encoding unit is configured to encode the text prompt information to obtain a text prompt feature;

[0012] A second encoding unit is configured to encode the multiple reference information respectively to obtain multiple semantic visual features, each reference information includes a reference image and a description text, the reference image includes a reference object instance, and the description text is used to describe the corresponding reference object instance;

[0013] A generating unit is configured to generate a target video based on the text prompt feature and the multiple semantic visual features.

[0014] In some embodiments, the second encoding unit is configured to, for any reference information, visually encode the reference image in the reference information to obtain the visual features of the reference object instance; semantically encode the descriptive text in the reference information to obtain the textual semantic features of the descriptive text; and fuse the visual features and the textual semantic features to obtain the semantic visual features of the reference information.

[0015] In some embodiments, the first encoding unit is configured to align the visual features and the text semantic features; and fuse the aligned visual features and the text semantic features based on a cross-attention mechanism.

[0016] In some embodiments, the generation unit is configured to fuse the video features of the diffusion model with the text prompt features to obtain intermediate fusion features, where the video features of the diffusion model are hidden features in the process of generating video; fuse the intermediate fusion features with the multiple semantic visual features in sequence to obtain video features; and decode the video features to obtain the target video.

[0017] In some embodiments, the generating unit is configured to fuse the video features of the diffusion model with the text prompt features through a cross attention mechanism to obtain the intermediate fusion features.

[0018] In some embodiments, the generation unit is configured to obtain multiple semantic visual features in sequence through a feature injection module; each time a semantic visual feature is obtained, the semantic visual feature is fused with the intermediate fusion feature to obtain an intermediate video feature; in response to the fusion being completed, the video feature is obtained.

[0019] In some embodiments, the first encoding unit is further configured to obtain input text prompt information; perform compliance verification on the text prompt information; and display prompt information if the text prompt information fails the verification.

[0020] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0021] one or more processors;

[0022] A memory for storing program codes executable by the processor;

[0023] The processor is configured to execute the program code to implement the above-mentioned video generation method.

[0024] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned video generating method.

[0025] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned video generating method is implemented.

[0026] The disclosed embodiment provides a video generation scheme, which can bind the features of the reference object instance and the features of the description text corresponding to each reference information by encoding multiple reference information respectively, and unbind the features of different reference information to avoid confusion between different reference information, so that the attributes of each reference object instance can be accurately restored in the generated video. In addition, since the above scheme does not rely on regional constraints, the generated video has diversity and motion smoothness.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0029] Figure 1 It is a schematic diagram of an implementation environment of a video generation method according to an exemplary embodiment.

[0030] Figure 2 The figure is a flowchart of a video generating method according to an exemplary embodiment.

[0031] Figure 3 The figure is a flowchart of another video generating method according to an exemplary embodiment.

[0032] Figure 4 The invention is a flowchart of encoding reference information according to an exemplary embodiment.

[0033] Figure 5 It is a flowchart of a video generation method provided according to an exemplary embodiment.

[0034] Figure 6 It is a block diagram of a video generating device according to an exemplary embodiment.

[0035] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0038] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the images involved in this disclosure are all obtained with full authorization.

[0039] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a video generation method according to an exemplary embodiment. Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0040] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and the application is used to generate a video according to information input by a user. The application is associated with the server 102, and the server 102 provides background services to the terminal 101.

[0041] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0042] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0043] Figure 2 is a flowchart of a video generation method according to an exemplary embodiment. Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0044] In step S201, the text prompt information is encoded to obtain text prompt features.

[0045] In the disclosed embodiment, the text prompt information is information input by the user, which is used to play a prompting role in the process of generating a video. The text prompt information includes at least one prompt word. By encoding the at least one prompt word, a text prompt feature can be obtained.

[0046] In step S202, multiple reference information are encoded respectively to obtain multiple semantic visual features.

[0047] In the disclosed embodiment, each reference information includes a pair of reference images and description text. The reference image includes a reference object instance, and the description text is used to describe the corresponding reference object instance. The reference object instance is a concept contained in the video to be generated, such as a person, an animal, a building, a landscape, and a style.

[0048] In step S203, a target video is generated based on the text prompt feature and multiple semantic visual features.

[0049] In the disclosed embodiment, multiple semantic visual features can guide the generation of video content, and text prompt features can obtain video content of different styles, roles or scenes. Therefore, based on the text prompt features and multiple semantic visual features, a target video that meets the above features can be generated.

[0050] The disclosed embodiment provides a video generation scheme, which can bind the features of the reference object instance and the features of the description text corresponding to each reference information by encoding multiple reference information respectively, and unbind the features of different reference information to avoid confusion between different reference information, so that the attributes of each reference object instance can be accurately restored in the generated video. In addition, since the above scheme does not rely on regional constraints, the generated video has diversity and motion smoothness.

[0051] In some embodiments, multiple reference information are encoded respectively to obtain multiple semantic visual features, including:

[0052] For any reference information, visually encode the reference image in the reference information to obtain the visual features of the reference object instance;

[0053] Semantically encode the description text in the reference information to obtain text semantic features of the description text;

[0054] The visual features and text semantic features are integrated to obtain the semantic visual features of the reference information.

[0055] In the disclosed embodiment, this operation can effectively integrate the visual content in the reference information with the semantic information of the descriptive text, and accurately extract the semantic visual features of the reference information, so that the electronic device can better understand the reference object instance, provide a more accurate and comprehensive basis for subsequent video generation tasks, and improve the processing effect.

[0056] In some embodiments, the visual features and the text semantic features are integrated to obtain the semantic visual features of the reference information, including:

[0057] Align visual features with textual semantic features;

[0058] Based on the cross-attention mechanism, the aligned visual features and textual semantic features are fused.

[0059] In the disclosed embodiment, by first aligning the visual features and the text semantic features and then fusing them based on the cross-attention mechanism, the relevant information of the image and the text can be more accurately and comprehensively combined, thereby providing a more accurate and effective feature basis for the subsequent generation of the video and improving the processing effect.

[0060] In some embodiments, generating a target video based on the text prompt feature and multiple semantic visual features includes:

[0061] The video features of the diffusion model are fused with the text prompt features to obtain the intermediate fusion features. The video features of the diffusion model are the hidden features in the process of generating the video.

[0062] The intermediate fusion features are sequentially fused with multiple semantic visual features to obtain video features;

[0063] Decode the video features to obtain the target video.

[0064] In the disclosed embodiment, by sequentially fusing the latent video features of the diffusion model when generating a video with the text prompt features and multiple semantic visual features, and obtaining the target video through decoding, by leveraging the multiple iteration advantages of the diffusion model, it is possible to fully integrate various feature information and effectively improve the accuracy, richness and conformity of the target video generation with user needs.

[0065] In some embodiments, the video features of the diffusion model are fused with the text prompt features to obtain intermediate fusion features, including:

[0066] Through the cross-attention mechanism, the video features and text prompt features of the diffusion model are fused to obtain the intermediate fusion features.

[0067] In the disclosed embodiment, the intermediate fusion features are obtained by fusing the video features and text prompt features of the diffusion model through the cross-attention mechanism, which enables the electronic device to accurately capture the relationship and interaction information between the two and effectively combine the relevant characteristics, thereby laying a good foundation for the subsequent generation of target videos that better meet the needs and have higher quality.

[0068] In some embodiments, the intermediate fusion feature is sequentially fused with a plurality of semantic visual features to obtain a video feature, including:

[0069] Through the feature injection module, multiple semantic visual features are obtained in sequence;

[0070] Whenever a semantic visual feature is obtained, the semantic visual feature is fused with the intermediate fusion feature to obtain the intermediate video feature;

[0071] In response to the fusion being completed, video features are obtained.

[0072] In the disclosed embodiment, by introducing a feature injection module into the diffusion model, multiple semantic visual features can be effectively acquired and integrated, ensuring that their features are deeply integrated with visual features and text prompt features. When the electronic device operates according to this process, video features with a high degree of integration and that can better meet the needs can be generated, thereby improving the quality and effect of target video generation.

[0073] In some embodiments, the method further comprises:

[0074] Get the input text prompt information;

[0075] Conduct compliance check on text prompt information;

[0076] If the text prompt message fails to pass the verification, the prompt message is displayed.

[0077] In the disclosed embodiment, by verifying the text prompt information, compliant content can be effectively screened out, avoiding subsequent process errors or invalid results due to non-compliant input, improving the accuracy and stability of the overall video generation process, and at the same time, by promptly prompting users to re-enter, optimizing the user interaction experience and ensuring the effectiveness of system operation.

[0078] Above Figure 2 The figure shows a flow chart of a video generation method of the present disclosure. The video generation scheme provided by the present disclosure is further described below. Figure 3 is a flowchart of another video generation method according to an exemplary embodiment. Figure 3 The method is performed by an electronic device and comprises the following steps:

[0079] In step S301, the input text prompt information is obtained.

[0080] In the disclosed embodiment, the text prompt information is used to guide and instruct the generation of video content. In simple terms, the text prompt information is to describe various aspects of the video to be generated in words, such as the style, scene, and scenario of the video.

[0081] For example, the text prompt information may include the theme of the video, such as "generate a video about the sunrise at the seaside". This theme clearly indicates the main scene and general content direction of the video. The text prompt information can also describe the style of the video, such as "show the city night view in the style of oil painting". Here, "oil painting style" is the text prompt information that prompts the style, so that the generated video screen has the texture of oil painting, such as the brushstroke effect, color intensity, etc. will be generated according to the characteristics of oil painting. The text prompt information can also include the plot development of the video. For example, "generate a video, the beginning is a little girl lost in the forest, and then she meets a talking elf, and the elf takes her to find the way home." Such text prompt information can construct a complete plot chain, including characters (little girl and elf), scenes (forest), and the beginning, development and ending of the plot.

[0082] In some embodiments, the electronic device can verify the text prompt information input by the user. Accordingly, the electronic device obtains the input text prompt information. Then, the electronic device performs compliance verification on the text prompt information. In the case where the text prompt information fails to pass the verification, the electronic device displays a prompt message to prompt the user that the text prompt information currently input is not compliant, and asks the user to re-enter. If the text prompt information passes the verification, the electronic device can execute subsequent processes, or display that the input is compliant. By verifying the text prompt information, compliant content can be effectively screened out, avoiding errors in subsequent processes or invalid results due to non-compliant input, improving the accuracy and stability of the overall video generation process, and at the same time, by promptly prompting the user to re-enter, the user interaction experience is optimized and the effectiveness of the system operation is guaranteed.

[0083] In some embodiments, the user can input the above text prompt information through the input box, or input the text prompt information through voice, or select template information close to their needs from the preset prompt information template, and then modify and supplement it. The embodiment of the present disclosure does not limit the input method of the text prompt information.

[0084] In step S302, the text prompt information is encoded to obtain text prompt features.

[0085] In the disclosed embodiment, since the text prompt information is usually a description in natural language, it is difficult for the electronic device to directly generate a video based on such natural language text. Therefore, the text prompt information is encoded and converted into text prompt features that can be understood and processed by a computer.

[0086] Optionally, the electronic device may encode the text prompt information by means of a text feature encoder, wherein the text feature encoder is used to extract text prompt words from the text prompt information, and then extract the extracted text prompt words into a complete text feature, namely, a text prompt feature.

[0087] Optionally, the text feature encoder can encode the text prompt information by word vector encoding. For example, the text feature encoder can convert each word in the text prompt information into a vector. Suppose the text prompt information is "Generate a happy birthday party video". The text feature encoder converts the words "happy", "birthday party", "video" and so on into corresponding word vectors. These word vectors are points in a high-dimensional space, and the dimensions of the vectors may be hundreds of dimensions. For example, the vector of the word "happy" may be represented as [0.1, 0.2, -0.3, ...], and these values ​​represent the position of the word in this high-dimensional space. Word vectors can capture the semantic information of words. For example, because "happy" and "pleasure" have similar semantics, the distance between word vectors in space will be relatively close.

[0088] Optionally, the text feature encoder may also be a deep learning encoding model, such as BERT, etc., which is not limited in the embodiments of the present disclosure.

[0089] In step S303, a plurality of reference information is obtained, each of which includes a reference image and a description text. The reference image includes a reference object instance, and the description text is used to describe the corresponding reference object instance.

[0090] In the embodiment of the present disclosure, the reference information is used to assist in generating a video. Each reference information consists of two parts: a reference image and a description text.

[0091] The reference image is used to intuitively present the visual content. For example, the reference image may be a photo of a cat, which clearly shows the cat's appearance, fur color, posture and other visual features. Each cat in the photo is a reference object instance, that is, the reference object instance is an individual object with actual form and characteristics.

[0092] The description text is used to describe the corresponding reference object instance in words. Taking the photo of a cat as an example, the description text may read "this is a white cat". Such description text can accurately convey the information of the reference object instance.

[0093] In step S304, multiple reference information are encoded respectively to obtain multiple semantic visual features.

[0094] In the embodiment of the present disclosure, for each reference information, the reference information is encoded separately to obtain the semantic visual features of the reference information, so that the semantic visual features corresponding to different reference information can be effectively distinguished.

[0095] In some embodiments, the semantic visual features are visual features with semantic information. Accordingly, for any reference information, the electronic device first visually encodes the reference image in the reference information to obtain the visual features of the reference object instance. Then, the electronic device semantically encodes the descriptive text in the reference information to obtain the text semantic features of the descriptive text. Finally, the electronic device fuses the visual features and the text semantic features to obtain the semantic visual features of the reference information. This operation can effectively fuse the visual content in the reference information with the semantic information of the descriptive text, accurately extract the semantic visual features of the reference information, so that the electronic device can better understand the reference object instance, provide a more accurate and comprehensive basis for the subsequent video generation task, and improve the processing effect.

[0096] Among them, visual coding is an operation that converts the visual information of an image into a form that can be understood and processed by computers. Visual coding uses specific algorithms and techniques to analyze the visual attributes of each pixel in the reference image, such as color, brightness, texture, etc., and then represents this information in a specific coding method.

[0097] For example, for a reference image containing a red apple, visual encoding can extract visual features such as the apple's shape (approximately round), color (specific hue value of red), and surface texture (smoothness, etc.). These features constitute the visual features of the reference object instance (this apple).

[0098] Among them, semantic coding is used to understand the semantic information conveyed by the descriptive text, such as analyzing the meaning of words in the text, the relationship between words, and the overall meaning of the sentence.

[0099] For example, continuing with the description text of the reference image containing a red apple, semantic coding can extract the fruit category semantics represented by the word "apple", the semantics describing the appearance of apples such as "bright color" and "smooth", and the position semantics involved in "placed on a wooden chopping board", etc. These constitute the text semantic features of the description text.

[0100] In some embodiments, the electronic device fuses the visual features and the text semantic features through a cross-attention mechanism. Accordingly, the electronic device first aligns the visual features and the text semantic features. Then, the electronic device fuses the aligned visual features and the text semantic features based on the cross-attention mechanism. By first aligning the visual features and the text semantic features, and then fusing them based on the cross-attention mechanism, the relevant information of the image and the text can be more accurately and comprehensively combined, thereby providing a more accurate and effective feature basis for the subsequent generation of the video and improving the processing effect.

[0101] When the electronic device obtains visual features from the reference image and text semantic features from the description text, these features may initially have different dimensions. The alignment operation can align the visual features with the text features in terms of dimension.

[0102] For example, matching certain key elements in visual features (such as the numerical value corresponding to color features) with the semantic codes corresponding to the words describing colors in text semantic features enables the elements in visual features and the elements in text semantic features to be compared with each other in a common "space" or "scale", ensuring that the object features seen from a visual perspective and the object features described in words can accurately correspond to each other.

[0103] Among them, the cross-attention mechanism is a special attention mechanism. The cross-attention mechanism involves the mutual attention between data of two different modalities (visual modality and text modality in the embodiment of the present disclosure). The cross-attention mechanism allows the visual features to "focus" on the relevant parts of the text semantic features, and also allows the text semantic features to "focus" on the corresponding parts of the visual features, so as to better capture the association and interaction information between the two. Optionally, the visual features and the text semantic features will be injected into the cross-attention module together, so as to inject each text semantic feature into the corresponding visual feature and obtain a visual feature with text semantics.

[0104] In some embodiments, the electronic device can encode multiple reference information separately through a multi-instance feature encoder. The multi-instance feature encoder includes a visual encoder and a text encoder. The visual encoder, such as the CLIP (Contrastive Language-Image Pretraining) model and multi-layer perceptual resampling, is used to extract the visual features of the reference image. The text encoder, such as the T5 (Text-To-Text Transfer Transformer) model, is used to extract text semantic features describing the text.

[0105] For example, see Figure 4 As shown, Figure 4 FIG. 1 is a flow chart of encoding reference information according to an exemplary embodiment. Figure 4 As shown. The reference image in the reference information is input into the visual encoder to obtain visual features. The description text in the reference information is input into the text encoder to obtain text semantic features. The visual features and semantic text features are input into the cross attention module to obtain semantic visual features.

[0106] In step S305, a target video is generated based on the text prompt feature and multiple semantic visual features.

[0107] In the disclosed embodiment, the electronic device injects the text prompt feature and multiple semantic visual features into the diffusion model, and based on the diffusion model, starts from the noise, fuses the various features, and obtains the target video.

[0108] In some embodiments, the electronic device first fuses the video features of the diffusion model with the text prompt features to obtain intermediate fusion features, and the video features of the diffusion model are the hidden features in the process of generating the video. Then, the electronic device fuses the intermediate fusion features with multiple semantic visual features in sequence to obtain video features. Finally, the electronic device decodes the video features to obtain the target video. Among them, the diffusion model can generate the target video through multiple iterations. For any iteration process, the video features obtained after the end of the previous iteration are the video features of the above-mentioned diffusion model. By fusing the hidden video features of the diffusion model when generating the video with the text prompt features and multiple semantic visual features in sequence, and decoding to obtain the target video, with the advantage of multiple iterations of the diffusion model, it is possible to fully integrate various feature information and effectively improve the accuracy, richness and fit with user needs of the target video generation.

[0109] In some embodiments, the electronic device can fuse the video features of the diffusion model and the text prompt features through a cross-attention mechanism to obtain an intermediate fusion feature. By fusing the video features of the diffusion model and the text prompt features through a cross-attention mechanism to obtain an intermediate fusion feature, the electronic device can accurately capture the association and interaction information between the two, effectively combine the relevant characteristics, and thus lay a good foundation for the subsequent generation of target videos that better meet the needs and have higher quality.

[0110] In some embodiments, a new feature injection module is introduced into the diffusion model, and the feature injection module is used to obtain and inject multiple semantic visual features into the diffusion model, so as to ensure that the features of the diffusion model are effectively integrated into the visual features and text prompt features. Correspondingly, the electronic device obtains multiple semantic visual features in sequence through the feature injection module. Each time a semantic visual feature is obtained, the electronic device can fuse the semantic visual feature with the intermediate fusion feature to obtain the intermediate video feature. Finally, in response to the completion of the fusion, the video feature is obtained. Among them, the core of the feature injection module is to introduce a new cross-attention module into the traditional Transformer Block, so that the diffusion model can adaptively learn the ability to distinguish different concepts according to the semantics in the concept features, ensuring that each concept in the generated video can be accurately restored. By introducing the feature injection module into the diffusion model, multiple semantic visual features can be effectively obtained and integrated, ensuring that its features are deeply integrated with the visual features and text prompt features. After the electronic device operates according to this process, a video feature with a high degree of integration that can better meet the needs can be generated, thereby improving the quality and effect of the target video generation.

[0111] It should be noted that, in order to make the video generation solution provided by the embodiment of the present disclosure easier to understand, see Figure 5 As shown, Figure 5 FIG. 1 is a flow chart of a video generation method provided according to an exemplary embodiment. Figure 5 As shown in the figure, the video generation scheme is implemented by a video generation model, which includes three parts: an input layer, a processing layer, and an output layer. The input layer is used to input text prompt information and multiple reference information. The processing layer includes a text feature encoder, a multi-instance feature encoder, and a diffusion model. The multi-instance feature encoder includes a visual encoder and a text encoder. The diffusion model includes a video diffusion model and a feature injection module. The output layer is used to output video features, and the video features are decoded by a hidden variable decoder to generate a target video.

[0112] The disclosed embodiment provides a video generation scheme, which can bind the features of the reference object instance and the features of the description text corresponding to each reference information by encoding multiple reference information respectively, and unbind the features of different reference information to avoid confusion between different reference information, so that the attributes of each reference object instance can be accurately restored in the generated video. In addition, since the above scheme does not rely on regional constraints, the generated video has diversity and motion smoothness.

[0113] Figure 6 FIG. 1 is a block diagram of a video generating device according to an exemplary embodiment. Figure 6 As shown, the device includes: a first encoding unit 601, a second encoding unit 602 and a generating unit 603.

[0114] The first encoding unit 601 is configured to encode the text prompt information to obtain a text prompt feature;

[0115] The second encoding unit 602 is configured to encode the multiple reference information respectively to obtain multiple semantic visual features, each reference information includes a reference image and a description text, the reference image includes a reference object instance, and the description text is used to describe the corresponding reference object instance;

[0116] The generating unit 603 is configured to generate a target video based on the text prompt feature and the multiple semantic visual features.

[0117] In some embodiments, the second encoding unit 602 is configured to, for any reference information, visually encode the reference image in the reference information to obtain visual features of the reference object instance; semantically encode the descriptive text in the reference information to obtain textual semantic features of the descriptive text; and fuse the visual features and textual semantic features to obtain semantic visual features of the reference information.

[0118] In some embodiments, the first encoding unit 601 is configured to align visual features and text semantic features; and fuse the aligned visual features and text semantic features based on a cross-attention mechanism.

[0119] In some embodiments, the generation unit 603 is configured to fuse the video features of the diffusion model with the text prompt features to obtain intermediate fusion features, where the video features of the diffusion model are hidden features in the process of generating video; fuse the intermediate fusion features with multiple semantic visual features in sequence to obtain video features; and decode the video features to obtain the target video.

[0120] In some embodiments, the generation unit 603 is configured to fuse the video features and the text prompt features of the diffusion model through a cross-attention mechanism to obtain an intermediate fusion feature.

[0121] In some embodiments, the generation unit 603 is configured to obtain multiple semantic visual features in sequence through a feature injection module; each time a semantic visual feature is obtained, the semantic visual feature is fused with the intermediate fusion feature to obtain an intermediate video feature; in response to the completion of the fusion, a video feature is obtained.

[0122] In some embodiments, the first encoding unit 601 is further configured to obtain input text prompt information; perform compliance verification on the text prompt information; and display the prompt information if the text prompt information fails the verification.

[0123] The disclosed embodiment provides a video generation device, which can bind the features of the reference object instance corresponding to each reference information and the features of the description text by encoding multiple reference information respectively, and unbind the features of different reference information to avoid confusion between different reference information, so that the attributes of each reference object instance can be accurately restored in the generated video. In addition, since the above scheme does not rely on regional constraints, the generated video has diversity and motion smoothness.

[0124] It should be noted that the video generation device provided in the above embodiment is only illustrated by the division of the above functional units. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the video generation device provided in the above embodiment and the video generation method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0125] Regarding the video generating device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0126] In the embodiments of the present disclosure, the electronic device may be a terminal or a server. When the electronic device is a terminal, the terminal is used as an execution subject to implement the technical solution provided in the embodiments of the present disclosure; when the electronic device is a server, the server is used as an execution subject to implement the technical solution provided in the embodiments of the present disclosure; or, the technical solution provided in the present disclosure is implemented through interaction between the terminal and the server. The embodiments of the present disclosure are not limited to this.

[0127] Figure 7 700 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 700 includes: a processor 701 and a memory 702 .

[0128] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0129] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the video generation method provided by the method embodiment of the present disclosure.

[0130] In some embodiments, the electronic device 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707 and a power supply 708.

[0131] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0132] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0133] The display screen 705 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, and the front panel of the electronic device 700 is set; in other embodiments, the display screen 705 can be at least two, which are respectively set on different surfaces of the electronic device 700 or are folded; in some other embodiments, the display screen 705 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0134] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0135] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0136] The power supply 708 is used to power various components in the electronic device 700. The power supply 708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0137] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the electronic device 700, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0138] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions, and the instructions can be executed by a processor 701 of an electronic device 700 to complete the above-mentioned video generation method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0139] A computer program product comprises a computer program, and when the computer program is executed by a processor, the video generating method is implemented.

[0140] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0141] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that: The method comprises: Encode the text prompt information to obtain text prompt features; Encoding the plurality of reference information respectively to obtain a plurality of semantic visual features, each reference information comprising a reference image and a description text, the reference image comprising a reference object instance, and the description text being used to describe the corresponding reference object instance; A target video is generated based on the text prompt feature and the plurality of semantic visual features.

2. The video generation method according to claim 1, characterized in that: The plurality of reference information are encoded respectively to obtain a plurality of semantic visual features, including: For any reference information, visually encode a reference image in the reference information to obtain visual features of the reference object instance; Performing semantic coding on the description text in the reference information to obtain text semantic features of the description text; The visual features and the text semantic features are integrated to obtain semantic visual features of the reference information.

3. The video generation method according to claim 2, characterized in that: The fusing of the visual features and the text semantic features to obtain the semantic visual features of the reference information includes: aligning the visual feature with the textual semantic feature; Based on the cross attention mechanism, the aligned visual features and the text semantic features are fused.

4. The video generation method according to claim 1, characterized in that: The generating a target video based on the text prompt feature and the plurality of semantic visual features comprises: The video feature of the diffusion model is fused with the text prompt feature to obtain an intermediate fusion feature, wherein the video feature of the diffusion model is a hidden layer feature in the process of generating the video; The intermediate fusion feature is sequentially fused with the plurality of semantic visual features to obtain a video feature; The video features are decoded to obtain the target video.

5. The video generation method according to claim 4, characterized in that: The step of fusing the video feature of the diffusion model with the text prompt feature to obtain an intermediate fusion feature includes: The video features of the diffusion model and the text prompt features are fused through a cross-attention mechanism to obtain the intermediate fusion features.

6. The video generation method according to claim 4, characterized in that: The step of fusing the intermediate fusion feature with the plurality of semantic visual features in sequence to obtain a video feature includes: Through the feature injection module, multiple semantic visual features are obtained in sequence; Whenever a semantic visual feature is obtained, the semantic visual feature is fused with the intermediate fusion feature to obtain an intermediate video feature; In response to the fusion being completed, the video feature is obtained.

7. The video generation method according to any one of claims 1 to 6, characterized in that: The method further comprises: Get the input text prompt information; Performing compliance verification on the text prompt information; If the text prompt information fails to pass the verification, the prompt information is displayed.

8. A video generating device, characterized in that: The device comprises: A first encoding unit is configured to encode the text prompt information to obtain a text prompt feature; A second encoding unit is configured to encode the multiple reference information respectively to obtain multiple semantic visual features, each reference information includes a reference image and a description text, the reference image includes a reference object instance, and the description text is used to describe the corresponding reference object instance; A generating unit is configured to generate a target video based on the text prompt feature and the multiple semantic visual features.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the video generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video generating method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.

Images