Video generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a video generation method and device, equipment, a storage medium and a program product, and is applied to the technical field of artificial intelligence. The video generation method comprises the following steps: acquiring user description information; obtaining a first image material matched with the user description information according to the user description information; filling the blank image sequence according to the user description information and the first image material to obtain a reference image sequence; and inputting the reference image sequence and the user description information into a video generation model to obtain a video output by the video generation model. According to the video generation method, the user description information and the reference image sequence filled with the first image material can be processed in combination with the video generation model, the video meeting the user requirement is accurately generated, the workload of later manual adjustment is greatly reduced, the efficiency of the whole video generation process is finally improved, and the user experience is improved. And thus, the problem of low video generation efficiency is effectively solved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a video generation method, apparatus, device, storage medium and program product. Background Technology

[0002] With the continuous evolution of computer technology, video generation technology is also constantly developing and updating. Currently, users can edit and create videos through image editing and splicing. However, existing video editing relies heavily on manual operation by users, which reduces the efficiency of video generation. Summary of the Invention

[0003] This application provides a video generation method, apparatus, device, storage medium, and program product to solve the problem of low video generation efficiency.

[0004] In a first aspect, embodiments of this application provide a video generation method, which includes: obtaining user description information; obtaining a first image material matching the user description information; filling a blank image sequence according to the first image material based on the user description information to obtain a reference image sequence; and inputting the reference image sequence and the user description information into a video generation model to obtain a video output by the video generation model.

[0005] In some possible implementations of this application, the user description information includes entity display time information and entity display location information; the above-mentioned filling the blank image sequence according to the user description information and the first image material to obtain the reference image sequence includes: determining the target image to be filled from the blank image sequence according to the entity display time information; determining the filling area of ​​the first image material in the target image according to the entity display location information; and filling the pixel data of the first image material into the filling area of ​​the target image to obtain the reference image sequence.

[0006] In some possible implementations of this application, the user description information includes entity display time information. Before performing the step of filling the blank image sequence according to the first image material based on the user description information to obtain a reference image sequence, the video generation method further includes: determining the total video duration based on the preset video duration and entity display time information; determining the number of blank images based on the total video duration and preset video frame rate; and creating a blank image sequence with random noise corresponding to the number of images.

[0007] In some possible implementations of this application's embodiments, the user description information includes entities, entity display time information, and entity display location information; the video generation model includes a random activation layer, an entity spatial sampling layer, an implicit representation fusion layer, a diffusing convolution layer, and an image synthesis layer; the above-mentioned inputting the reference image sequence and user description information into the video generation model to obtain the video output by the video generation model includes: inputting the reference image sequence and user description information into the video generation model; through the random activation layer, calculating the random activation value of the entity in each reference image in the reference image sequence based on the mapping relationship between the entity display time information and the corresponding time of each reference image in the reference image sequence; through the entity spatial sampling layer, generating atoms according to a preset resolution, and processing each reference image in the reference image sequence... The image is segmented into pixel blocks, and spatial sampling is performed on the segmented pixel blocks to extract pixel features corresponding to the entity. An implicit representation fusion layer is used to fuse the latent representations corresponding to the entity in each reference image in the reference image sequence with the random activation values ​​corresponding to the entity, resulting in a fused implicit representation. The latent representation corresponding to the entity is determined by the pixel features corresponding to the entity. A diffusing convolutional layer is used to perform convolutional diffusing processing on the fused implicit representation to obtain the diffusion feature map of the entity in the corresponding reference image. An image synthesis layer is used to synthesize the diffusion feature maps of the entity in the corresponding reference image to obtain single-frame video frames corresponding to each reference image. The output video is composed of the single-frame video frames corresponding to each reference image in the reference image sequence in sequence.

[0008] In some possible implementations of this application, a random activation layer is used to calculate the random activation value of the entity in each reference image in the reference image sequence based on the mapping relationship between the entity display time information and the corresponding time of each reference image in the reference image sequence. This includes: generating a time activation function value corresponding to each reference image based on the entity display time information and the corresponding time of each reference image in the reference image sequence, where the time activation function value is used to characterize whether the entity is activated at the reference image; generating the transformation trajectory position of the entity at each reference image based on the time activation function value, time linear interpolation term, and random perturbation term corresponding to each reference image; and determining the random activation value of the entity in each reference image in the reference image sequence based on the transformation trajectory position of the entity and preset amplitude control parameters.

[0009] In some possible implementations of this application, the user description information includes entity description information; obtaining a first image material matching the user description information according to the user description information includes: matching the entity description information with the image description information of each image material in the image material library to obtain matching information between the entity description information and the image description information of each image material; obtaining the first image material according to the matching information between the entity description information and the image description information of each image material.

[0010] In some possible implementations of this application, obtaining the first image material based on the matching information between entity description information and the image description information of each image material includes: if the matching information between entity description information and the image description information of each image material includes target matching information, determining the image material corresponding to the target matching information as the first image material, wherein the value of the target matching information is greater than or equal to a preset threshold; if the matching information between entity description information and the image description information of each image material does not include target matching information, displaying prompt information on a first interface, wherein the prompt information is used to prompt the user to provide the first image material; and obtaining the first image material based on the user's input in response to the prompt information.

[0011] Secondly, embodiments of this application provide a video generation apparatus, which includes: The first acquisition module is used to acquire user description information; The second acquisition module is used to acquire the first image material that matches the user description information based on the user description information; The fill module is used to fill the blank image sequence according to the user description information and the first image material to obtain the reference image sequence; The information processing module is used to input the reference image sequence and user description information into the video generation model to obtain the video output by the video generation model.

[0012] Thirdly, embodiments of this application provide an electronic device, the device comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a video generation method as described in any of the first aspects.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the video generation method as described in any of the first aspects.

[0014] Fifthly, embodiments of this application provide a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the video generation method as described in any of the first aspects.

[0015] The video generation method, apparatus, device, storage medium, and program product of this application embodiment obtain first image materials based on user description information, eliminating the need for manual screening of first image materials that match user description information. This targeted matching method not only reduces ineffective human resource investment but also ensures the consistency between the first image materials and user needs (i.e., user description information) through matching logic, thereby significantly reducing the human resource consumption in the image material preparation stage. Subsequently, by filling the blank image sequence with the first image materials according to the user description information to obtain a reference image sequence, the initial construction of a serialized visual foundation from static images is achieved. Furthermore, by using a video generation model to simultaneously process the reference image sequence and user description information, under the dual guidance of visual content and semantic intent, videos that better meet user needs can be generated more accurately. In this way, the workload of manual video adjustments in the later stages is effectively reduced, the overall efficiency of the entire video generation process is optimized, and the problem of high human resource consumption in the video production process is effectively solved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating some embodiments of the video generation method provided in this application is shown; Figure 2 The diagram shows an interface of the entity form interface in a video generation method provided in some embodiments of this application; Figure 3 A flowchart illustrating a specific implementation of step 120 provided in some embodiments of this application is shown; Figure 4 The diagram illustrates a specific implementation of step 1202 provided in some embodiments of this application; Figure 5 The diagram illustrates a specific implementation of step 130 provided in some embodiments of this application; Figure 6 A flowchart illustrating a specific implementation of step 140 provided in some embodiments of this application is shown; Figure 7 This paper illustrates a schematic diagram of the model processing flow of the video generation model in some embodiments of the video generation method provided in this application; Figure 8 A flowchart illustrating a specific implementation of step 1401 provided in some embodiments of this application is shown; Figure 9 The present application provides schematic diagrams of the structure of a video generation apparatus according to some embodiments; Figure 10 The diagram shows a schematic representation of the structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0018] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0019] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0020] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0021] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0022] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first provides a detailed description of the relevant technologies involved: Videos are widely used in diverse scenarios such as advertising design, product promotion, and service recommendation. The video production process can specifically include stages such as material acquisition, editing and integration, and post-production optimization. Currently, these stages mainly rely on manual operation supplemented by professional design software. However, although existing design software provides many auxiliary tools, the entire video production process, from material processing to video generation, requires a significant investment of manpower, resulting in substantial labor costs. Furthermore, it demands a high level of professional expertise from operators, ultimately reducing video generation efficiency.

[0023] To address the problems in the aforementioned related technologies, embodiments of this application provide a video generation method, apparatus, device, storage medium, and program product. The following description, in conjunction with the appendix... Figure 1 To be continued Figure 8 The video generation method provided in this application will be described in detail through specific embodiments and application scenarios.

[0024] Figure 1 The diagram illustrates a flowchart of a video generation method provided in some embodiments of this application. For example... Figure 1 As shown, the video generation method may include steps 110 to 140.

[0025] Step 110: Obtain user description information; Step 120: Obtain first image material matching the user description information; Step 130: Fill the blank image sequence according to the first image material based on the user description information to obtain a reference image sequence; Step 140: Input the reference image sequence and user description information into the video generation model to obtain the video output by the video generation model.

[0026] Therefore, by obtaining the first image material based on user description information, the process of manually selecting first image materials that match user description information is eliminated. This targeted matching method of obtaining image materials not only reduces ineffective human resource investment, but also ensures the consistency between the first image material and user needs, i.e., user description information, through matching logic, thus significantly reducing the human resource consumption in the image material preparation stage. Subsequently, by filling the blank image sequence with the first image material according to the user description information to obtain the reference image sequence, the initial construction of the serialized visual foundation from static images is achieved. Furthermore, by using the video generation model to simultaneously process the reference image sequence and user description information, under the dual guidance of visual content and semantic intent, videos that better meet user needs can be generated more accurately. In this way, the workload of manual video adjustments in the later stage is effectively reduced, the overall efficiency of the entire video generation process is optimized, and the problem of high human resource consumption in the video production process is effectively solved.

[0027] The steps described above are explained in detail below.

[0028] First, regarding step 110, the user description information involved in this embodiment refers to the set of information provided by the user to express their video generation needs. The user description information can take forms including, but not limited to, text, voice, and images. The video generation method of this embodiment can be applied to a video generation system. In one example, the user can input a text description through a text input box provided by the video generation system, and the video generation system obtains the user description information through the user's input text description. In another example, the user can record a voice description through a microphone provided by the video generation system, and the video generation system can convert the user's input voice into text using a voice recognition tool to obtain the user description information.

[0029] Secondly, to reduce the computational load of calculating the first image material, before performing step 120 above, entity description information and entity display information can be extracted from the user description information. Entity description information includes entity identification information and entity attribute information, while entity display information includes entity display time information, entity display location information, and associated feature information. For example, entity identification information may include, but is not limited to: entity number, entity; entity attribute information may include, but is not limited to: entity attribute information before transformation, entity attribute information after transformation; entity display time information refers to the set of time parameters in the user description information used to define when the entity appears and disappears in the video, and may include, but is not limited to: entity display time, entity display start time, entity display end time; entity display location information refers to the set of spatial parameters in the user description information used to define the entity's display location in the image, and may include entity location information before transformation and entity location information after transformation. Entity location information before transformation may include, but is not limited to: entity coordinates, width, and height before transformation; entity location information after transformation may include, but is not limited to: entity coordinates, width, and height after transformation; associated feature information of entity display may include, but is not limited to: transformation method, associated entity, association method, whether it is a background, and operation.

[0030] The system includes the following information: Entity ID and Entity name to identify the object to be transformed (e.g., a person). Entity appearance before transformation (e.g., female; red top; front view). Entity coordinates, width, and height before transformation (e.g., the entity's position coordinates and dimensions in space). Entity display start time and end time (e.g., 0s start, 5s end, meaning the transformation is completed within 5 seconds). Transformation method (e.g., running, indicating a running dynamic). Entity appearance after transformation (e.g., female; red top; back view). Entity coordinates, width, and height after transformation (e.g., the entity's position and dimensions after transformation). Associated entities, association method, and background (e.g., setting association logic with other entities and whether it's used as a background layer). Operations provide modification and deletion functions for easy configuration adjustments.

[0031] In some embodiments of this application, the user can use, for example... Figure 2 The presented "Add / Modify Entity" form interface allows you to configure the transformation parameters of entities in animation and interactive scenes. Specifically, by filling in the corresponding fields on this interface, you can specify to the video generation system: which entity to target, when to perform the transformation, how to perform the transformation, and which entities to interact with.

[0032] Secondly, regarding step 120, the first image material refers to the original image material that matches the user's description information, which can serve as the base image for subsequent blank image filling. For example, the first image material can be an image material from an image material library or an image material uploaded by the user. The image material sources in the image material library may include, but are not limited to: publicly available labeled image libraries, publicly available images with added tags in internal enterprise applications, images with tag information in publicly available internet applications, and image libraries with tag information authorized by partner companies.

[0033] In some embodiments of this application, the user description information includes entity description information. To quickly and accurately obtain first image material matching the user description information, such as... Figure 3 As shown, step 120 above may specifically include steps 1201 and 1202.

[0034] Step 1201: Match the entity description information with the image description information of each image in the image material library to obtain the matching information between the entity description information and the image description information of each image.

[0035] The entity description information can include entity identification information and entity attribute information. Entity attribute information can include attribute information before and after entity transformation. Attribute information before transformation can include, but is not limited to: color before transformation, shape before transformation, angle before transformation, and associated scene before transformation. Attribute information after transformation can include, but is not limited to: color after transformation, shape after transformation, angle after transformation, and associated scene after transformation. Image description information refers to the attribute tags corresponding to each image in the image library, used to describe the image content. Matching information refers to the similarity result generated after comparing the entity description information with the image description information of a certain image, such as a matching score; the higher the score, the higher the similarity. Matching methods can include, but are not limited to, at least one of the following: keyword matching and semantic similarity matching.

[0036] Step 1202: Obtain the first image material based on the matching information between the entity description information and the image description information of each image material.

[0037] For example, a matching score threshold can be set to directly filter out image materials with a matching score greater than or equal to the preset matching score threshold. If there are multiple image materials with a matching score greater than or equal to the preset matching score threshold, the image material ranked first in the matching score from high to low is selected as the first image material. Alternatively, if there are multiple image materials with a matching score greater than or equal to the preset matching score threshold, one image material is randomly selected from the multiple image materials as the first image material.

[0038] Therefore, by matching entity description information with image description information in a pre-built image material library, manual search time can be reduced and rapid matching can be achieved. Subsequently, the first image material can be filtered and obtained based on the matching information, ensuring that the first image material is highly relevant to the user's needs, i.e., the user description information, and ultimately achieving the effect of efficiently and accurately obtaining the first image material.

[0039] In some embodiments of this application, such as Figure 4 As shown, step 1202 above may specifically include steps 12021 to 12023.

[0040] Step 12021: If the matching information between the entity description information and the image description information of each image material includes target matching information, the image material corresponding to the target matching information is determined as the first image material, and the value of the target matching information is greater than or equal to a preset threshold.

[0041] In this context, target matching information refers to specific matching information in the matching information between entity description information and image description information of image materials, where the numerical value (such as a matching score) is greater than or equal to a preset matching score threshold. This serves as the basis for determining whether an image material meets the requirements. For example, if the preset matching score threshold can be set to 80 points, and an image material has a matching score of 85 points, then the matching information corresponding to that image material is the target matching information. The preset threshold is a pre-set matching score threshold set by the system or user before performing the matching filter. It is used to quickly distinguish between image materials that meet the requirements and those that do not. The preset threshold can be adjusted according to the actual scenario.

[0042] Step 12022: If the target matching information is not included in the matching information between the entity description information and the image description information of each image material, a prompt message is displayed on the first interface. The prompt message is used to prompt the user to provide the first image material.

[0043] The first interface refers to the visual operation interface in the video generation system used to interact with users and receive user input. Specifically, it may include a prompt text area and operation buttons, such as uploading image materials and selecting image materials.

[0044] For example, after traversing the matching information of all image materials, if the matching score of all image materials is less than the preset matching score threshold, the video generation system can automatically bring up the first interface and display clear prompt information in the prompt text area of ​​the first interface; then, the user can upload the first image material by clicking the upload image material operation button.

[0045] Step 12023: Obtain the first image material based on the user's input in response to the prompt information.

[0046] For example, after a user uploads an image, the video generation system can first verify the image quality of the image, and after the quality verification is passed, automatically mark the image as the first image.

[0047] Therefore, when a matching image is found in the image library, the first image is automatically selected, eliminating the need for manual comparison by the user and significantly saving time in selecting image materials. When no matching image is found in the image library, the first image can be uploaded through user interaction, effectively avoiding the limitation of insufficient image coverage in the image library and improving the overall smoothness of video generation.

[0048] In some embodiments of this application, after extracting entity description information and entity association information, the entity description information can be converted into a corresponding vector expression, which can be represented by the following formula (1).

[0049] ...(1) in, Represents an entity, Represents entity attributes, Indicates association, Indicates the association method. Indicates the position of the entity before the transformation. , , , Let x, y, w, and h represent the entity's coordinates before transformation, respectively. Indicates the position of the entity after transformation. , , , Let x, y, w, and h represent the transformed coordinates of the entity, respectively. Indicates the time of physical display. Indicates the start time of the physical display. Indicates the end time of the physical display.

[0050] Based on this, step 1201 above may specifically include traversal. ,for , obtain It then matches these images with the image tags in the image resource library, filters out basic image resources that meet the criteria from the image resource library, and generates an initial list of image resources to be selected. ,for (Corresponding tags:) And according to the degree of matching Select suitable first image materials, wherein the degree of matching between entity description information and image description information of image materials can be calculated by the following formula (2).

[0051] ...(2) Among them, match( , ) indicates a judgment Is it included? If Included In this context, match( , If ) is 1, then Not included In this context, match( , ) is 0, Represents statistics The quantity.

[0052] Then, according to Sort and retrieve To ensure the consistency of the generated video, one of the first 10 source images can be randomly selected as the first source image. If no matching image can be found, a prompt message will be displayed on the first interface to inform the user that no matching image was found in the image library, and to instruct the user to upload the first image themselves.

[0053] It is worth noting that before determining the first source image, a step of determining the image quality may also be included. Specifically, the image quality q(x) of the source image x is... It can be calculated using the following formula (3).

[0054] (3) Where W represents the width of the source image x, and h represents the height of the source image x. The horizontal sampling amplitude parameter is configured. The vertical sampling amplitude parameter is configured. and These represent the number of sampling blocks in the horizontal and vertical directions, respectively.

[0055] Then, after initially determining the first source image, it is necessary to verify the quality of the first source image. Specifically, this may include calculating the quality score of the first image source using the formula (3) above. ;Will Compared with the preset quality score threshold, in If the quality score is greater than or equal to a preset threshold, the first image material is used in the subsequent step of generating a reference image sequence by filling blank image sequences; if If the quality score is below the preset threshold, the first image material will be re-selected from the candidate image materials until the final first image material meets the preset quality requirements.

[0056] Furthermore, regarding step 130, the blank image sequence refers to a collection of multiple blank images of the same size and format arranged in chronological order.

[0057] In some embodiments of this application, before performing step 130 above, the video generation method may further include the process of determining the number of blank images in the blank image sequence and generating the blank image sequence. The user description information includes entity display information, and the entity display information includes entity display time information. Based on this, as... Figure 5 As shown, the video generation method described above may further include: determining the total video duration based on a preset video duration and entity display time information; determining the number of blank images based on the total video duration and a preset video frame rate; and creating a sequence of blank images with random noise corresponding to the number of images.

[0058] The preset video duration refers to the total duration limit of the generated video, for example, 15 seconds. Entity display time information refers to the time parameters in the user description information used to define the period during which the entity is displayed in the video, including but not limited to the entity display start time, entity display end time, and entity display duration; for example, entity A is displayed from 1 second to 6 seconds, with a total duration of 5 seconds. The total video duration refers to the complete duration of the final generated video from start to finish. The preset video frame rate refers to the pre-set video frame rate, representing the number of image frames contained in the video per second. Blank image sequence. , where n is the number of blank images, for example, ,in, This is the preset video frame rate. Random noise values ​​refer to image data with randomly distributed pixel values ​​generated by Gaussian noise generation algorithms or uniform noise distribution algorithms, used to fill blank images.

[0059] For example, if the set of all entities displayed in the video is called the entity set D, then D = { , ,…, (k≥1, k is the total number of entities); iterate through ,for Get the maximum value of the entity display end time. Considering limitations in video generation capabilities and computing power, excessively long video generation times would require enormous computational resources. Therefore, the preset video length is limited to 15 seconds, thus limiting the total video length generated. , where r(t) is a random decimal generation function less than t, used to add a small random fluctuation to the total video duration while controlling the total duration; express The end time of the physical exhibition and It must be less than or equal to 15 seconds.

[0060] Therefore, the total video duration determined by combining the preset video length with entity display time information effectively avoids situations where the total video length is too short, resulting in incomplete entity display, or too long, causing invalid frame redundancy, ensuring that the total video duration meets the user's expectations. Subsequently, based on the conversion logic between the total video duration and the preset video frame rate, the time dimension duration can be converted into the number of images in the spatial dimension, ensuring that the number of blank images matches the total video duration. Then, creating blank images with random noise corresponding to the number provides basic texture differences for subsequent entity feature fusion. In this way, the blank image sequence can provide an accurate and suitable carrier for subsequent entity filling, helping the video generation model output a video that meets the user's expectations.

[0061] It is worth noting that the resolution parameters of the generated blank image include the horizontal width. and vertical height The following conditions must be met: and Each is an integer multiple of the resolution a of the generated atoms; , The minimum value is 512. In this embodiment, the resolution-split atomic mass a is uniformly set to 16.

[0062] In some embodiments of this application, such as Figure 5 As shown, step 130 above may specifically include steps 1301 to 1303.

[0063] Step 1301: Determine the target image to be filled from the blank image sequence based on the entity display time information.

[0064] For example, extract the entity display time from the entity display time information. The entity display starts at time T1 and ends at time T2. The blank image sequence is numbered sequentially from 1 to m, with each number corresponding to a video time; for example, the first blank image corresponds to 0 seconds, the 30th blank image to 1 second, and the 60th blank image to 2 seconds. Subsequently, the entity display time is selected. The corresponding blank image is the target image to be filled, for example, T1=0s. =5s, filter out the first to the 150th blank images as the target images to be filled.

[0065] Step 1302: Determine the filling area of ​​the first image material in the target image based on the entity display location information.

[0066] For example, firstly, the entity coordinates, width, and height corresponding to the current entity are extracted from the entity display location information, and the width and height of the first image material are read. For instance, if the current entity is the entity before transformation, the entity's coordinates, width, and height before transformation are extracted; if the current entity is the entity after transformation, the entity's coordinates, width, and height after transformation are extracted. Secondly, the image resolution of the target image is obtained, and the reference coordinates are calculated. If the entity coordinates are the absolute coordinates of the upper left corner, these entity coordinates are directly used as the upper left corner of the filling area, and the target coordinates of the lower right corner of the filling area are calculated based on the entity width and height. Finally, the filling area is determined based on the entity coordinates and the target coordinates.

[0067] Step 1303: Fill the target image with the pixel data of the first image material to obtain the reference image sequence.

[0068] For example, the pixel data of the first image material is written point by point into the filling area of ​​the corresponding target image according to the sequence number of the target image, and the filled target image is integrated with the blank image that was not selected as the target image according to the original sequence number, so as to form a continuous reference image sequence of blank image and entity-filled image, and the display time and display position of the entity in the sequence are completely matched with the user description information.

[0069] Therefore, by using entity display time information, the system ensures that entities are displayed in the target image within the blank image sequence, achieving dynamic temporal synthesis. By using entity display position information, the system can accurately control the display area and layout of entities within the target image. Thus, by using entity display time and position information to fill the pixel data of the first image material into the target image to obtain a reference image sequence, precise matching with user-described information can be achieved.

[0070] In some embodiments of this application, when there are multiple entities, traversal is performed. ,for The corresponding frame image at time t upper fill area The filling logic in the middle can be represented by the following formula (4).

[0071] ... (4) in, To fill the coefficient matrix, Let be the number of entities at the same location at time t. for The initial fill coordinates of the solid. These represent the starting horizontal and vertical coordinates for filling. These represent the width and height of the fill, respectively.

[0072] In this way, by controlling the transparency of entities through the fill coefficient matrix and automatically adjusting the fill intensity according to the number of entities k at the same location, visual saturation can be avoided.

[0073] Then, in step 140, the reference image sequence and user description information are input into the video generation model, and the video generation model processes the reference image sequence and user description information to obtain the video output by the video generation model.

[0074] In some embodiments of this application, before performing step 1401, the video generation method may further include inputting user description information into a word2vec model, using the word2vec model to vectorize the user description information to obtain a text vector containing entity relationship transformation descriptions, and then inputting the text vector corresponding to the user description information and a reference image sequence into the video generation model for processing. The video generation model includes a random activation layer, an entity space sampling layer, an implicit representation fusion layer, a diffusing convolutional layer, and an image synthesis layer. Based on this, as... Figure 6 As shown, step 140 above may specifically include steps 1401 to 1405.

[0075] Step 1401: Input the reference image sequence and user description information into the video generation model. Through the random activation layer, calculate the random activation value of the entity in each reference image in the reference image sequence based on the mapping relationship between the entity display time information and the corresponding time of each reference image in the reference image sequence.

[0076] For example, such as Figure 7 As shown, the video generation model described above can also include an entity text processing layer, which comprises an LSTM (Long Short-Term Memory) network, fully connected layers, and linear layers. This entity text processing layer is used to process the entity text information of each entity in the entity set D. Independent processing channels are allocated for each entity, one channel per entity. Specifically, the text vector corresponding to the user description information is input into the entity text processing layer and processed sequentially as follows: First, an LSTM layer with ReLU activation is used to perform ReLU activation on the attributes in the entity text information of each entity to extract the temporal correlation of attribute features; second, three fully connected layers with ReLU activation and dropout regularization are used to progressively optimize the feature text feature dimension and suppress overfitting; finally, a linear layer with ReLU activation and dropout regularization is used to obtain the entity text feature representation of each entity.

[0077] In some embodiments of this application, such as Figure 8 As shown, step 1401 can specifically include steps 14011 to 14013.

[0078] Step 14011: Through a random activation layer, based on the entity display time information and the time corresponding to each reference image in the reference image sequence, a time activation function value corresponding to each reference image is generated. The time activation function value is used to characterize whether the entity is activated at the reference image.

[0079] For example, based on the mapping relationship between the reference image number j and time t' in the reference image sequence, it is determined whether the time t' corresponding to each reference image is within the entity. Within the display time range (i.e. If time t' is at the entity Within the display time range, the event activation function value of the reference image at time t' is 1, indicating that an entity needs to be activated in the reference image; if time t' is not within the entity's display time range, the event activation function value of the reference image at time t' is 1. Within the display time range, the event activation function value of the reference image corresponding to time t' is 0, indicating that no entity needs to be activated in the reference image.

[0080] The determination logic of the above time activation function value can be expressed by the following formula (5).

[0081] ... (5) Step 14012: Generate the transformation trajectory position of the entity at each reference image based on the time activation function value, time linear interpolation term, and random perturbation term corresponding to each reference image.

[0082] For example, the transformed trajectory position of an entity at each reference image can be represented by the following formula (6).

[0083] ... (6) in, This represents the time difference between two adjacent reference images. Indicates generating values ​​not greater than A random decimal, where seed is an entity. Random seed generation, Rand( , , ) represents an entity At the time corresponding to the current reference image The changing trajectory position, Represents the time linear interpolation term. Represents the random disturbance term. Representing entities The position of the trajectory changes in the previous time step at the current time step.

[0084] It is worth noting here that each entity It corresponds to a unique seed and remains unchanged during training.

[0085] Step 14013: Determine the random activation value of the entity in each reference image in the reference image sequence based on the entity's transformation trajectory position and preset amplitude control parameters.

[0086] For example, the random activation value of an entity in each reference image in the reference image sequence can be determined by the following formula (7).

[0087] ... (7) Where W is a preset amplitude control parameter (weight matrix) used to adjust the overall intensity of the activation value.

[0088] Therefore, by calculating the random activation value of the reference image corresponding to each entity at each moment in the reference image sequence based on the time correspondence between the entity display time and the reference image sequence, it is possible to introduce controllable differences while preserving the continuity between video frames. Furthermore, by strictly limiting the entity activation range through the time activation function value, it is possible to effectively avoid screen jumps or tearing caused by excessive differences, thus balancing the stability and diversity of video generation.

[0089] Step 1402: Atoms are generated according to a preset resolution through the entity space sampling layer. Each reference image in the reference image sequence is segmented into pixel blocks, and the segmented pixel blocks are spatially sampled to extract the pixel features corresponding to the entity.

[0090] The preset resolution for generating atoms can be set to 16, i.e., a=16.

[0091] For example, the entity space sampling layer can specifically perform the following operations: First, generate atoms at a preset resolution, and then sample each reference image in the reference image sequence. The image is divided into a×a pixel blocks; secondly, for each reference image... Further processing according to color channel c (such as RGB) yields a dimension of The input data I, where, , , n is the total number of entities, I is a×a pixel block; then, the entity space sampling mechanism is used for each pixel block to extract the entity space color channel sampling result of each pixel block, that is, to extract the pixel features associated with the entity. The sampling logic can be expressed by the following formula (8).

[0092] ... (8) in, Representing entities The sampling result of the pixel block at coordinates x, y in color channel c, where c is the color channel. Represents the entity space color channel sampling matrix. This is a function for determining entity association, representing... and Is there a connection? The value is 1 when there is a correlation, and 0 otherwise, to ensure that only features of related entities are merged. Represents the base value of the original pixel block.

[0093] In some embodiments of this application, the video generation model described above may further include a convolutional dimensionality reduction network. This convolutional dimensionality reduction network is used to perform feature compression and optimization on the pixel block sampling results obtained in step 1402. The specific process of this convolutional dimensionality reduction network is as follows: First, seven 3×3×3 convolutional kernels are used to perform spatial convolution between frames and time, with a stride of 1 in both the horizontal and vertical directions. Planar traversal is performed only between frames, and the number of output channels is k×3, where k is the total number of entities, and each entity corresponds to three dedicated color channels. Second, the sampling results are input into the network, and through feature aggregation layer-by-layer convolution, each entity is obtained. At time t' (corresponding to the reference image) ), the potential representation of color channel c ,in, The dimension is .

[0094] Step 1403: Through the implicit representation fusion layer, the latent representation corresponding to the entity in each reference image in the reference image sequence is fused with the random activation value corresponding to the entity to obtain the fused implicit representation. The latent representation corresponding to the entity is determined by the pixel features corresponding to the entity.

[0095] For example, first, adjust each entity At time t' (corresponding to the reference image) random activation value The dimension that makes it compatible with the entity At time t' (corresponding to the reference image) Potential representation of ) Dimensional consistency; secondly, for the aligned and Element-wise addition is performed to obtain the fused implicit representation of the entity, completing the transformation of the entity's implicit representation. In this way, the static spatial features (latent representation) of the entity are combined with the dynamic temporal features (random activation values), providing a feature foundation that contains both spatial attributes and temporal variations for subsequent diffusion.

[0096] Step 1404: Perform convolutional diffusion processing on the fused implicit representation through a diffusion convolutional layer to obtain the diffusion feature map of the entity in the corresponding reference image.

[0097] The network configuration of the diffuse convolutional layer is consistent with that of the dimensionality reduction network.

[0098] For example, the fused implicit representation can be extended and smoothly transitioned by a diffusion convolutional layer, which can specifically include updating the fused implicit representation layer by layer by the following formula (9).

[0099] ... (9) in, Representing entities The implicit representation of the corresponding diffusion entity is the color channel c value at coordinates x, y, after diffusing the x, y values ​​from the j-th diffusion layer to the (j+1)-th diffusion layer. , indicating the direction of diffusion. is the diffusion factor for the c-th color channel of the j-th layer, used to control the diffusion intensity.

[0100] Finally, after diffusion is complete, an image restoration operation is performed to restore the original image resolution, resulting in the diffusion feature map of the entity in the corresponding reference image.

[0101] Step 1405: Through the image synthesis layer, the diffusion feature maps of entities in the corresponding reference images are synthesized to obtain single-frame video frames corresponding to each reference image, and the video is output. The video is composed of single-frame video frames corresponding to each reference image in the reference image sequence in sequence.

[0102] For example, firstly, each reference image is synthesized according to the following formula (10). The corresponding output single-frame video frame The pixel value.

[0103] ... (10) in, is the pixel coordinate, and c is the color channel.

[0104] Subsequently, all the synthesized single-frame video frames are sorted in temporal order according to the reference image sequence to form a complete video and output it.

[0105] Therefore, by sampling entity space and fusing implicit representations, the generated video frames are ensured to conform to both the entity space attributes of the reference image and the dynamic laws of the entities described by the user. The combination of convolutional dimensionality reduction and diffusing convolution can effectively avoid problems such as blurred entity edges and frame jumps. At the same time, the controllable differences introduced by the random activation mechanism enhance the diversity of the video. In this way, the output video can accurately match the user's spatiotemporal needs for entity display.

[0106] In some embodiments of this application, in order to ensure that the above video generation model conforms to entity features and dynamic laws, it is necessary to construct a high-quality labeled dataset. The specific steps are as follows: Step 1: Select training data. Specifically, in order to ensure that the training data covers multiple application scenarios and multiple entity types, the sources of training data may include, but are not limited to: non-confidential videos from the enterprise's internal business systems, publicly authorized videos on the Internet, and compliant videos provided by partners.

[0107] Step 2: Entity labeling and tracking. Specifically, entity recognition and labeling are performed frame by frame in the video. For example, professional labeling tools can be used, and labelers can manually label the entity objects in each video frame and record the entity attributes.

[0108] Step 3: Entity dynamic tracking. For the labeled entities, the target tracking algorithm in OpenCV is used to automatically generate the motion trajectory of the entities in continuous video frames, and obtain preliminary trajectory data of the entity position changing over time.

[0109] Step 4: Manual verification and supplementary annotation. The annotator verifies the automatically generated trajectory data, corrects tracking deviations, and adds entity transformation logic markers (such as "triggering conditions for entity A to move from rest"); associated entity relationships (such as "entity A and entity B are marked as associated when they interact, otherwise they are not associated"); background / entity differentiation (marking whether a pixel area belongs to the background or an entity to avoid background interference).

[0110] Step 5: Output the dataset, specifically, the final structured training dataset consisting of "original video + entity annotations for each video frame + dynamic trajectory + association relationships".

[0111] After obtaining the labeled dataset based on steps 1 to 5 above, a video generation model is trained based on this labeled dataset. The parameters are optimized using a loss function to make the video generated by the model approximate the features of real videos. The specific process is as follows: Using manually labeled entity descriptions as input, a video generation model is trained. During training, the loss between the predicted video P' and the original video P is calculated. The backpropagation algorithm (such as the Adam optimizer) is used to update all parameters of the video generation model, including but not limited to: the weights W of the random activation layer and the sampling matrix of the entity space sampling layer. Repeat the iteration until loss. Convergence, i.e. loss Once the model stops shrinking or the number of training iterations reaches 500, a preliminary training completed model is obtained.

[0112] In some embodiments of this application, the loss function during the training process of the video generation model can be represented by the following formula (11).

[0113] ... (11) in, To output video width, To output the video height, This represents the total number of frames in the video. This represents the image of the q-th frame of the video, with coordinates (i...). The value of the pixel at position j in the c color channel.

[0114] In some embodiments of this application, to adapt to new scenarios and new entity types, the model needs to be continuously optimized through incremental training, as follows: First, the training dataset is periodically checked for new data, such as newly labeled video resources. If new data exists, the model update process is triggered. Second, based on the currently trained model parameters, the new data is incrementally trained according to the above training process. For example, the descriptive information of the new data is used as input, the loss is calculated and backpropagation is performed, and only the model parameters are fine-tuned to ensure that the model learns the features and dynamic patterns of new entities while retaining its original capabilities. Subsequently, after each update, the model version is saved, and the update time and the type of new data are recorded for easy traceability and rollback. If the new data causes a performance degradation, it can be restored to the previous version. Finally, based on the trained model, a video is generated according to the user description information input by the user.

[0115] In some embodiments of this application, the video generation method can also provide an online preview function for video P', allowing users to view whether the entity display time, location, and dynamic effects meet expectations; if the user confirms that the requirements are met, video P' can be downloaded and applied to business scenarios such as video copywriting design and scene demonstration.

[0116] Based on the video generation method provided in the above embodiments, this application also provides specific implementations of the video generation apparatus. Please refer to the following embodiments.

[0117] See Figure 9 The video generation apparatus 20 provided in this application embodiment includes: The first acquisition module 201 is used to acquire user description information; The second acquisition module 202 is used to acquire a first image material that matches the user description information based on the user description information; The filling module 203 is used to fill the blank image sequence according to the user description information and the first image material to obtain the reference image sequence; The information processing module 204 is used to input the reference image sequence and user description information into the video generation model to obtain the video output by the video generation model.

[0118] Therefore, the second acquisition module 202 acquires the first image material based on the user description information acquired by the first acquisition module 201, eliminating the need for manual screening of the first image material that matches the user description information. This targeted matching method not only reduces ineffective manpower input but also ensures the consistency between the first image material and the user's needs, i.e., the user description information, through matching logic, thus significantly reducing manpower consumption in the image material preparation stage. Subsequently, the filling module 203 fills the blank image sequence with the first image material according to the user description information to obtain a reference image sequence, realizing the initial construction of the serialized visual foundation from static images. Furthermore, the information processing module 204 uses the video generation model to simultaneously process the reference image sequence and user description information. Under the dual guidance of visual content and semantic intent, it can more accurately generate videos that better meet user needs. In this way, the workload of manual video adjustment in the later stage is effectively reduced, the overall efficiency of the entire video generation process is optimized, and the problem of high manpower consumption in the video production process is effectively solved.

[0119] In some embodiments of this application, the user description information includes entity display time information and entity display location information; the filling module 203 can be specifically used to: determine the target image to be filled from the blank image sequence according to the entity display time information; determine the filling area of ​​the first image material in the target image according to the entity display location information; fill the pixel data of the first image material into the filling area of ​​the target image to obtain a reference image sequence.

[0120] In some embodiments of this application, the user description information includes entity display time information. The video generation device 20 may further include a creation module, which is used to determine the total video duration based on the preset video duration and entity display time information before the filling module performs the step of filling a blank image sequence according to the user description information and the first image material to obtain a reference image sequence; determine the number of blank images based on the total video duration and the preset video frame rate; and create a blank image sequence with random noise corresponding to the number of images.

[0121] In some embodiments of this application, the user description information includes an entity, entity display time information, and entity display location information; the video generation model includes a random activation layer, an entity spatial sampling layer, an implicit representation fusion layer, a diffusion convolution layer, and an image synthesis layer; the aforementioned information processing module 204 is specifically used to: input the reference image sequence and user description information into the video generation model; through the random activation layer, calculate the random activation value of the entity in each reference image in the reference image sequence based on the mapping relationship between the entity display time information and the corresponding time of each reference image in the reference image sequence; through the entity spatial sampling layer, generate atoms according to a preset resolution, perform pixel block segmentation on each reference image in the reference image sequence, and process the segmentation... Spatial sampling is performed on the subsequent pixel blocks to extract the pixel features corresponding to the entity. An implicit representation fusion layer is used to fuse the latent representations corresponding to the entities in each reference image in the reference image sequence with the random activation values ​​corresponding to the entities, resulting in a fused implicit representation. The latent representation corresponding to the entity is determined by the pixel features corresponding to the entity. A diffusing convolutional layer is used to perform convolutional diffusing processing on the fused implicit representation to obtain the diffusion feature map of the entity in the corresponding reference image. An image synthesis layer is used to synthesize the diffusion feature maps of the entities in the corresponding reference images, resulting in single-frame video frames corresponding to each reference image. The output video is composed of the single-frame video frames corresponding to each reference image in the reference image sequence, arranged sequentially.

[0122] In some embodiments of this application, the information processing module 204 is specifically used to: generate a time activation function value corresponding to each reference image based on the entity display time information and the time corresponding to each reference image in the reference image sequence through a random activation layer, wherein the time activation function value is used to characterize whether the entity is activated at the reference image; generate the transformation trajectory position of the entity at each reference image based on the time activation function value, time linear interpolation term and random perturbation term corresponding to each reference image; and determine the random activation value of the entity at each reference image in the reference image sequence based on the transformation trajectory position of the entity and a preset amplitude control parameter.

[0123] In some embodiments of this application, the user description information includes entity description information; the second acquisition module 202 is used to: match the entity description information with the image description information of each image material in the image material library to obtain matching information between the entity description information and the image description information of each image material; and acquire the first image material based on the matching information between the entity description information and the image description information of each image material.

[0124] In some embodiments of this application, the second acquisition module 202 is configured to: determine the image material corresponding to the target matching information as the first image material when the matching information between the entity description information and the image description information of each image material includes target matching information, wherein the value of the target matching information is greater than or equal to a preset threshold; display prompt information on the first interface when the matching information between the entity description information and the image description information of each image material does not include target matching information, wherein the prompt information is used to prompt the user to provide the first image material; and acquire the first image material based on the user's input in response to the prompt information.

[0125] The various modules of the video generation apparatus 20 provided in this application embodiment can realize Figures 1 to 8 The functions of each step in the provided video generation method, and the corresponding technical effects they achieve, will not be elaborated here for the sake of brevity.

[0126] Figure 10 The diagram shows a hardware structure schematic of an electronic device provided in some embodiments of this application.

[0127] The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0128] Specifically, the processor 301 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0129] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 302 is non-volatile solid-state memory.

[0130] In certain embodiments, memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 302 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the video generation method in the above embodiments according to this application.

[0131] The processor 301 implements any of the video generation methods described in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0132] In one example, the electronic device may also include a communication interface 303 and a bus 310. For example, Figure 3 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 310 and complete communication with each other.

[0133] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0134] Bus 310 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0135] The electronic device can execute the video generation method described in the embodiments of this application, thereby achieving a combination Figures 1 to 9 The video generation method and apparatus described.

[0136] Furthermore, in conjunction with the video generation methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the video generation methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0137] Furthermore, in conjunction with the video generation methods in the above embodiments, this application embodiment can provide a computer program product for implementation. This program product is stored in a storage medium and may specifically include a computer program or instructions. When executed by a processor, the computer program or instructions implement any of the video generation methods in the above embodiments. This program product is executed by at least one processor to implement the various processes of the video generation method embodiments described above, and can achieve the same technical effects. To avoid repetition, further details are omitted here.

[0138] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0139] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0140] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0141] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0142] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method of video generation, the method comprising: The method comprises: obtaining user description information; obtaining first image material matched with the user description information according to the user description information; filling a blank image sequence according to the first image material to obtain a reference image sequence according to the user description information; inputting the reference image sequence and the user description information into a video generation model to obtain a video output by the video generation model.

2. The method of claim 1, wherein, The user description information comprises entity display information, and the entity display information comprises entity display time information and entity display position information; filling a blank image sequence according to the first image material to obtain a reference image sequence according to the user description information comprises: determining a target image to be filled from the blank image sequence according to the entity display time information; determining a filling area of the first image material in the target image according to the entity display position information; filling pixel data of the first image material into the filling area in the target image to obtain the reference image sequence.

3. The method according to claim 1 or 2, characterized in that, The user description information comprises entity display information, and the entity display information comprises entity display time information; before filling a blank image sequence according to the first image material to obtain a reference image sequence according to the user description information, the method further comprises: determining a total video duration according to a preset video duration and the entity display time information; determining the number of images of the blank image according to the total video duration and a preset video frame rate; creating the blank image sequence with random noise corresponding to the number of images.

4. The method of claim 1, wherein, The user description information comprises entity display information, and the entity display information comprises entity display time information and entity display position information of an entity; the video generation model comprises a random activation layer, an entity space sampling layer, an implicit representation fusion layer, a diffusion convolution layer and an image synthesis layer; inputting the reference image sequence and the user description information into the video generation model to calculate random activation values of the entity in each reference image in the reference image sequence based on a mapping relationship between the entity display time information and the corresponding time of each reference image in the reference image sequence through the random activation layer; generating atoms according to a preset resolution through the entity space sampling layer, performing pixel block segmentation on each reference image in the reference image sequence, and performing spatial sampling on the segmented pixel blocks to extract pixel feature determinations corresponding to the entity; performing feature fusion on the latent representation corresponding to the entity in each reference image in the reference image sequence and the random activation value corresponding to the entity through the implicit representation fusion layer to obtain fused implicit representations, and the latent representation corresponding to the entity is determined by the pixel feature determination corresponding to the entity; performing convolution diffusion processing on the fused implicit representations through the diffusion convolution layer to obtain diffusion feature maps of the entity in the corresponding reference image; ​ The image synthesis layer synthesizes the diffusion feature map of the entity in the corresponding reference image to obtain a single-frame video frame corresponding to each reference image in the sequence of reference images, and outputs the video composed of the single-frame video frames corresponding to each reference image in the sequence of reference images.

5. The method of claim 4, wherein, The random activation layer calculates the random activation value of the entity in each reference image in the sequence of reference images based on the mapping relationship between the entity display time information and the time corresponding to each reference image in the sequence of reference images, and the method comprises the following steps: The random activation layer generates a time activation function value corresponding to each reference image according to the entity display time information and the time corresponding to each reference image in the sequence of reference images, and the time activation function value is used to represent whether the entity is activated at the reference image; According to the time activation function value corresponding to each reference image, the time linear interpolation term and the random disturbance term, the transformation trajectory position of the entity at each reference image is generated. According to the transformation trajectory position of the entity and the preset amplitude control parameter, the random activation value of the entity in each reference image in the sequence of reference images is determined.

6. The method of claim 1, wherein, The user description information comprises entity description information. The first image material matching the user description information is obtained according to the user description information, which comprises the following steps: The entity description information is matched with the image description information of each image material in the image material library to obtain the matching information of the entity description information and the image description information of each image material. The first image material is obtained according to the matching information of the entity description information and the image description information of each image material.

7. The method of claim 6, wherein, The first image material is obtained according to the matching information of the entity description information and the image description information of each image material, which comprises the following steps: In the case that the target matching information is included in the matching information of the entity description information and the image description information of each image material, the image material corresponding to the target matching information is determined as the first image material, and the value of the target matching information is greater than or equal to a preset threshold; In the case that the target matching information is not included in the matching information of the entity description information and the image description information of each image material, a prompt information is displayed in the first interface, and the prompt information is used to prompt the user to provide the first image material; The first image material is obtained according to the input of the user to the prompt information.

8. A video generating apparatus characterized by comprising: The device comprises: The first acquisition module is configured to acquire user description information. The second acquisition module is configured to acquire a first image material matching the user description information according to the user description information. The filling module is configured to fill a blank image sequence according to the first image material according to the user description information to obtain a sequence of reference images. The information processing module is configured to input the sequence of reference images and the user description information into a video generation model to obtain a video output by the video generation model.

9. An electronic device, comprising: The device comprises a processor and a memory storing computer program instructions; The processor implements the video generation method as claimed in any one of claims 1-7 when executing the computer program instructions.

10. A computer-readable storage medium, characterized in that, The computer program instructions stored on the computer readable storage medium implement the video generation method as claimed in any one of claims 1-7 when executed by the processor.

11. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device, so that the electronic device can execute the video generation method as claimed in any one of claims 1-7.

Images