Video generation method and apparatus, and electronic device and computer-readable medium

By extracting the structural information of the reference video and generating a target video with a consistent structure, the problem of uncontrollable video generation structure in the prior art is solved, the video generation effect is improved, the flicker phenomenon is reduced, and better video quality is achieved.

WO2024234998A1PCT designated stage expired Publication Date: 2024-11-21BEIJING ZITIAO NETWORK TECH CO LTD

Patent Information

Application Number
PCT/CN2024/090563
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-04-29
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When the existing video generation method generates video based on constraint information, the video structure information is uncontrollable, resulting in poor generation effect and flickering problems.

Method used

By obtaining the reference video and constraint text, the structure information in the reference video is extracted, and the target video is generated using this information, so that the frame image data structure information of the target video is consistent with the frame image data structure information of the reference video, thereby achieving controllable structure video generation.

Benefits of technology

It effectively avoids adverse effects caused by uncontrollable video data structure, improves video generation effect, reduces flickering, and improves video quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024090563_21112024_PF_FP_ABST
    Figure CN2024090563_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present disclosure are a video generation method and apparatus, and an electronic device and a computer-readable medium. The method comprises: when a reference video and constraint text are acquired, first performing structural information extraction processing on each reference image in the reference video to obtain structural information corresponding to each reference image, such that the structural information can represent a pixel distribution state carried by the reference image; and then generating a target video according to the structural information and the constraint text, such that the target video satisfies the constraint text, and structural information of each image data frame in the target video is kept consistent with the structural information of the corresponding reference image frame in the reference video. In this way, video generation processing can be performed according to the constraint text and a structure constraint carried by the reference video, such that adverse effects caused by the structure of generated video data being uncontrollable can be effectively avoided, and thus a video generation effect can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Video generation method, device, electronic device, and computer-readable medium

[0001] This application claims priority to Chinese patent application No. 202310539974.6 filed on May 12, 2023, the entire text of which is incorporated herein by reference as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to a video generation method, device, electronic device, and computer-readable medium. Background Art

[0003] In some application scenarios, it may be necessary to generate a video based on some constraint information (for example, text similar to "a robot shooting a basketball") so that the video satisfies the constraint information.

[0004] However, how to meet the above requirements is a technical problem that needs to be solved urgently.

[0005] Summary of the Invention

[0006] The present disclosure provides a video generation method, device, electronic device, and computer-readable medium, which can meet the above-mentioned video generation requirements.

[0007] In order to achieve the above objectives, the technical solutions provided by the present disclosure are as follows:

[0008] The present disclosure provides a video generation method, the method comprising:

[0009] Acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images;

[0010] Performing structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images;

[0011] A target video is generated according to the structural information and the constraint text.

[0012] In one possible implementation manner, the at least two reference image frames include a guide image and an image to be processed; a time sequence position corresponding to the guide image in the reference video is earlier than a time sequence position corresponding to the image to be processed in the reference video;

[0013] The target video generation process includes:

[0014] Determining image representation data of a first frame image in the target video based on the constraint text, structural information corresponding to the guidance image, and noise data corresponding to the guidance image;

[0015] The target video is generated according to the constraint text, the structural information corresponding to the at least two frames of reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0016] In one possible implementation, generating the target video based on the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes:

[0017] Performing text feature extraction on the constraint text to obtain text representation data;

[0018] Determining structural representation data based on structural information corresponding to the at least two frames of reference images;

[0019] The target video is generated according to the text representation data, the structure representation data, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0020] In one possible implementation, the process of determining the structural characterization data includes:

[0021] Inputting the structural information corresponding to the at least two frames of reference images into a pre-built structural feature extraction model to obtain the structural representation data output by the structural feature extraction model;

[0022] or,

[0023] The process of determining the structural characterization data includes:

[0024] The at least two reference image frames and the structural information corresponding to the at least two reference image frames are spliced ​​together to obtain the structural representation data.

[0025] In a possible implementation manner, the at least two reference image frames include M image groups, where M is a positive integer;

[0026] Generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes:

[0027] Initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data; m is a positive integer, 1≤m≤M;

[0028] Determining image representation data of the mth group of video images in the target video based on the constraint text, structural information corresponding to the mth image group, the image constraint representation data, and noise data corresponding to non-first frame images in the mth image group;

[0029] Using the image representation data that satisfies the preset screening condition in the image representation data of the mth group of video images, updating m, and continuing the step of determining the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group;

[0030] After the M image groups are traversed, the target video is determined, where the target video is determined based on the image representation data of the video image.

[0031] In a possible implementation manner, the image representation data of the first frame image is determined by using a first generator;

[0032] The target video is generated by a second generator;

[0033] The first generator and the second generator are determined based on the same generator.

[0034] In one possible implementation, the process of determining noise data corresponding to the image to be processed includes:

[0035] Generate a random noise;

[0036] The random noise is corrected using the noise data corresponding to the previous frame of the image to be processed to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame of the image.

[0037] In one possible implementation, the correction process includes:

[0038] Determining a non-motion state representation position based on a difference between the image to be processed and the previous frame image;

[0039] The value of the non-motion state representation position in the noise data corresponding to the previous frame image is used to update the value of the non-motion state representation position in the random noise to obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state representation position in the noise data corresponding to the previous frame image.

[0040] In one possible implementation, the target video is generated using a pre-built generator;

[0041] The construction process of the generator includes:

[0042] Obtaining a training video and text information corresponding to the training video, wherein the training video includes at least two frames of training images;

[0043] Performing structural information extraction processing on each of the training images to obtain structural information corresponding to each of the training images;

[0044] The generator is trained using the text information corresponding to the training video and the structural information corresponding to each of the training images.

[0045] In one possible implementation, the training video includes N frames of training images, where N is a positive integer;

[0046] The training process of the generator includes:

[0047] Performing encoding processing on each of the training images to obtain encoding results of each of the training images;

[0048] Performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N;

[0049] Determining prediction information using the generator, text information corresponding to the training video, structural information corresponding to each of the training images, an encoding result of the first training image frame, and a noise addition result of the second training image frame to a noise addition result of the Nth training image frame;

[0050] The generator is updated according to the prediction information, and the step of determining the prediction information by using the generator, the text information corresponding to the training video, the structural information corresponding to each of the training images, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image is continued until a preset stop condition is reached.

[0051] In one possible implementation, the process of determining the text information corresponding to the training video includes:

[0052] Perform text extraction processing on the first training image frame in the training video to obtain text information corresponding to the training video.

[0053] In a possible implementation manner, the structural information is a depth map, an edge map, or a posture map.

[0054] In one possible implementation, the process of obtaining the constraint text includes:

[0055] Obtaining constraint description data, where the constraint description data is image data, or video data, or audio data;

[0056] Perform text extraction processing on the constraint description data to obtain the constraint text.

[0057] The present disclosure provides a video generation device, comprising:

[0058] An acquisition unit, configured to acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images;

[0059] an extraction unit, configured to perform structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images;

[0060] A generating unit is used to generate a target video according to the structural information and the constraint text.

[0061] The present disclosure provides an electronic device, the device comprising: a processor and a memory;

[0062] The memory is used to store instructions or computer programs;

[0063] The processor is configured to execute the instructions or computer programs in the memory, so that the electronic device executes the video generation method provided by the present disclosure.

[0064] The present disclosure provides a computer-readable medium having instructions or a computer program stored therein. When the instructions or the computer program are executed on a device, the device executes the video generation method provided by the present disclosure.

[0065] The present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, wherein the computer program contains program code for executing the video generation method provided by the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] FIG1 is a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0068] FIG2 is a schematic diagram of a video generation process provided by an embodiment of the present disclosure;

[0069] FIG3 is a schematic structural diagram of a video generation device provided by an embodiment of the present disclosure;

[0070] FIG4 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] Research has found that some video generation schemes typically directly generate videos based on pre-set constraints. These schemes generate videos from scratch, making the structural information of the generated videos uncontrollable. This can lead to flickering in the generated videos, resulting in poor video generation quality.

[0072] To better improve video generation effects, the present disclosure provides a video generation method, which includes: after obtaining a reference video (e.g., a video to be edited) and a constraint text (e.g., the text "a robot is shooting a basketball"), first performing structural information extraction processing on each reference image in the reference video to obtain structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (e.g., which pixels are used to describe the background and which pixels are used to describe the foreground object, etc.); then, based on the structural information and the constraint text, generating a target video, so that the target video satisfies the constraint text and the structural information of each frame of image data in the target video is consistent with the structural information of the corresponding frame reference image in the reference video (e.g., the structural information of the first frame of image data in the target video is consistent with the structural information of the first frame of reference image in the reference video, etc.). In this way, video generation processing can be performed based on the constraint text and the structural constraints carried by the reference video, thereby effectively avoiding adverse effects caused by uncontrollable structure of the generated video data (e.g., flickering problems), thereby effectively improving the video generation effect.

[0073] In addition, the present disclosure does not limit the execution subject of the above video generation method. For example, the video generation method provided in the embodiment of the present disclosure can be applied to a device with data processing function, such as a terminal device or a server. For another example, the video generation method provided in the embodiment of the present disclosure can also be implemented with the help of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.

[0074] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0075] To better understand the technical solutions provided by the present disclosure, the video generation method provided by the present disclosure is described below with reference to some accompanying drawings. As shown in FIG1 , the video generation method provided by an embodiment of the present disclosure includes steps S1-S3 below. FIG1 is a flow chart of a video generation method provided by an embodiment of the present disclosure.

[0076] S1: Obtain a reference video and a constraint text, where the reference video includes at least two frames of reference images.

[0077] Among them, the reference video refers to the video data required to refer to when generating a new video, and the reference video can be used to provide structural constraints for the generation process of the new video, so that the structural state of each frame image data in the final generated new video is consistent with the structural state of the corresponding frame image data in the reference video.

[0078] Furthermore, the present disclosure does not limit the implementation of the reference video described above. For example, in some application scenarios (e.g., editing an existing video), the reference video may refer to the video data that needs to be edited. In another example, in other application scenarios, the reference video may refer to video data selected by the user based on video generation requirements to provide structural constraints. For another example, the reference video may refer to Video 1 shown in FIG2 .

[0079] In addition, for the reference video mentioned above, the reference video includes at least two frames of reference images, wherein the reference images refer to image data existing in the reference video; and the present disclosure does not limit the number of the reference images.

[0080] Constraint text refers to the text information that needs to be referenced when generating a new video (for example, the text information "a robot is shooting a basketball"). The constraint text is used to describe the constraint information set for the generation process of the new video through a character string so that the new video finally generated can meet the constraint text.

[0081] In addition, the present disclosure does not limit the implementation method of the above constraint text. For example, in some application scenarios, the constraint text may refer to the constraint information expressed in the form of a string set by the user for the generation process of a new video (for example, the prompt text shown in Figure 2).

[0082] Research has found that most data types other than text data (e.g., image data, video data, audio data, etc.) can be expressed using text data. Based on this, the present disclosure also provides a possible implementation of the above-mentioned constraint text acquisition process. In this implementation, the constraint text acquisition process may specifically include the following steps 1 and 2.

[0083] Step 1: Get the constraint description data first.

[0084] The constraint description data is used to describe the video generation constraints in a data format that is not a text data format. Furthermore, this disclosure does not limit the implementation of the constraint description data described above; for example, the constraint description data may be any other type of data besides text data. For ease of understanding, the following three scenarios are described as examples.

[0085] In case 1, in some application scenarios (e.g., generating video data in a certain style), it may be necessary to use image data to express the constraint information configured for the new video (e.g., style constraints). Based on this, it can be seen that in one possible implementation, the constraint description data above can be implemented using image data, so that the constraint description data can describe the constraint information in an image data format.

[0086] Case 2: In other application scenarios, it may be necessary to use video data to express the constraint information configured for the new video. Based on this, it can be seen that in one possible implementation, the constraint description data above can be implemented using video data, so that the constraint description data can describe the constraint information in a video data format.

[0087] Case 3: In some application scenarios, it may be necessary to use audio data (e.g., user voice data) to express the constraint information configured for the new video. Based on this, it can be seen that in one possible implementation, the constraint description data above can be implemented using audio data, so that the constraint description data can describe the constraint information in an audio data format.

[0088] Step 2: Perform text extraction on the constraint description data above to obtain the constraint text.

[0089] The text extraction process is used to extract text data from the above constraint description data, so that the text data can describe the constraint information carried by the constraint description data through a character string.

[0090] In addition, the present disclosure does not limit the implementation method of the above-mentioned "text extraction processing". For example, when the constraint description data is image data, the "text extraction processing" can be implemented by using any known or future method that can convert image data into text data, such as the BLIP (Bootstrapping Language-Image Pre-training) model. For another example, when the constraint description data is video data, the "text extraction processing" can be implemented by using any known or future method that can convert video data into text data. For another example, when the constraint description data is audio data, the "text extraction processing" can be implemented by using any known or future method that can convert audio data into text data.

[0091] Based on the relevant content of steps one to two above, it can be seen that in some application scenarios, the above constraint text can be determined by performing text extraction processing on the constraint description data, which can effectively enrich the expression method of constraint information, thereby helping to enrich the application scenarios of the video generation method provided by the present disclosure.

[0092] Based on the relevant content of S1 above, it can be seen that in some application scenarios (for example, a scenario of editing an existing video, etc.), if you want to generate a new video, you need to first obtain a reference video and a constraint text, so that the reference video and the constraint text are both used to describe the constraints required in the video generation process, so that the generation process of the new video can be completed under the two constraints of the reference video and the constraint text.

[0093] S2: Perform structural information extraction processing on each reference image to obtain structural information corresponding to each reference image.

[0094] The structural information corresponding to a reference image is used to describe the structural state of the reference image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.).

[0095] In addition, the present disclosure does not limit the implementation of the above structural information. For example, it can be implemented using any known or future data information that can describe the structural state of image data (for example, the depth map shown in Figure 2). For ease of understanding, the following three cases are used as examples for explanation.

[0096] Case 1: In some application scenarios, a depth map can be used to describe the structural state of image data. Based on this, it can be seen that in one possible implementation, the structural information can be implemented using a depth map (for example, the depth map shown in FIG2 ).

[0097] Case 2: In other application scenarios, the structural state of image data can be described with the help of an edge map. Based on this, it can be seen that in a possible implementation, the structural information can be implemented using an edge map.

[0098] Case 3: In some application scenarios, the structural state of image data can be described with the help of a posture graph. Based on this, it can be seen that in a possible implementation method, the structural information can be implemented using a posture graph.

[0099] The above-mentioned “structural information extraction processing” is used to extract structural information from an image data; and the present disclosure does not limit the implementation method of the “structural information extraction processing”. For example, when the structural information is implemented using a depth map, the “structural information extraction processing” can be implemented using any known or future method that can perform depth extraction processing on an image data (for example, the depth extraction method shown in FIG2 ). For another example, when the structural information is implemented using an edge map, the “structural information extraction processing” can be implemented using any known or future method that can perform edge extraction processing on an image data. For another example, when the structural information is implemented using a posture map, the “structural information extraction processing” can be implemented using any known or future method that can perform posture extraction processing on an image data.

[0100] Based on the relevant content of S2 above, it can be known that after obtaining a reference video (for example, video 1 shown in Figure 2), structural information extraction processing (for example, depth extraction processing) can be performed on each reference image frame in the reference video to obtain structural information corresponding to each reference image (for example, the N-frame depth map shown in Figure 2), so that this structural information can represent the structural constraints carried by the reference video, so that a new video that conforms to the structural constraints can be generated based on this structural information (for example, a new video including the N-frame generated images shown in Figure 2). Wherein, N is a positive integer.

[0101] S3: Generate a target video based on the constraint text and the structural information corresponding to each reference image.

[0102] The target video refers to the new video that is finally generated (for example, a new video including the N frames of generated images shown in FIG. 2 ).

[0103] In addition, the number of frames of image data in the target video is the same as the number of frames of the reference image in the above “at least two reference image frames”. For example, when the above reference video includes N frames of reference images, the target video also includes N frames of image data.

[0104] In addition, the structural information of each frame of image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video above. That is, the structural information of the first frame of image data in the target video is consistent with the structural information of the first frame reference image in the reference video; the structural information of the second frame of image data in the target video is consistent with the structural information of the second frame reference image in the reference video; the structural information of the third frame of image data in the target video is consistent with the structural information of the third frame reference image in the reference video; ... (and so on); the structural information of the Nth frame of image data in the target video is consistent with the structural information of the Nth frame reference image in the reference video. Here, "consistent" means that the similarity between the two data is relatively high (for example, the similarity reaches a preset similarity threshold, or is exactly the same, etc.).

[0105] Furthermore, the target video satisfies the aforementioned constraint text. For example, when the constraint text is "A robot shooting a basketball," the target video describes the motion of a robot shooting a basketball. For another example, when the constraint text defines a certain image style (e.g., the constraint text is extracted from image data of a certain style), each frame of image data in the target video belongs to this style, so that subsequent stylized processing of the video data can be achieved based on the constraint text and the video generation method provided by the present disclosure.

[0106] Furthermore, the present disclosure is not limited to the implementation method of S3 above. For example, it can be implemented by any known or future method that can generate video data based on the constrained text and the structural information corresponding to each reference image (for example, by means of a pre-built machine learning model that can generate video data based on the constrained text and the structural information corresponding to each reference image).

[0107] It can be seen that, under one possible implementation, S3 above can specifically be: inputting the above constraint text and the structural information corresponding to each reference image into a pre-built machine learning model to obtain a target video output by the machine learning model, so that the target video satisfies the constraint text, and the structural information of each frame image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video. It should be noted that the present disclosure does not limit the implementation method of the machine learning model. For example, the machine learning model can be implemented using a generator based on a diffusion model. In addition, the present disclosure does not limit the construction process of the machine learning model.

[0108] Based on the relevant contents of S1 to S3 above, it can be seen that for the video generation method provided by the embodiment of the present disclosure, after obtaining a reference video (for example, a video that needs to be edited) and a constraint text (for example, the text "a robot is shooting a basketball"), the structural information of each reference image in the reference video is first extracted to obtain the structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (for example, which pixels are used to describe the background and which pixels are used to describe the foreground object, etc.); then, based on the structural information and the constraint text, a target video is generated so that the target video satisfies the constraint text and the structural information of each frame image data in the target video is consistent with the structural information of the corresponding frame reference image in the reference video (for example, the structural information of the first frame image data in the target video is consistent with the structural information of the first frame reference image in the reference video, etc.). In this way, video generation processing can be performed based on the constraint text and the structural constraints carried by the reference video, thereby effectively avoiding adverse effects caused by the uncontrollable structure of the generated video data (for example, the existence of flickering problems), and thus effectively improving the video generation effect.

[0109] In fact, in order to better improve the video generation effect, the first frame image in the new video can be used as a constraint to guide the generation process of all frame image data in the new video. Based on this, the present disclosure also provides a possible implementation of the above S3. Under this implementation, when the above at least two reference images include a guiding image (for example, the first frame reference image) and an image to be processed (for example, other reference images other than the first frame reference image), and the corresponding temporal arrangement position of the guiding image in the reference video is earlier than the corresponding temporal arrangement position of the image to be processed in the reference video, the S3 can specifically include the following steps 11-12.

[0110] Step 11: Determine the image representation data of the first frame image in the target video based on the above constraint text, the structural information corresponding to the above guidance image, and the noise data corresponding to the guidance image.

[0111] The guiding image refers to a reference image that exists in the reference video and is used to guide the generation process of non-first frame images in the target video.

[0112] In addition, the present disclosure is not limited to the above-mentioned guiding image. For example, it may refer to the first frame reference image in the above-mentioned reference video.

[0113] The "noise data corresponding to the guidance image" mentioned above refers to the noise data required for reference when determining the first frame of the target video. For example, the "noise data corresponding to the guidance image" may refer to the "noise data corresponding to the first depth map frame" shown in Figure 2.

[0114] In addition, the present disclosure does not limit the implementation of the above “noise data corresponding to the guidance image”. For example, it may refer to randomly generated noise data.

[0115] The above-mentioned “image representation data of the first frame image in the target video” is used to represent the first frame image data in the target video; and the present disclosure does not limit the implementation method of the “image representation data of the first frame image in the target video”, for example, it can be implemented using implicit coding (for example, the “implicit coding of the first frame image” shown in Figure 2).

[0116] In addition, the present disclosure does not limit the implementation method of the above step 11. For example, it can be implemented by using any known or future method that can generate image representation data (for example, implicit coding) of a new image based on text information, structural information corresponding to image data, and noise data corresponding to the image data.

[0117] For example, in some application scenarios, the above step 11 can be specifically: inputting the above constraint text, the structural information corresponding to the above guidance image, and the noise data corresponding to the guidance image into a pre-constructed first generator to obtain the image representation data of the first frame image in the target video output by the first generator.

[0118] The first generator is used to generate image representation data (e.g., hidden coding) of a new image based on text information, structural information corresponding to image data, and noise data corresponding to the image data; and the present disclosure does not limit the implementation method of the first generator. For example, the first generator can be implemented using any known or future image generator. For another example, the first generator can be implemented using a generator constructed using steps 41 to 43 below. For another example, the first generator can be implemented using the generator shown in Figure 2.

[0119] In fact, in a possible implementation, the above step 11 can also be implemented using the following steps 111 to 113.

[0120] Step 111: Perform text feature extraction on the above constraint text to obtain text representation data, so that the text representation data is used to represent the semantic information carried by the constraint text.

[0121] It should be noted that the present disclosure does not limit the implementation method of the above step 111. For example, it can be implemented by using any known or future method that can perform text feature extraction processing on a text data, such as the Contrastive Language-Image Pre-Training (CLIP) model.

[0122] Step 112: Determine structural representation data corresponding to the guidance image based on the structural information corresponding to the guidance image, so that the structural representation data can represent the structural state of the guidance image.

[0123] It should be noted that the present disclosure does not limit the implementation of the above step 112. For ease of understanding, the following is an explanation with reference to three examples.

[0124] Example 1: In a possible implementation, step 112 above may specifically be: directly determining the structural information corresponding to the above guidance image as the structural representation data corresponding to the guidance image.

[0125] Example 2: In another possible implementation, the above step 112 may specifically be: inputting the structural information corresponding to the above guidance image into a pre-built structural feature extraction model to obtain the structural representation data corresponding to the guidance image output by the structural feature extraction model.

[0126] Among them, the structural feature extraction model is used to perform feature extraction processing on structural information; and the present disclosure does not limit the implementation method of the structural feature extraction model. For example, the structural feature extraction model can be implemented using any known or future method that can perform feature extraction processing on structural information (for example, a certain encoding network or the feature extraction module shown in Figure 2, etc.). For example, in some application scenarios, for the structural feature extraction model, when there are two-dimensional (2D) convolutions and / or 2D attention layers in the structural feature extraction model, a 1D convolution can be added after each 2D convolution, and a 1D temporal attention layer can be added after each 2D attention layer (similar to the module with flames in Figure 2) to expand the function of the structural feature extraction model from an image processing function to a video processing function, so that the structural feature extraction model with expanded functions can be used to perform feature extraction processing on structural information corresponding to at least one image data.

[0127] In addition, the present disclosure does not limit the construction process of the structural feature extraction model described above. For example, it can be implemented using any known or future model construction method. For example, in some application scenarios, when the generator constructed using steps 41 to 43 below (for example, the generator shown in Figure 2) includes the structural feature extraction model, the structural feature extraction model can be constructed according to the process shown in steps 41 to 43 below.

[0128] Example 3. In another possible implementation, the above step 112 may specifically be: splicing the above guidance image with the structural information corresponding to the guidance image to obtain the structural representation data corresponding to the guidance image, so that the structural representation data can better represent the structural state of the guidance image.

[0129] Step 113: Determine the image representation data of the first frame in the target video based on the above text representation data, the structure representation data corresponding to the above guidance image, and the noise data corresponding to the guidance image.

[0130] It should be noted that the present disclosure does not limit the implementation of step 113. For example, it can be implemented by any known or future method that can implement step 113. For ease of understanding, the following is an explanation with reference to examples.

[0131] As an example, in one possible implementation, the above step 113 can specifically be: inputting the above text representation data, the structural representation data corresponding to the above guidance image, and the noise data corresponding to the guidance image into a pre-built denoising module to obtain the above "image representation data of the first frame image in the target video" output by the denoising module.

[0132] It should be noted that the present disclosure does not limit the implementation of the denoising module described above. For example, the denoising module can be implemented using a denoising module in any known or future diffusion model. For another example, in some application scenarios, the denoising module can be implemented using a denoising module in a generator constructed using steps 41 to 43 below.

[0133] Based on the relevant content of steps 111 to 113 above, it can be seen that in some application scenarios, the generator shown in Figure 2 can be used to generate image representation data of the first frame image in the target video above, so that the image representation data can subsequently better guide the generation process of all frame image data in the target video.

[0134] Step 12: Generate a target video based on the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0135] The image to be processed refers to a reference image existing in the reference video and used to generate a non-first frame image in the target video.

[0136] In addition, the present disclosure is not limited to the above-mentioned image to be processed. For example, it may refer to other reference images existing in the above-mentioned reference video except the first frame reference image.

[0137] The noise data corresponding to the image to be processed refers to the noise data required for reference when determining the non-first frame image in the target video.

[0138] In addition, the present disclosure does not limit the implementation of the above “noise data corresponding to the image to be processed”. For example, it may refer to randomly generated noise data.

[0139] Research has found that in some application scenarios (for example, when using the denoising module in the diffusion model for video generation processing), the video generation effect is not only related to the pre-set constraint information (for example, the constraint text mentioned above, etc.), but also affected by the noise data. In other words, when the constraint information is fixed, different noise data can produce completely different video generation results. It can be seen that in order to better improve the video generation effect, certain constraints can be imposed on the noise data (for example, for the noise data corresponding to two adjacent frames, the noise values ​​corresponding to the pixels used to represent the non-motion state should remain consistent, etc.).

[0140] Based on the content of the above paragraph, the present disclosure also provides a possible implementation method of the process of determining the "noise data corresponding to the image to be processed" mentioned above. In this implementation method, for any image to be processed, the process of determining the noise data corresponding to the image to be processed can specifically include the following steps 21-22.

[0141] Step 21: Generate a random noise (ie, randomly generated noise data).

[0142] It should be noted that the present disclosure does not limit the implementation of step 21.

[0143] Step 22: Using the noise data corresponding to the previous frame of the image to be processed, the above random noise is corrected to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame of the image.

[0144] The previous frame of the image to be processed refers to a reference image in the reference video that is temporally positioned earlier than and adjacent to the temporal position of the image to be processed. For example, when the image to be processed is the hth reference frame in the reference video, the previous frame of the image to be processed refers to the h-1th reference frame in the reference video. Here, h is a positive integer, and h-1 ≥ 1.

[0145] In addition, the present disclosure does not limit the implementation of the above step 22. For example, it may specifically include the following steps 221 and 222.

[0146] Step 221: Determine a non-motion state representation position based on a difference between the image to be processed and a previous frame of the image to be processed.

[0147] Among them, the non-motion state representation position refers to the pixel point position whose pixel value has not changed or has changed relatively little between two adjacent frame images, so that the non-motion state representation position can represent the pixel point used to represent the non-motion state of the image to be processed relative to the previous frame image of the image to be processed.

[0148] In addition, the present disclosure does not limit the determination process of the above-mentioned non-motion state representation position. For example, it can be specifically as follows: first, the image to be processed is subtracted from the previous frame image of the image to be processed; then, it is determined whether the difference corresponding to each pixel point is less than a preset threshold value. If it is less than the preset threshold value, it can be determined that the pixel value of the pixel point has hardly changed, so the position of the pixel point can be determined as the non-motion state representation position; if it is not less than the preset threshold value, it can be determined that the pixel value of the pixel point has changed significantly, so the position of the pixel point can be determined as the motion state representation position. The motion state representation position refers to the pixel point position where the pixel value has changed significantly between two adjacent frames of images, so that the motion state representation position can represent the pixel point for representing the motion state of the image to be processed relative to the previous frame image of the image to be processed.

[0149] Step 222: Use the value of the non-motion state representation position in the noise data corresponding to the previous frame image of the image to be processed to update the value of the non-motion state representation position in the random noise, and obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state representation position in the noise data corresponding to the previous frame image.

[0150] In the present disclosure, after determining the non-motion state representation position based on the difference between the image to be processed and the previous frame image of the image to be processed, the value of the non-motion state representation position in the above random noise can be directly replaced with the value of the corresponding non-motion state representation position in the noise data corresponding to the previous frame image to obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the corresponding non-motion state representation position in the noise data corresponding to the previous frame image. In this way, it can be ensured that the noise values ​​corresponding to the pixel points used to represent the non-motion state between adjacent frames are consistent, but the pixel points used to represent the motion state correspond to different noise values, so that the noise data corresponding to adjacent frames meet certain constraints between the adjacent frames (for example, non-motion state constraints), which is conducive to improving the smoothness of the video, thereby helping to reduce the occurrence of video flickering, and further helping to improve the video generation effect.

[0151] Based on the relevant contents of steps 21 to 22 above, it can be known that in some application scenarios, when the above reference video includes N frames of reference images, for the first frame of reference image, the randomly generated noise data 1 can be directly determined as the noise data corresponding to the first frame of reference image (that is, the above "noise data corresponding to the guidance image"); for the second frame of reference image, after determining the non-motion state representation position and the motion state representation position of the second frame of reference image compared with the first frame of reference image, and determining the randomly generated noise data 2, the values ​​at each non-motion state representation position in the noise data 2 can be directly replaced with the values ​​at the corresponding positions in the noise data 1, and the values ​​at each motion state representation position in the noise data 2 are kept unchanged to obtain the noise data corresponding to the second frame of reference image; ... (and so on), which is conducive to improving the smoothness of the generated video data.

[0152] Based on the content in the previous paragraph, it can be seen that for any image to be processed, it can be ensured that the values ​​at some positions in the noise data corresponding to the image to be processed are consistent with the values ​​at corresponding positions in the noise data corresponding to the previous frame image of the image to be processed. In this way, the purpose of constraining the noise data can be achieved, which is conducive to better improving the video generation effect.

[0153] In addition, the present disclosure does not limit the implementation of the above step 12. For ease of understanding, some possible implementations are described below.

[0154] In one possible implementation, step 12 above may specifically be as follows: first input the above constraint text, the structural information corresponding to the above at least two frames of reference images, the image representation data of the above first frame image, and the noise data corresponding to the image to be processed into a pre-constructed second generator (for example, the generator shown in FIG2 ), and obtain the image representation data of each frame of image data in the target video output by the second generator (for example, the N latent codes shown in FIG2 ); then determine the target video based on the image representation data of these image data (for example, perform decoding processing on the image representation data of each image data separately to obtain each image data, and generate the target video based on these image data, etc.).

[0155] The second generator is used to generate image representation data (e.g., implicit coding) of all image data in a new video based on a text message, structural information corresponding to multiple image data, image representation data of the first frame of image data in the multiple image data, and noise data corresponding to non-first frame image data in the image data; and the present disclosure does not limit the implementation method of the second generator. For example, it can be implemented using any known or future video generator. For another example, the second generator can be implemented using the generator shown in Figure 2.

[0156] In addition, the present disclosure does not limit the association relationship between the above second generator and the above first generator, for example, the two are different generators. For example, in one possible implementation, the above second generator and the above first generator are determined based on the same generator (for example, the generator shown in Figure 2), so that the second generator is the same as the first generator, so that step 11 implemented based on the first generator and step 12 implemented based on the second generator have better coordination, and then the above "image representation data of the first frame image in the target video" determined based on the first generator can better guide the generation process of all frame image data in the target video, which is conducive to improving the video generation effect.

[0157] In fact, in a possible implementation, the above step 12 may also specifically include the following steps 121 to 123.

[0158] Step 121: Perform text feature extraction on the above constraint text to obtain text representation data, so that the text representation data is used to represent the semantic information carried by the constraint text.

[0159] It should be noted that the relevant content of step 121 can be found in step 111 above, and for the sake of brevity, it will not be repeated here.

[0160] Step 122: Determine structure representation data based on structure information corresponding to at least two frames of reference images, so that the structure representation data can represent the structure states of these reference images.

[0161] It should be noted that the present disclosure does not limit the implementation of the above step 122. For example, the implementation of the above step 122 is similar to the implementation of the above step 112. For the sake of brevity, it is not repeated here. For ease of understanding, the following two examples are used for explanation.

[0162] Example 1: In one possible implementation, step 122 may specifically include inputting the structural information corresponding to the at least two reference images into a pre-established structural feature extraction model, obtaining structural representation data output by the structural feature extraction model, such that the structural representation data can represent the structural states of the reference images. It should be noted that the relevant details of the structural feature extraction model are described above and are not further elaborated here for the sake of brevity.

[0163] Example 2: In another possible implementation, the above step 122 may specifically be: splicing the above at least two frames of reference images and the structural information corresponding to the at least two frames of reference images to obtain structural representation data, so that the structural representation data can better represent the structural state of these reference images.

[0164] It should be noted that the present disclosure does not limit the implementation of the stitching process in the above section. For example, it may specifically include stitching each reference image frame with the structural information corresponding to the reference image frame to obtain structural representation data, such that the structural representation data includes the stitching result corresponding to each reference image frame. The stitching result corresponding to a reference image frame is obtained by stitching the reference image frame with the structural information corresponding to the reference image frame.

[0165] Based on the relevant content of step 122 above, it can be known that after obtaining the structural information corresponding to at least two frames of reference images above, the structural representation data corresponding to these reference images can be determined based on this structural information, so that the structural representation data can represent the structural state of these reference images.

[0166] Step 123: Generate a target video based on the above text representation data, the above structure representation data, the above image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0167] It should be noted that the present disclosure does not limit the implementation of step 123. For example, it can be implemented by any known or future method that can implement step 123. For ease of understanding, the following is an explanation with reference to examples.

[0168] As an example, in one possible implementation, the above step 123 can be specifically as follows: inputting the above text representation data, the above structure representation data, the image representation data of the above first frame image, and the noise data corresponding to the image to be processed into a pre-built denoising module to obtain image representation data of each frame image data in the target video output by the denoising module (for example, the N latent codes shown in Figure 2); and then determining the target video based on the image representation data of these image data (for example, decoding the N latent codes shown in Figure 2 separately to obtain N frames of generated images, and determining the target video based on these generated images).

[0169] It should be noted that the present disclosure does not limit the association between the denoising module in the above paragraph and the denoising module involved in step 113 above. For example, the two can be implemented using different denoising modules. For another example, in some application scenarios, the two can be implemented using the same denoising module.

[0170] It can be seen that in some application scenarios, when the denoising module involved in step 123 above and the denoising module involved in step 113 above are implemented using the same denoising module, the denoising module can have image generation function and video generation function. Therefore, in order to ensure the processing performance of the denoising module, some improvements can be made to certain denoising modules (especially, denoising modules with image generation function), and these improvements are specifically as follows: ① Add some learnable timing parameters after the 2D convolution and / or 2D attention layer in the denoising module, that is, add 1D convolution after each 2D convolution and / or add 1D temporal attention layer after each 2D attention layer (that is, the module with flame in Figure 2) to expand the processing function of the denoising module from image generation function to video generation function, so that the denoising module can process at least one frame of image data, so that the denoising module can uniformly implement the image / video generation process. ② The present disclosure provides a space-time attention mechanism (for example, the self-attention mechanism shown in Figure 2) for the 1D temporal attention layer, and the space-time attention mechanism is specifically as follows: Qi (Query) corresponding to the i-th frame image data will perform attention processing on Ki (Key) and Vi (Value) corresponding to each frame image data in N frames of image data (i = 1, ...N), and perform fine-grained modeling between different frames of the video, thereby enhancing the consistency between video frames.

[0171] Based on the relevant content of steps 121 to 123 above, it can be seen that in some application scenarios, the generator shown in Figure 2 can be used to generate the above target video so that the target video not only meets the constraint information carried by the above reference video, but also meets the constraint information carried by the above constraint text.

[0172] Based on the relevant content of steps 11 to 12 above, it can be known that in some application scenarios, the image representation data of the first frame image in the target video can be generated based on the first frame reference image in the reference video above (for example, the implicit code of the first frame image shown in Figure 2); then, under the guidance of the image representation data of the first frame image, all the image data in the target video can be determined. In this way, the generation process of all frame image data in the new video can be guided by using the first frame image in the new video as a constraint condition, which is conducive to improving the video generation effect.

[0173] In fact, in some application scenarios, it is possible to achieve a video generation process of any length (especially a relatively long video) by iteratively generating multiple video segments (for example, the auto-regression method shown in FIG2 ). Based on this, the present disclosure also provides a possible implementation of S3 above. In this implementation, when the at least two reference image frames above include M image groups, M is a positive integer, and the first image group includes the above guidance image, S3 may specifically include the following steps 31 to 36.

[0174] Step 31: Determine the image representation data of the first frame image in the target video based on the above constraint text, the structural information corresponding to the above guidance image (that is, the first frame image data in the first image group), and the noise data corresponding to the guidance image.

[0175] It should be noted that the relevant content of step 31 can be found in step 11 above, and for the sake of brevity, it will not be repeated here.

[0176] Step 32: Initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data, where m is a positive integer, 1≤m≤M.

[0177] An image group is used to represent all reference images involved in a round of video segment generation.

[0178] In addition, the present disclosure does not limit the implementation method of the above M image groups. For example, when the above reference video includes G frame reference images, G is a positive integer, and G = 1 + M × (J-1), the 1st frame reference image to the Jth frame reference image can be divided into the first image group, so that the first image group can perform video segment generation processing under the guidance of the generated image corresponding to the 1st frame reference image; the Jth frame reference image to the 2J-1th frame reference image can be divided into the second image group, so that the second image group can perform video segment generation processing under the guidance of the generated image corresponding to the Jth frame reference image; ... (and so on). Wherein, J is a positive integer, and the present disclosure does not limit J. For example, J = N shown in Figure 2.

[0179] It should be noted that the present disclosure does not limit the division rule of the above M image groups. For example, the division rule can be set according to actual application scenarios.

[0180] m is used to represent the sequence number of the image group to be used in the current round of video segment generation; and the initial value of m is 1, that is, in the first round of video segment generation, a new video segment needs to be generated based on the first image group.

[0181] Image constraint representation data refers to the image guidance information (e.g., implicit coding) required for the current round of video segment generation (i.e., when generating a new video segment based on the mth image group). For example, when m = 1, the image constraint representation data is the "image representation data of the first frame of the target video" mentioned above. That is, the video segment generation process in the first round is achieved under the guidance of this "image representation data of the first frame of the target video."

[0182] Step 33: Determine the image representation data of the mth group of video images in the target video based on the above constraint text, the structural information corresponding to the mth image group, the above image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group.

[0183] Among them, the structural information corresponding to the mth image group includes the structural information corresponding to all reference images in the mth image group, so that the "structural information corresponding to the mth image group" is used to represent the structural state of all reference images in the mth image group.

[0184] The non-first frame images in the mth image group refer to reference images present in the mth image group, excluding the reference image that is positioned earliest in time sequence. For example, when the mth image group includes the 1st to the Jth reference frames, the 2nd, 3rd, ..., and Jth reference frames may be considered as non-first frames in the mth image group.

[0185] The mth group of video images is used to represent all image data in a new video segment generated based on the mth image grouping.

[0186] The image representation data of the mth group of video images is used to represent image information carried by all image data in a new video segment generated based on the mth image grouping.

[0187] It should be noted that the present disclosure does not limit the implementation of step 33 above. For example, the implementation of step 33 is similar to any implementation of step 12 above. For the sake of brevity, it will not be repeated here.

[0188] Step 34: Determine whether all M image groups have been traversed. If so, execute the following step 36; if not, execute the following step 35.

[0189] Step 35: If it is determined that M image groups have not been traversed, the image representation data of the mth group of video images that meets the preset screening conditions is used to update the image constraint representation data, update m, and return to continue executing the above step 33 and subsequent steps.

[0190] The preset filtering conditions are used to select image guidance information (e.g., implicit coding) required for use in the next round of video segment generation. The present disclosure does not limit the preset filtering conditions; for example, the preset filtering conditions can be set based on the application scenario. For ease of understanding, the following examples are provided for illustration.

[0191] As an example, when the first image group includes the 1st frame reference image to the Jth frame reference image, the second image group includes the Jth frame reference image to the 2J-1th frame reference image, and so on (and so on), the above preset filtering conditions can specifically be: the image representation data with the latest arrangement position in the image representation data of the above mth group of video images (that is, the image representation data of the last frame of video image in the mth group of video images).

[0192] In addition, the present disclosure does not limit the implementation method of the above step of "updating the image constraint representation data using the image representation data that meets the preset filtering conditions in the image representation data of the mth group of video images". For example, it can be specifically: determining the image representation data that meets the preset filtering conditions in the image representation data of the mth group of video images as new image constraint representation data.

[0193] In addition, the present disclosure does not limit the implementation of "updating m" in the above paragraph. For example, it can be implemented using the following formula (1). m'=m+1 (1)

[0194] Where m' represents the updated value of m.

[0195] Based on the relevant content of step 35 above, it can be known that for the current round of video segment generation process, if it is determined that the image representation data of M groups of video images have not been obtained, it can be determined that there are still some image groups that have not been traversed, so the image representation data of the last frame of video image can be extracted from the image representation data of the mth group of video images above, and the image representation data of the last frame of video image can be used to update the image constraint representation data, so that the updated image constraint representation data is the image representation data of the last frame of video image, and m also needs to be updated so that the above step 33 and its subsequent steps can be re-executed based on the updated image constraint representation data and the updated m, so that the next round of video segment generation process can be started.

[0196] Step 36: If it is determined that all M image groups have been traversed, a target video is determined. The target video is determined based on the image representation data of the above video image.

[0197] It should be noted that the present disclosure does not limit the implementation method of the step "determining the target video" in the previous paragraph. For example, if the image representation data with the rearmost arrangement position in the image representation data of the dth group of video images and the image representation data with the frontmost arrangement position in the image representation data of the d+1th group of video images are both used to represent the same video image, d is a positive integer, d≤M-1, then the image representation data with the rearmost arrangement position in the image representation data of the 1st group of video images can be deleted first, and the image representation data with the rearmost arrangement position in the image representation data of the 2nd group of video images can be deleted, and so on (and so on), and the image representation data with the rearmost arrangement position in the image representation data of the M-1th group of video images can be deleted; then, the image representation data of these M groups of video images are spliced ​​in sequence to obtain an image representation data sequence; finally, the image representation data sequence is decoded to obtain the target video.

[0198] For another example, if the image representation data with the rearmost arrangement position in the image representation data of the d-th group of video images and the image representation data with the frontmost arrangement position in the image representation data of the d+1-th group of video images are both used to represent the same video image, d is a positive integer, d≤M-1, then after obtaining the image representation data of the m-th group of video images, the image representation data of the d-th group of video images are decoded to obtain the m-th image sequence, 1≤m≤M; then, the image data with the rearmost arrangement position in the 1st image sequence is deleted, the image data with the rearmost arrangement position in the 2nd image sequence is deleted, ... (and so on), and the image data with the rearmost arrangement position in the M-1-th image sequence is deleted; finally, these M image sequences are spliced ​​in order to obtain the target video.

[0199] Based on the relevant content of steps 31 to 36 above, it can be seen that in some application scenarios (for example, scenarios where long videos need to be generated), the long video generation process can be completed by iteratively generating video segments (for example, the autoregressive generation of longer videos as shown in Figure 2). This can effectively avoid overcoming the difficulty of generating long videos, thereby helping to improve the video generation effect.

[0200] In fact, in some application scenarios, the target video generation process described above can be implemented with the help of a pre-built generator (e.g., the generator shown in Figure 2). Based on this, the present disclosure also provides a construction process of the generator, which can specifically include the following steps 41 to 43.

[0201] Step 41: Obtain a training video and text information corresponding to the training video, where the training video includes at least two frames of training images.

[0202] The training video refers to the video data required for constructing the generator; and the present disclosure is not limited to the training video, for example, it can be the video 2 shown in Figure 2. In addition, the present disclosure does not limit the number of the training videos.

[0203] The training images refer to the image data present in the above training video; and the present disclosure does not limit the number of the training images.

[0204] The text information corresponding to the training video is used to describe the content described in the training video; and the present disclosure does not limit the determination process of the "text information corresponding to the training video", for example, it can be implemented through manual annotation.

[0205] In fact, to reduce the difficulty of training, this disclosure also provides a possible implementation of the process for obtaining the "text information corresponding to the training video" mentioned above. Specifically, this process can include performing text extraction on the first training image frame in the training video to obtain the text information corresponding to the training video. It should be noted that this disclosure does not limit the implementation method of this text extraction process; for example, it can be implemented using BLIP.

[0206] Step 42: Perform structural information extraction processing on each training image to obtain structural information corresponding to each training image.

[0207] The structural information corresponding to a training image is used to describe the structural state of the training image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.).

[0208] It should be noted that the relevant content of the above “structural information corresponding to the training image” is similar to the relevant content of the above “structural information corresponding to the reference image”, and for the sake of brevity, it will not be repeated here.

[0209] Step 43: Use the text information corresponding to the training video and the structural information corresponding to each training image to train the video generator.

[0210] It should be noted that the present disclosure does not limit the implementation of the above step 43. For example, it can be implemented by using any known or future model training method (such as a diffusion model training method).

[0211] Research has found that training tasks can affect the training process of the generator described above. Therefore, to further reduce the difficulty of training, the task of "generating a new video using the first frame of image data from a certain video as a condition" can be included as a training task required for the generator's construction. Based on this, the present disclosure also provides a possible implementation of step 43 described above. In this implementation, when the training video includes N training image frames, where N is a positive integer, step 43 can specifically include steps 431-434 described below.

[0212] Step 431: perform encoding processing on each training image to obtain an encoding result of each training image.

[0213] The encoding result of the n-th training image is used to represent the image information carried by the n-th training image, where n is a positive integer and n≤N.

[0214] In addition, the present disclosure does not limit the above-mentioned process of determining the "encoding result of the nth training image". For example, it can be implemented using any known or future encoding method (for example, any encoder, the encoder in the diffusion model, or the encoder shown in Figure 2, etc.).

[0215] Step 432: performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N.

[0216] The noise addition result of the i-th frame training image is determined by performing noise addition processing on the encoding result of the i-th frame training image.

[0217] In addition, the present disclosure does not limit the method for determining the "noise addition result of the i-th frame training image" above. For example, it can be implemented using any known or future noise addition method (for example, the noise addition module in the diffusion model or the noise addition method shown in Figure 2, etc.).

[0218] Step 433: Determine prediction information using the generator, text information corresponding to the training video, structural information corresponding to each training image, the encoding result of the first frame of training image, and the noise addition result of the second frame of training image to the noise addition result of the Nth frame of training image.

[0219] Among them, the prediction information refers to the prediction information determined by the generator after the text information corresponding to the training video, the structural information corresponding to each training image, the encoding result of the first frame training image, and the noise addition result of the second frame training image are input into the generator.

[0220] In addition, the present disclosure does not limit the prediction information. For example, in some application scenarios, the prediction information may include N predicted noises (e.g., the predicted noises shown in FIG2 ), so that the prediction performance of the generator can be determined based on the similarity between the predicted noises and the actual added noise. For another example, in other application scenarios, the prediction information may include N generated image data, so that the prediction performance of the generator can be determined based on these generated image data.

[0221] Step 434: Determine whether the preset stop condition is met. If so, end the training process; if not, update the generator based on the above prediction information, and return to continue executing the above step 433 and its subsequent steps.

[0222] Among them, the preset stopping condition refers to the condition that needs to be achieved at the end of the training process of the generator; and the present disclosure does not limit the preset stopping condition. For example, it can be: the loss of the generator is lower than the first threshold, or the loss change rate of the generator is lower than the second threshold (that is, the generator reaches convergence), or the number of updates of the generator reaches the third threshold.

[0223] It should be noted that the "loss of the generator" in the previous paragraph is used to describe the predictive performance of the generator; and the present disclosure does not limit the determination process of the "loss of the generator". For example, it can be implemented using any known or future method that can measure the predictive performance of the generator (for example, the loss function of the diffusion model, etc.).

[0224] It should also be noted that the present disclosure does not limit the implementation of the above step of "updating the generator according to the above prediction information", which can be set based on actual application scenarios.

[0225] Based on the relevant content of steps 431 to 434 above, it can be seen that in some application scenarios, the task of "generating a new video based on the first frame of image data in a certain video data as a condition" can be used as a training task to be completed in the construction process of the above generator. In this way, only a small amount of training data is needed for the generator to learn how to generate video data. In addition, in some application scenarios, multiple first frames can also be used as indicator information to achieve autoregressive generation of longer videos, no longer limited to single video generation. For example, if 8 frames are generated for the first time, in order to generate a longer continuous video, the 8th frame can be used as the beginning of the next 8 frames, so that an 8+8 frame video can be generated. It can be seen that by adopting this method, theoretically, infinitely long videos can be generated, and they are no longer limited to single generation, which is conducive to better improving the video generation effect.

[0226] Based on the relevant content of steps 41 to 43 above, it can be seen that in some application scenarios, for the construction process of the generator, as shown in Figure 2, the depth map of each training image in the training video can be extracted, and each training image in the training data can be encoded in turn, and the encoding results of each training image in the training video except the first frame training image can be denoised, so that the generator can complete the training task of "generating a new video based on the first frame image data in a certain video data as a condition" based on this information, so that the constructed generator can infer all image frames in the new video based on the first frame image, constraint text, depth map and other information in any video.

[0227] It should be noted that in order to better improve the construction effect of the generator, the present disclosure can also combine multiple image data and multiple video data to complete the construction process for the generator, so that the constructed generator has both image generation function and video generation function; and in this construction process, if the number of frames is 1, it can be determined that the training data is image data, and the training data can be directly denoised, and then the generator can be trained using the noisy training data; if the number of frames is greater than 1, it can be determined that the training data is video data, and the generator can be trained using the method shown in Figure 2.

[0228] Based on the video generation method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide a video generation device, which is explained and illustrated below in conjunction with Figure 3. Figure 3 is a schematic diagram of the structure of a video generation device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the video generation device provided in the embodiments of the present disclosure, please refer to the relevant content of the video generation method above.

[0229] As shown in FIG3 , the video generation device 300 provided by the embodiment of the present disclosure includes:

[0230] An acquisition unit 301 is configured to acquire a reference video and a constraint text, wherein the reference video includes at least two reference image frames;

[0231] An extraction unit 302 is configured to perform structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images;

[0232] The generating unit 303 is configured to generate a target video according to the structural information and the constraint text.

[0233] In one possible implementation manner, the at least two reference image frames include a guide image and an image to be processed; a time sequence position corresponding to the guide image in the reference video is earlier than a time sequence position corresponding to the image to be processed in the reference video;

[0234] The generating unit 303 includes:

[0235] A first determining subunit is configured to determine image representation data of a first frame image in the target video based on the constraint text, structural information corresponding to the guide image, and noise data corresponding to the guide image;

[0236] The first generating subunit is used to generate the target video based on the constraint text, the structural information corresponding to the at least two frames of reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0237] In a possible implementation manner, the first generating subunit includes:

[0238] A first extraction subunit is used to perform text feature extraction processing on the constraint text to obtain text representation data;

[0239] A second determining subunit is configured to determine structural representation data based on structural information corresponding to the at least two reference image frames;

[0240] The second generating subunit is used to generate the target video based on the text representation data, the structure representation data, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0241] In a possible implementation manner, the second determining subunit is specifically configured to input the structural information corresponding to the at least two frames of reference images into a pre-built structural feature extraction model to obtain the structural representation data output by the structural feature extraction model.

[0242] In a possible implementation manner, the second determining subunit is specifically configured to: perform splicing processing on the at least two reference image frames and the structural information corresponding to the at least two reference image frames to obtain the structural representation data.

[0243] In one possible implementation, the at least two frames of reference images include M image groups, where M is a positive integer; the first generating subunit is specifically used to: initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data; m is a positive integer, 1≤m≤M; determine the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; update the image constraint representation data and m using the image representation data of the mth group of video images that meets the preset filtering conditions, and continue to execute the step of determining the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; until the M image groups are traversed, the target video is determined, and the target video is determined based on the image representation data of the video image.

[0244] In one possible implementation, the image representation data of the first frame image is determined using a first generator; the target video is generated using a second generator; and the first generator and the second generator are determined based on the same generator.

[0245] In a possible implementation manner, the video generating device 300 further includes:

[0246] A noise generating unit, used for generating a random noise;

[0247] The noise correction unit is used to correct the random noise using the noise data corresponding to the previous frame image of the image to be processed to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame image.

[0248] In one possible implementation, the noise correction unit is specifically used to: determine the non-motion state representation position based on the difference between the image to be processed and the previous frame image; use the value of the non-motion state representation position in the noise data corresponding to the previous frame image to update the value of the non-motion state representation position in the random noise, and obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state representation position in the noise data corresponding to the previous frame image.

[0249] In one possible implementation, the target video is generated using a pre-built generator;

[0250] The construction process of the generator includes:

[0251] Obtaining a training video and text information corresponding to the training video, wherein the training video includes at least two frames of training images;

[0252] Performing structural information extraction processing on each of the training images to obtain structural information corresponding to each of the training images;

[0253] The generator is trained using the text information corresponding to the training video and the structural information corresponding to each of the training images.

[0254] In one possible implementation, the training video includes N frames of training images, where N is a positive integer;

[0255] The training process of the generator includes:

[0256] Performing encoding processing on each of the training images to obtain encoding results of each of the training images;

[0257] Performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N;

[0258] Determining prediction information using the generator, text information corresponding to the training video, structural information corresponding to each of the training images, an encoding result of the first training image frame, and a noise addition result of the second training image frame to a noise addition result of the Nth training image frame;

[0259] The generator is updated according to the prediction information, and the step of determining the prediction information by using the generator, the text information corresponding to the training video, the structural information corresponding to each of the training images, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image is continued until a preset stop condition is reached.

[0260] In a possible implementation manner, the process of determining the text information corresponding to the training video includes: performing text extraction processing on the first training image frame in the training video to obtain the text information corresponding to the training video.

[0261] In a possible implementation manner, the structural information is a depth map, an edge map, or a posture map.

[0262] In a possible implementation manner, the acquiring unit 301 includes:

[0263] A data acquisition subunit, configured to acquire constraint description data, wherein the constraint description data is image data, video data, or audio data;

[0264] The text extraction subunit is used to perform text extraction processing on the constraint description data to obtain the constraint text.

[0265] Based on the relevant content of the above-mentioned video generation device 300, it can be known that for the video generation device 300 provided in the embodiment of the present disclosure, after obtaining a reference video (for example, a video that needs to be edited) and a constraint text (for example, the text "a robot is shooting a basketball"), the video generation device 300 first performs structural information extraction processing on each reference image in the reference video to obtain structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (for example, which pixels are used to describe the background and which pixels are used to describe the foreground object, etc.); then, based on the structural information and the constraint text, a target video is generated so that the target video satisfies the constraint text and the structural information of each frame image data in the target video is consistent with the structural information of the corresponding frame reference image in the reference video (for example, the structural information of the first frame image data in the target video is consistent with the structural information of the first frame reference image in the reference video, etc.). In this way, video generation processing can be performed based on the constraint text and the structural constraints carried by the reference video, thereby effectively avoiding adverse effects caused by uncontrollable structure of the generated video data (for example, flickering problems), and thus effectively improving the video generation effect.

[0266] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the video generation method provided by the embodiment of the present disclosure.

[0267] Referring to FIG4 , a schematic diagram of the structure of an electronic device 400 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG4 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0268] As shown in Figure 4, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. Various programs and data required for the operation of electronic device 400 are also stored in RAM 403. Processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0269] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 400 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0270] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0271] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0272] The embodiments of the present disclosure further provide a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the video generation method provided in the embodiments of the present disclosure.

[0273] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0274] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0275] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0276] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0277] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0278] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0279] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0280] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0281] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0282] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0283] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0284] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0285] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0286] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, comprising: Acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images; Performing structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images; A target video is generated according to the structural information and the constraint text.

2. The method according to claim 1, wherein: The at least two frames of reference images include a guide image and an image to be processed; the time sequence arrangement position corresponding to the guide image in the reference video is earlier than the time sequence arrangement position corresponding to the image to be processed in the reference video; The target video generation process includes: Determining image representation data of a first frame image in the target video according to the constraint text, structural information corresponding to the guide image, and noise data corresponding to the guide image; The target video is generated according to the constraint text, the structural information corresponding to the at least two frames of reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

3. The method according to claim 2, wherein: The step of generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes: Performing text feature extraction processing on the constraint text to obtain text representation data; Determining structural representation data according to structural information corresponding to the at least two frames of reference images; The target video is generated according to the text representation data, the structure representation data, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

4. The method according to claim 3, wherein: The process of determining the structural characterization data includes: Inputting the structural information corresponding to the at least two frames of reference images into a pre-constructed structural feature extraction model to obtain the structural representation data output by the structural feature extraction model; or, The process of determining the structural characterization data includes: The at least two reference images are compared with the structural information corresponding to the at least two reference images. The structure characterization data is obtained by splicing.

5. The method according to claim 2, wherein: The at least two reference image frames include M image groups, where M is a positive integer; The step of generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes: Initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data; m is a positive integer, 1≤m≤M; Determine the image representation data of the mth group of video images in the target video according to the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; Using the image representation data that meets the preset screening condition in the image representation data of the mth group of video images, updating the image constraint representation data, updating m, and continuing to perform the step of determining the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; After the M image groups are traversed, the target video is determined, and the target video is determined according to the image representation data of the video image.

6. The method according to any one of claims 2 to 5, wherein: The image representation data of the first frame image is determined by using a first generator; The target video is generated by a second generator; The first generator and the second generator are determined based on the same generator.

7. The method according to any one of claims 2 to 5, wherein: The process of determining the noise data corresponding to the image to be processed includes: Generate a random noise; The random noise is corrected using the noise data corresponding to the previous frame image of the image to be processed to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame image.

8. The method according to claim 7, wherein: The correction process comprises: Determining a non-motion state representation position according to a difference between the image to be processed and the previous frame image; The value of the non-motion state characterization position in the random noise is updated by using the value of the non-motion state characterization position in the noise data corresponding to the previous frame image, and the noise data corresponding to the image to be processed is obtained, so that the value of the non-motion state characterization position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state characterization position in the noise data corresponding to the previous frame image.

9. The method according to any one of claims 1 to 8, wherein: The target video is generated using a pre-built generator; The construction process of the generator includes: Acquire a training video and text information corresponding to the training video, wherein the training video includes at least two frames of training images; Performing structural information extraction processing on each of the training images to obtain structural information corresponding to each of the training images; The generator is trained using text information corresponding to the training video and structural information corresponding to each of the training images.

10. The method according to claim 9, wherein: The training video includes N frames of training images, where N is a positive integer; The training process of the generator includes: Performing encoding processing on each of the training images to obtain encoding results of each of the training images; Performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N; Determine prediction information by using the generator, text information corresponding to the training video, structural information corresponding to each of the training images, an encoding result of the first frame of the training image, and a noise addition result of the second frame of the training image to a noise addition result of the Nth frame of the training image; The generator is updated according to the prediction information, and the step of determining the prediction information by using the generator, the text information corresponding to the training video, the structural information corresponding to each of the training images, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image is continued until a preset stop condition is reached.

11. The method according to claim 9, wherein: The process of determining the text information corresponding to the training video includes: Perform text extraction processing on the first frame of training image in the training video to obtain text information corresponding to the training video.

12. The method according to any one of claims 1 to 11, wherein: The structural information is a depth map, an edge map, or a posture map.

13. The method according to any one of claims 1 to 12, wherein: The process of obtaining the constraint text includes: Obtaining constraint description data, where the constraint description data is image data, or video data, or audio data; The constraint description data is subjected to text extraction processing to obtain the constraint text.

14. A video generating device, comprising: An acquisition unit is configured to acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images; an extraction unit, configured to perform structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images; The generating unit is configured to generate a target video according to the structural information and the constraint text.

15. An electronic device, comprising: Processor and memory; The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer programs in the memory so that the electronic device performs the method according to any one of claims 1 to 13.

16. A computer readable medium, wherein: The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Method and device for generating a video, equipment and medium

    CN111415662A

  • Video generation method and device, generation model training method and device, medium and equipment

    CN111933110A

  • Video synthesis method and device, computer equipment and storage medium

    CN114390217A

  • Systems and Methods for Animation Generation

    US20210027511A1

  • Video generation method and related apparatus

    US20230089566A1

Cited By

  • Scene change controllable text video method, system and equipment

    CN119583907A

  • Video generation method and device and electronic equipment

    CN119741222A