Video generation method and device, electronic equipment and computer readable medium

By extracting the structural information of the reference video and generating a consistent target video, the problem of uncontrollable video structure in the prior art is solved, the video generation effect is improved, and flickering is avoided.

CN118945415APending Publication Date: 2024-11-12BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310539974.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing video generation technology is difficult to effectively generate videos that meet constraint information, resulting in uncontrollable video structure and possible adverse effects such as flickering, affecting the generation effect.

Method used

By obtaining the reference video and constraint text, the structure information in the reference video is extracted, and the target video is generated using this information, so that the frame image data structure information of the target video is consistent with the corresponding frame image data of the reference video, and the problem of uncontrollable structure is avoided .

Benefits of technology

It effectively avoids adverse effects caused by uncontrollable video data structure, improves the video generation effect, and ensures the structural consistency of the generated video while satisfying the constraint text.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945415A_ABST
    Figure CN118945415A_ABST
Patent Text Reader

Abstract

The invention discloses a video generation method and device, electronic equipment and a computer readable medium, and the method comprises the steps: carrying out the structural information extraction processing of each reference image in a reference video after obtaining the reference video and a constraint text, and obtaining the structural information corresponding to each reference image, the structure information can represent the pixel distribution state carried by the reference image; and generating a target video according to the structure information and the constraint text, so that the target video meets the constraint text, and the structure information of each frame of image data in the target video is kept consistent with the structure information of the corresponding frame of reference image in the reference video. In this way, video generation processing can be performed according to the constraint text and the structural constraint carried by the reference video, so that adverse effects caused by uncontrollable structure of the generated video data can be effectively avoided, and the video generation effect can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a video generation method, device, electronic device, and computer-readable medium. Background Art

[0002] In some application scenarios, it may be necessary to generate a video based on some constraint information (for example, text similar to "a robot is shooting a basketball") so that the video satisfies the constraint information.

[0003] However, how to meet the above requirements is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a video generation method, device, electronic device, and computer-readable medium that can meet the above-mentioned video generation requirements.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides a video generation method, the method comprising:

[0007] Acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images;

[0008] Performing structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images;

[0009] A target video is generated according to the structural information and the constraint text.

[0010] In a possible implementation manner, the at least two frames of reference images include a guide image and an image to be processed; a time sequence arrangement position corresponding to the guide image in the reference video is earlier than a time sequence arrangement position corresponding to the image to be processed in the reference video;

[0011] The target video generation process includes:

[0012] Determining image representation data of a first frame image in the target video according to the constraint text, structural information corresponding to the guide image, and noise data corresponding to the guide image;

[0013] The target video is generated according to the constraint text, the structural information corresponding to the at least two frames of reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0014] In a possible implementation manner, generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes:

[0015] Performing text feature extraction processing on the constraint text to obtain text representation data;

[0016] Determining structural representation data according to structural information corresponding to the at least two frames of reference images;

[0017] The target video is generated according to the text representation data, the structure representation data, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0018] In a possible implementation manner, the process of determining the structural characterization data includes:

[0019] Inputting the structural information corresponding to the at least two frames of reference images into a pre-constructed structural feature extraction model to obtain the structural representation data output by the structural feature extraction model;

[0020] or,

[0021] The process of determining the structural characterization data includes:

[0022] The at least two reference images and the structural information corresponding to the at least two reference images are spliced ​​to obtain the structural representation data.

[0023] In a possible implementation manner, the at least two reference image frames include M image groups, where M is a positive integer;

[0024] The step of generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes:

[0025] Initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data; m is a positive integer, 1≤m≤M;

[0026] Determine the image representation data of the mth group of video images in the target video according to the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group;

[0027] Using the image representation data that meets the preset screening condition in the image representation data of the mth group of video images, updating the image constraint representation data, updating m, and continuing to perform the step of determining the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group;

[0028] After the M image groups are traversed, the target video is determined, and the target video is determined according to the image representation data of the video image.

[0029] In a possible implementation manner, the image representation data of the first frame image is determined by using a first generator;

[0030] The target video is generated by a second generator;

[0031] The first generator and the second generator are determined based on the same generator.

[0032] In a possible implementation manner, the process of determining the noise data corresponding to the image to be processed includes:

[0033] Generate a random noise;

[0034] The random noise is corrected using the noise data corresponding to the previous frame image of the image to be processed to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame image.

[0035] In a possible implementation manner, the correction process includes:

[0036] Determining a non-motion state representation position according to a difference between the image to be processed and the previous frame image;

[0037] The value of the non-motion state characterization position in the random noise is updated by using the value of the non-motion state characterization position in the noise data corresponding to the previous frame image, and the noise data corresponding to the image to be processed is obtained, so that the value of the non-motion state characterization position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state characterization position in the noise data corresponding to the previous frame image.

[0038] In a possible implementation, the target video is generated using a pre-built generator;

[0039] The construction process of the generator includes:

[0040] Acquire a training video and text information corresponding to the training video, wherein the training video includes at least two frames of training images;

[0041] Performing structural information extraction processing on each of the training images to obtain structural information corresponding to each of the training images;

[0042] The generator is trained using text information corresponding to the training video and structural information corresponding to each of the training images.

[0043] In a possible implementation manner, the training video includes N frames of training images, where N is a positive integer;

[0044] The training process of the generator includes:

[0045] Performing encoding processing on each of the training images to obtain encoding results of each of the training images;

[0046] Performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N;

[0047] Determine prediction information by using the generator, text information corresponding to the training video, structural information corresponding to each of the training images, an encoding result of the first frame of the training image, and a noise addition result of the second frame of the training image to a noise addition result of the Nth frame of the training image;

[0048] The generator is updated according to the prediction information, and the step of determining the prediction information by using the generator, the text information corresponding to the training video, the structural information corresponding to each of the training images, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image is continued until a preset stop condition is reached.

[0049] In a possible implementation manner, the process of determining the text information corresponding to the training video includes:

[0050] Perform text extraction processing on the first frame of training image in the training video to obtain text information corresponding to the training video.

[0051] In a possible implementation manner, the structural information is a depth map, or an edge map, or a posture map.

[0052] In a possible implementation manner, the process of obtaining the constraint text includes:

[0053] Obtaining constraint description data, where the constraint description data is image data, or video data, or audio data;

[0054] The constraint description data is subjected to text extraction processing to obtain the constraint text.

[0055] The present application provides a video generation device, comprising:

[0056] An acquisition unit, used to acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images;

[0057] An extraction unit, configured to perform structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images;

[0058] A generating unit is used to generate a target video according to the structural information and the constraint text.

[0059] The present application provides an electronic device, the device comprising: a processor and a memory;

[0060] The memory is used to store instructions or computer programs;

[0061] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the video generation method provided in the present application.

[0062] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the video generation method provided by the present application.

[0063] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the video generation method provided by the present application.

[0064] Compared with the prior art, this application has at least the following advantages:

[0065] In the technical solution provided by the present application, after obtaining a reference video (for example, a video that needs to be edited) and a constraint text (for example, the text "a robot is shooting a basketball"), firstly, a structural information extraction process is performed on each reference image in the reference video to obtain the structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.); then, based on the structural information and the constraint text, a target video is generated, so that the target video satisfies the constraint text, and the structural information of each frame image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video (for example, the structural information of the first frame image data in the target video and the structural information of the first frame reference image in the reference video are relatively similar, etc.), so that video generation processing can be performed based on the constraint text and the structural constraints carried by the reference video, thereby effectively avoiding adverse effects caused by the uncontrollable structure of the generated video data (for example, the existence of flickering problems), and thus effectively improving the video generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A flowchart of a video generation method provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a video generation process provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] Through research, it is found that for some video generation schemes, these video generation schemes usually directly generate videos based on pre-set constraint information. It can be seen that these video generation schemes generate videos from scratch, so that the structural information of the videos generated by these video generation schemes is uncontrollable, so that the videos generated by these video generation schemes may have flickering problems, which leads to poor video generation effects.

[0072] In order to better improve the video generation effect, the present application provides a video generation method, which includes: after obtaining a reference video (for example, a video that needs to be edited) and a constraint text (for example, the text of "a robot is shooting a basketball"), first performing structural information extraction processing on each reference image in the reference video to obtain the structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.); then generating a target video based on the structural information and the constraint text, so that the target video satisfies the constraint text, and the structural information of each frame image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video (for example, the structural information of the first frame image data in the target video is consistent with the structural information of the first frame reference image in the reference video, etc.), so that video generation processing can be achieved based on the constraint text and the structural constraints carried by the reference video, thereby effectively avoiding adverse effects caused by the uncontrollable structure of the generated video data (for example, the existence of flickering problems), and thus effectively improving the video generation effect.

[0073] In addition, the present application does not limit the execution subject of the above video generation method. For example, the video generation method provided in the embodiment of the present application can be applied to a device with a data processing function such as a terminal device or a server. For another example, the video generation method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer, etc. The server can be an independent server, a cluster server or a cloud server.

[0074] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] In order to better understand the technical solution provided by the present application, the video generation method provided by the present application is described below with reference to some drawings. Figure 1 As shown, the video generation method provided in the embodiment of the present application includes the following S1-S3. Figure 1 A flowchart of a video generation method provided in an embodiment of the present application.

[0076] S1: Obtain a reference video and a constraint text, where the reference video includes at least two frames of reference images.

[0077] Among them, the reference video refers to the video data required to refer to when generating a new video, and the reference video can be used to provide structural constraints for the generation process of the new video, so that the structural state of each frame image data in the finally generated new video is consistent with the structural state of the corresponding frame image data in the reference video.

[0078] In addition, the present application does not limit the implementation method of the reference video above. For example, in some application scenarios (e.g., a scenario of editing an existing video), the reference video may refer to the video data that needs to be edited. In another example, in other application scenarios, the reference video may refer to the video data selected by the user based on the video generation requirements and used to provide structural constraints. In another example, the reference video may refer to Figure 2 Video 1 shown.

[0079] In addition, for the reference video above, the reference video includes at least two frames of reference images, wherein the reference image refers to image data existing in the reference video; and the present application does not limit the number of the reference images.

[0080] Constraint text refers to text information that needs to be referenced when generating a new video (for example, the text information "a robot is shooting a basketball"), and the constraint text is used to describe the constraint information set for the generation process of the new video through a character string, so that the finally generated new video can satisfy the constraint text.

[0081] In addition, the present application does not limit the implementation method of the above constraint text. For example, in some application scenarios, the constraint text may refer to constraint information expressed in the form of a string set by a user for a new video generation process (for example, Figure 2 prompt text shown).

[0082] Through research, it is found that most types of data other than text data (such as image data, video data, audio data, etc.) can be expressed with the help of text data. Based on this, the present application also provides a possible implementation of the above constraint text acquisition process, in which the constraint text acquisition process can specifically include the following steps 1 to 2.

[0083] Step 1: Get the constraint description data first.

[0084] The constraint description data is used to describe the video generation constraints in a certain data format, and the data format is not a text data format. In addition, the present application does not limit the implementation method of the constraint description data above, for example, it can be any other data except text data. For ease of understanding, three cases are used as examples for explanation below.

[0085] Case 1: In some application scenarios (e.g., generating video data in a certain style), it may be necessary to use image data to express the constraint information (e.g., style constraints, etc.) configured for the new video. Based on this, it can be seen that in a possible implementation, the constraint description data above can be implemented using image data, so that the constraint description data can describe the constraint information in an image data format.

[0086] Case 2: In some other application scenarios, it may be necessary to use video data to express the constraint information configured for the new video. Based on this, it can be seen that in a possible implementation, the constraint description data above can be implemented using video data, so that the constraint description data can describe the constraint information in the video data format.

[0087] Case 3: In some application scenarios, it may be necessary to use audio data (such as user voice data, etc.) to express the constraint information configured for the new video. Based on this, it can be seen that in a possible implementation, the constraint description data above can be implemented using audio data, so that the constraint description data can describe the constraint information in an audio data format.

[0088] Step 2: Perform text extraction on the above constraint description data to obtain constraint text.

[0089] The text extraction process is used to extract text data from the above constraint description data, so that the text data can describe the constraint information carried by the constraint description data through a character string.

[0090] In addition, the present application does not limit the implementation method of the above “text extraction processing”. For example, when the constraint description data is image data, the “text extraction processing” can be implemented by any existing or future method that can convert an image data into text data, such as the BLIP (Bootstrapping Language-Image Pre-training) model. For another example, when the constraint description data is video data, the “text extraction processing” can be implemented by any existing or future method that can convert a video data into text data. For another example, when the constraint description data is audio data, the “text extraction processing” can be implemented by any existing or future method that can convert an audio data into text data.

[0091] Based on the relevant content of steps one to two above, it can be known that in some application scenarios, the above constraint text can be determined by performing text extraction processing on the constraint description data, which can effectively enrich the expression method of constraint information, thereby helping to enrich the application scenarios of the video generation method provided in this application.

[0092] Based on the relevant content of S1 above, it can be known that in some application scenarios (for example, a scenario in which an existing video is edited, etc.), if you want to generate a new video, you need to first obtain a reference video and a constraint text, so that the reference video and the constraint text are both used to describe the constraints required in the video generation process, so that the generation process of the new video can be completed under the two constraints of the reference video and the constraint text.

[0093] S2: Perform structural information extraction processing on each reference image to obtain structural information corresponding to each reference image.

[0094] The structural information corresponding to a reference image is used to describe the structural state of the reference image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.).

[0095] In addition, the present application does not limit the implementation method of the above structural information. For example, it can adopt any existing or future data information that can describe the structural state of an image data (for example, Figure 2 For ease of understanding, three cases are used as examples for explanation.

[0096] Case 1: In some application scenarios, a depth map can be used to describe the structural state of an image data. Based on this, it can be seen that in a possible implementation, the structural information can be expressed in the form of a depth map (for example, Figure 2 The depth map shown, etc.) is implemented.

[0097] Case 2: In some other application scenarios, the structural state of an image data can be described with the help of an edge map. Based on this, it can be seen that in a possible implementation, the structural information can be implemented using an edge map.

[0098] Case 3: In some application scenarios, the structural state of image data can be described with the help of a gesture graph. Based on this, it can be seen that in a possible implementation method, the structural information can be implemented using a gesture graph.

[0099] The above “structural information extraction process” is used to extract structural information from an image data; and the present application does not limit the implementation method of the “structural information extraction process”. For example, when the structural information is implemented using a depth map, the “structural information extraction process” can adopt any existing or future method that can perform depth extraction processing on an image data (for example, Figure 2 For another example, when the structural information is implemented using an edge map, the "structural information extraction process" can be implemented using any existing or future method that can perform edge extraction process on an image data. For another example, when the structural information is implemented using a posture map, the "structural information extraction process" can be implemented using any existing or future method that can perform posture extraction process on an image data.

[0100] Based on the relevant content of S2 above, it can be seen that after obtaining the reference video (for example, Figure 2 After the video 1 shown in the figure, the structural information extraction process (for example, depth extraction process) can be performed on each frame reference image in the reference video to obtain the structural information corresponding to each reference image (for example, Figure 2 N frames of depth map shown), so that these structural information can represent the structural constraints carried by the reference video, so that a new video that meets the structural constraints can be generated based on these structural information (for example, including Figure 2 The N frames shown generate a new video of the image. Wherein N is a positive integer.

[0101] S3: Generate a target video based on the constraint text and the structural information corresponding to each reference image.

[0102] The target video refers to the new video that is finally generated (for example, including Figure 2The N frames shown generate a new video of the image).

[0103] In addition, the number of frames of the image data in the target video is the same as the number of frames of the reference image in the above “at least two reference image frames”. For example, when the above reference video includes N reference image frames, the target video also includes N image data frames.

[0104] In addition, the structural information of each frame of image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the above reference video. That is, the structural information of the first frame of image data in the target video is consistent with the structural information of the first frame of reference image in the reference video; the structural information of the second frame of image data in the target video is consistent with the structural information of the second frame of reference image in the reference video; the structural information of the third frame of image data in the target video is consistent with the structural information of the third frame of reference image in the reference video; ... (and so on); the structural information of the Nth frame of image data in the target video is consistent with the structural information of the Nth frame of reference image in the reference video. Among them, the "consistent" means that the similarity between the two data is relatively high (for example, the similarity reaches a preset similarity threshold, or is exactly the same, etc.).

[0105] In addition, the target video satisfies the above constraint text. For example, when the constraint text is the text "a robot is shooting a basketball", the target video is used to describe the movement process of a robot shooting a basketball. For another example, when the constraint text defines a certain image style (for example, the constraint text is extracted from image data of a certain style), each frame of image data in the target video belongs to this style, so that the stylized processing of a video data can be realized based on the constraint text and the video generation method provided by the present application.

[0106] Furthermore, the present application is not limited to the implementation method of S3 above. For example, it can be implemented by any existing or future method that can generate video data based on the constrained text and the structural information corresponding to each reference image (for example, with the help of a pre-built machine learning model that can generate video data based on the constrained text and the structural information corresponding to each reference image).

[0107] It can be seen that, under a possible implementation, S3 above can specifically be: input the above constraint text and the structural information corresponding to each reference image into a pre-constructed machine learning model to obtain a target video output by the machine learning model, so that the target video satisfies the constraint text, and the structural information of each frame image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video. It should be noted that the present application does not limit the implementation method of the machine learning model. For example, the machine learning model can be implemented using a generator based on a diffusion model. In addition, the present application does not limit the construction process of the machine learning model.

[0108] Based on the relevant contents of S1 to S3 above, it can be known that for the video generation method provided in the embodiment of the present application, after obtaining a reference video (for example, a video that needs to be edited) and a constraint text (for example, the text of "a robot is shooting a basketball"), firstly perform structural information extraction processing on each reference image in the reference video to obtain structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.); then, based on the structural information and the constraint text, generate a target video so that the target video satisfies the constraint text, and the structural information of each frame image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video (for example, the structural information of the first frame image data in the target video is consistent with the structural information of the first frame reference image in the reference video, etc.), so that the video generation processing can be realized under the structural constraints carried by the constraint text and the reference video, so that the adverse effects caused by the uncontrollable structure of the generated video data (for example, the existence of flickering problems) can be effectively avoided, and the video generation effect can be effectively improved.

[0109] In fact, in order to better improve the video generation effect, the first frame image in the new video can be used as a constraint to guide the generation process of all frame image data in the new video. Based on this, the present application also provides a possible implementation of S3 above, in which, when at least two reference images above include a guide image (e.g., the first frame reference image) and an image to be processed (e.g., other reference images except the first frame reference image), and the corresponding time sequence arrangement position of the guide image in the reference video is prior to the corresponding time sequence arrangement position of the image to be processed in the reference video, S3 can specifically include the following steps 11-12.

[0110] Step 11: Determine the image representation data of the first frame image in the target video based on the above constraint text, the structural information corresponding to the above guidance image, and the noise data corresponding to the guidance image.

[0111] The guiding image refers to a reference image existing in the above reference video and used to guide the generation process of non-first frame images in the target video.

[0112] In addition, the present application is not limited to the above-mentioned guiding image. For example, it may refer to the first frame reference image in the above-mentioned reference video.

[0113] The “noise data corresponding to the guidance image” mentioned above refers to the noise data required for reference when determining the first frame image in the target video. For example, the “noise data corresponding to the guidance image” may refer to Figure 2 The “noise data corresponding to the depth map of the first frame” is shown.

[0114] In addition, the present application does not limit the implementation method of the above-mentioned "noise data corresponding to the guidance image", for example, it may refer to randomly generated noise data.

[0115] The above “image representation data of the first frame image in the target video” is used to represent the first frame image data in the target video; and the present application does not limit the implementation method of the “image representation data of the first frame image in the target video”, for example, it can adopt implicit coding (for example, Figure 2 It is implemented as shown in the “hidden coding of the first frame image”).

[0116] In addition, the present application does not limit the implementation method of the above step 11. For example, it can be implemented by using any existing or future method that can generate image representation data (e.g., hidden coding) of a new image based on a text information, structural information corresponding to an image data, and noise data corresponding to the image data.

[0117] For example, in some application scenarios, the above step 11 can be specifically: input the above constraint text, the structural information corresponding to the above guidance image, and the noise data corresponding to the guidance image into a pre-constructed first generator to obtain the image representation data of the first frame image in the target video output by the first generator.

[0118] The first generator is used to generate image representation data (e.g., hidden coding) of a new image based on text information, structural information corresponding to image data, and noise data corresponding to the image data; and the present application does not limit the implementation method of the first generator. For example, the first generator can be implemented using any existing or future image generator. For another example, the first generator can be implemented using a generator constructed using steps 41 to 43 below. For another example, the first generator can be implemented using Figure 2 The generator shown is implemented.

[0119] In fact, in a possible implementation, the above step 11 may also be implemented using the following steps 111 to 113.

[0120] Step 111: Perform text feature extraction processing on the above constraint text to obtain text representation data, so that the text representation data is used to represent the semantic information carried by the constraint text.

[0121] It should be noted that the present application does not limit the implementation method of the above step 111. For example, it can be implemented by adopting any existing or future method that can perform text feature extraction and processing on a text data, such as the Contrastive Language-Image Pre-Training (CLIP) model.

[0122] Step 112: Determine the structural representation data corresponding to the guidance image according to the structural information corresponding to the guidance image, so that the structural representation data can represent the structural state of the guidance image.

[0123] It should be noted that the present application does not limit the implementation method of the above step 112. For ease of understanding, it is described below with reference to three examples.

[0124] Example 1: In a possible implementation manner, the above step 112 may specifically be: directly determining the structural information corresponding to the above guidance image as the structural representation data corresponding to the guidance image.

[0125] Example 2: In another possible implementation, the above step 112 may specifically be: inputting the structural information corresponding to the above guidance image into a pre-built structural feature extraction model to obtain structural representation data corresponding to the guidance image output by the structural feature extraction model.

[0126] The structural feature extraction model is used to perform feature extraction processing on structural information; and the present application does not limit the implementation method of the structural feature extraction model. For example, the structural feature extraction model can adopt any existing or future method that can perform feature extraction processing on structural information (for example, a certain encoding network or Figure 2 For example, in some application scenarios, for the structural feature extraction model, when there are two-dimensional (2D) convolutions and / or 2D attention layers in the structural feature extraction model, a 1D convolution can be added after each 2D convolution, and a 1D temporal attention layer can be added after each 2D attention layer (similar to Figure 2A module with flame in the middle) is used to expand the function of the structural feature extraction model from an image processing function to a video processing function, so that the structural feature extraction model with expanded function can be used to perform feature extraction processing on the structural information corresponding to at least one image data.

[0127] In addition, the present application does not limit the construction process of the structural feature extraction model above. For example, it can be implemented using any existing or future model construction method. For example, in some application scenarios, when using the generator constructed by steps 41 to 43 below (for example, Figure 2 When the structural feature extraction model is included in the generator shown in FIG. 1 , the structural feature extraction model can be constructed according to the process shown in steps 41 to 43 below.

[0128] Example 3, in another possible implementation, the above step 112 may specifically be: splicing the above guidance image with the structural information corresponding to the guidance image to obtain the structural representation data corresponding to the guidance image, so that the structural representation data can better represent the structural state of the guidance image.

[0129] Step 113: Determine the image representation data of the first frame image in the target video according to the above text representation data, the structure representation data corresponding to the above guidance image, and the noise data corresponding to the guidance image.

[0130] It should be noted that the present application does not limit the implementation of step 113. For example, it can be implemented by any existing or future method that can implement step 113. For ease of understanding, the following is an explanation with reference to examples.

[0131] As an example, in one possible implementation, the above step 113 can specifically be: inputting the above text representation data, the structural representation data corresponding to the above guidance image, and the noise data corresponding to the guidance image into a pre-built denoising module to obtain the above "image representation data of the first frame image in the target video" output by the denoising module.

[0132] It should be noted that the present application does not limit the implementation method of the above denoising module. For example, the denoising module can be implemented by using a denoising module in any existing or future diffusion model. For another example, in some application scenarios, the denoising module can be implemented by using a denoising module in a generator constructed using steps 41 to 43 below.

[0133] Based on the above steps 111 to 113, it can be seen that in some application scenarios, Figure 2The generator shown generates image representation data of the first frame image in the target video above, so that the image representation data can better guide the generation process of all frame image data in the target video later.

[0134] Step 12: Generate a target video based on the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0135] The image to be processed refers to a reference image existing in the above reference video and used to generate a non-first frame image in the target video.

[0136] In addition, the present application does not limit the above-mentioned image to be processed. For example, it may refer to other reference images existing in the above-mentioned reference video except the first frame reference image.

[0137] The noise data corresponding to the image to be processed refers to the noise data required to be referenced when determining the non-first frame image in the target video.

[0138] In addition, the present application does not limit the implementation method of the above “noise data corresponding to the image to be processed”, for example, it may refer to randomly generated noise data.

[0139] Through research, it is found that in some application scenarios (for example, when using the denoising module in the diffusion model for video generation processing), the video generation effect is not only related to the pre-set constraint information (for example, the constraint text mentioned above, etc.), but also affected by the noise data. That is, when the constraint information is fixed, different noise data can produce completely different video generation results. It can be seen that in order to better improve the video generation effect, certain constraints can be imposed on the noise data (for example, for the noise data corresponding to two adjacent frames, the noise values ​​corresponding to the pixels used to represent the non-motion state should remain consistent, etc.).

[0140] Based on the content of the above paragraph, the present application also provides a possible implementation method of the process of determining the above "noise data corresponding to the image to be processed". In this implementation method, for any image to be processed, the process of determining the noise data corresponding to the image to be processed can specifically include the following steps 21-22.

[0141] Step 21: Generate a random noise (ie, randomly generated noise data).

[0142] It should be noted that the present application does not limit the implementation method of step 21.

[0143] Step 22: Use the noise data corresponding to the previous frame image of the image to be processed to correct the above random noise to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame image.

[0144] The previous frame image of the image to be processed refers to a reference image existing in the above reference video, whose temporal arrangement position is earlier than the temporal arrangement position of the image to be processed and whose temporal arrangement position is adjacent to the temporal arrangement position of the image to be processed. For example, when the image to be processed refers to the h-th frame reference image in the reference video, the previous frame image of the image to be processed refers to the h-1-th frame reference image in the reference video. Where h is a positive integer, and h-1≥1.

[0145] In addition, the present application does not limit the implementation method of the above step 22. For example, it may specifically include the following steps 221-222.

[0146] Step 221: Determine the non-motion state representation position according to the difference between the image to be processed and the previous frame image of the image to be processed.

[0147] Among them, the non-motion state representation position refers to the pixel point position where the pixel value between two adjacent frame images has not changed or has changed relatively little, so that the non-motion state representation position can represent the pixel point used to represent the non-motion state of the image to be processed relative to the previous frame image of the image to be processed.

[0148] In addition, the present application does not limit the determination process of the above non-motion state characterization position. For example, it can be specifically as follows: first, the image to be processed is subtracted from the previous frame of the image to be processed; then, it is determined whether the difference corresponding to each pixel point is less than a preset threshold value. If it is less than the preset threshold value, it can be determined that the pixel value of the pixel point has hardly changed, so the position of the pixel point can be determined as the non-motion state characterization position; if it is not less than the preset threshold value, it can be determined that the pixel value of the pixel point has changed significantly, so the position of the pixel point can be determined as the motion state characterization position. Among them, the motion state characterization position refers to the pixel point position where the pixel value has changed significantly between two adjacent frames of images, so that the motion state characterization position can represent the pixel point for characterizing the motion state of the image to be processed relative to the previous frame of the image to be processed.

[0149] Step 222: Use the value of the non-motion state representation position in the noise data corresponding to the previous frame image of the image to be processed to update the value of the non-motion state representation position in the random noise, and obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state representation position in the noise data corresponding to the previous frame image.

[0150] In the present application, after determining the non-motion state representation position based on the difference between the image to be processed and the previous frame image of the image to be processed, the value of the non-motion state representation position in the above random noise can be directly replaced with the value of the corresponding non-motion state representation position in the noise data corresponding to the previous frame image to obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the corresponding non-motion state representation position in the noise data corresponding to the previous frame image. In this way, it can be ensured that the noise values ​​corresponding to the pixel points used to represent the non-motion state between adjacent frames are consistent, but the pixel points used to represent the motion state correspond to different noise values, so that the noise data corresponding to the adjacent frames meet certain constraints between the adjacent frames (for example, the constraint of the non-motion state), which is conducive to improving the smoothness of the video, thereby helping to reduce the occurrence of video flickering, and further helping to improve the video generation effect.

[0151] Based on the relevant contents of steps 21 to 22 above, it can be known that in some application scenarios, when the reference video above includes N frames of reference images, for the first frame of reference image, the randomly generated noise data 1 can be directly determined as the noise data corresponding to the first frame of reference image (that is, the above "noise data corresponding to the guidance image"); for the second frame of reference image, after determining the non-motion state representation position and the motion state representation position of the second frame of reference image compared with the first frame of reference image, and determining the randomly generated noise data 2, the value of each non-motion state representation position in the noise data 2 can be directly replaced by the value of the corresponding position in the noise data 1, and the value of each motion state representation position in the noise data 2 is kept unchanged to obtain the noise data corresponding to the second frame of reference image; ... (and so on), which is conducive to improving the smoothness of the generated video data.

[0152] Based on the above content, it can be seen that for any image to be processed, it can be ensured that the values ​​at some positions in the noise data corresponding to the image to be processed are consistent with the values ​​at corresponding positions in the noise data corresponding to the previous frame of the image to be processed. In this way, the purpose of constraining the noise data can be achieved, which is conducive to better improving the video generation effect.

[0153] In addition, the present application does not limit the implementation method of the above step 12. For ease of understanding, some possible implementation methods are described below.

[0154] In a possible implementation manner, the above step 12 may specifically be: firstly input the above constraint text, the structural information corresponding to the above at least two reference images, the image representation data of the above first frame image, and the noise data corresponding to the image to be processed into a pre-constructed second generator (for example, Figure 2 Generator shown in the figure), and obtain the image representation data of each frame of image data in the target video output by the second generator (for example, Figure 2 Then, based on the image representation data of these image data, the target video is determined (for example, the image representation data of each image data is decoded separately to obtain each image data, and the target video is generated based on these image data, etc.).

[0155] The second generator is used to generate image representation data (for example, hidden coding) of all image data in a new video based on a text message, structural information corresponding to multiple image data, image representation data of the first frame image data in the multiple image data, and noise data corresponding to non-first frame image data in the image data; and the present application does not limit the implementation method of the second generator. For example, it can be implemented using any existing or future video generator. For another example, the second generator can be used Figure 2 The generator shown is implemented.

[0156] In addition, the present application does not limit the association relationship between the second generator and the first generator, for example, the two are different generators. For example, in a possible implementation, the second generator and the first generator are based on the same generator (for example, Figure 2 The generator shown in the figure) is determined so that the second generator is the same as the first generator, so that there is better coordination between step 11 implemented based on the first generator and step 12 implemented based on the second generator, and further the above "image representation data of the first frame image in the target video" determined based on the first generator can better guide the generation process of all frame image data in the target video, which is beneficial to improving the video generation effect.

[0157] In fact, in a possible implementation, the above step 12 may also specifically include the following steps 121 to 123.

[0158] Step 121: Perform text feature extraction processing on the above constraint text to obtain text representation data, so that the text representation data is used to represent the semantic information carried by the constraint text.

[0159] It should be noted that the relevant content of step 121 can be found in the above step 111, and for the sake of brevity, it will not be repeated here.

[0160] Step 122: Determine structure representation data according to structure information corresponding to at least two frames of reference images, so that the structure representation data can represent the structure states of these reference images.

[0161] It should be noted that the present application does not limit the implementation of the above step 122. For example, the implementation of the above step 122 is similar to the implementation of the above step 112. For the sake of brevity, it will not be repeated here. For ease of understanding, two examples are combined for explanation below.

[0162] Example 1, in a possible implementation, the above step 122 may specifically be: inputting the structural information corresponding to the above at least two frames of reference images into a pre-built structural feature extraction model, and obtaining the structural representation data output by the structural feature extraction model, so that the structural representation data can represent the structural state of these reference images. It should be noted that the relevant content of the structural feature extraction model can be found above, and for the sake of brevity, it will not be repeated here.

[0163] Example 2, in another possible implementation, the above step 122 can specifically be: splicing the above at least two frames of reference images and the structural information corresponding to the at least two frames of reference images to obtain structural representation data, so that the structural representation data can better represent the structural state of these reference images.

[0164] It should be noted that the present application does not limit the implementation method of the above-mentioned stitching process. For example, it can be specifically: stitching each frame of reference image with the structural information corresponding to the frame of reference image to obtain structural representation data, so that the structural representation data includes the stitching result corresponding to each frame of reference image. Among them, the stitching result corresponding to a frame of reference image is obtained by stitching the reference image with the structural information corresponding to the reference image.

[0165] Based on the relevant content of step 122 above, it can be known that after obtaining the structural information corresponding to at least two frames of reference images above, the structural representation data corresponding to these reference images can be determined based on the structural information so that the structural representation data can represent the structural state of these reference images.

[0166] Step 123: Generate a target video based on the above text representation data, the above structure representation data, the above image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0167] It should be noted that the present application does not limit the implementation of step 123. For example, it can be implemented by any existing or future method that can implement step 123. For ease of understanding, the following is an explanation with reference to examples.

[0168] As an example, in a possible implementation, the above step 123 may specifically be: inputting the above text representation data, the above structure representation data, the image representation data of the above first frame image, and the noise data corresponding to the image to be processed into a pre-built denoising module, and obtaining the image representation data of each frame image data in the target video output by the denoising module (for example, Figure 2 Then, based on the image representation data of these image data, the target video is determined (for example, Figure 2 The N hidden codes shown are decoded separately to obtain N frames of generated images, and the target video is determined based on these generated images).

[0169] It should be noted that the present application does not limit the association between the denoising module in the above paragraph and the denoising module involved in the above step 113. For example, the two can be implemented using different denoising modules. For another example, in some application scenarios, the two can be implemented using the same denoising module.

[0170] It can be seen that in some application scenarios, when the denoising module involved in step 123 above and the denoising module involved in step 113 above are implemented using the same denoising module, the denoising module may have an image generation function and a video generation function. Therefore, in order to ensure the processing performance of the denoising module, some improvements may be made to certain denoising modules (especially, denoising modules with image generation functions), and these improvements are specifically as follows: ① Add some learnable timing parameters after the 2D convolution and / or 2D attention layer in the denoising module, that is, add a 1D convolution after each 2D convolution and / or add a 1D timing attention layer after each 2D attention layer (i.e. Figure 2 Module with flame in the middle) to realize the expansion of the processing function of the denoising module from the image generation function to the video generation function, so that the denoising module can process at least one frame of image data, so that the denoising module can uniformly realize the image / video generation process. ② This application provides a spatial-temporal attention mechanism for the 1D temporal attention layer (for example, Figure 2The self-attention mechanism shown in the figure is based on space-time attention mechanism: Qi (Query) corresponding to the i-th frame image data will perform attention processing on Ki (Key) and Vi (Value) corresponding to each frame image data in N frames image data (i=1,...N), and perform fine-grained modeling between different frames of the video, thereby enhancing the consistency between video frames.

[0171] Based on the above steps 121 to 123, it can be seen that in some application scenarios, Figure 2 The generator shown generates the above target video so that the target video not only satisfies the constraint information carried by the above reference video, but also satisfies the constraint information carried by the above constraint text.

[0172] Based on the relevant contents of steps 11 to 12 above, it can be known that in some application scenarios, the image representation data of the first frame image in the target video can be generated based on the first frame reference image in the reference video above (for example, Figure 2 Then, under the guidance of the image representation data of the first frame image, all the image data in the target video is determined. In this way, the generation process of all the frame image data in the new video can be guided by using the first frame image in the new video as a constraint condition, which is beneficial to improving the video generation effect.

[0173] In fact, in some application scenarios, multiple video segments can be generated iteratively (for example, Figure 2 Based on this, the present application also provides a possible implementation of S3 above. In this implementation, when the above at least two reference images include M image groups, M is a positive integer, and the first image group includes the above guidance image, S3 may specifically include the following steps 31 to 36.

[0174] Step 31: Determine the image representation data of the first frame image in the target video based on the above constraint text, the structural information corresponding to the above guidance image (that is, the first frame image data in the first image group), and the noise data corresponding to the guidance image.

[0175] It should be noted that the relevant content of step 31 can be found in step 11 above, and for the sake of brevity, it will not be repeated here.

[0176] Step 32: Initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data, where m is a positive integer, 1≤m≤M.

[0177] Among them, an image group is used to represent all reference images involved in a round of video segment generation process.

[0178] In addition, the present application does not limit the implementation method of the above M image groups. For example, when the above reference video includes G frame reference images, G is a positive integer, and G=1+M×(J-1), the 1st frame reference image to the Jth frame reference image can be divided into the 1st image group, so that the 1st image group can perform video segment generation processing under the guidance of the generated image corresponding to the 1st frame reference image; the Jth frame reference image to the 2J-1th frame reference image are divided into the 2nd image group, so that the 2nd image group can perform video segment generation processing under the guidance of the generated image corresponding to the Jth frame reference image; ... (and so on). Wherein, J is a positive integer, and the present application does not limit J. For example, J= Figure 2 N shown.

[0179] It should be noted that the present application does not limit the division rule of the above M image groups. For example, the division rule can be set according to the actual application scenario.

[0180] m is used to represent the serial number of the image group to be used in the current round of video segment generation process; and the initial value of m is 1, that is, in the first round of video segment generation process, a new video segment needs to be generated based on the first image group.

[0181] The image constraint representation data refers to the image guidance information (e.g., implicit coding) required in the current round of video segment generation process (i.e., when generating a new video segment based on the mth image group). For example, when m=1, the image constraint representation data is the above “image representation data of the first frame image in the target video”, that is, the video segment generation process in the first round is achieved under the guidance of the “image representation data of the first frame image in the target video”.

[0182] Step 33: Determine the image representation data of the mth group of video images in the target video based on the above constraint text, the structural information corresponding to the mth image group, the above image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group.

[0183] The structural information corresponding to the mth image group includes the structural information corresponding to all reference images in the mth image group, so that the "structural information corresponding to the mth image group" is used to indicate the structural status of all reference images in the mth image group.

[0184] The non-first frame image in the mth image group refers to other reference images existing in the mth image group except for the reference frame image that is the most forward in the time sequence arrangement. For example, when the mth image group includes the 1st reference frame image to the Jth reference frame image, the 2nd reference frame image, the 3rd reference frame image, ..., and the Jth reference frame image can be respectively regarded as the non-first frame image in the mth image group.

[0185] The mth group of video images is used to represent all image data in a new video segment generated based on the mth image grouping.

[0186] The image representation data of the mth group of video images is used to represent the image information carried by all image data in a new video segment generated based on the mth image grouping.

[0187] It should be noted that the present application does not limit the implementation method of the above step 33. For example, the implementation method of the step 33 is similar to any implementation method of the above step 12. For the sake of brevity, it will not be repeated here.

[0188] Step 34: Determine whether all M image groups have been traversed. If so, execute the following step 36; if not, execute the following step 35.

[0189] Step 35: If it is determined that the M image groups have not been traversed, the image constraint representation data is updated using the image representation data of the mth group of video images that meets the preset screening conditions, m is updated, and the process returns to continue to execute the above step 33 and its subsequent steps.

[0190] The preset filtering condition is used to filter out the image guidance information (e.g., hidden coding) required for use in the next round of video segment generation process; and the present application does not limit the preset filtering condition, for example, the preset filtering condition can be set according to the application scenario. For ease of understanding, the following is an example.

[0191] As an example, when the first image group includes the 1st frame reference image to the Jth frame reference image, the second image group includes the Jth frame reference image to the 2J-1th frame reference image, and so on (and so on), the above preset filtering condition can specifically be: the image representation data with the latest arrangement position in the image representation data of the mth group of video images above (that is, the image representation data of the last frame of the video image in the mth group of video images).

[0192] In addition, the present application does not limit the implementation method of the above step of "updating the image constraint representation data using the image representation data in the image representation data of the mth group of video images that meets the preset filtering conditions". For example, it can specifically be: determining the image representation data in the image representation data of the mth group of video images that meets the preset filtering conditions as new image constraint representation data.

[0193] In addition, the present application does not limit the implementation method of "updating m" in the above paragraph. For example, it can be implemented using the following formula (1).

[0194] m'=m+1 (1)

[0195] Where m' represents the updated m value.

[0196] Based on the relevant content of step 35 above, it can be known that for the current round of video segment generation process, if it is determined that the image representation data of M groups of video images have not been obtained, it can be determined that there are still some image groups that have not been traversed, so the image representation data of the last frame of video image can be extracted from the image representation data of the mth group of video images above, and the image representation data of the last frame of video image can be used to update the image constraint representation data, so that the updated image constraint representation data is the image representation data of the last frame of video image, and m needs to be updated so that the above step 33 and its subsequent steps can be re-executed based on the updated image constraint representation data and the updated m, so that the next round of video segment generation process can be started.

[0197] Step 36: If it is determined that the M image groups have been traversed, a target video is determined, where the target video is determined based on the image representation data of the above video image.

[0198] It should be noted that the present application does not limit the implementation method of the step "determining the target video" in the previous paragraph. For example, if the image representation data with the latest arrangement position in the image representation data of the dth group of video images and the image representation data with the latest arrangement position in the image representation data of the d+1th group of video images are both used to represent the same video image, d is a positive integer, d≤M-1, then the image representation data with the latest arrangement position in the image representation data of the 1st group of video images can be deleted first, and the image representation data with the latest arrangement position in the image representation data of the 2nd group of video images can be deleted, and ... (and so on), and the image representation data with the latest arrangement position in the image representation data of the M-1th group of video images can be deleted; then, the image representation data of these M groups of video images are spliced ​​in sequence to obtain an image representation data sequence; finally, the image representation data sequence is decoded to obtain the target video.

[0199] For another example, if the image representation data with the rearmost arrangement position in the image representation data of the d-th group of video images and the image representation data with the frontmost arrangement position in the image representation data of the d+1-th group of video images are both used to represent the same video image, d is a positive integer, d≤M-1, then after obtaining the image representation data of the m-th group of video images, the image representation data of the d-th group of video images are decoded to obtain the m-th image sequence, 1≤m≤M; then, the image data with the rearmost arrangement position in the 1st image sequence is deleted, the image data with the rearmost arrangement position in the 2nd image sequence is deleted, and so on (and so on), and the image data with the rearmost arrangement position in the M-1-th image sequence is deleted; finally, these M image sequences are spliced ​​in order to obtain the target video.

[0200] Based on the relevant contents of steps 31 to 36 above, it can be known that in some application scenarios (for example, scenarios where long videos need to be generated), the generation process of long videos can be completed by iteratively generating video segments (for example, Figure 2 This can effectively avoid overcoming the difficulty of generating long videos, thereby helping to improve the video generation effect.

[0201] In fact, in some application scenarios, the generation process of the target video above can be assisted by a pre-built generator (for example, Figure 2 Based on this, the present application also provides a construction process of the generator, which may specifically include the following steps 41 to 43.

[0202] Step 41: Obtain a training video and text information corresponding to the training video, where the training video includes at least two frames of training images.

[0203] The training video refers to the video data required to build the generator; and this application does not limit the training video, for example, it can be Figure 2 Video 2 shown. In addition, this application does not limit the number of the training videos.

[0204] The training images refer to the image data present in the above training video; and this application does not limit the number of the training images.

[0205] The text information corresponding to the training video is used to describe the content described in the training video; and the present application does not limit the determination process of the "text information corresponding to the training video", for example, it can be implemented through manual annotation.

[0206] In fact, in order to reduce the difficulty of training, this application also provides a possible implementation method of the acquisition process of the above "text information corresponding to the training video", which can be specifically: performing text extraction processing on the first frame of the training image in the training video to obtain the text information corresponding to the training video. It should be noted that this application does not limit the implementation method of the text extraction processing, for example, it can be implemented using BLIP.

[0207] Step 42: Perform structural information extraction processing on each training image to obtain structural information corresponding to each training image.

[0208] The structural information corresponding to a training image is used to describe the structural state of the training image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.).

[0209] It should be noted that the relevant content of the above “structural information corresponding to the training image” is similar to the relevant content of the above “structural information corresponding to the reference image”, and for the sake of brevity, it will not be repeated here.

[0210] Step 43: Train the video generator using the text information corresponding to the training video and the structural information corresponding to each training image.

[0211] It should be noted that the present application does not limit the implementation method of the above step 43. For example, it can be implemented by adopting any existing or future model training method (for example, a diffusion model training method).

[0212] Through research, it is found that the training task can affect the training process of the above generator. Therefore, in order to better reduce the difficulty of training, the task of "generating a new video based on the first frame of image data in a certain video data as a condition" can be used as a training task to be completed in the construction process of the above generator. Based on this, the present application also provides a possible implementation of step 43 above. In this implementation, when the above training video includes N frames of training images, and N is a positive integer, step 43 can specifically include the following steps 431-434.

[0213] Step 431: perform encoding processing on each training image to obtain an encoding result of each training image.

[0214] The encoding result of the nth training image is used to represent the image information carried by the nth training image, where n is a positive integer and n≤N.

[0215] In addition, the present application does not limit the above "encoding result of the nth training image" determination process, for example, it can adopt any existing or future encoding method (for example, any encoder, encoder in the diffusion model or Figure 2The encoder shown in the figure is implemented.

[0216] Step 432: performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N.

[0217] The noise addition result of the i-th frame training image is determined by performing noise addition processing on the encoding result of the i-th frame training image.

[0218] In addition, the present application does not limit the determination method of the above “noise addition result of the i-th frame training image”. For example, it can adopt any existing or future noise addition method (for example, the noise addition module in the diffusion model or Figure 2 It is implemented by the noise addition method shown in the figure, etc.

[0219] Step 433: Determine prediction information using the generator, text information corresponding to the training video, structural information corresponding to each training image, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image.

[0220] Among them, the prediction information refers to the prediction information determined by the generator after the text information corresponding to the training video, the structural information corresponding to each training image, the encoding result of the first frame training image, and the noise addition result of the second frame training image are input into the generator.

[0221] In addition, the present application does not limit the prediction information. For example, in some application scenarios, the prediction information may include N prediction noises (for example, Figure 2 The predicted noise shown in the figure) is used to determine the prediction performance of the generator based on the similarity between the predicted noise and the actual added noise. For example, in other application scenarios, the prediction information may include N generated image data, so that the prediction performance of the generator can be determined based on these generated image data.

[0222] Step 434: Determine whether the preset stop condition is met. If so, end the training process; if not, update the generator according to the above prediction information, and return to continue executing the above step 433 and its subsequent steps.

[0223] Among them, the preset stopping condition refers to the condition that needs to be achieved at the end of the training process of the generator; and the present application does not limit the preset stopping condition. For example, it can be: the loss of the generator is lower than the first threshold, or the loss change rate of the generator is lower than the second threshold (that is, the generator reaches convergence), or the number of updates of the generator reaches the third threshold.

[0224] It should be noted that the "loss of the generator" in the previous paragraph is used to describe the predictive performance of the generator; and the present application does not limit the determination process of the "loss of the generator". For example, it can be implemented by any existing or future method that can measure the predictive performance of the generator (for example, the loss function of the diffusion model, etc.).

[0225] It should also be noted that the present application does not limit the implementation method of the above step "updating the generator according to the above prediction information", which can be set based on the actual application scenario.

[0226] Based on the relevant contents of steps 431 to 434 above, it can be known that in some application scenarios, the task of "generating a new video based on the first frame of image data in a certain video data as a condition" can be used as a training task to be completed in the construction process of the generator above. In this way, only a small amount of training data is needed for the generator to learn how to generate video data. In addition, in some application scenarios, multiple first frames can also be used as indication information to achieve autoregressive generation of longer videos, no longer limited to single video generation. For example, 8 frames are generated for the first time. In order to generate a longer continuous video, the 8th frame can be used as the beginning of the next 8 frames, so that a video of 8+8 frames can be generated. It can be seen that by adopting this method, it is theoretically possible to generate infinitely long videos, no longer limited to a single generation, which is conducive to better improving the video generation effect.

[0227] Based on the relevant contents of steps 41 to 43 above, it can be seen that in some application scenarios, for the construction process of the generator, Figure 2 As shown, the depth map of each training image in the training video can be extracted, and each training image in the training data can be encoded in turn, and the encoding results of each training image in the training video except the first frame training image can be denoised, so that the generator can complete the training task of "generating a new video based on the first frame image data in a certain video data as a condition" based on this information, so that the constructed generator can infer all image frames in the new video based on the first frame image, constraint text, depth map and other information in any video.

[0228] It should be noted that, in order to better improve the construction effect of the generator, the present application can also combine multiple image data and multiple video data to complete the construction process for the generator, so that the constructed generator has both image generation function and video generation function; and in this construction process, if the frame number is 1, it can be determined that the training data is image data, then the training data can be directly denoised, and then the generator can be trained using the noisy training data; if the frame number is greater than 1, it can be determined that the training data is video data, then the generator can be trained using the noisy training data. Figure 2 The generator can be trained in the manner shown.

[0229] Based on the video generation method provided in the embodiment of the present application, the embodiment of the present application also provides a video generation device. Figure 3 Explain and illustrate. Figure 3 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present application. It should be noted that for the technical details of the video generation device provided in an embodiment of the present application, please refer to the relevant content of the video generation method above.

[0230] like Figure 3 As shown, the video generating device 300 provided in the embodiment of the present application includes:

[0231] An acquisition unit 301 is used to acquire a reference video and a constraint text, wherein the reference video includes at least two reference image frames;

[0232] An extraction unit 302, configured to perform structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images;

[0233] The generating unit 303 is used to generate a target video according to the structural information and the constraint text.

[0234] In a possible implementation manner, the at least two frames of reference images include a guide image and an image to be processed; a time sequence arrangement position corresponding to the guide image in the reference video is earlier than a time sequence arrangement position corresponding to the image to be processed in the reference video;

[0235] The generating unit 303 includes:

[0236] A first determination subunit is used to determine the image representation data of the first frame image in the target video according to the constraint text, the structural information corresponding to the guide image, and the noise data corresponding to the guide image;

[0237] The first generating subunit is used to generate the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0238] In a possible implementation manner, the first generating subunit includes:

[0239] A first extraction subunit is used to perform text feature extraction processing on the constraint text to obtain text representation data;

[0240] A second determining subunit, configured to determine structural representation data according to structural information corresponding to the at least two frames of reference images;

[0241] The second generating subunit is used to generate the target video according to the text representation data, the structure representation data, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

[0242] In a possible implementation manner, the second determining subunit is specifically used to: input the structural information corresponding to the at least two frames of reference images into a pre-constructed structural feature extraction model to obtain the structural representation data output by the structural feature extraction model.

[0243] In a possible implementation manner, the second determining subunit is specifically configured to: perform splicing processing on the at least two reference image frames and the structural information corresponding to the at least two reference image frames to obtain the structural representation data.

[0244] In one possible implementation, the at least two frames of reference images include M image groups, where M is a positive integer; the first generating subunit is specifically used to: initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data; m is a positive integer, 1≤m≤M; determine the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; update the image constraint representation data and m using the image representation data in the image representation data of the mth group of video images that meets the preset screening conditions, and continue to execute the step of determining the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; until the M image groups are traversed, the target video is determined, and the target video is determined based on the image representation data of the video image.

[0245] In one possible implementation, the image representation data of the first frame image is determined using a first generator; the target video is generated using a second generator; and the first generator and the second generator are determined based on the same generator.

[0246] In a possible implementation manner, the video generating device 300 further includes:

[0247] A noise generating unit, used for generating a random noise;

[0248] The noise correction unit is used to correct the random noise using the noise data corresponding to the previous frame image of the image to be processed to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame image.

[0249] In one possible implementation, the noise correction unit is specifically used to: determine the non-motion state representation position based on the difference between the image to be processed and the previous frame image; use the value of the non-motion state representation position in the noise data corresponding to the previous frame image to update the value of the non-motion state representation position in the random noise, and obtain the noise data corresponding to the image to be processed, so that the value of the non-motion state representation position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state representation position in the noise data corresponding to the previous frame image.

[0250] In a possible implementation, the target video is generated using a pre-built generator;

[0251] The construction process of the generator includes:

[0252] Acquire a training video and text information corresponding to the training video, wherein the training video includes at least two frames of training images;

[0253] Performing structural information extraction processing on each of the training images to obtain structural information corresponding to each of the training images;

[0254] The generator is trained using text information corresponding to the training video and structural information corresponding to each of the training images.

[0255] In a possible implementation manner, the training video includes N frames of training images, where N is a positive integer;

[0256] The training process of the generator includes:

[0257] Performing encoding processing on each of the training images to obtain encoding results of each of the training images;

[0258] Performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N;

[0259] Determine prediction information by using the generator, text information corresponding to the training video, structural information corresponding to each of the training images, an encoding result of the first frame of the training image, and a noise addition result of the second frame of the training image to a noise addition result of the Nth frame of the training image;

[0260] The generator is updated according to the prediction information, and the step of determining the prediction information by using the generator, the text information corresponding to the training video, the structural information corresponding to each of the training images, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image is continued until a preset stop condition is reached.

[0261] In a possible implementation manner, the process of determining the text information corresponding to the training video includes: performing text extraction processing on the first frame of training image in the training video to obtain the text information corresponding to the training video.

[0262] In a possible implementation manner, the structural information is a depth map, or an edge map, or a posture map.

[0263] In a possible implementation manner, the acquisition unit 301 includes:

[0264] A data acquisition subunit, used to acquire constraint description data, wherein the constraint description data is image data, or video data, or audio data;

[0265] The text extraction subunit is used to perform text extraction processing on the constraint description data to obtain the constraint text.

[0266] Based on the relevant content of the above-mentioned video generating device 300, it can be known that for the video generating device 300 provided in the embodiment of the present application, after obtaining a reference video (for example, a video that needs to be edited) and a constraint text (for example, the text of "a robot is shooting a basketball"), the structural information of each reference image in the reference video is first extracted to obtain the structural information corresponding to each reference image, so that the structural information can represent the pixel distribution state carried by the reference image (for example, which pixels are used to describe the background, which pixels are used to describe the foreground object, etc.); then, based on the structural information and the constraint text, a target video is generated so that the target video satisfies the constraint text, and the structural information of each frame of image data in the target video is respectively consistent with the structural information of the corresponding frame reference image in the reference video (for example, the structural information of the first frame of image data in the target video is consistent with the structural information of the first frame of reference image in the reference video, etc.), so that video generation processing can be achieved under the structural constraints carried by the constraint text and the reference video, thereby effectively avoiding adverse effects caused by the uncontrollable structure of the generated video data (for example, the existence of flickering problems), and thus effectively improving the video generation effect.

[0267] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the video generation method provided in the embodiment of the present application.

[0268] See also Figure 4 , which shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0269] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0270] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0271] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0272] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0273] The embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the video generation method provided in the embodiment of the present application.

[0274] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0275] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0276] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0277] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0278] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0279] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0280] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0281] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0282] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0283] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0284] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0285] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0286] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0287] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images; Performing structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images; A target video is generated according to the structural information and the constraint text.

2. The method according to claim 1, characterized in that: The at least two frames of reference images include a guide image and an image to be processed; the time sequence arrangement position corresponding to the guide image in the reference video is earlier than the time sequence arrangement position corresponding to the image to be processed in the reference video; The target video generation process includes: Determining image representation data of a first frame image in the target video according to the constraint text, structural information corresponding to the guide image, and noise data corresponding to the guide image; The target video is generated according to the constraint text, the structural information corresponding to the at least two frames of reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

3. The method according to claim 2, characterized in that The step of generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes: Performing text feature extraction processing on the constraint text to obtain text representation data; Determining structural representation data according to structural information corresponding to the at least two frames of reference images; The target video is generated according to the text representation data, the structure representation data, the image representation data of the first frame image, and the noise data corresponding to the image to be processed.

4. The method according to claim 3, characterized in that The process of determining the structural characterization data includes: Inputting the structural information corresponding to the at least two frames of reference images into a pre-constructed structural feature extraction model to obtain the structural representation data output by the structural feature extraction model; or, The process of determining the structural characterization data includes: The at least two reference images and the structural information corresponding to the at least two reference images are spliced ​​to obtain the structural representation data.

5. The method according to claim 2, characterized in that: The at least two reference image frames include M image groups, where M is a positive integer; The step of generating the target video according to the constraint text, the structural information corresponding to the at least two reference images, the image representation data of the first frame image, and the noise data corresponding to the image to be processed includes: Initialize m to 1, and use the image representation data of the first frame image to initialize the image constraint representation data; m is a positive integer, 1≤m≤M; Determine the image representation data of the mth group of video images in the target video according to the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; Using the image representation data that meets the preset screening condition in the image representation data of the mth group of video images, updating the image constraint representation data, updating m, and continuing to perform the step of determining the image representation data of the mth group of video images in the target video based on the constraint text, the structural information corresponding to the mth image group, the image constraint representation data, and the noise data corresponding to the non-first frame images in the mth image group; After the M image groups are traversed, the target video is determined, and the target video is determined according to the image representation data of the video image.

6. The method according to claim 2, characterized in that The image representation data of the first frame image is determined by using a first generator; The target video is generated by a second generator; The first generator and the second generator are determined based on the same generator.

7. The method according to claim 2, characterized in that The process of determining the noise data corresponding to the image to be processed includes: Generate a random noise; The random noise is corrected using the noise data corresponding to the previous frame image of the image to be processed to obtain the noise data corresponding to the image to be processed, so that part of the noise data corresponding to the image to be processed is consistent with part of the noise data corresponding to the previous frame image.

8. The method according to claim 7, characterized in that The correction process comprises: Determining a non-motion state representation position according to a difference between the image to be processed and the previous frame image; The value of the non-motion state characterization position in the random noise is updated by using the value of the non-motion state characterization position in the noise data corresponding to the previous frame image, and the noise data corresponding to the image to be processed is obtained, so that the value of the non-motion state characterization position in the noise data corresponding to the image to be processed is consistent with the value of the non-motion state characterization position in the noise data corresponding to the previous frame image.

9. The method according to claim 1, characterized in that: The target video is generated using a pre-built generator; The construction process of the generator includes: Acquire a training video and text information corresponding to the training video, wherein the training video includes at least two frames of training images; Performing structural information extraction processing on each of the training images to obtain structural information corresponding to each of the training images; The generator is trained using text information corresponding to the training video and structural information corresponding to each of the training images.

10. The method according to claim 9, characterized in that The training video includes N frames of training images, where N is a positive integer; The training process of the generator includes: Performing encoding processing on each of the training images to obtain encoding results of each of the training images; Performing noise addition processing on the encoding result of the i-th frame training image to obtain the noise addition result of the i-th frame training image; i is a positive integer, 2≤i≤N; Determine prediction information by using the generator, text information corresponding to the training video, structural information corresponding to each of the training images, an encoding result of the first frame of the training image, and a noise addition result of the second frame of the training image to a noise addition result of the Nth frame of the training image; The generator is updated according to the prediction information, and the step of determining the prediction information by using the generator, the text information corresponding to the training video, the structural information corresponding to each of the training images, the encoding result of the first frame of the training image, and the noise addition result of the second frame of the training image to the noise addition result of the Nth frame of the training image is continued until a preset stop condition is reached.

11. The method according to claim 9, characterized in that The process of determining the text information corresponding to the training video includes: Perform text extraction processing on the first frame of training image in the training video to obtain text information corresponding to the training video.

12. The method according to claim 1, characterized in that The structural information is a depth map, an edge map, or a posture map.

13. The method according to claim 1, characterized in that The process of obtaining the constraint text includes: Obtaining constraint description data, where the constraint description data is image data, or video data, or audio data; The constraint description data is subjected to text extraction processing to obtain the constraint text.

14. A video generating device, characterized in that: include: An acquisition unit, used to acquire a reference video and a constraint text, wherein the reference video includes at least two frames of reference images; An extraction unit, configured to perform structural information extraction processing on each of the reference images to obtain structural information corresponding to each of the reference images; A generating unit is used to generate a target video according to the structural information and the constraint text.

15. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 13.

16. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 13.