Methods, systems, and devices for generating long videos using multi-text video temporal location retrieval

Abstract

This application provides a method, system, and device for generating long videos using multi-text video temporal location retrieval. The method includes: retrieving target video segments from a video library that match each text description based on a text retrieval sequence, obtaining multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions; adjusting the text retrieval sequence into a new text description sequence with the same subject and theme, and editing the multiple target video segments according to the new text description sequence to obtain multiple processed video segments with the same subject and theme; and smoothly transitioning and connecting the multiple processed video segments according to the video sequence order based on the noise characteristics of adjacent processed video segments to obtain a long temporal video, wherein the video sequence order is determined based on the new text description sequence. This method uses less memory and generates long temporal videos that better conform to physical laws.

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to a method, system and device for generating long videos using multi-text video temporal location retrieval. Background Technology

[0002] Long video generation is a task that generates video content with a relatively long time span based on given text guidance, corresponding to a continuous text scene. The length of the video content needs to be significantly longer than a short video clip with only one scene. It has broad application prospects and value.

[0003] However, existing long-video generation methods primarily rely on fitting loss functions to existing video / image data to train models suitable for video generation, resulting in memory consumption that increases with the length of the generated video. Furthermore, due to inadequate or missing fits to the training data, generated video content is prone to ignoring real-world physics (e.g., a table moving). In addition, regarding video continuity, the overall consistency of generated video content remains weaker than that of real video due to discontinuous / unnatural transitions between frames.

[0004] Therefore, how to generate longer videos that better follow physical laws using less memory is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a long video generation method, system and device for multi-text video temporal location retrieval, so as to overcome the above problems or at least partially solve the above problems.

[0006] A first aspect of this application discloses a long video generation method for multi-text video temporal location retrieval, the method comprising:

[0007] Based on the text retrieval sequence, target video segments that match each text description are retrieved from the video library to obtain multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions;

[0008] The text retrieval sequence is adjusted into a new text description sequence with the same subject and theme, and the multiple target video segments are edited according to the new text description sequence to obtain multiple processed video segments with the same subject and theme.

[0009] Based on the noise characteristics of two adjacent processed video segments, the multiple processed video segments are smoothly connected in the order of the video sequence to obtain a long time-series video. The order of the video sequence is determined according to the new text description sequence, and the noise characteristics represent the content information of the processed video.

[0010] Optionally, based on the text retrieval sequence, target video segments matching each text description are retrieved from the video library, including:

[0011] Each text description in the text retrieval sequence is converted into a text embedding feature vector;

[0012] According to a preset time length, each video in the video library is divided into multiple video segments, and features are extracted from each video segment to obtain a video feature vector with the same dimension as the text embedding feature vector.

[0013] The text embedding feature vector is matched with the video feature vector to obtain a video feature vector that matches the text embedding feature vector, and the video segment corresponding to the matched video feature vector is determined as the target video segment that conforms to the text description.

[0014] Optionally, the text retrieval sequence can be adjusted into a new text description sequence with the same subject and topic, including:

[0015] Each text description in the text retrieval sequence is adjusted one by one to obtain multiple text descriptions with the same subject and theme;

[0016] Add the same style description to each of the multiple text descriptions to obtain a new text description sequence with the same subject.

[0017] Optionally, the plurality of target video segments are edited according to the new text description sequence to obtain a plurality of processed video segments with the same subject and theme, including:

[0018] Using a generative diffusion model, the multiple target video segments are sequentially edited based on the new text descriptions in the new text description sequence to obtain multiple processed video segments with the same subject and theme.

[0019] The generative diffusion model has the ability to generate corresponding videos based on text descriptions.

[0020] Optionally, a generative diffusion model is used to sequentially edit the multiple target video segments based on the new text descriptions in the new text description sequence, resulting in multiple processed video segments with the same subject and theme, including:

[0021] Forward noise addition is performed on each frame of the target video segment to obtain noise features with image information;

[0022] Based on the new text description corresponding to the new text description sequence, the noise features are subjected to reverse denoising processing to obtain processed video clips with the same subject and theme.

[0023] The multiple target video segments are edited using the method described above to obtain multiple processed video segments with the same subject and theme.

[0024] Optionally, based on the corresponding new text description in the new text description sequence, the noise features are subjected to reverse denoising processing to obtain processed video segments with the same subject and theme, including:

[0025] By utilizing the noise features of adjacent frames of each frame, the noise features of each frame are corrected to obtain the corrected noise features.

[0026] Based on the corrected noise characteristics, the corrected image for each frame is obtained;

[0027] Based on the corrected image, processed video clips with the same subject and theme are obtained.

[0028] Optionally, based on the noise characteristics of two adjacent processed video segments, the plurality of processed video segments are smoothly transitioned and connected in the order of the video sequence to obtain a long time-series video, including:

[0029] Based on the video sequence order, the first noise feature of the last frame of the previous processed video segment is selected, and the second noise feature of the first frame of the next processed video segment is selected.

[0030] The first noise feature and the second noise feature are weighted to obtain a weighted noise feature sequence.

[0031] The weighted noise feature sequence is processed using a generative diffusion model to generate an intermediate frame image sequence;

[0032] The previous processed video segment and the next processed video segment are joined together based on the intermediate frame image sequence, and a long time sequence video is obtained by joining the multiple processed video segments.

[0033] Optionally, the generative diffusion model is trained according to the following steps:

[0034] Construct a training dataset, which includes: multiple images of the target subject and text descriptions for each image;

[0035] The generative diffusion model is trained using the training dataset so that it learns the image features of the target subject.

[0036] A second aspect of this application discloses a long video generation system for multi-text video temporal location retrieval, the system comprising:

[0037] The video retrieval module is used to retrieve target video segments that match each text description from the video library based on a text retrieval sequence, thereby obtaining multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions.

[0038] The video editing module is used to adjust the text retrieval sequence into a new text description sequence with the same subject and theme, and to edit the multiple target video segments according to the new text description sequence to obtain multiple processed video segments with the same subject and theme.

[0039] The video stitching module is used to smoothly transition and stitch together the multiple processed video segments according to the video sequence order based on the noise characteristics of two adjacent processed video segments, to obtain a long time-series video. The video sequence order is determined based on the new text description sequence, and the noise characteristics represent the content information of the processed video.

[0040] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the long video generation method for multi-text video temporal location retrieval described in the first aspect of this application.

[0041] The embodiments of this application have the following advantages:

[0042] In this embodiment, target video segments matching each text description are retrieved from a video library according to a text retrieval sequence, resulting in multiple target video segments corresponding to the text retrieval sequence, where the text retrieval sequence includes multiple text descriptions. The text retrieval sequence is then adjusted to a new text description sequence with the same subject, and the multiple target video segments are edited according to the new text description sequence to obtain multiple processed video segments with the same subject and theme. Based on the noise characteristics of adjacent processed video segments, the multiple processed video segments are smoothly connected according to the video sequence order to obtain a long time-series video, where the video sequence order is determined according to the new text description sequence, and the noise characteristics characterize the content information of the processed video.

[0043] This application embodiment enhances the adherence of the generated video to real-world physical laws by retrieving target video segments that match the text description from a video library and using existing video segments to generate a long video. Multiple target video segments are edited based on new text description sequences with the same subject, maintaining consistency between the processed video segments and the original real video (i.e., the target video) in terms of video content. This results in segmented generation of video content (i.e., processed video segments) with consistent subject matter and video content style, which can then be used to generate long-sequence videos. Furthermore, noise features characterize the content information of the processed videos, and connecting the processed videos based on these noise features improves the continuity between different processed videos.

[0044] Since this application embodiment processes video segments and generates long temporal videos by combining and transitioning multiple content segments, the method effectively segments the temporal video generation task. Its memory usage does not increase with the video length, maintaining a low memory overhead throughout the process. Compared to other long video generation methods where memory usage increases with video length, this method demonstrates greater versatility. Thus, the long video generation method for multi-text video temporal location retrieval provided in this application embodiment uses less memory and generates long temporal videos that better follow physical laws. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the steps of a long video generation method for multi-text video temporal location retrieval provided in an embodiment of this application;

[0047] Figure 2 This is an overall architecture diagram of a long video generation method for multi-text video temporal location retrieval provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of a long video generation system for multi-text video temporal location retrieval provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] Existing video generation methods mainly include: generative diffusion model-based methods and generative adversarial network (GAN)-based methods. Generative diffusion model-based methods define a Markov chain of diffusion steps, gradually adding random noise to the data, and then learning the inverse diffusion process to construct the desired data samples—text indicators—corresponding to the image content from the noise. GAN-based methods train the model by having a generative model and a discriminative model compete against each other, adjusting the model parameters to obtain a generative model that makes it impossible for the discriminative model to determine whether the generated image is realistic. Furthermore, by combining the generative model with the Transofromer architecture, the temporal coherence of the generated content is enhanced, thus completing the long video generation task.

[0052] Since the aforementioned video generation methods all rely on fitting a loss function to existing video / image data to train a model suitable for video generation, memory usage increases with the duration of the generated video. Furthermore, due to inadequate or missing fits to the training data, the generated video content is prone to ignoring real-world physics (e.g., a table moving). In addition, regarding video continuity, the overall consistency of the generated video content remains weaker than that of real video due to the discontinuous / unnatural transitions between frames.

[0053] Therefore, in order to overcome the limitations of existing video generation methods, this application provides a long video generation method based on multi-text video temporal location retrieval. This method retrieves target video segments that match the text description from an existing video library, uses existing video segments as references to enhance the generation video's adherence to the physical laws of the real world, and adjusts the target video segments to maintain consistency with the original real video content. This results in segmented generation of video content with a consistent theme and video content style for long video generation. At the same time, the entire process of this method maintains a low memory footprint.

[0054] The following description, in conjunction with the accompanying drawings, details the long video generation method for multi-text video temporal location retrieval provided in the application embodiments.

[0055] Reference Figure 1 As shown, Figure 1This is a flowchart illustrating the steps of a long video generation method for multi-text video temporal location retrieval provided in an embodiment of this application. Figure 1 As shown, the long video generation method for multi-text video temporal location retrieval provided in this application embodiment may include steps S110 to S130:

[0056] Step S110: Based on the text retrieval sequence, retrieve target video segments from the video library that match each text description to obtain multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions.

[0057] In this embodiment, the text retrieval sequence corresponds to the script for generating a long-time sequence video, and the events that should occur in the long-time sequence video. For example, in the long-time sequence video, event A should occur first, then event B, and so on. The format of each text description in the text retrieval sequence is "a certain subject is doing something".

[0058] The video library contains a large number of videos from the real physical world, and the target video segment is a temporal segment within a specific video. In other words, for each text description in the text retrieval sequence, not only must videos matching the text description be retrieved from the video library, but the specific temporal segment matching the text description (i.e., the target video segment) must also be located within that video. In some implementations, retrieving the target video segment matching each text description from the video library includes: searching the video library based on each text description in the text retrieval sequence, identifying target videos matching the text description, and locating the corresponding temporal segment from the target videos as the target video segment matching the text description.

[0059] Step S120: Adjust the text retrieval sequence to a new text description sequence with the same subject and theme, and edit the multiple target video segments according to the new text description sequence to obtain multiple processed video segments with the same subject and theme.

[0060] In this embodiment, the subject and theme of each text description in the text retrieval sequence are not necessarily the same. Therefore, the subjects and themes of multiple retrieved target video clips are also not necessarily the same. In order to obtain multiple processed video clips with the same subject and theme, the subject and theme in the text retrieval sequence are adjusted to obtain a new text description sequence with the same subject and theme. For example, the description text "Person A is cooking" and the description text "Person B is playing football" are uniformly modified to "Person C is cooking" and "Person C is playing football".

[0061] After obtaining the new text description sequence, multiple target video segments are edited according to the new text description sequence. That is, the main body and theme of multiple target video segments are edited according to the new text description sequence, and finally multiple processed video segments with the same main body and the same theme are obtained.

[0062] In some optional embodiments, to obtain multiple processed video clips with a unified style, the same style description is added to the descriptive text. Specifically, adjusting the text retrieval sequence into a new text description sequence with the same subject and theme includes: adjusting each text description in the text retrieval sequence one by one to obtain multiple text descriptions with the same subject and theme; adding the same style description to each of the multiple text descriptions to obtain a new text description sequence with the same subject and theme. For example, adding a style description of "watercolor painting style" to each of the multiple text descriptions with the same subject and theme, and then using the new text descriptions carrying the style description to adjust the target video clips, ultimately obtaining multiple processed video clips with the same subject, theme, and consistent style.

[0063] In this embodiment, multiple processed videos are obtained by editing multiple target video segments according to the new text description sequence, rather than directly generating videos. Therefore, the processed video segments maintain the consistency of video content with the target videos, which strengthens the adherence of the processed video segments to the physical laws of the real world.

[0064] Step S130: Based on the noise characteristics of two adjacent processed video segments, the multiple processed video segments are smoothly connected in the order of the video sequence to obtain a long time-series video. The order of the video sequence is determined according to the new text description sequence, and the noise characteristics represent the content information of the processed video.

[0065] In this embodiment, each processed video segment corresponds to a new text description. Therefore, the video sequence order of multiple processed video segments is determined based on the new text description sequence and then concatenated to obtain a long-time video script corresponding to the new text description sequence. For two adjacent processed video segments, the two processed videos are concatenated based on their noise characteristics. Since noise characteristics can characterize the content information of the processed video, concatenating the processed videos based on noise characteristics results in better continuity between different processed videos.

[0066] Based on the above implementation process, by retrieving target video segments from the video library that match the text descriptions, and using existing video segments to generate long videos, the generated videos are better aligned with the physical laws of the real world. Multiple target video segments are edited according to new text description sequences of the same subject, maintaining consistency between the processed video segments and the original real videos (i.e., the target videos). This results in segmented generation of video content (i.e., processed video segments) with consistent subject matter and video content style, which can then be used to generate long-sequence videos. Furthermore, noise features characterize the content information of the processed videos, and connecting the processed videos based on these noise features improves the continuity between different processed videos.

[0067] Since this application embodiment processes video segments and generates long temporal videos by combining and transitioning multiple content segments, the method effectively segments the temporal video generation task. Its memory usage does not increase with the video length, maintaining a low memory overhead throughout the process. Compared to other long video generation methods where memory usage increases with video length, this method demonstrates greater versatility. Thus, the long video generation method for multi-text video temporal location retrieval provided in this application embodiment uses less memory and generates long temporal videos that better follow physical laws.

[0068] The following describes this application in detail with reference to the above implementation steps.

[0069] In an optional embodiment, based on the text retrieval sequence, target video segments matching each text description are retrieved from the video library, including steps A1 to A3:

[0070] Step A1: Convert each text description in the text retrieval sequence into a text embedding feature vector.

[0071] Step A2: According to the preset time length, each video in the video library is divided into multiple video segments, and features are extracted from each video segment to obtain a video feature vector with the same dimension as the text embedding feature vector.

[0072] Step A3: Match the text embedding feature vector with the video feature vector to obtain a video feature vector that matches the text embedding feature vector, and determine the video segment corresponding to the matched video feature vector as the target video segment that conforms to the text description.

[0073] In this embodiment, a text embedding feature vector refers to a numerical representation containing the meaning of the text. Converting each text description in the text retrieval sequence into a text embedding feature vector can be achieved by using a network with text feature extraction capabilities to extract features from each text description in the text retrieval sequence, thereby converting each text description into a text embedding feature vector. In some implementations, the network with text feature extraction capabilities can be a text editor, which converts each text description in the text retrieval sequence into a text embedding feature vector in the latent space.

[0074] In order to find video segments (i.e., sequential segments) that match each text description from the video library, for each video in the video library, the video is divided into multiple video segments of a preset time length (where the preset time is flexibly determined according to the processing speed and processing accuracy), and then the target video segment that matches each text description is found from the segmented video segments.

[0075] Feature extraction can be performed on each video segment using a pre-trained visual module (e.g., a C3D visual model) to obtain multiple video feature vectors f with the same dimension as the text embedding vector. ij Among them, the video feature vector f ij This represents the video feature vector derived from multiple video segments corresponding to the start time of the i-th video and the end time of the j-th video.

[0076] For each text description, the text embedding feature vector corresponding to the text description is matched with the video feature vector of the video segment. If the text embedding feature vector matches the video feature vector, it means that the video segment corresponding to the video feature vector is a video segment that matches the text description, and therefore the video segment is determined as the target video segment.

[0077] Matching the text embedding feature vector with the video feature vector can be achieved by calculating the similarity distance between the text embedding feature vector and the video feature vector. In some embodiments, a cross-modal bidirectional matching network is used to complete the video temporal localization task, that is, to calculate the similarity distance between the text embedding feature vector and the video feature vector for matching, and finally obtain the target video segment that best matches the description of each text. Here, the similarity distance can be any distance that can measure the similarity between two feature vectors. For example, the similarity distance can be cosine distance, Mahalanobis distance, etc., and this application does not limit it.

[0078] In this embodiment, since the videos in the video library are segmented into video segments for processing, each matching calculation only processes the video feature vector corresponding to a specific video segment. Compared to frame-by-frame processing, this improves processing efficiency and reduces memory usage, and its memory usage does not increase with the video length. Furthermore, compared to processing the entire video directly, this method can not only identify videos that match the text description but also locate specific temporal segments that match the text description, improving the accuracy of video retrieval. Therefore, this method maintains low memory consumption and demonstrates greater versatility compared to other long video generation methods where memory usage increases with video length.

[0079] In one optional embodiment, editing the plurality of target video segments according to the new text description sequence to obtain a plurality of processed video segments with the same subject and the same theme includes: using a generative diffusion model to sequentially edit the plurality of target video segments according to the new text description in the new text description sequence to obtain a plurality of processed video segments with the same subject and the same theme; wherein, the generative diffusion model has the ability to generate corresponding videos based on text descriptions.

[0080] In this embodiment, since the new text description sequence has the same theme, a generative diffusion model can generate multiple processed video segments with the same subject and theme corresponding to the new text description sequence. To obtain multiple processed video segments with better continuity, a generative diffusion model is used in conjunction with the noise features of each frame to edit the target video segments, ultimately resulting in processed video segments with better continuity.

[0081] Furthermore, using a generative diffusion model, the multiple target video segments are sequentially edited based on the new text descriptions in the new text description sequence to obtain multiple processed video segments with the same subject and theme, including steps B1 to B3:

[0082] Step B1: Perform forward noise processing on each frame of the target video segment to obtain noise features with image information.

[0083] The noise features of each frame can represent the image information in the image. The DDIM (Denoising Diffusion Implicit Models) inversion method is used to perform forward noise processing on each frame of the target video segment, thereby obtaining noise features with image information.

[0084] Step B2: Based on the new text description corresponding to the new text description sequence, perform reverse denoising on the noise features to obtain processed video segments with the same subject and theme.

[0085] Specifically, the DDIM sampling method using the generative diffusion model is used to reverse-denoise the noise features of each frame of the image by utilizing the corresponding new text description in the new text description sequence to obtain the frame-by-frame edited image. The frame-by-frame edited images are then aggregated to obtain processed video clips with the same subject and theme.

[0086] In one alternative embodiment, considering that simply editing each frame of the image may result in discontinuous image content between adjacent frames, causing fragmentation of the video content, a preceding frame feature interpolation and a cross-window attention mechanism are used to process each frame of the image.

[0087] Specifically, based on the new text description corresponding to the new text description sequence, the noise features are subjected to reverse denoising processing to obtain processed video segments with the same subject and theme. This includes: using the noise features of adjacent frames of each frame to correct the noise features of each frame to obtain corrected noise features; obtaining corrected frames based on the corrected noise features; and obtaining processed video segments with the same subject and theme based on the corrected images.

[0088] In this context, adjacent frames refer to the preceding frames or the frames immediately before and after the current frame. By utilizing the noise characteristics of adjacent frames, the noise characteristics of each frame are corrected, ensuring temporal continuity between the noise characteristics of the inter-frame images and the edited images, thereby reducing the sense of disjointedness in the video content.

[0089] Step B3: Edit the multiple target video segments according to the above method to obtain multiple processed video segments with the same subject and theme.

[0090] Each target video segment is processed according to the methods described in steps B1 and B2 above, ultimately resulting in multiple processed video segments with the same subject and theme.

[0091] In this embodiment, multiple target video segments are edited based on a new text description sequence with the same subject, maintaining the consistency of video content between the processed video segments and the original real video (i.e., the target video). Video content with the same subject and video content style is generated in segments for generating long-sequence videos. Compared with directly generated video content, the processed video segments generated by this method follow physical laws better.

[0092] In an optional embodiment, based on the noise characteristics of two adjacent processed video segments, the plurality of processed video segments are smoothly transitioned and connected in the order of the video sequence to obtain a long time-series video, including steps C1 to C4:

[0093] Step C1: Based on the video sequence order, select the first noise feature of the last frame of the previous processed video segment, and select the second noise feature of the first frame of the next processed video segment.

[0094] Step C2: Perform weighted processing based on the first noise feature and the second noise feature to obtain a weighted noise feature sequence.

[0095] Step C3: Process the weighted noise feature sequence using a generative diffusion model to generate an intermediate frame image sequence.

[0096] Step C4: Connect the previous processed video segment and the next processed video segment according to the intermediate frame image sequence, and obtain a long time sequence video by connecting the multiple processed video segments.

[0097] In this embodiment, the intermediate frame image sequence refers to the image sequence used to connect the last frame of the previous processed video segment and the first frame of the next processed video segment. To ensure better video connection and avoid a sense of discontinuity between the two processed video segments, a weighted processing method is used. This method utilizes the first noise feature of the last frame of the previous processed video segment and the second noise feature of the first frame of the next processed video segment to construct the noise features corresponding to the intermediate frame image sequence (i.e., a weighted noise feature sequence). This weighted noise feature sequence simultaneously contains image information from both the last frame of the previous processed video segment and the first frame of the next processed video segment. Therefore, the intermediate frame image sequence generated based on the weighted noise features can effectively connect the two segments, thereby obtaining a long-time sequence video by connecting multiple processed video segments.

[0098] Specifically, the weighted noise feature sequence includes multiple weighted noise features (e.g., at least 16 weighted noise features in the weighted noise feature sequence). In some embodiments, a weighted processing is performed based on the first noise feature and the second noise feature to obtain the weighted noise feature sequence, including: transitioning the weight of the first noise feature in the weighted noise feature sequence from 100% to 0% through a linear / spherical linear method, and transitioning the weight of the second noise feature in the weighted noise feature sequence from 0% to 100%, thereby obtaining a weighted noise feature sequence composed of multiple weighted noise features. Each weighted noise feature corresponds to the generation of an intermediate frame image, and the sequence of intermediate frame images generated by the weighted noise feature sequence constitutes a stitched video, which is used to complete the stitching of the preceding and following videos.

[0099] In one optional embodiment, since the subjects of the target video segments retrieved from the video library based on the text retrieval sequence are different, for example, the subjects include the elderly, children, young people, etc., and the embodiments of this application expect to generate long-term videos with the same subject, for a specific subject, the pre-trained generative diffusion model may be unable to effectively generate videos with the same subject because the specific subject (e.g., a cartoon character or a movie character) has not appeared in the training data.

[0100] To address this issue, the generative diffusion model described in this application embodiment is trained according to the following steps:

[0101] Step D1: Construct a training dataset, which includes: multiple images of the target subject and text descriptions for each image.

[0102] Step D2: Train the generative diffusion model using the training dataset so that the generative diffusion model learns the image features of the target subject.

[0103] In this embodiment, the target subject refers to a specific subject, and the generative diffusion model refers to a pre-trained generative diffusion model capable of generating videos of general subjects. For the target subject, multiple images of the target subject, along with text descriptions of each image, are selected as a training dataset. This training dataset is then input into the generative diffusion model for fine-tuning, enabling the model to learn the image features of the target subject. Subsequently, the generative diffusion model can generate the target subject from images or videos based on these image features.

[0104] It is understandable that if the subject in the generated long-term video is widely recognized, and the generative diffusion model has already been exposed to image data of that subject during pre-training, then it is not necessary to fine-tune the generative diffusion model using the above training steps. If the long-term video generation is for a specific target subject, then the generative diffusion models used in the above steps of "using the generative diffusion model to sequentially edit the multiple target video segments according to the new text descriptions in the new text description sequence" and "using the generative diffusion model to process the weighted noise features" are both generative diffusion models trained using the above training steps. The generative diffusion model trained according to the above training steps can normally produce videos with the target subject, ultimately resulting in a long-term video with the target subject.

[0105] For example, Figure 2 This is an overall architecture diagram of a long video generation method for multi-text video temporal location retrieval provided in this application embodiment. The method mainly includes three stages: video segment location, video segment editing, and video segment splicing. In some embodiments, it also includes a model training and fine-tuning stage. Specifically, in the video segment location stage, each text description in the text retrieval sequence is converted into a text embedding feature vector; according to a preset time length, each video in the video library is divided into multiple video segments, and features are extracted from each video segment to obtain a video feature vector with the same dimension as the text embedding feature vector; finally, the text embedding feature vector is matched with the video feature vector to obtain a video feature vector that matches the text embedding feature vector, and the video segment corresponding to the matched video feature vector is determined as the target video segment that conforms to the text description.

[0106] During the video clip editing stage, the text retrieval sequence is adjusted to a new text description sequence with the same subject. A generative diffusion model is then used to sequentially edit the multiple target video clips based on the new text descriptions in the new text description sequence. For each target video clip, forward noise processing is performed on each frame to obtain noise features with image information. Based on the corresponding new text descriptions in the new text description sequence, reverse denoising processing is performed on the noise features to obtain processed video clips with the same subject and theme. Finally, multiple processed video clips with the same subject and theme are obtained.

[0107] In the video segment stitching stage, based on the noise characteristics of two adjacent processed video segments, the multiple processed video segments are smoothly stitched together in the order of the video sequence to obtain a long temporal video. For two adjacent processed video segments, a first noise feature of the last frame of the preceding processed video segment and a second noise feature of the first frame of the following processed video segment are selected. The first and second noise features are weighted to obtain weighted noise features. A generative diffusion model is used to process the weighted noise features to generate intermediate frame images. Finally, the preceding and following processed video segments are stitched together based on the intermediate frame images, resulting in a long temporal video.

[0108] For generating long-time-series videos of a target subject (a specific subject), the model training and fine-tuning stage is also included. In the model training and fine-tuning stage, the generative diffusion model is trained using multiple images of the target subject so that the generative diffusion model learns the image features of the target subject. Thus, the generative diffusion model can generate the target subject in an image or video based on the image features of the target subject.

[0109] The long video generation method based on multi-text video temporal localization retrieval provided in this application is orthogonal to existing generative diffusion model methods, and can improve the temporal consistency of generated video content based on related work. By retrieving target video segments that match the text description from the video library and using existing video segments to generate long videos, the generated videos are better adapted to the physical laws of the real world. Multiple target video segments are edited according to new text description sequences with the same subject, maintaining the consistency of video content between the processed video segments and the original real video (i.e., the target video). This results in segmented generation of video content (i.e., processed video segments) with consistent subject and video content style, which can be used to generate long temporal videos. Furthermore, noise features characterize the content information of the processed videos, and connecting the processed videos based on noise features improves the continuity between different processed videos. Because this method processes video segments and generates long temporal videos by combining and transitioning multiple segments, it effectively segments the temporal video generation task. Its memory usage does not increase with video length, maintaining a consistently low memory overhead throughout the process. Compared to other long video generation methods where memory usage increases with video length, this method demonstrates greater versatility. Thus, the long video generation method for multi-text video temporal location retrieval provided in this application uses less memory and generates long temporal videos that better conform to physical laws.

[0110] This application also provides a long video generation system for multi-text video temporal location retrieval, referring to... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a long video generation system for multi-text video temporal location retrieval provided in an embodiment of this application. The system includes:

[0111] The video retrieval module 310 is used to retrieve target video segments that match each text description from the video library according to the text retrieval sequence, and obtain multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions;

[0112] The video editing module 320 is used to adjust the text retrieval sequence into a new text description sequence with the same subject and theme, and to edit the multiple target video segments according to the new text description sequence to obtain multiple processed video segments with the same subject and theme.

[0113] The video stitching module 330 is used to smoothly transition and stitch together the multiple processed video segments according to the video sequence order based on the noise characteristics of two adjacent processed video segments, to obtain a long time-series video. The video sequence order is determined according to the new text description sequence, and the noise characteristics characterize the content information of the processed video.

[0114] In one optional embodiment, the video retrieval module includes:

[0115] The text feature module is used to convert each text description in the text retrieval sequence into a text embedding feature vector;

[0116] The video feature module is used to divide each video in the video library into multiple video segments according to a preset time length, and to extract features from each video segment to obtain a video feature vector with the same dimension as the text embedding feature vector.

[0117] The feature matching module is used to match the text embedding feature vector with the video feature vector to obtain a video feature vector that matches the text embedding feature vector, and to determine the video segment corresponding to the matched video feature vector as the target video segment that conforms to the text description.

[0118] In one optional embodiment, the video editing module includes:

[0119] The text adjustment module is used to adjust each text description in the text retrieval sequence one by one to obtain multiple text descriptions with the same subject and theme;

[0120] The style addition module is used to add the same style description to the multiple text descriptions respectively, so as to obtain a new text description sequence with the same subject and theme.

[0121] In one optional embodiment, the video editing module includes:

[0122] The generative diffusion model editing module is used to use a generative diffusion model to sequentially edit the multiple target video segments according to the new text descriptions in the new text description sequence, so as to obtain multiple processed video segments with the same subject and the same theme.

[0123] The generative diffusion model has the ability to generate corresponding videos based on text descriptions.

[0124] In one optional embodiment, the generative diffusion model editing module includes:

[0125] A forward noise-adding module is used to perform forward noise-adding processing on each frame of the target video segment to obtain noise features with image information;

[0126] The reverse denoising module is used to perform reverse denoising on the noise features based on the new text description corresponding to the new text description sequence, so as to obtain processed video segments with the same subject and the same theme.

[0127] Multiple video segment processing modules are used to edit the multiple target video segments according to the above method to obtain multiple processed video segments with the same subject and theme.

[0128] In one optional embodiment, the reverse denoising module includes:

[0129] The feature correction module is used to correct the noise features of each frame image by utilizing the noise features of adjacent frames of each frame image, and obtain the corrected noise features.

[0130] An image generation module is used to obtain the corrected image for each frame based on the corrected noise features;

[0131] The video clip generation module is used to obtain processed video clips with the same subject and theme based on the corrected image.

[0132] In one optional embodiment, the video splicing module includes:

[0133] The selection module is used to select, according to the video sequence order, the first noise feature of the last frame of the previous processed video segment, and the second noise feature of the first frame of the next processed video segment.

[0134] The weighting module is used to perform weighted processing based on the first noise feature and the second noise feature to obtain a weighted noise feature sequence;

[0135] The intermediate frame image generation module is used to process the weighted noise feature sequence using a generative diffusion model to generate an intermediate frame image sequence.

[0136] The preceding and following video stitching module is used to stitch the previous processed video segment and the following processed video segment according to the intermediate frame image sequence, and to obtain a long time sequence video by stitching the multiple processed video segments.

[0137] In an optional embodiment, the system further includes a model training module for training the generative diffusion model, the model training module comprising:

[0138] The building module is used to build a training dataset, which includes: multiple images of the target subject and text descriptions for each image;

[0139] The learning module is used to train the generative diffusion model using the training dataset, so that the generative diffusion model learns the image features of the target subject.

[0140] This application also provides an electronic device, see embodiments thereof. Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 includes a memory 410 and a processor 420. The memory 410 and the processor 420 are connected via a bus for communication. The memory 410 stores a computer program that can run on the processor 420 to implement the steps of the long video generation method for multi-text video temporal positioning retrieval described in the embodiments of this application.

[0141] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0142] This application describes embodiments of methods and systems according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0146] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0147] The above provides a detailed description of the long video generation method, system, and device for multi-text video temporal location retrieval provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating long videos using multi-text video temporal location retrieval, characterized in that, The method includes: Based on the text retrieval sequence, target video segments that match each text description are retrieved from the video library to obtain multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions; The text retrieval sequence is adjusted into a new text description sequence with the same subject and theme, and the multiple target video segments are edited according to the new text description sequence to obtain multiple processed video segments with the same subject and theme. This includes: using a generative diffusion model to sequentially edit the multiple target video segments according to the new text description in the new text description sequence to obtain multiple processed video segments with the same subject and theme; wherein, the generative diffusion model has the ability to generate corresponding videos based on text descriptions; Based on the noise characteristics of two adjacent processed video segments, the multiple processed video segments are smoothly connected in the order of the video sequence to obtain a long time-series video. The order of the video sequence is determined according to the new text description sequence, and the noise characteristics represent the content information of the processed video. Specifically, a generative diffusion model is used to sequentially edit the multiple target video segments based on the new text descriptions in the new text description sequence, resulting in multiple processed video segments with the same subject and theme, including: Forward noise addition is performed on each frame of the target video segment to obtain noise features with image information; Based on the new text description corresponding to the new text description sequence, the noise features are subjected to reverse denoising processing to obtain processed video segments with the same subject and theme. This includes: using the noise features of adjacent frames of each frame to correct the noise features of each frame to obtain corrected noise features; obtaining corrected frames based on the corrected noise features; and obtaining processed video segments with the same subject and theme based on the corrected images. The multiple target video segments are edited using the method described above to obtain multiple processed video segments with the same subject and theme.

2. The method according to claim 1, characterized in that, Based on the text retrieval sequence, retrieve target video segments from the video library that match each text description, including: Each text description in the text retrieval sequence is converted into a text embedding feature vector; According to a preset time length, each video in the video library is divided into multiple video segments, and features are extracted from each video segment to obtain a video feature vector with the same dimension as the text embedding feature vector. The text embedding feature vector is matched with the video feature vector to obtain a video feature vector that matches the text embedding feature vector, and the video segment corresponding to the matched video feature vector is determined as the target video segment that conforms to the text description.

3. The method according to claim 1, characterized in that, Adjusting the text retrieval sequence into a new text description sequence with the same subject and topic includes: Each text description in the text retrieval sequence is adjusted one by one to obtain multiple text descriptions with the same subject and theme; By adding the same style description to each of the multiple text descriptions, a new text description sequence with the same subject and theme is obtained.

4. The method according to claim 1, characterized in that, Based on the noise characteristics of two adjacent processed video segments, the multiple processed video segments are smoothly transitioned and connected in the order of the video sequence to obtain a long time-series video, including: Based on the video sequence order, the first noise feature of the last frame of the previous processed video segment is selected, and the second noise feature of the first frame of the next processed video segment is selected. The first noise feature and the second noise feature are weighted to obtain a weighted noise feature sequence. The weighted noise feature sequence is processed using a generative diffusion model to generate an intermediate frame image sequence; The previous processed video segment and the next processed video segment are joined together based on the intermediate frame image sequence, and a long time sequence video is obtained by joining the multiple processed video segments.

5. The method according to claim 1, characterized in that, The generative diffusion model is trained according to the following steps: Construct a training dataset, which includes: multiple images of the target subject and text descriptions for each image; The generative diffusion model is trained using the training dataset so that it learns the image features of the target subject.

6. A long video generation system for multi-text video temporal location retrieval, characterized in that, The system includes: The video retrieval module is used to retrieve target video segments that match each text description from the video library based on a text retrieval sequence, thereby obtaining multiple target video segments corresponding to the text retrieval sequence, wherein the text retrieval sequence includes multiple text descriptions. A video editing module is used to adjust the text retrieval sequence into a new text description sequence with the same subject and theme, and to edit the multiple target video segments according to the new text description sequence to obtain multiple processed video segments with the same subject and theme. This includes: using a generative diffusion model to sequentially edit the multiple target video segments according to the new text descriptions in the new text description sequence to obtain multiple processed video segments with the same subject and theme; wherein the generative diffusion model has the ability to generate corresponding videos based on text descriptions. The video stitching module is used to smoothly stitch together the multiple processed video segments according to the video sequence order based on the noise characteristics of two adjacent processed video segments, to obtain a long time-series video. The video sequence order is determined based on the new text description sequence, and the noise characteristics represent the content information of the processed video. Specifically, a generative diffusion model is used to sequentially edit the multiple target video segments based on the new text descriptions in the new text description sequence, resulting in multiple processed video segments with the same subject and theme, including: Forward noise addition is performed on each frame of the target video segment to obtain noise features with image information; Based on the new text description corresponding to the new text description sequence, the noise features are subjected to reverse denoising processing to obtain processed video segments with the same subject and theme. This includes: using the noise features of adjacent frames of each frame to correct the noise features of each frame to obtain corrected noise features; obtaining corrected frames based on the corrected noise features; and obtaining processed video segments with the same subject and theme based on the corrected images. The multiple target video segments are edited using the method described above to obtain multiple processed video segments with the same subject and theme.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the long video generation method for multi-text video temporal positioning and retrieval as described in any one of claims 1-5.

Images