Search-based text video method

Through text feature extraction and video library retrieval, combined with motion feature extraction and model fine-tuning, the problems of inaccurate motion features, high calculation costs and poor video authenticity in the existing technology are solved, and high-quality video generation is achieved, suitable for a variety of purposes Application scenario.

CN119011969AInactive Publication Date: 2024-11-22ZHEJIANG UNIV

Patent Information

Application Number
CN202411054722.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing text-to-video generation technologies have significant challenges in motion feature accuracy, computational cost and video authenticity, making it difficult to generate high-quality dynamic videos.

Method used

By extracting text features of the input text, searching matching similar videos in the text-video library using semantic vectors, extracting keyframes and motion features, and fine-tuning the pre-trained text-video generation model to generate the final video.

Benefits of technology

It improves the accuracy of the motion characteristics of the video, reduces the calculation cost, improves the authenticity and visual effects of the video, and is suitable for entertainment, education and virtual environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011969A_ABST
    Figure CN119011969A_ABST
Patent Text Reader

Abstract

The invention discloses a retrieval-based text video method, which comprises the following steps of: (1) carrying out text feature extraction on an input text prompt P to obtain a semantic vector; (2) searching in a text-video library by using the semantic vector, matching similar text-video pairs, and selecting a reference video most similar to the input text prompt P; (3) extracting a key frame and a key area thereof from the reference video, and extracting motion features; (4) finely adjusting a pre-trained text-video generation model by using the extracted motion features; and (5) inputting the text prompt P into the fine-tuned text-video generation model to generate a final video. According to the invention, the accuracy of the motion features of the generated video can be improved, and the authenticity of the video is improved while the calculation cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of video generation, and in particular relates to a retrieval-based method for generating videos. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technology, text-to-video generation technology has received widespread attention. Traditional video generation technology mainly relies on pre-trained deep learning models to generate corresponding video content by inputting text descriptions.

[0003] For example, a Chinese patent document with publication number CN114598926A discloses a video generation method, which preprocesses the text to be inferred to obtain a text identification sequence of the text to be inferred; the text identification sequence of the text to be inferred is input into a trained neural network video generation model to generate a video corresponding to the text to be inferred.

[0004] A Chinese patent document with publication number CN117041459A discloses a text-based video generation method, which pre-establishes a video resource material library, receives text input by a user and determines various materials required for the video to be generated and the styles of various materials, converts and analyzes the text to obtain dependency data of various materials, obtains various target materials from the video resource material library based on the styles of various materials and through preset style matching rules, packages the dependency data and various target materials, obtains data of each layer of the video, and renders the target video based on the data of each layer of the video.

[0005] However, existing technologies still face significant challenges in generating videos with realistic and dynamic consistency. For example, many existing methods cannot generate videos with high-quality motion features or require a lot of computing resources and time for model training.

[0006] Existing text-to-video generation technologies mainly include the following methods:

[0007] Pre-trained model-based methods: These methods generate videos through pre-trained deep learning models, but they are limited in capturing motion features. Untrained methods: Video generation is performed by using cross-attention mechanisms of key frames or feature extraction such as depth maps and optical flows without relying on annotations of target video data. Video editing methods: These methods generate new videos by inverting the noise of the input video and subtitles describing the target video, but it is difficult to achieve high-quality dynamic generation while retaining the features of the original video.

[0008] In general, the existing text-to-video generation technologies have the following main problems:

[0009] 1. Inaccurate motion features: Existing methods often perform poorly in generating motion features of videos, resulting in unrealistic dynamics of generated videos.

[0010] 2. High computational cost: Many video generation methods require a lot of computing resources and time for training, which is costly.

[0011] 3. Poor video authenticity: There is a large gap between the generated video and the real video, making it difficult to achieve good results in the application. Summary of the invention

[0012] The present invention provides a retrieval-based video generation method, which can improve the accuracy of motion features of generated videos, reduce calculation costs and improve the authenticity of videos.

[0013] A retrieval-based video method comprises the following steps:

[0014] (1) Extract text features from the input text prompt P to obtain a semantic vector;

[0015] (2) Using semantic vectors to search in the text-video library, match similar text-video pairs, and select the reference video that is most similar to the input text prompt P;

[0016] (3) Extract key frames and key areas from the reference video and extract motion features;

[0017] (4) Fine-tune the pre-trained text-to-video generation model using the extracted motion features;

[0018] (5) The text prompt P is input into the fine-tuned text-to-video generation model to generate the final video.

[0019] The present invention utilizes existing video resources as references and integrates dynamic information in the synthesis process to achieve high-quality video generation. It can solve the problems of poor motion feature accuracy and high generation cost of existing video generation models and is suitable for applications in the fields of entertainment, education, and virtual environments.

[0020] Furthermore, the specific process of step (1) is as follows:

[0021] (1-1) Use the pre-trained language model to abstract the input text prompt P into a sentence-level semantic vector

[0022] (1-2) Deconstruct and reorganize the input text prompt P into multiple independent clauses, each of which can represent a complete and independent motion information in the original text prompt; and use the pre-trained language model to extract the word vectors of these clauses, thereby obtaining the atomic-level semantic vector set S of P. P .

[0023] In step (1-2), the generation of atomic semantic vectors includes: parsing the input text prompt P into a dependency tree to establish the relationship between words, splitting P into several independent clauses according to the rules; for a single clause, using a pre-trained language model to accurately extract the semantic vector s of the clause's motion, action subject, and action recipient s=(v mot ,v atr ,v rec )∈S P ; Among all the clauses, the clause containing the core motion semantics of the original text is denoted as s core .

[0024] The specific process of step (2) is as follows:

[0025] (2-1) Calculate the cosine similarity between the sentence-level semantic vector of the input text prompt P and the semantic vector of the video description text in the text-video library to preliminarily screen the text-video pairs;

[0026] (2-2) Design a ranking score to evaluate the motion semantic similarity between the text prompt P and the video description text of the initially screened text-video pairs, and select the optimal text-video pair as the reference video based on the ranking score.

[0027] In step (2-2), the ranking score is calculated as follows:

[0028]

[0029] Among them, t i is the text description t of the i-th video in the dataset i , t i The sentence-level semantic vector, S i T i The set of atomic semantic units, sim(·,·) represents the cosine similarity; the matching score F between semantic units is defined as:

[0030]

[0031] Among them, α, β and γ are the coefficients of their respective parameters, and T i The action and action performer word vectors of the core semantic unit, and are the action and action performer word vectors of the P core semantic unit; sim′(·,·) represents the similarity calculation function of the semantic unit, which is defined as follows:

[0032]

[0033] in,

[0034] The specific process of step (3) is as follows:

[0035] (3-1) Based on the atomic semantic units of the text prompt P and the atomic semantic units of the text in the reference video, find a subset of semantic units whose elements closely match;

[0036] (3-2) Extract keywords from the atomic semantic units in the above semantic unit subset, use an open target detection method to detect targets corresponding to the keywords in the reference video, and obtain key frames and their bounding boxes.

[0037] In step (3-2), the open target detection method includes: detecting the picture frames where the keyword corresponding target appears in the video and the bounding boxes where it roughly appears, and integrating the detection results to obtain the key time periods and key image areas in the video.

[0038] The specific process of step (4) is as follows:

[0039] (4-1) Using the temporal attention adaptation method to perform frame distillation on the motion features of the extracted key frames, the most important motion features are extracted;

[0040] (4-2) The motion features extracted by distillation are used to fine-tune the pre-trained text-to-video generation model to enhance its generation capability. The text-to-video generation model adopts the T2V model.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. Improve the accuracy of motion features of generated videos: By referring to the dynamic information in real videos, the generated videos are more realistic in motion features.

[0043] 2. Reduce computing costs: Utilize existing video resources to reduce the computing resources and time required for model training, thereby reducing generation costs.

[0044] 3. Improve the authenticity of the video: The generated video is closer to the real video in terms of visual effects and dynamic consistency, and has a wider range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The present invention is a flowchart of a retrieval-based video method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0047] like Figure 1 As shown, a retrieval-based video method includes the following steps:

[0048] Step 1: Feature Search

[0049] 1. Text feature extraction:

[0050] First, feature extraction is performed on the input text prompt P. Specifically, text feature extraction is divided into two levels: sentence-level semantic vector and atomic semantic vector:

[0051] (1) Sentence-level semantic vector: To represent the semantics of the entire sentence, we use a pre-trained language model to abstract the input text P into a dense vector representation.

[0052] (2) Atomized semantic vector: In order to represent the fine-grained semantics of P, irrelevant semantics (such as various adjectives and time states) are removed and the core semantic structure is extracted. The text is decomposed into m atomic semantic units S P ={s} m Each unit s=(v mot ,v atr ,v rec ) represents an independent subject in the original semantics, respectively representing movement (v mot ), participants(v atr ) and the receiver (v rec ) vector.

[0053] 2. Video search:

[0054] Next, the extracted text features are used to match with the pre-constructed text-video dataset D to select the reference video V that is most similar to the input prompt P. r .

[0055] (1) Rough screening: First, calculate the semantic vector of the input text P With each video description text T in the dataset D i The semantic vector Based on the calculation results, we perform a rough screening to exclude candidates that do not meet the threshold conditions.

[0056] (2) Sorting: After the initial screening, a ranking score Score (T i ,P) to evaluate the text T i The motion semantic similarity between and prompt P. The score is calculated as follows:

[0057]

[0058] Among them, t i is the text description t of the i-th video in the dataset i , t i The sentence-level semantic vector, S i T i The set of atomic semantic units, sim(·,·) represents the cosine similarity; the matching score F between semantic units is defined as:

[0059]

[0060] Among them, α, β and γ are the coefficients of their respective parameters, and T i The action and action performer word vectors of the core semantic unit, and are the action and action performer word vectors of the P core semantic unit; sim′(·,·) represents the similarity calculation function of the semantic unit, which is defined as follows:

[0061]

[0062] in,

[0063] Finally, the best text-video pair (Tr, Vr) is selected according to the ranking score.

[0064] Step 2: Model Tuning and Video Generation

[0065] 1. Motion Extraction:

[0066] Extract the most representative visual frame {f}n from the reference video Vr and refine its motion features. The selected frames should represent the dynamic content of the video as much as possible to ensure that the subsequently generated videos have similar motion features:

[0067] (1) Atomized semantic unit S based on a given input text P P , and the atomic semantic unit S of the text in the retrieved text-video pair (Tr, Vr) r , find S r With S P A subset of closely matching elements

[0068] (2) For each s i ∈{s r} ′ , extract its original text as keywords.

[0069] (3) Using open object detection, using the keywords obtained in (2), detect the picture frames and their approximate bounding boxes where the keywords in the video Vr correspond to the target, and obtain the detection results.

[0070] (4) Integrate the detection results to obtain the key time periods and key image areas in the video. Based on these analyses, the key frame set {f}n is finally cropped and obtained.

[0071] 2. Model tuning:

[0072] Using the extracted motion keyframe features {f}n, we extract its motion information and fine-tune the pre-trained T2V model to enhance its generation capability. Specifically, we:

[0073] (1) Frame distillation: distill the extracted visual frame {f}n to extract the most important motion features. The specific method involved is the temporal attention adaptation method mentioned in the reference (VMC: Video motion customization using temporal attention adaptation for text-to-video diffusion models)

[0074] (2) Model fine-tuning: Using the motion features extracted by distillation, the pre-trained T2V model G θ Fine-tune and obtain This process ensures that the generated model can more accurately capture and reproduce the motion characteristics in the reference video.

[0075] 3. Video generation:

[0076] Finally, the tuned model Used to generate the final video V gen The input text prompt P is fed into the tuned generator To generate high-quality videos that match the input prompts. The videos generated in this way should be optimal in terms of motion characteristics and visual effects.

[0077] Through the above implementation, the present invention realizes high-quality text-to-video generation, solves the problems of the prior art in motion characteristics, computational cost and video authenticity, and provides new ideas and methods for the development of video generation technology.

[0078] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A retrieval-based video method, characterized in that: The following steps are involved: (1) Extract text features from the input text prompt P to obtain a semantic vector; (2) Using semantic vectors to search in the text-video library, match similar text-video pairs, and select the reference video that is most similar to the input text prompt P; (3) Extract key frames and key areas from the reference video and extract motion features; (4) Fine-tune the pre-trained text-to-video generation model using the extracted motion features; (5) The text prompt P is input into the fine-tuned text-to-video generation model to generate the final video.

2. The retrieval-based video method according to claim 1, characterized in that: The specific process of step (1) is as follows: (1-1) Use the pre-trained language model to abstract the input text prompt P into a sentence-level semantic vector (1-2) Deconstruct and reorganize the input text prompt P into multiple independent clauses, each of which can represent a complete and independent motion information in the original text prompt; and use the pre-trained language model to extract the word vectors of these clauses, thereby obtaining the atomic-level semantic vector set S of P. P .

3. The retrieval-based video method according to claim 2, characterized in that: In step (1-2), the generation of atomic semantic vectors includes: parsing the input text prompt P into a dependency tree to establish the relationship between words, splitting P into several independent clauses according to the rules; for a single clause, using a pre-trained language model to accurately extract the semantic vector s of the clause's motion, action subject, and action recipient s=(v mot ,v atr ,v rec )∈S P ; Among all the clauses, the clause containing the core motion semantics of the original text is denoted as s core .

4. The retrieval-based video method according to claim 2, characterized in that: The specific process of step (2) is as follows: (2-1) Calculate the cosine similarity between the sentence-level semantic vector of the input text prompt P and the semantic vector of the video description text in the text-video library to preliminarily screen the text-video pairs; (2-2) Design a ranking score to evaluate the motion semantic similarity between the text prompt P and the video description text of the initially screened text-video pairs, and select the optimal text-video pair as the reference video based on the ranking score.

5. The retrieval-based video method according to claim 4, characterized in that: In step (2-2), the ranking score is calculated as follows: Among them, T i is the text description T of the i-th video in the dataset i , T i The sentence-level semantic vector, S i T i The set of atomic semantic units, sim(·,·) represents the cosine similarity; the matching score F between semantic units is defined as: Among them, α, β and γ are the coefficients of their respective parameters, and T i The action and action performer word vectors of the core semantic unit, and are the action and action performer word vectors of the P core semantic unit; sim′(·,·) represents the similarity calculation function of the semantic unit, which is defined as follows: in, 6. The retrieval-based video method according to claim 1, characterized in that: The specific process of step (3) is as follows: (3-1) Based on the atomic semantic units of the text prompt P and the atomic semantic units of the text in the reference video, find a subset of semantic units whose elements closely match; (3-2) Extract keywords from the atomic semantic units in the above semantic unit subset, use an open target detection method to detect targets corresponding to the keywords in the reference video, and obtain key frames and their bounding boxes.

7. The retrieval-based video method according to claim 6, characterized in that: In step (3-2), the open target detection method includes: detecting the picture frames where the keyword corresponding target appears in the reference video and the bounding boxes where it roughly appears, and integrating the detection results to obtain the key time periods and key image areas in the video.

8. The retrieval-based video method according to claim 1, characterized in that: The specific process of step (4) is as follows: (4-1) Using the temporal attention adaptation method to perform frame distillation on the motion features of the extracted key frames, the most important motion features are extracted; (4-2) The motion features extracted by distillation are used to fine-tune the pre-trained text-to-video generation model to enhance its generation capability. The text-to-video generation model adopts the T2V model.

Citation Information

Patent Citations

  • Text video retrieval optimization method based on text-to-image technology

    CN118377933A

  • Bidirectional image-text retrieval method based on multi-view joint embedding space

    WO2019007041A1

Cited By

  • Graphics video generation method based on non-training strategy and multi-subject attention alignment

    CN120050481A