Video generation method and device based on time sequence similarity, electronic equipment and medium

Abstract

The invention relates to a video generation method based on time sequence similarity, and is applied to the field of video generation. Specifically, the video generation method based on the time sequence similarity comprises the following steps: acquiring generated video information; generating a key video frame sequence with frame rate information based on the text information, wherein the key video sequence is separated by a plurality of frame marks; performing recursive interpolation processing on the key video frame sequence, fusing front and back key frame features through a preset attention mechanism, and generating an initial intermediate frame between adjacent key frames; extracting visual features of a plurality of key frames and intermediate frames included in the key video frame sequence, calculating cosine similarity of the visual features of adjacent frames, and adjusting intermediate frame generation parameters based on the similarity to optimize inter-frame coherence so as to obtain an optimized intermediate frame; and combining the key video frame sequence with the optimized middle frame to generate a complete video clip which is consistent with the text information and has a coherent inter-frame time sequence.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of video generation, and more particularly, to a video generation method and device based on time sequence similarity, an electronic device, and a medium. BACKGROUND

[0002] In the field of video generation, with the development of artificial intelligence technology, users have an increasing demand for generating video clips based on text information. However, in the prior art, when generating a video, it is often difficult to ensure that the generated video clip has good coherence in visual effect. Specifically, the transition between video frames may not be natural enough, resulting in poor overall fluency of the video. This problem not only affects the user experience, but also limits the application range of video generation technology in actual scenarios.

[0003] Therefore, how to improve the visual coherence of a video generated based on text has become a problem to be solved. SUMMARY

[0004] An object of embodiments of the present disclosure is to provide a new technical solution for improving the visual coherence of a video generated based on text.

[0005] According to a first aspect of the present disclosure, a video generation method based on time sequence similarity is provided, comprising:

[0006] obtaining generated video information, the generated video information comprising text information;

[0007] generating a key video frame sequence based on the text information at a frame rate information, the key video sequence being separated by a plurality of frame markers; wherein the key video frame sequence is used to represent the content and structure of a video clip, and the frame marker is used to define the time sequence boundary in the key video frame sequence;

[0008] performing a recursive interpolation process on the key video frame sequence, generating an initial intermediate frame between adjacent key frames by fusing the features of the front and rear key frames through a preset attention mechanism;

[0009] extracting the visual features of the plurality of key frames and the intermediate frames included in the key video frame sequence, calculating the cosine similarity of the visual features of adjacent frames, adjusting the intermediate frame generation parameters based on the similarity to optimize the inter-frame coherence, and obtaining the optimized intermediate frame;

[0010] combining the key video frame sequence and the optimized intermediate frame to generate a complete video clip consistent with the text information and having inter-frame time sequence coherence.

[0011] Optionally, the generating the key video frame sequence based on the text information and the frame rate information comprises: obtaining a reference frame, the reference frame being an initial image containing a scene related to the text information; performing cross-modal fusion on a reference visual feature corresponding to the reference frame and a text feature corresponding to the text information to generate a fusion feature; and sequentially outputting the key video frame sequence corresponding to the text information based on the fusion feature and according to the frame rate information.

[0012] Optionally, the performing the recursive interpolation processing on the key video frame sequence to generate initial intermediate frames between adjacent key frames by fusing features of front and rear key frames through a preset attention mechanism comprises: determining any two key frames that are continuous in time in the key video frame sequence as a to-be-interpolated frame pair, the to-be-interpolated frame pair comprising a preceding key frame and a following key frame; performing one-way attention mechanism on the to-be-interpolated frame pair to aggregate feature information of the preceding key frame, and performing bidirectional attention mechanism to fuse feature association of the preceding key frame and the following key frame; and recursively generating the initial intermediate frames between the preceding key frame and the following key frame based on the fused feature information, a number of the initial intermediate frames being determined according to the frame rate information and a time interval of the two key frames.

[0013] Optionally, the extracting visual features of the key frames and the intermediate frames included in the key video frame sequence, calculating cosine similarity of visual features of adjacent frames, and adjusting intermediate frame generation parameters based on the similarity to optimize frame continuity to obtain optimized intermediate frames comprises: extracting visual features of the preceding key frame, the following key frame and the initial intermediate frames included in the key video frame sequence; calculating cosine similarity of visual features of adjacent frame pairs that are continuous in time, the adjacent frame pairs comprising the preceding key frame and a first initial intermediate frame that is immediately adjacent thereto, between two adjacent initial intermediate frames, and the last initial intermediate frame and the following key frame; if the cosine similarity of any of the adjacent frame pairs is lower than a preset threshold, adjusting feature fusion weight parameters of the initial intermediate frames, and regenerating the initial intermediate frames in a corresponding interval to obtain the optimized intermediate frames; and if the cosine similarity of all the adjacent frame pairs is higher than or equal to the preset threshold, determining the initial intermediate frames as the optimized intermediate frames.

[0014] Optionally, the generating a complete video clip that is consistent with the text information and is continuous in frame timing by combining the key video frame sequence and the optimized intermediate frames comprises: reading each key frame in the key video frame sequence in time sequence, inserting the optimized intermediate frames between the corresponding key frames to form a continuous frame sequence; performing timing smoothing processing on the continuous frame sequence to eliminate visual jumps at insertion boundaries to obtain a video clip that is continuous in frame timing; and synthesizing the video clip that is continuous in frame timing to obtain the complete video clip.

[0015] Optionally, in the extraction of the visual features of the key video frame sequence and the intermediate frames, the CLIP encoder is used to extract visual features frame by frame, and the obtained visual features are input into the LSTM network to aggregate the time sequence information along the time axis, to generate a feature vector for representing the inter-frame time sequence correlation, which is used to optimize the generation parameters of the initial intermediate frames.

[0016] Optionally, after generating the complete video segment consistent with the text information, the CLIP encoder is used to extract visual features frame by frame from the complete video segment, and the visual features are subjected to cosine similarity calculation with the text features to verify the overall consistency of the complete video segment.

[0017] According to a second aspect of the present disclosure, a video generation device based on time sequence similarity is also provided, comprising: an acquisition module configured to acquire generation video information, the generation video information comprising text information; a first generation module configured to generate a key video frame sequence based on the text information at a frame rate information, the key video sequence being separated by a plurality of frame markers; wherein the key video frame sequence is used to represent the content and structure of a video segment, and the frame marker is used to define the time sequence boundary in the key video frame sequence; an interpolation module configured to perform recursive interpolation processing on the key video frame sequence, to generate initial intermediate frames between adjacent key frames by fusing the features of the front and rear key frames through a preset attention mechanism; a feature extraction module configured to extract visual features of a plurality of key frames included in the key video frame sequence and the intermediate frames, to calculate the cosine similarity of the visual features of adjacent frames, and to adjust the intermediate frame generation parameters based on the similarity to optimize the inter-frame continuity, to obtain optimized intermediate frames; and a second generation module configured to generate a complete video segment consistent with the text information and having inter-frame time sequence continuity by combining the key video frame sequence and the optimized intermediate frames.

[0018] According to a third aspect of the present disclosure, an electronic device is also provided, comprising a memory and a processor, the memory being configured to store a computer program; and the processor being configured to execute the computer program to implement the method according to the first aspect of the present disclosure.

[0019] According to a fourth aspect of the present disclosure, a computer readable storage medium is also provided, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the method according to the first aspect of the present disclosure.

[0020] One beneficial effect of the embodiments of the present disclosure is that by combining text information and frame rate information to generate a sequence of key video frames, and using recursive interpolation processing and attention mechanism to generate intermediate frames, the visual coherence of the video generated according to the text is ensured; by calculating the cosine similarity of the visual features of adjacent frames and adjusting the frame generation parameters accordingly, the visual difference between adjacent key video frames is reduced, making the video play more smooth and natural, and further improving the coherence of the video.

[0021] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0023] Figure 1 is a flowchart of a method for generating a video based on timing similarity according to an embodiment;

[0024] Figure 2 is a schematic flowchart of a strategy for generating a complete video segment in layers according to an embodiment;

[0025] Figure 3 is a flowchart of a calculation of a timing consistency score according to an embodiment;

[0026] Figure 4 is a block schematic diagram of a device for generating a video based on timing similarity according to an embodiment;

[0027] Figure 5 is a schematic diagram of a hardware structure of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0028] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting of the scope of the present disclosure unless otherwise specifically stated.

[0029] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the disclosure and its applications or uses.

[0030] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0031] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Thus, other examples of the example embodiments can have different values.

[0032] It should be noted that like reference numerals and letters refer to like items throughout the attached drawings, and therefore once an item is defined in one drawing, further discussion of the same item in subsequent drawings is not required.

[0033] 1. Introduction to Video Generation and Text-to-Video Generation Tasks

[0034] Text-to-Video Generation is a cutting-edge research direction in the fields of computer vision and natural language processing in recent years, aiming to automatically generate video content that matches a given textual description. With the development of deep learning techniques, especially Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and autoregressive models like Transformers, text-to-video generation not only promotes the integration of visual understanding and natural language processing technologies but also provides great potential for content creation in entertainment, advertising, education, Virtual Reality (VR), and other fields. The background of this task can be traced back to the development of Text-to-Image technology, especially methods based on Conditional Generative Adversarial Networks (cGANs) and Diffusion Models. Image generation tasks generate static images from textual descriptions, laying the foundation for text-to-video generation. However, the complexity of video generation tasks in the temporal dimension presents challenges, requiring models not only to understand and generate visual content at a single moment but also to capture temporal changes and dynamic scenes. Therefore, video generation tasks are more difficult than image generation tasks, involving multiple aspects of image generation such as scene understanding, action recognition, dynamic change generation, and also dealing with the temporal structure and long-term dependencies of videos.

[0035] Currently, research on text-to-video generation mainly focuses on two aspects: generation quality and temporal consistency. Generation quality refers to whether the video content is consistent with the given text description, while temporal consistency focuses on whether the temporal evolution of the video is coherent and whether the changes in actions and scenes conform to the temporal order of the text. To achieve these goals, researchers have adopted a variety of advanced generation models and techniques, such as generative adversarial networks (GANs) for temporal modeling. In terms of generation methods, the earliest text-to-video generation methods were mostly based on GAN structures, generating images for each frame through adversarial training and incorporating textual constraints during the generation process. However, generating single-frame images cannot capture the temporal information of the video. Therefore, with the development of temporal modeling methods, more and more research has begun to combine temporal networks, such as 3D convolutional neural networks (3D-CNN) and long short-term memory networks (LSTM), to handle the temporal dependencies between video frames. Furthermore, the introduction of the Transformer model has further promoted the development of the video generation field, especially in capturing long-range temporal dependencies and fusing cross-modal information, where the Transformer has shown excellent performance. Transformer-based architectures, such as VideoBERT, can establish more complex and deeper connections between text descriptions and video content. Another important direction in video generation is multimodal learning. A key issue in video generation tasks is how to effectively fuse the multimodal features of text and video. CLIP (Contrastive Language-Image Pretraining), as a multimodal learning method, maps text and images to a shared vector space, enabling contrastive learning of text and visual information. This approach provides a new perspective for video generation. CLIP-based text-to-video models encode video frames and text into the CLIP model, evaluate the similarity between video and text, and thus guide the video generation process, ensuring consistency between the generated video and the input text. Currently, diffusion-based generation methods have also attracted widespread attention. Diffusion models, through a step-by-step image generation process, can better generate detailed and high-quality video content. Compared to traditional GAN ​​and VAE methods, diffusion models have a more stable training process during generation, can avoid mode collapse, and generate more diverse and coherent visual content.

[0036] 2. Introduction to the Large Model and the Transferformer Model

[0037] Large Models are an important development direction in the field of artificial intelligence in recent years, referring to deep learning models with tens of billions or even hundreds of billions of parameters. With the explosive growth of computing power and data volume, large models have shown superior performance over traditional small-scale models in multiple tasks, especially in natural language processing (NLP), computer vision (CV), and speech recognition. The core advantage of large models is their ability to capture deeper patterns and more complex features in data, enabling breakthroughs in a variety of tasks. One of the typical representatives of large models is the Transformers architecture, first proposed by Google and published in a paper titled "Attention Is All You Need" in 2017, marking the birth of the Transformer model. The Transformer model relies on the self-attention mechanism to handle the global dependencies of input data, breaking through the limitations of traditional recurrent neural networks (RNN) and convolutional neural networks (CNN), and showing significant advantages in processing long texts and long time series. The Transformer architecture quickly achieved excellent results in various NLP tasks, and many variants (such as BERT, GPT series, T5, BART, etc.) have emerged in succession, becoming the mainstream in the current natural language processing field.

[0038] With the development of technology, more and more large models have emerged in the field of AI. GPT-3 (released by OpenAI) is one of the most famous language models so far, with a parameter size of 175 billion. The huge parameter size of GPT-3 enables it to perform zero-shot learning, that is, without training for specific tasks, the model can generate high-quality text output. Its capabilities are not limited to generating text, but also can perform translation, question answering, summarization and other natural language processing tasks. This scale of model has promoted the popularization of artificial intelligence applications, but also raised discussions about the computational cost, training efficiency and interpretability of large models. With the continuous evolution of large models, the increase in parameter size has also become an important standard for measuring their performance. For example, Google's PaLM (Pathways Language Model) model has a parameter size of 540 billion, while Microsoft and OpenAI jointly developed Gpt-4 model is said to have a parameter size of nearly 100 billion in some versions. In the field of computer vision, the parameter size of models such as CLIP and DALL·E is also gradually increasing, making them achieve higher accuracy in image generation and image-text multimodal tasks. The training and application of large models face significant challenges, first of all, the demand for computing resources is extremely large, training such a large model requires a large amount of GPU / TPU resources and efficient distributed computing platforms. Secondly, the inference speed and storage requirements of large models will also bring difficulties to actual deployment. In addition, how to improve the training efficiency of large models, reduce energy consumption and carbon emissions, and how to deal with the problems of interpretability and ethics of large models, are all hotspots of current research. In summary, large models represent an important trend in the development of current artificial intelligence, through their large parameter size and complex architecture, they have pushed the performance of various tasks to a new level.

[0039] 3、Clip model combined with video timing information

[0040] CLIP (Contrastive Language-Image Pretraining) is a powerful multi-modal learning model proposed by OpenAI, aiming to understand both image and text modalities by jointly learning the relationship between them. The core idea of CLIP is to train a unified model that can handle various tasks between images and texts by conducting contrastive learning on a large amount of image and text data. The main advantage of this model is that it can map visual and linguistic information into a common vector space, enabling efficient retrieval and comparison between the two modalities. The training method of CLIP is based on contrastive learning, specifically, CLIP inputs pairs of images and texts simultaneously, and then transforms them into vector representations through a visual encoder (usually CNN or Transformer) and a text encoder (usually Transformer). By maximizing the similarity between images and their corresponding texts, and minimizing the similarity between images and unrelated texts, CLIP learns how to map images and texts into a shared embedding space during training. This approach enables CLIP to perform zero-shot learning, i.e., completing tasks such as image classification, text description generation, image-text retrieval, etc., without additional training for specific tasks. A notable feature of the CLIP model is its ability to directly handle various natural language descriptions, making the model highly flexible. For example, given a text query, CLIP can find the most matching image from an image library, and vice versa. Due to its efficient contrastive learning framework, CLIP outperforms many traditional single-modal learning methods and can be widely applied across modalities.

[0041] In the task of video generation, the influence of temporal information and text prompts on the generation of video content is crucial. Videos not only contain visual content of static images, but also involve changes and dynamic interpretations in the time dimension, so temporal information plays a crucial role in video generation. Text prompts provide semantic guidance for the model to generate video content, describing the scenes, actions, characters, and other information that the video should present. In this context, the combination of temporal information and text prompts can effectively affect the quality, coherence, and consistency of video output. Compared with images, videos have dynamic changes along the time axis. Each frame of image not only depends on the content of the previous frame, but also needs to present the evolution in the time dimension. Therefore, the influence of temporal information on video generation is reflected in the following aspects:

[0042] 1) Dynamic scenes and actions: The temporal nature of videos determines the presentation of dynamic scenes and actions. For example, when describing "a person running in a park," the temporal information of the video needs to capture the continuity of the running action (from the start to the end of the run), and each frame should show the gradual change of the action. If the temporal information is not accurate, it may cause the generated video to have incoherent, jumping or stuttering actions.

[0043] 2) Object positions and changes: Temporal information also affects the movement and position changes of objects in the video. In dynamic videos, the position, size, speed and angle of objects change over time. Temporal modeling helps the model understand the relationship between different frames and generate smooth and natural transitions, avoiding unreasonable motion trajectories.

[0044] 3) Video rhythm and fluidity: The rhythm in the video (such as fast motion or slow transition) needs to be modeled through temporal information. Temporal information helps the generation model identify changes in the rhythm of the video, so as to generate the expected fluidity. For example, fast chase scenes and slow-motion replays are significantly different in terms of timing, and the generation model needs to accurately understand and present these differences.

[0045] 4) Time dependence: The core problem of temporal information in video generation is time dependence. That is, how to use the information at the previous moment to influence the generation at the current moment, ensuring that each frame in the video has a causal relationship. Effective temporal modeling can ensure that the frames before and after have natural and logical connections in terms of content, action and scene.

[0046] Text prompts provide semantic guidance for video generation, helping the model understand what content, scenes, characters, actions, etc. the video should show. The impact of text on video is mainly reflected in the following aspects:

[0047] Scene and background setting: Text prompts usually include scene descriptions such as "on the beach" or "on the city street", which indicate the setting of the background and environment during video generation. The model generates a background and scene that matches the text prompt. If the text describes a "cold winter", the model will generate a scene with snow and ice, rather than a sunny beach.

[0048] Actions and characters: Text prompts describe the behavior or action of characters, such as "running", "dancing" or "flying", which need to be accurately time modeled through temporal information. The text not only gives the type of action, but also further refines the state or details of the action (for example, "running fast" or "dancing gracefully"), which will affect the speed, amplitude of the character's action and its interaction with the background.

[0049] Emotion and Atmosphere: The text may also contain descriptions of emotions or atmospheres, such as "a happy gathering" or "a tense chase." The details of the emotions or atmospheres are not only reflected in the expressions and actions of the characters, but also affect the color tone, lighting, and camera usage in the video. The model needs to understand these emotional information and embody it in the visual content generation.

[0050] Details and Features: The text description often involves specific details, such as "a man wearing a red coat" or "a lot of dogs playing in the park." These details require the generation model to accurately map the features in the text description when generating the video. For example, the model will generate a specific character wearing a red coat, or generate a scene with multiple dogs playing in the park.

[0051] The combination of temporal information and text prompt words is the most challenging part of the video generation model. The generation process needs to consider both the consistency in the time dimension and the accuracy in the semantics. Their interaction is reflected in the following aspects:

[0052] Consistency and Coherence: When generating a video, temporal information ensures that each frame of the video is consistent with the previous and next frames, while text prompt words ensure that the video content meets the given semantics. The coordination of the two guarantees the coherence and logic of the video. For example, if the text prompt word describes a person eating dinner, the temporal information ensures that the action process from "picking up utensils" to "eating food" is naturally connected.

[0053] Action and Time Matching: The action description in the text needs to be coordinated with the temporal information to ensure that the action unfolds reasonably in time. For example, when the text describes "a person running," the temporal information needs to ensure that the sequence of actions such as changes in pace and adjustments in running speed are synchronized with the text description, avoiding temporal misalignment in the generation process.

[0054] Dynamic Semantic Expression: The "change" or "evolution" in the text prompt word needs to be expressed through temporal information. For example, when describing "the sun rises from the east," temporal information can help the model generate the dynamic changes in the process of the sun rising, while the text description provides semantic guidance for such changes.

[0055] Figure 1 is a flowchart of a video generation method based on temporal similarity according to an embodiment. The video generation method based on temporal similarity can be executed by any electronic device that can execute the method.

[0056] As shown in Figure 1 The video generation method based on temporal similarity of the present embodiment can include the following steps S110-S150:

[0057] Step S110, obtain the video generation information.

[0058] The generated video information is a basic data set for guiding the video generation process, which contains core description information related to the target video content, and provides content basis for subsequent generation of key video frame sequence, intermediate frame and complete video segment. Among them, the text information is used to represent the specific description of the user to the video content, which can include one or more elements such as scene setting, object feature, action process and emotional atmosphere, and is the core input of the video generation system to understand the user demand and convert it into visual content. The content directly determines the theme, plot and visual style of the generated video. By obtaining the text information, the system can clearly determine the direction and target of video generation, and ensure that the final generated video segment is consistent with the user's expectation.

[0059] In step S120, a key video frame sequence is generated based on the text information and frame rate information, and the key video sequence is separated by multiple frame markers.

[0060] Among them, the key video frame sequence is used to represent the content and structure of the video segment, and the frame marker is used to define the time sequence boundary in the key video frame sequence. In addition, the multiple frame markers can be one frame marker or multiple frame markers.

[0061] Optionally, the frame rate information and the multiple frame markers can be obtained when generating the complete video segment, such as the generated video information obtained including the frame rate information and the multiple frame markers, or can be pre-set by the user, and the present disclosure does not make specific limitation thereto.

[0062] Specifically, the key video frame sequence is generated by extracting the key scenes and actions in the text information, and each key frame represents an important moment or scene in the video. The multiple frame markers are used to mark the time interval between these key frames, helping to determine the rhythm and structure of the video, and thus providing a time reference for subsequent generation of intermediate frames.

[0063] In an optional embodiment, the step S120 of generating a key video frame sequence based on the text information and frame rate information can include the following steps S1211-S1213:

[0064] In step S1211, a reference frame is obtained, which is an initial image containing a scene related to the text information.

[0065] In this embodiment, the reference frame is used to provide an initial visual basis for generating a complete video, helping the video generation model (such as CLIP model) to better understand the specific scene described in the text, and thus generating key video frames highly matched with the text information.

[0066] Optionally, the reference frame can be retrieved from a preset image library based on the text information, the similarity of the text features and the visual features of the images in the image library is calculated by using the CLIP model, and the image with the highest matching degree is selected as the reference frame; or an initial image meeting the scene requirements can be directly generated as the reference frame according to the text description by using a text-driven image generation model; or an image containing the target scene can be directly uploaded by the user as the reference frame, and the present disclosure does not make specific limitations thereon.

[0067] In step S1212, the reference visual features corresponding to the reference frame and the text features corresponding to the text information are cross-modal fused to generate fused features.

[0068] The frame rate information is a frame number parameter contained in the video generation process per second, which is used to control the fluency and time accuracy of the video, and determines the time interval and output density of the key frame sequence.

[0069] The fused features are feature vectors obtained by combining the visual features of the reference frame and the semantic features of the text information, which are used to guide the generation of the key video frame sequence.

[0070] In step S1213, based on the fused features, the key video frame sequence corresponding to the text information is sequentially output according to the frame rate information.

[0071] In combination Figure 2 As shown in the figure, it is assumed that the text information input by the user is “a lion is drinking water”, and the frame rate information is set to 25 frames per second. According to the text information, the reference frame is retrieved from the preset image library, the similarity of the text features and the visual features of the images is calculated by using the CLIP model, and an image containing a lion, a water source and a natural environment is selected as the reference frame to provide a basic visual scene for video generation; then the visual features of the lion's posture, the water source details and the environment background in the reference frame are extracted, and the action semantics of “the lion drinking water” in the text are analyzed, and the two types of features are cross-modal fused to form fused features containing both scene details and action logic; then based on the fused features and according to the frame rate of 25 frames per second, the key video frame sequence is sequentially generated on the time axis: first, the picture of the lion approaching the water source, then the picture of the lion lowering his head to approach the water surface, and finally the picture of the lion drinking water, and the time interval between these key frames is divided by the frame marker, and the time interval between the adjacent key frames is set to 2 seconds according to the rhythm of the lion drinking water, forming a key video frame sequence that can completely present the content and structure of the text description.

[0072] In this embodiment, by obtaining the reference frame related to the text information, cross-modal fusing the reference visual features and the text features, generating the fused features, and sequentially outputting the key video frame sequence according to the frame rate, the key frame is highly matched with the text information, the accuracy and continuity of the video generation are improved, and the problem of mismatch between the video generation and the text description in the prior art is solved.

[0073] In another alternative implementation, the key video frame sequence is generated according to the text information without first obtaining the reference frame. Specifically, the text information can be input into a pre-trained generation model (such as a video generation model of the Transformer), which can directly generate a key video frame sequence matching the text content. This method omits the steps of obtaining the reference frame and cross-modal fusion, and directly generates key frames through text driving, thereby simplifying the process.

[0074] In step S130, recursive interpolation processing is performed on the key video frame sequence to generate initial intermediate frames between adjacent key frames by fusing the features of the front and rear key frames through a preset attention mechanism.

[0075] The recursive interpolation processing on the key video frame sequence can be understood as a process of generating intermediate transition frames between two adjacent key frames in stages through gradual refinement, i.e., first generating a preliminary intermediate frame between the two key frames, then further interpolating a more refined transition frame based on the generated intermediate frame and the original key frame, until the required number of frames is reached to ensure the continuity of the transition between frames.

[0076] The preset attention mechanism is used to enhance the feature association between key frames, so that the generated intermediate frames can better reflect the transition effect between the two frames. In the present disclosure, the preset attention mechanism can include a unidirectional attention mechanism and a bidirectional attention mechanism, wherein the unidirectional attention mechanism is used to aggregate the feature information of the preceding key frame, and the bidirectional attention mechanism is used to fuse the feature association between the preceding key frame and the following key frame.

[0077] In an alternative implementation, step S130 can include steps S1311-S1313 as follows:

[0078] In step S1311, any two key frames that are continuous in time in the key video frame sequence are determined as a pair of frames to be interpolated, which includes a preceding key frame and a following key frame.

[0079] From the generated key video frame sequence, two key frames adjacent in time sequence are sequentially selected as a group of interpolation processing objects, wherein the preceding key frame is referred to as the preceding key frame, and the following key frame is referred to as the following key frame, which clearly defines the start and end boundaries for the generation of subsequent intermediate frames.

[0080] In step S1312, the unidirectional attention mechanism is used to aggregate the feature information of the preceding key frame for the pair of frames to be interpolated, and the bidirectional attention mechanism is used to fuse the feature association between the preceding key frame and the following key frame.

[0081] The feature information is visual features of the key frames, which can be extracted by a deep learning model (such as a CLIP model) and can represent the visual content (such as the position, color, and texture of an object) in the video frames (key frames). For example, the feature information can include one or more of the position, action pose, and background environment of the characters in the key frames.

[0082] In processing the pair of frames to be interpolated, on the one hand, the core visual features in the previous key frame are extracted and integrated through a one-way attention mechanism, ensuring the progression in time sequence; on the other hand, the feature connection between the previous key frame and the subsequent key frame is established through a bidirectional attention mechanism, thereby generating a feature vector that can reflect the transition between the two frames.

[0083] In step S1313, the initial intermediate frames are recursively generated between the previous key frame and the subsequent key frame based on the fused feature information, and the number of initial intermediate frames is determined according to the frame rate information and the time interval of the two key frames.

[0084] Using the aforementioned fused feature information, transition frames are generated between the two key frames in stages: first, preliminary intermediate frames are generated, and then further interpolation is performed based on the features of the preliminary intermediate frames and the original key frames until intermediate frames that meet the quantity requirement are generated; the specific number is calculated from the frame rate and the time interval, for example, when the frame rate is 25 frames per second and the interval between the two key frames is 2 seconds, the number of intermediate frames to be generated is (25x2)-2 = 48 frames (minus the 2 key frames themselves).

[0085] Continuing the above example, the user inputs the text information as "a lion is drinking water", and provides a reference image of a lion drinking water. First, the reference image is processed by the image marker to generate 20 to 400 image markers per frame, which contain the key visual features of the image; the vectorization autoencoder (such as VQ-VAE) is used to highly disperse the image markers to extract the key visual features.

[0086] In stage 1: key video frame sequence generation, the system generates a key video frame sequence according to the text marker and the frame rate information. For example, the system may generate three key frames representing the scenes of the lion approaching the water source, the lion lowering its head to approach the water surface, and the lion drinking water.

[0087] In stage 2: recursive interpolation optimization processing, the system performs recursive interpolation processing on these key frames. Specifically, the system determines "the lion approaching the water source" and "the lion drinking water" as the pair of frames to be interpolated, where "the lion approaching the water source" is the previous key frame and "the lion drinking water" is the subsequent key frame.

[0088] Then, a unidirectional attention mechanism is adopted for the pair of frames to be interpolated to aggregate feature information of the preceding key frame (the lion approaches the water source), and a bidirectional attention mechanism is adopted to fuse the feature association between the preceding key frame and the following key frame (the lion is drinking water). Based on the fused feature information, initial intermediate frames are recursively generated between the two key frames. If the frame rate is 25 frames per second and the time interval of the two key frames is 2 seconds, then (25*2)-2=48 initial intermediate frames will be generated, which will gradually show the process of the lion approaching the water source and starting to drink water, ensuring the smoothness and continuity of the video.

[0089] In this embodiment, by adopting unidirectional and bidirectional attention mechanisms for the pair of frames to be interpolated, the key frame feature information can be more accurately aggregated and fused, so that the initial intermediate frames can better reflect the transition between the key frames, and the inter-frame continuity of the complete video is significantly improved.

[0090] In an embodiment, when extracting the visual features of the sequence of key video frames and the intermediate frames, the CLIP model is used to extract visual features frame by frame, and the obtained visual features are input into the LSTM network to aggregate the time sequence information along the time axis to generate a feature vector for representing the time sequence association between frames, which is used to optimize the generation parameters of the initial intermediate frames.

[0091] That is, the visual features of the key frames and the intermediate frames are extracted frame by frame by the CLIP model, and then these features are input into the LSTM network to capture the time sequence relationship between frames, and the generated feature vector is used to adjust and optimize the generation parameters of the initial intermediate frames, thereby improving the quality of the generated video.

[0092] For example, after determining the two key frames of "the lion approaches the water source" and "the lion lowers its head to approach the water surface", as shown in Figure 3 , the two key frames are input into the LSTM model as inputs, the time sequence association information in the sequence of key frames is captured by the LSTM model, and then the standardized visual features containing the lion's limb posture, the water source ripple pattern, and the environmental light and shadow are extracted by the CLIP model; then, relying on the time sequence information output by the LSTM and the visual features extracted by the CLIP, the unidirectional attention mechanism extracts the core static features such as the starting position of the lion in the preceding key frame and the basic texture of the water source, and the bidirectional attention mechanism extracts the dynamic association such as the moving track of the lion's head, the shift of the lion's body center of gravity, and the ripple changes of the water source caused by the lion's approach, to generate a cross-modal feature vector; finally, based on the cross-modal feature, 48 initial intermediate frames are recursively generated between the two key frames, and these intermediate frames gradually present the continuous process of the lion's forelimbs approaching, the lion's head slowly sinking, and the water source ripple gradually spreading, and the cross-modal feature matrix supports the inter-frame transition, ensuring the smoothness and continuity of the video.

[0093] In another alternative embodiment, a deep learning model (such as a Transformer or a GAN) is used to interpolate the sequence of key video frames. The model automatically extracts key frame features and learns inter-frame correlations through its internal self-attention mechanism, thereby directly generating initial intermediate frames. The number of generated intermediate frames is automatically adjusted according to the frame rate information and the time interval between key frames, improving the efficiency of generating complete video segments.

[0094] In step S140, visual features of the plurality of key frames and intermediate frames included in the sequence of key video frames are extracted, the cosine similarity of the visual features of adjacent frames is calculated, the intermediate frame generation parameters are adjusted based on the similarity to optimize the inter-frame continuity, and the optimized intermediate frames are obtained.

[0095] The intermediate frame generation parameters are a set of configuration items that control the generation of intermediate frame content. For example, they can include motion trajectory parameters of elements within a video frame (such as the step size of the lion's head movement, the angle of limb swing), visual detail rendering parameters (such as the speed of water ripple diffusion, the amplitude of light and shadow transition), and time sequence correlation weight parameters (weight distribution when fusing features of previous and subsequent frames).

[0096] The optimized intermediate frames can be understood as intermediate frames that are more visually matched to the previous and subsequent key frames and have smoother transitions after adjusting the intermediate frame generation parameters.

[0097] In an alternative embodiment, step S140 can include steps S1411-S1414 as follows:

[0098] In step S1411, the visual features of the preceding key frame, the subsequent key frame, and the initial intermediate frames included in the sequence of key video frames are extracted.

[0099] In step S1412, the cosine similarity of the visual features of adjacent frame pairs that are continuous in time is calculated. The adjacent frame pairs include the preceding key frame and the first initial intermediate frame, the adjacent two initial intermediate frames, and the last initial intermediate frame and the subsequent key frame.

[0100] In step S1413, if the cosine similarity of any adjacent frame pair is lower than a preset threshold, the feature fusion weight parameter of the initial intermediate frame generation is adjusted, and the initial intermediate frame in the corresponding interval is regenerated to obtain the optimized intermediate frame.

[0101] The feature fusion weight parameter controls the proportion of each type of feature of the preceding key frame and the subsequent key frame in the fusion process during intermediate frame generation. For example, the weight proportion of object pose features, scene detail features, and motion trajectory features during fusion. By adjusting the weight, the influence of a specific feature on the transition effect of the intermediate frame can be strengthened or weakened.

[0102] Step S1414, if the cosine similarity of all adjacent frame pairs is higher than or equal to the preset threshold, the initial intermediate frame is determined as the optimized intermediate frame.

[0103] The preset threshold is a critical value for judging whether the transition of the visual features of adjacent frames is natural, and is usually set according to the complexity of the video scene. For example, the dynamic scene can be set to 0.7, and the static scene can be set to 0.8. When the similarity is higher than the preset threshold, it indicates that the feature change between adjacent frames is continuous and logical.

[0104] For example, in the scene of "a lion is drinking water", the body contour of the previous key frame "the lion approaches the pit", the head angle of the latter key frame "the lion bends down to drink water", and the action detail features of the 48 initial intermediate frames between the two frames are extracted; the similarity of adjacent frame pairs is calculated, such as the similarity of the previous key frame and the first intermediate frame (the lion's forelimb steps), the similarity between the intermediate frames (the change of the head sinking amplitude), and the similarity of the last intermediate frame and the latter key frame; if the similarity of a certain group of intermediate frame pairs (the head sinking 10° and 15°) is 0.65, which is lower than the preset threshold 0.7, the feature fusion weight parameter is adjusted, the weight proportion of the head motion trajectory feature is increased, and the interval intermediate frame is regenerated; if the similarity of all frame pairs is ≥0.7, the 48 initial intermediate frames are determined as the optimized intermediate frames, ensuring that the lion's drinking action is smooth.

[0105] In this embodiment, by calculating the cosine similarity of the visual features of the key frames and the intermediate frames, and adjusting the intermediate frame generation parameters according to the similarity, the coherence between video frames can be improved, ensuring that the generated video is more visually smooth and natural, avoiding frame skipping and incoherence.

[0106] In another alternative embodiment, a deep learning model such as a convolutional neural network (CNN) combined with a recurrent neural network (RNN) is used to directly train and optimize the key frames and initial intermediate frames end-to-end, to automatically learn the visual features and timing relationship between key frames, and to optimize the generation of intermediate frames by minimizing the visual difference loss function between key frames, reducing the dependence on feature extraction and parameter adjustment.

[0107] Step 150, combining the key video frame sequence and the optimized intermediate frame, a complete video segment consistent with the text information and with coherent timing between frames is generated.

[0108] The video generation method based on time sequence similarity provided by the present disclosure generates a key video frame sequence by combining text information and frame rate information, and generates intermediate frames by using recursive interpolation processing and an attention mechanism, thereby ensuring the visual coherence of the video generated according to the text; the visual difference between adjacent key video frames is reduced by calculating the cosine similarity of the visual features of adjacent frames and adjusting the frame generation parameters accordingly, so that the video is more smooth and natural when played, and the coherence of the video is further improved.

[0109] In an optional implementation, step S150 can include steps S1511-S1513 as follows:

[0110] Step S1511 reads each key frame in the key video frame sequence in time sequence, inserts the optimized intermediate frames between the corresponding key frames, and forms a continuous frame sequence.

[0111] Step S1512 performs time sequence smoothing processing on the continuous frame sequence to eliminate visual jumps at the insertion boundaries, and obtains a video segment with time sequence coherence between frames.

[0112] Step S1513 synthesizes the video segment with time sequence coherence between frames, and obtains a complete video segment.

[0113] The time sequence smoothing processing can be any technical means capable of reducing or eliminating visual discontinuity or jump phenomenon that may be generated in the video frame sequence due to the insertion of intermediate frames. For example, the time sequence smoothing processing can be a gradual adjustment of the pixel values of adjacent frames by using an inter-frame interpolation algorithm, so that the color, brightness and other visual properties present a continuous transition between frames; it can also be a curve fitting of the object motion trajectory to correct the sudden change of the action amplitude in the intermediate frame, so as to ensure that the dynamic changes such as limb swing and position movement conform to the natural motion law; it can also be the use of an image processing algorithm to adjust the transition between frames to ensure the smoothness of the video when played.

[0114] For example, the key frames "lion approaches the water pit", "lion lowers head to drink water", "lion raises head after drinking water" and the like are read in time sequence, the optimized 48 intermediate frames are inserted between the corresponding key frames respectively, and a continuous frame sequence from "approaching" to "lowering head" to "raising head" is formed; the transition parameter of the sharpness of the lion's limbs is adjusted for the boundary between the key frame and the first intermediate frame to avoid sudden change from the clear outline of the key frame to the dynamic blur of the intermediate frame; the smoothing coefficient of the head sinking trajectory is optimized for the interval with large action amplitude in the intermediate frame sequence, so that the angle change between each frame is uniform; the continuous frame sequence after smoothing processing is synthesized into a complete video segment, and finally a complete video segment of the smooth action of the lion from approaching the water pit to completing the drinking is presented.

[0115] In this embodiment, by precisely inserting the optimized intermediate frames between the key frames and performing timing smoothing processing, the visual jump is effectively eliminated, a video segment consistent with the text information and coherent in inter-frame timing is generated, and the visual fluency and coherence of the generated complete video segment can be improved.

[0116] In an alternative embodiment, after generating a complete video segment consistent with the text information, the CLIP model is used to extract visual features from each frame of the complete video segment, and the visual features are calculated with the text features for cosine similarity to verify the overall consistency of the complete video segment.

[0117] Specifically, the image encoder of the CLIP model can be used to extract the features of each frame of the video, while the pre-trained CLIP text encoder is used to extract the features of the text information; then the cosine similarity between the video frame features of the complete video segment and the text features is calculated to evaluate the matching degree of the video frame and the text description. This method not only ensures the high consistency of the video content and the text information, but also provides an effective evaluation means for the quality of the video generation by quantifying the similarity score.

[0118] In another alternative embodiment, a generator network of GAN is used to automatically generate preliminary intermediate frames from the text information and key frames, and then a discriminator network is used to evaluate the quality of the generated intermediate frames to ensure the consistency and coherence of the preliminary intermediate frames and the key frames in vision, and then the generated intermediate frames are further refined, such as enhancing details through convolution layers and reducing noise through pooling layers, to improve the visual quality of the video, and finally the refined key frames and intermediate frames are combined to form the final video segment.

[0119] Figure 4 is the principle block diagram of the video generation device based on timing similarity according to an embodiment. As Figure 4As shown, the video generation apparatus 400 based on the time sequence similarity can include an acquisition module 410 configured to acquire generation video information, the generation video information including text information; a first generation module 420 configured to generate a key video frame sequence based on the text information at frame rate information, the key video sequence being separated by a plurality of frame markers; wherein the key video frame sequence is used to represent the content and structure of a video segment, and the frame marker is used to define the time sequence boundary in the key video frame sequence; an interpolation module 430 configured to perform recursive interpolation processing on the key video frame sequence, generate an initial intermediate frame between adjacent key frames by fusing the features of the front and rear key frames through a preset attention mechanism; a feature extraction module 440 configured to extract the visual features of a plurality of key frames and the intermediate frames included in the key video frame sequence, calculate the cosine similarity of adjacent frame visual features, adjust the intermediate frame generation parameters based on the similarity to optimize the inter-frame coherence, and obtain the optimized intermediate frame; and a second generation module 450 configured to generate a complete video segment consistent with the text information and the inter-frame time sequence coherence by combining the key video frame sequence and the optimized intermediate frame.

[0120] In an optional implementation, the first generation module 420 is further configured to acquire a reference frame, the reference frame being an initial image containing a scene related to the text information; perform cross-modal fusion on the reference visual features corresponding to the reference frame and the text features corresponding to the text information to generate fusion features; and output the key video frame sequence corresponding to the text information in sequence based on the fusion features and the frame rate information.

[0121] In an optional implementation, the interpolation module 430 is further configured to determine any two key frames that are continuous in time in the key video frame sequence as a to-be-interpolated frame pair, the to-be-interpolated frame pair including a preceding key frame and a following key frame; aggregate the feature information of the preceding key frame by using a one-way attention mechanism for the to-be-interpolated frame pair, and simultaneously fuse the feature association of the preceding key frame and the following key frame by using a bidirectional attention mechanism; and recursively generate the initial intermediate frame between the preceding key frame and the following key frame based on the fused feature information, the number of the initial intermediate frames being determined according to the frame rate information and the time interval of the two key frames.

[0122] In an optional implementation, the feature extraction module 440 is further configured to extract visual features of the preceding key frame, the following key frame and the initial intermediate frame included in the sequence of key video frames; calculate cosine similarity of visual features of a pair of time-continuous adjacent frames, the pair of adjacent frames including the preceding key frame and the first adjacent initial intermediate frame, between two adjacent initial intermediate frames, and the last initial intermediate frame and the following key frame; if the cosine similarity of any pair of adjacent frames is lower than a preset threshold, adjust the feature fusion weight parameter generated by the initial intermediate frame, regenerate the initial intermediate frame in the corresponding interval to obtain the optimized intermediate frame; if the cosine similarity of all pairs of adjacent frames is higher than or equal to the preset threshold, determine the initial intermediate frame as the optimized intermediate frame.

[0123] In an optional implementation, the second generation module 450 is further configured to read each key frame in the sequence of key video frames in chronological order, insert the optimized intermediate frame between the corresponding key frames to form a continuous frame sequence; perform time smoothing processing on the continuous frame sequence to eliminate visual jump at the insertion boundary to obtain a video segment with time-sequential coherence between frames; and synthesize the video segment with time-sequential coherence between frames to obtain the complete video segment.

[0124] In an optional implementation, the feature extraction module 440 is further configured to extract visual features frame by frame using a CLIP encoder, and input the obtained visual features into an LSTM network to aggregate time sequence information along a time axis to generate a feature vector for representing time sequence correlation between frames, the feature vector being used to optimize the generation parameter of the initial intermediate frame.

[0125] In an optional implementation, the feature extraction module 440 is further configured to extract visual features frame by frame from the complete video segment using a CLIP encoder, and perform cosine similarity calculation on the visual features and the text features to verify overall consistency of the complete video segment.

[0126] The video generation device 400 based on time sequence similarity can be any electronic device capable of performing the video generation method based on time sequence similarity.

[0127] Figure 5 is a hardware structure schematic diagram of an electronic device according to another embodiment.

[0128] As shown in Figure 5 the electronic device 500 includes a processor 510 and a memory 520, the memory 520 being configured to store an executable computer program, and the processor 510 being configured to execute the method of any method embodiment as described above according to the control of the computer program.

[0129] In some embodiments, the processor 510 can be configured to control overall operations of the electronic device 500. For example, the processor 510 can execute instructions to implement all or part of the steps of the methods in any of the preceding embodiments of the disclosure, so that one or more of operations such as voice communication, data communication, database operation, display control, component control, multimedia processing, etc. can be implemented, where the components can include sensors, cameras, earphones, input / output devices, etc., which can be internal components of the electronic device itself or external components connected to the electronic device wirelessly or by wire, and the multimedia can include one or more of voice, image, video, text, etc.

[0130] The electronic device 500 can be any electronic device capable of performing the time-series similarity based video generation method.

[0131] The modules of the electronic device 400 described above can be implemented by the processor 510 executing computer programs stored in the memory 510, or by other structures, which are not limited herein.

[0132] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions stored therein.

[0133] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanism that stores data in a form of a modulated data signal such as a carrier wave or other transportable signal, and any suitable combination of the foregoing. The computer readable storage medium is not, however, a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission media (e.g., light pulse passing through a fiber optic cable), or an electrical signal through a wire.

[0134] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0135] Computer readable program instructions for carrying out operations of the present disclosure can be assembly-level instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0136] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0137] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all. The computer readable storage medium can also have instructions stored thereon or therein which may

[0138] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0139] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0140] Having described above several embodiments of the disclosure, any modifications and variations that fall within the scope of the described embodiments are also contemplated by the inventor(s). As such, the foregoing description is not intended to limit the scope of the disclosure, and it is recognized that modifications are contemplated which can provide one or more benefits and which are within the scope of the disclosure. The disclosure is defined by the appended claims.

Claims

1. A video generation method based on timing similarity, comprising: obtaining generation video information, the generation video information comprising text information; generating a key video frame sequence based on the text information and frame rate information, the key video sequence being separated by a plurality of frame markers; wherein the key video frame sequence is used to represent the content and structure of a video segment, and the frame markers are used to define the timing boundaries in the key video frame sequence; performing recursive interpolation processing on the key video frame sequence, generating initial intermediate frames between adjacent key frames by fusing the features of the front and rear key frames through a preset attention mechanism; extracting visual features of the key frames and the intermediate frames included in the key video frame sequence, calculating the cosine similarity of the visual features of adjacent frames, adjusting the intermediate frame generation parameters based on the similarity to optimize the inter-frame coherence, and obtaining optimized intermediate frames; combining the key video frame sequence and the optimized intermediate frames to generate a complete video segment consistent with the text information and having inter-frame timing coherence.

2. The method of claim 1, wherein, The generating a key video frame sequence based on the text information and frame rate information comprises: obtaining a reference frame, the reference frame being an initial image containing a scene related to the text information; performing cross-modal fusion on the reference visual features corresponding to the reference frame and the text features corresponding to the text information to generate fusion features; based on the fusion features, outputting the key video frame sequence corresponding to the text information in sequence according to the frame rate information.

3. The method of claim 1, wherein, The performing recursive interpolation processing on the key video frame sequence, generating initial intermediate frames between adjacent key frames by fusing the features of the front and rear key frames through a preset attention mechanism comprises: any two key frames in the key video frame sequence that are continuous in time are determined as a pair of frames to be interpolated, the pair of frames to be interpolated comprising a preceding key frame and a following key frame; a one-way attention mechanism is used to aggregate the feature information of the preceding key frame, and a bidirectional attention mechanism is used to fuse the feature association of the preceding key frame and the following key frame; based on the fused feature information, the initial intermediate frames are recursively generated between the preceding key frame and the following key frame, and the number of the initial intermediate frames is determined according to the frame rate information and the time interval of the two key frames.

4. The method of claim 3, wherein, The extracting visual features of the key frames and the intermediate frames included in the key video frame sequence, calculating the cosine similarity of the visual features of adjacent frames, adjusting the intermediate frame generation parameters based on the similarity to optimize the inter-frame coherence, and obtaining optimized intermediate frames comprises: extracting visual features of the preceding key frame, the following key frame and the initial intermediate frames included in the key video frame sequence; calculating the cosine similarity of the visual features of adjacent frame pairs that are continuous in time, the adjacent frame pairs comprising: the preceding key frame and the first initial intermediate frame, adjacent two initial intermediate frames, and the last initial intermediate frame and the following key frame; If the cosine similarity of any of the adjacent frame pairs is lower than a preset threshold, the feature fusion weight parameters of the initial intermediate frame generation are adjusted, and the initial intermediate frame of the corresponding interval is regenerated to obtain the optimized intermediate frame. If the cosine similarity of all adjacent frame pairs is higher than or equal to a preset threshold, then the initial intermediate frame is determined as the optimized intermediate frame.

5. The method of claim 1, wherein, The step of combining the key video frame sequence with the optimized intermediate frames to generate a complete video segment that is consistent with the text information and has a coherent inter-frame temporal sequence includes: Read each key frame in the key video frame sequence in chronological order, and insert the optimized intermediate frame between the corresponding key frames to form a continuous frame sequence. Perform temporal smoothing on the continuous frame sequence to eliminate visual jumps at the insertion boundaries and obtain video segments with coherent inter-frame temporal sequence. The inter-frame temporally coherent video segments are synthesized to obtain the complete video segment.

6. The method according to any one of claims 1 to 5, characterized in that, When extracting the visual features of the key video frame sequence and the intermediate frames, the CLIP model is used to extract the visual features frame by frame, and the obtained visual features are input into the LSTM network to aggregate temporal information along the time axis to generate a feature vector that characterizes the temporal correlation between frames. The feature vector is used to optimize the generation parameters of the initial intermediate frames.

7. The method of claim 6, wherein, After generating the complete video segment that is consistent with the text information, the CLIP model is used to extract visual features frame by frame from the complete video segment, and the visual features and the text features are calculated with cosine similarity to verify the overall consistency of the complete video segment.

8. A video generation apparatus based on temporal similarity, comprising: The acquisition module is used to acquire generated video information, which includes text information; The first generation module is used to generate a key video frame sequence based on the text information and frame rate information. The key video sequence is separated by multiple frame markers. The key video frame sequence is used to characterize the content and structure of the video segment, and the frame markers are used to define the temporal boundaries in the key video frame sequence. The interpolation module is used to perform recursive interpolation processing on the key video frame sequence, and to generate an initial intermediate frame between adjacent key frames by fusing the features of the preceding and following key frames through a preset attention mechanism. The feature extraction module is used to extract the visual features of multiple key frames and intermediate frames included in the key video frame sequence, calculate the cosine similarity of the visual features of adjacent frames, and adjust the intermediate frame generation parameters based on the similarity to optimize the inter-frame coherence, thereby obtaining the optimized intermediate frame. The second generation module is used to combine the key video frame sequence with the optimized intermediate frames to generate a complete video segment that is consistent with the text information and has a coherent inter-frame temporal sequence.

9. An electronic device comprising a memory and a processor, the memory being configured to store a computer program; the processor being configured to execute the computer program to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, the computer program, when executed by a processor, implementing the method according to any one of claims 1-7.

Images