A Text-Based Drone-View Video Generation Method Based on Tail Frame Modeling and Motion Guidance

Abstract

This invention specifically relates to a method for generating UAV-perspective videos based on tail-frame modeling and motion guidance, comprising: receiving a sequence of UAV-perspective text descriptions input by a user; using a pre-trained large language model to decompose the complex scene description into multiple text prompts for single events; inputting the first prompt into a video diffusion model to generate an initial video segment; extracting the last frame as a reference frame and generating initial latent codes for subsequent video segments using a latent initialization method based on the last frame; introducing a dynamic noise adjustment strategy during latent initialization, controlling the diversity of UAV motion through a noise scheduling function; and employing a structure-guided sampling strategy, updating the latent codes through gradient guidance during denoising to maintain temporal consistency. This invention improves the temporal coherence and visual realism of the video, achieving continuous multi-event video generation with a single framework, and is compatible with various UAV motion modes.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and to a method for generating video from text, particularly a method for generating UAV-view video from text based on tail frame modeling and motion guidance. Background Technology

[0002] Currently, text-to-video generation methods are mainly divided into three categories: the first is based on generative adversarial networks (GANs), the second is based on autoregressive transformers (ARTs), and the third is based on diffusion models. GAN-based methods excel at generating high-quality single frames but struggle to maintain temporal consistency over long time sequences. ARTs model motion dynamics using discrete representations but require large-scale training data and computational resources. With breakthroughs in diffusion probability models, text-to-video generation methods based on diffusion models have become mainstream. The latent diffusion model proposed in the paper "R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, Dec. 2022" lays an important foundation for video generation. The paper "Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen. Latent video diffusion models for high-fidelity long videogeneration. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 11234-11248, Sep. 2023" further expands the application of latent diffusion models in video generation, achieving long video generation through spatiotemporal modeling. However, existing methods mainly target video generation tasks with a single event or a single cue word.The paper "J. Gu, S. Wang, S. Zhao, H. Lu, T. Zhang, X. Wu, Z. Xu, S. Zhang, W. Jiang, YG Xu, and H. Xu. Reuse and diffuse: Iterative denoising for text-to-video generation. arXiv preprint arXiv:2309.03549, 2023." directly reuses the previous initial latent code to extend the video through an iterative denoising process, but this can easily lead to drastic changes in visual content and inconsistent backgrounds. The paper "FY Wang, W. Chen, G. Song, HJ Ye, Y. Liu, and H. Li. Gen-L-Video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264, 2023." utilizes overlapping frames between consecutive cues for temporal co-denoising, but the overlapping denoising process can introduce unwanted content. Especially when dealing with multi-event video generation tasks, existing methods face the dual challenges of maintaining temporal consistency and ensuring smooth semantic transitions. How to effectively combine the generative capabilities of diffusion models with the advantages of multi-event temporal modeling still requires further exploration, and designing a single framework to achieve coherent video generation under multiple cue word conditions remains a key problem that urgently needs to be solved.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a method for generating UAV perspective videos based on tail frame modeling and motion guidance, aiming to solve the problem in the prior art that a single model is difficult to generate continuous UAV perspective videos of multiple events.

[0005] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0006] According to a first aspect of the present invention, a method for generating text-based UAV perspective video based on tail frame modeling and motion guidance is provided, the method comprising: Step 1: Receive the drone perspective text description sequence input by the user. Each text description corresponds to a drone flight event. The input text is preprocessed by the natural language processing module to identify action keywords and scene keywords. The keywords are used to constrain the event decomposition, single event prompt word generation and time sequence organization of subsequent complex scene descriptions to ensure that the text decomposition results are consistent with the input flight actions and scene semantics. Step 2: Using a pre-trained large language model, semantic parsing and event decomposition are performed on the complex drone scene description based on the action keywords and scene keywords extracted in Step 1. The description is decomposed into multiple text prompt words for single events, and the long text is transformed into a sequence of phrases with temporal logic through prompt engineering. Step 3: Input the first text prompt word generated in Step 2 into the pre-trained text-to-video diffusion model to generate the initial drone video clip; Step 4: Extract the last frame of the initial drone video clip as a reference frame, and generate the initial latent code for subsequent video clips using the latent initialization method based on the last frame. Step 5: Introduce a dynamic noise adjustment strategy during the potential initialization process, and control the diversity of UAV motion through a noise scheduling function; Step 6: Employ a structure-guided sampling strategy to update the latent code through gradient guidance during the denoising process, thereby constraining inter-frame geometric changes; Step 7: Repeat steps 4-6 for each subsequent text prompt to generate the corresponding drone video clips in sequence, forming a complete long video sequence; Step 8: Smoothly connect adjacent video segments using a time-domain interpolation algorithm to eliminate abrupt changes at the splicing points; Step 9: Post-process the generated UAV video using the motion consistency constraint module, extract optical flow information between consecutive frames, estimate the motion parameters of the UAV camera, and constrain the motion continuity between adjacent frames using the motion smoothing loss function; Step 10: Calculate the optical flow field based on the pixel differences between adjacent frames, and calculate the inter-frame motion field through the optical flow estimation module; Step 11: Apply Gaussian weights to smooth the motion parameters and use a motion smoothing filter to reduce camera shake; Step 12: Maintain the consistency of moving objects through target detection and tracking algorithms to prevent objects from deforming or being lost during flight; Step 13: Use a multi-head self-attention mechanism to associate the information of consecutive frames in the video at the feature level to capture global spatiotemporal dependencies; Step 14: Combine shallow detail features with deep semantic features and perform feature fusion through convolution operations; Step 15: Constrain the color consistency of adjacent frames through color histogram features to maintain the uniformity of lighting and tone in long video sequences; Step 16: Combine CLIPScore, optical flow consistency and temporal stability metrics to comprehensively evaluate the generated video and select the best quality result; Step 17: Generate drone-view video with high resolution, smooth motion trajectories, and a coherent narrative structure; Step 18: Feed back successful generated cases to the system to optimize the performance of subsequent low-altitude UAV perspective video generation tasks.

[0007] In some exemplary embodiments, the last frame perception latent initialization method in step 4 specifically includes: Step 4-1: Copy and expand the last frame of the previous video segment into a frame sequence of the same length as the new video segment, and use it as the visual prior for generating the new segment; Step 4-2: Use DDIM inversion technology to convert the expanded frame sequence into the initial latent code, and map the image information in the pixel space back to the latent noise space through the reverse diffusion process; Step 4-3: Constrain the consistency between the first frame and the reference frame of the new video segment by using the perceptual inversion loss function of the last frame, and ensure that the starting state of the newly generated video is smoothly connected with the ending state of the previous segment in terms of semantics and pixels. Step 4-4: Update the latent code based on gradient descent and optimize the initial noise vector by minimizing the perceptual inversion loss function of the last frame.

[0008] In some exemplary embodiments, the DDIM inversion technique calculation formula in step 4-2 is as follows:

[0009] in, These are the denoised state observations. The current state observation value, and At time step and The cumulative noise scheduling coefficient.

[0010] In some exemplary embodiments, the calculation formula for the perceptual inversion loss function of the last frame in step 4-3 is as follows:

[0011] in, For the current number The first frame potential representation of each segment, This is the latent representation of the last frame of the previous segment. This represents the square of the L2 norm.

[0012] In some exemplary embodiments, the dynamic noise adjustment strategy in step 5 specifically includes: Step 5-1: Generate a dynamic noise vector independently for each frame, introducing randomness to simulate small disturbances during drone flight; Step 5-2: Calculate the noise weighting coefficients and dynamically adjust the balance between rigid structure and motion diversity according to the frame sequence position; Step 5-3: Weighted fusion of dynamic noise and predicted noise to generate the final mixed noise distribution.

[0013] In some exemplary embodiments, the motion consistency constraint module in step 9 specifically includes: Step 9-1: Extract optical flow information between consecutive frames of the generated video to estimate the motion parameters of the UAV camera; Step 9-2: Constrain the motion continuity between adjacent frames using the motion smoothing loss function, penalizing drastic non-physical motion; Step 9-3: Track the trajectory of the detected moving object and distinguish between foreground motion and background motion; Step 9-4: Enhance motion consistency through spatiotemporal attention mechanisms to optimize motion flow globally.

[0014] In some exemplary embodiments, the quality assessment module of step 16 uses the following assessment metrics:

[0015] in, , , , where CLIPScore represents the semantic matching degree between text and video, FlowConsistenc represents the inter-frame motion consistency, and TemporalStability represents the temporal visual stability.

[0016] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the text generation method for UAV perspective video based on tail frame modeling and motion guidance described in the first aspect above.

[0017] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the text generation method for UAV perspective video based on tail frame modeling and motion guidance described in the first aspect above.

[0018] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the text-generating UAV perspective video method based on tail frame modeling and motion guidance described in the first aspect above by executing the executable instructions.

[0019] The proposed method for generating UAV-view videos from text based on tail frame modeling and motion guidance integrates the core advantages of tail frame modeling and motion guidance technologies. It solves the technical problems of traditional text-based video generation methods, such as difficulty in generating continuous UAV-view videos from multiple events, poor temporal consistency, and unsmooth visual transitions. It has significant technical effects and practical value in many aspects. 1. Enhance temporal coherence and visual realism: By using a latent initialization strategy to perceive the last frame, semantic and pixel-level smooth transitions between the first and last frames of video segments are achieved. Combined with a structure-guided sampling strategy to constrain geometric changes between frames, scene collapse is effectively prevented, significantly improving the temporal consistency and image realism of drone-view videos.

[0020] 2. Balancing motion diversity and visual stability: A dynamic noise adjustment strategy is introduced, which simulates the small disturbances in the drone's flight through a noise scheduling function, giving the video dynamic changes that conform to physical laws; combined with a motion consistency constraint module, the physical rationality of the flight trajectory is constrained through operations such as optical flow estimation and motion smoothing filtering, so as to avoid image jitter and motion distortion while ensuring motion diversity.

[0021] 3. Achieve continuous generation of multiple events within a single framework: Collaborative pre-trained large language model and text-to-video diffusion model intelligently decompose complex drone scene text descriptions into temporally coherent single event prompts. By cyclically generating video segments and smoothly splicing them through temporal interpolation, multi-event, long-sequence drone perspective video generation is completed within a single framework, compatible with various typical drone motion modes such as fixed-point hovering, straight-line flight, and circling flight.

[0022] 4. Enhance video semantics and detail: Capture global spatiotemporal dependencies in the video through a multi-head self-attention mechanism, combine shallow details and deep semantic features with multi-scale feature fusion technology, and ensure continuous trajectory of moving objects and uniform lighting and color tone through object detection and tracking and color consistency correction. While maintaining global semantic consistency, improve local motion details and visual quality.

[0023] 5. The model is self-optimizing and adaptable to practical applications: The optimal generated results are selected through comprehensive evaluation of multiple indicators to ensure the high standard of the output video, and successful generated cases are fed back to the system to realize continuous model optimization. The videos generated by the method have high resolution, smooth motion trajectory and coherent narrative structure, which can be directly adapted to the practical application needs of fields such as urban planning. It has strong promotion and significant practical value.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 This is a flowchart of the method for generating UAV-view videos based on tail frame modeling and motion guidance, according to the present invention.

[0027] Figure 2 This is a schematic diagram of the overall architecture of the text generation method for UAV perspective video based on tail frame modeling and motion guidance designed in this invention.

[0028] Figure 3 This is a schematic diagram of the last frame sensing potential initialization module designed in this invention.

[0029] Figure 4 This is a schematic diagram of the dynamic noise adjustment strategy designed in this invention.

[0030] Figure 5 This is a schematic diagram of the structure-guided sampling mechanism designed in this invention.

[0031] Figure 6 This is a structural diagram of the motion consistency constraint module designed in this invention.

[0032] Figure 7 This is a schematic diagram of a multi-event UAV perspective video frame sequence generated by the method of this invention. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] To overcome the limitations of existing text-to-video generation methods in applying them to multi-view dynamic scene modeling, this invention provides a text-to-video generation method based on last-frame modeling and motion guidance for UAV perspectives. This method employs a last-frame perceptual latent initialization technique, maintaining visual coherence between video segments through dynamic noise adjustment and latent code inversion. Secondly, a structure-guided sampling mechanism is established, ensuring smooth transitions in motion trajectories through progressive latent code updates and inter-frame consistency constraints. Simultaneously, six-DOF motion parametric modeling is introduced, combining attention mechanisms and deformable convolutions to accurately capture the unique motion patterns of the UAV. Then, multimodal parameters are input into the UAV kinematics and camera projection models, and inversely solved to generate multi-view continuous videos conforming to physical laws. Finally, a spatiotemporal consistency enhancement module is constructed, combining physical constraint algorithms to compensate for motion distortion, achieving high-fidelity generation of multi-view UAV videos.

[0036] Figure 1 The flowchart of the method of the present invention includes the following processes: receiving a sequence of text descriptions from the user's perspective of the UAV; decomposing the scene using a large language model to generate initial video segments; generating subsequent video segments through a loop of latent perception initialization, dynamic noise adjustment, and structure-guided sampling in the last frame; smoothing the transition through temporal interpolation; constructing a motion consistency constraint module to perform optical flow estimation, motion smoothing, target tracking, and attention enhancement processing on the video; and finally performing multi-scale feature fusion and color correction to output a high-quality UAV perspective video. The method specifically includes the following steps: Step 1: Receive the sequence of text descriptions from the user's perspective of the drone. Each text description corresponds to a drone flight event. The system preprocesses the input text through the natural language processing module to identify action keywords (such as "hovering" and "diving") and scene keywords to ensure that the text descriptions accurately correspond to the events captured by the drone during flight, providing structured input for subsequent generation. Step 2: As Figure 2The diagram shows how a pre-trained Large Language Model (LLM) is used to decompose a complex drone scene description into multiple text prompts for a single event. Prompt engineering is then used to transform the long text into a sequence of phrases with temporal logic, ensuring that each prompt accurately corresponds to an independent motion phase in the video, thereby improving the accuracy of semantic decomposition and the temporal coherence of video generation. Step 3: Input the first text prompt word generated in Step 2 into the pre-trained text-to-video diffusion model, generate initial noise through Gaussian distribution sampling, and perform multi-step denoising by combining text embedding features to generate the initial drone video clip; Step 4: Extract the last frame of the initial drone video clip as a reference frame, and generate the initial latent code of the subsequent video clips through the latent initialization method of the last frame perception to ensure the visual coherence between video clips; generate initial noise through Gaussian distribution sampling, and perform multi-step denoising by combining text embedding features to generate high visual quality initial drone video clips, providing a foundation for the generation of subsequent clips; Further, refer to Figure 3 Step 4, the final frame of the perception potential initialization, specifically includes the following steps: Step 4-1: Copy and expand the last frame of the previous video segment into a frame sequence of the same length as the new video segment, forming a repeating tensor in the time dimension, which serves as a visual prior for the generation of the new segment, ensuring visual coherence between video segments in both the time and spatial dimensions. Step 4-2: Use DDIM inversion technology to convert the expanded frame sequence into initial latent code. Through a reverse diffusion process, map the image information in the pixel space back to the latent noise space, providing an optimized initial state for subsequent denoising generation. (1) in, These are the denoised state observations. The current state observation value, and At time step and The cumulative noise scheduling coefficient.

[0037] Step 4-3: Constrain the consistency between the first frame and the reference frame of the new video segment using the perceptual inversion loss function of the last frame. This ensures a smooth semantic and pixel-wise transition between the starting state of the newly generated video and the ending state of the previous segment, avoiding visual jumps. The formula is described as follows: (2) in, For the current number The first frame latent representation of each segment (i.e., the initial code obtained by inversion). For the previous segment (the first) The last frame latent representation of (i.e., the last frame generated by sampling) of (a number of segments). This represents the square of the L2 norm.

[0038] Step 4-4: Update the latent code based on gradient descent. Optimize the initial noise vector by minimizing the above loss function to ensure a natural transition in the generated content and maintain temporal continuity. The formula is described as follows: (3) in, This is the learning rate.

[0039] Step 5: For the video clip generated in Step 4, refer to... Figure 4 In the potential initialization process, a dynamic noise adjustment strategy is introduced to control the diversity of drone motion through a noise scheduling function. Specifically, the last frame of the previous video segment is copied and expanded into a frame sequence of the same length as the new video segment. The DDIM inversion technique is used to convert the expanded frame sequence into the initial potential code, and the consistency between the first frame of the new video segment and the reference frame is constrained by the last frame perceptual inversion loss function to ensure the visual coherence between video segments. (4) in, For the first Frame noise intensity, It is a monotonically decreasing function. The attenuation coefficient is... This represents the total number of frames in the video.

[0040] Furthermore, the dynamic noise adjustment strategy in step 5 specifically includes: Step 5-1: Generate a dynamic noise vector independently for each frame, introducing randomness to simulate minute disturbances during drone flight, enhancing motion diversity and realism. (5) in, Current time The dynamic noise vector, It is a normal distribution function. Represents the identity matrix. This represents the variance correlation coefficient.

[0041] Step 5-2: Calculate the noise weighting coefficients and dynamically adjust the balance between rigid structure and motion diversity according to the frame sequence position to avoid excessive repetition or disordered changes. (6) in, For the first The noise weighting coefficients of the frame, These are noise adjustment parameters.

[0042] Step 5-3: Weighted fusion of dynamic noise and predicted noise to generate the final mixed noise distribution, injecting reasonable randomness while maintaining content consistency: (7) in, For the first Mixed noise in frames, For the first Predicted noise in frames.

[0043] Step 6: For the initial potential code obtained in Step 5, refer to... Figure 5 Furthermore, a structure-guided sampling strategy is adopted. During the denoising process, latent code is updated through gradient guidance, and inter-frame geometric structure change constraints are used to prevent scene collapse and improve the temporal consistency of the video. The structure-guided sampling loss function is defined as: (8) in, For frame indexing; Step 7: Repeat steps 4-6 for each subsequent text prompt word to generate video clips corresponding to each text prompt word, forming a complete long video sequence, ensuring the coherence of multi-event videos and the integrity of the narrative structure, and generating corresponding drone video clips; Step 8: Smoothly connect adjacent video segments using a temporal interpolation algorithm to eliminate abrupt changes at the splicing points, improving video smoothness and visual continuity. The interpolation formula is defined as: (9) in, and For adjacent video clips, For interpolation weights; Step 9: Reference Figure 6 The generated UAV video is post-processed using a motion consistency constraint module. First, optical flow information between consecutive frames is extracted, and motion parameters of the UAV camera are estimated. Then, the motion continuity between adjacent frames is constrained by the motion smoothing loss function to ensure the physical rationality and visual stability of the flight trajectory. Furthermore, the motion consistency constraint module in step 9 specifically includes: Step 9-1: Extract optical flow information between consecutive frames and estimate the motion parameters of the UAV camera (such as rotation matrix and translation vector) to provide basic data for motion smoothness analysis; Step 9-2: Within the currently generated short segment, constrain the motion continuity between adjacent frames using a motion smoothing loss function, penalizing violent non-physical motions to ensure a natural and smooth flight trajectory. (10) in, For the first The camera motion matrix of the frame. This represents the Frobenius norm.

[0044] Step 9-3: Track the trajectory of the detected moving objects, distinguish between foreground motion and background motion, and improve the semantic consistency of objects in the video; Step 9-4: Enhance motion consistency through spatiotemporal attention mechanisms, optimize motion flow globally, and avoid local motion conflicts.

[0045] Step 10: Calculate the optical flow field based on the pixel differences between adjacent frames for subsequent motion analysis and consistency constraints, further improving the accuracy of the trajectory of moving objects in the video. The inter-frame motion field is calculated through the optical flow estimation module. (11) in, and For adjacent frames, For optical flow field.

[0046] Step 11: For the video generated after steps 9 and 10, the motion parameters are further smoothed using Gaussian weights to simulate the stabilization effect of a professional drone gimbal. Applying a motion smoothing filter reduces camera shake, thus improving video stability. The steps can be described as follows: (12) in, For the first Frame motion parameters, Use Gaussian weights.

[0047] Step 12: The video generated after further optimization in steps 8-11 needs to maintain the consistency of moving objects through object detection and tracking algorithms. By detecting and tracking moving objects in the video, deformation or loss of objects during flight is prevented, improving the semantic consistency and continuity of object trajectories. The tracking loss function is defined as: (13) in, For the first An object in The bounding box of time.

[0048] Step 13: For the objects detected and tracked in Step 12, a multi-head self-attention mechanism is further used to associate the information of the preceding and following frames of the video at the feature level, capturing global spatiotemporal dependencies and enhancing the spatiotemporal consistency and motion smoothness of the video. The attention mechanism is used to enhance spatiotemporal consistency; the formula for the attention mechanism is: (14) in, , , These represent the query, key, and value matrices, respectively. Furthermore, the attention mechanism used in step 13 employs a multi-head self-attention mechanism, specifically including: (15) in, To capture the spatiotemporal dependencies of different subspaces in parallel.

[0049] Step 14: Combine shallow detail features with deep semantic features, and perform feature fusion through convolution to improve the visual quality and detail representation of the video. The multi-scale feature fusion formula is as follows: (16) in, , , They are characterized as low, medium, and high levels, respectively.

[0050] Step 15: For generated videos that meet physical laws and time constraints, it is also necessary to constrain the color consistency of adjacent frames using color histogram features to maintain the uniformity of lighting and tone in the long video sequence and improve the overall visual effect of the video. The color consistency correction algorithm applied is as follows: (17) in, This represents the characteristics of the color histogram.

[0051] Step 16: For the video generated in the previous 15 steps, the generated video is comprehensively evaluated by combining indicators such as CLIPScore, optical flow consistency and temporal stability, and the best quality result is selected. The best generated result is selected through the quality evaluation module to ensure the high standard of the output video. Furthermore, the quality assessment module in step 16 uses the following assessment metrics: (18) in, , , The weighting coefficients are used. By comprehensively evaluating indicators such as CLIPScore, optical flow consistency, and temporal stability, the generated videos are evaluated from multiple dimensions to select the results with high visual quality, smooth motion, and optimal semantic compliance, ensuring that the output videos meet practical application standards.

[0052] Step 17: Output the final multi-event drone perspective video, which has high resolution, smooth motion trajectory and coherent narrative structure to meet the needs of practical applications; Step 18: Save the generated video file and update the model parameters. Feed back successful generated cases to the system to optimize subsequent generation tasks and achieve continuous improvement and optimization of the model.

[0053] refer to Figure 7 The method of this invention effectively improves the temporal coherence and visual realism of UAV perspective video generation, and can realize the generation of multi-event continuous video with a single frame. It is compatible with a variety of typical UAV motion modes such as fixed-point hovering, straight-line flight, and circling flight, which facilitates its promotion and use in fields such as military simulation and urban planning.

[0054] This invention provides a method for generating UAV-view videos based on last-frame modeling and motion guidance. The method employs a last-frame perceptual latent initialization technique, maintaining visual coherence between video segments through dynamic noise adjustment and latent code inversion. Secondly, a structure-guided sampling mechanism is established, ensuring smooth transitions in motion trajectories through progressive latent code updates and inter-frame consistency constraints. Simultaneously, six-DOF motion parametric modeling is introduced, combining attention mechanisms and deformable convolutions to accurately capture UAV-specific motion patterns. Then, multimodal parameters are input into the UAV kinematics and camera projection models, and inversely solved to generate multi-view continuous videos conforming to physical laws. Finally, a spatiotemporal consistency enhancement module is constructed, combining physical constraint algorithms to compensate for motion distortion, achieving high-fidelity generation of multi-view UAV videos.

[0055] The method of the present invention is further illustrated below through simulation: 1. Simulation conditions.

[0056] The method of this invention is a simulation performed using Anaconda software on an AMD Ryzen 7 5700X 8-Core Processor CPU, 12GB of memory, an Nvidia RTX4070 graphics card, and a Windows 11 operating system.

[0057] 2. Simulation content.

[0058] The simulation used a self-built Wensheng UAV video dataset for fine-tuning the model, and VBench was selected as the test benchmark.

[0059] To demonstrate the effectiveness of the proposed method, several text-to-video algorithms were selected as comparison algorithms and tested on the VPench benchmark after fine-tuning a self-built text-to-image UAV video dataset. Comparisons were made with CogVideo, LVDM, Gen-L-Video, etc. CogVideo is a text-to-video generation method based on a pre-trained text-to-image model. LVDM (Latent Video Diffusion Models) is a video diffusion generation model based on a latent space. Gen-L-Video is a temporal denoising method for long video generation, and other recent state-of-the-art methods such as the Promptist learner optimizer, which utilizes reinforcement learning to explore different cues under a reward function. In the aforementioned literature, CogVideo was proposed in the paper "W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers. arXiv preprint arXiv:2205.15868, Aug. 2022."; LVDM was proposed in the paper "Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen. Latent Video Diffusion Models for High-Fidelity Long Video Generation. arXiv preprint arXiv:2211.13221, Dec. 2022."; and Gen-L-Video was proposed in the paper "FY Wang, W. Chen, G. Song, HJ Ye, Y. Liu, and H. Li. Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising. arXiv preprint The paper was proposed in arXiv:2305. 18264, Jun. 2023.Promptist was proposed in the paper "W. Mo, T. Zhang, Y. Bai, B. Su, J.-R. Wen, and Q. Yang, Dynamic prompt optimizing for text-to-image generation In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, pp:6913–6923, Dec. 2024". LaVie-PAE was proposed in the paper "Y. Hao, Z. Chi, L. Dong, and F. Wei, Optimizing prompts for text-to-image generation, Advances in Neural Information Processing Systems, vol. 36, pp. 66923–66939, Jun. 2023".

[0060] DroneT2V (Ours) is the result obtained by the method of this invention. FVD (Fréchet Video Distance) is used to evaluate video quality (lower is better), CLIPSIM is used to evaluate text-video consistency (higher is better), and Temp-Con (Temporal Consistency) is used to evaluate temporal consistency (higher is better). The comparison results are shown in Table 1.

[0061] As shown in Table 1, in drone aerial photography scenarios, the performance of the present invention is significantly better than other comparative algorithms. The lower FVD and higher Temp-Con results indicate that the video generated by the method of the present invention maintains high-quality images while having better motion coherence and temporal stability.

[0062] Table 1

[0063] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0064] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0065] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. A method for generating drone-view videos based on tail-frame modeling and motion guidance, the method comprising: receiving a video sequence; determining a tail-frame for the video sequence; generating a tail-frame video sequence based on the tail-frame; and generating a drone-view video based on the tail-frame video sequence. The method includes: Step 1: Receive the sequence of text descriptions from the user's perspective of the drone. Each text description corresponds to a drone flight event. The input text is preprocessed by the natural language processing module to identify action keywords and scene keywords. Step 2: Using a pre-trained large language model, semantic parsing and event decomposition are performed on the complex drone scene description based on the action keywords and scene keywords extracted in Step 1. The description is decomposed into multiple text prompt words for single events, and the long text is transformed into a sequence of phrases with temporal logic through prompt engineering. Step 3: Input the first text prompt word generated in Step 2 into the pre-trained text-to-video diffusion model to generate the initial drone video clip; Step 4: Extract the last frame of the initial drone video clip as a reference frame, and generate the initial latent code for subsequent video clips using the latent initialization method based on the last frame. Step 5: Introduce a dynamic noise adjustment strategy during the potential initialization process, and control the diversity of UAV motion through a noise scheduling function; Step 6: Employ a structure-guided sampling strategy to update the latent code through gradient guidance during the denoising process, thereby constraining inter-frame geometric changes; Step 7: Repeat steps 4-6 for each subsequent text prompt to generate the corresponding drone video clips in sequence, forming a complete long video sequence; Step 8: Smoothly connect adjacent video segments using a time-domain interpolation algorithm to eliminate abrupt changes at the splicing points; Step 9: Post-process the generated UAV video using the motion consistency constraint module, extract optical flow information between consecutive frames, estimate the motion parameters of the UAV camera, and constrain the motion continuity between adjacent frames using the motion smoothing loss function; Step 10: Calculate the optical flow field based on the pixel differences between adjacent frames, and calculate the inter-frame motion field through the optical flow estimation module; Step 11: Apply Gaussian weights to smooth the motion parameters and use a motion smoothing filter to reduce camera shake; Step 12: Maintain the consistency of moving objects through target detection and tracking algorithms to prevent objects from deforming or being lost during flight; Step 13: Use a multi-head self-attention mechanism to associate the information of consecutive frames in the video at the feature level to capture global spatiotemporal dependencies; Step 14: Combine shallow detail features with deep semantic features and perform feature fusion through convolution operations; Step 15: Constrain the color consistency of adjacent frames through color histogram features to maintain the uniformity of lighting and tone in long video sequences; Step 16: Combine CLIPScore, optical flow consistency and temporal stability metrics to comprehensively evaluate the generated video and select the best quality result; Step 17: Generate drone-view video with high resolution, smooth motion trajectories, and a coherent narrative structure; Step 18: Feed back successful generated cases to the system to optimize the performance of subsequent low-altitude UAV perspective video generation tasks.

2. The tail-frame modeling and motion-guided text generation drone perspective video method of claim 1, wherein, The final frame perception latent initialization method in step 4 specifically includes: Step 4-1: Copy and expand the last frame of the previous video segment into a frame sequence of the same length as the new video segment, and use it as the visual prior for generating the new segment; Step 4-2: Use DDIM inversion technology to convert the expanded frame sequence into the initial latent code, and map the image information in the pixel space back to the latent noise space through the reverse diffusion process; Step 4-3: Constrain the consistency between the first frame and the reference frame of the new video segment by using the perceptual inversion loss function of the last frame, and ensure that the starting state of the newly generated video is smoothly connected with the ending state of the previous segment in terms of semantics and pixels. Step 4-4: Update the latent code based on gradient descent and optimize the initial noise vector by minimizing the perceptual inversion loss function of the last frame.

3. The tail-frame modeling and motion-guided text-based drone perspective video generation method of claim 2, wherein, The calculation formula for the DDIM inversion technique in step 4-2 is as follows: wherein, is the denoised state observation, is the current state observation, and are the accumulated noise schedule coefficients at time steps and respectively.

4. The method for generating UAV-view video based on tail frame modeling and motion guidance according to claim 2, characterized in that, The formula for calculating the perceptual inversion loss function in the last frame of step 4-3 is as follows: in, For the current number The first frame potential representation of each segment, This is the latent representation of the last frame of the previous segment. This represents the square of the L2 norm.

5. The method for generating UAV-view video based on tail frame modeling and motion guidance according to claim 1, characterized in that, The dynamic noise adjustment strategy in step 5 specifically includes: Step 5-1: Generate a dynamic noise vector independently for each frame, introducing randomness to simulate small disturbances during drone flight; Step 5-2: Calculate the noise weighting coefficients and dynamically adjust the balance between rigid structure and motion diversity according to the frame sequence position; Step 5-3: Weighted fusion of dynamic noise and predicted noise to generate the final mixed noise distribution.

6. The method for generating UAV-view video based on tail frame modeling and motion guidance according to claim 1, characterized in that, The motion consistency constraint module in step 9 specifically includes: Step 9-1: Extract optical flow information between consecutive frames of the generated video to estimate the motion parameters of the UAV camera; Step 9-2: Constrain the motion continuity between adjacent frames using the motion smoothing loss function, penalizing drastic non-physical motion; Step 9-3: Track the trajectory of the detected moving object and distinguish between foreground motion and background motion; Step 9-4: Enhance motion consistency through spatiotemporal attention mechanisms to optimize motion flow globally.

7. The method for generating UAV-view video based on tail frame modeling and motion guidance according to claim 1, characterized in that, The quality assessment module in step 16 uses the following assessment metrics: in, , , , where CLIPScore represents the semantic matching degree between text and video, FlowConsistenc represents the inter-frame motion consistency, and TemporalStability represents the temporal visual stability.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the text generation method for UAV perspective video based on tail frame modeling and motion guidance as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the text generation method for UAV perspective video based on tail frame modeling and motion guidance as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the text-generating UAV perspective video method based on tail frame modeling and motion guidance according to any one of claims 1 to 7 by executing the executable instructions.

Images