Method and device for resolving image animation to dance step air route of unmanned aerial vehicle group and medium

Abstract

The invention discloses a method, device and medium for converting an image animation into an unmanned aerial vehicle group dance step air route, and belongs to the field of unmanned aerial vehicle cluster control, and the method comprises the steps: 1, obtaining an original video corresponding to an image animation to be displayed of an unmanned aerial vehicle group; 2, performing frame insertion processing on the original video to obtain a smooth video; 3, preprocessing the obtained video, and mapping discrete points in each frame of two-dimensional image in the video into a three-dimensional space coordinate system to form a dot matrix distribution diagram of the unmanned aerial vehicle group; and step 4, for the dot matrix distribution diagrams of two adjacent frames of unmanned aerial vehicle groups, solving positions and moving paths of each unmanned aerial vehicle in the dot matrix distribution diagram of the previous frame of unmanned aerial vehicle group and the corresponding unmanned aerial vehicle in the dot matrix distribution diagram of the next frame of unmanned aerial vehicle group through a nearest neighbor matching algorithm in combination with a linear distribution algorithm. And taking the obtained position and moving path of each unmanned aerial vehicle as the dance step air route of the unmanned aerial vehicle group. According to the method, the image animation can be efficiently and accurately converted into the dance step air route of the unmanned aerial vehicle group.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) swarm control technology, and in particular to a method for resolving video animation into a UAV swarm dance path. Background Technology

[0002] Currently, drone swarm performances have become an indispensable visual element in large-scale events and celebrations. However, traditional drone swarm performances require professional designers to choreograph fixed flight paths and dance moves for the drones, relying on the experience of professionals and lacking the necessary flexibility and ability to adjust in real time. Existing methods require professionals to manually set the position and movements of each drone, involving complex programming and design, which is not only time-consuming but also lacks precision. Furthermore, existing methods lack flexibility and struggle to adapt to complex and ever-changing animation effects, limiting the visual expressiveness and artistry of the performance. At the same time, the editing and design process of existing drone swarm performances is inefficient, making it difficult to quickly respond to changing user needs and implement new performance solutions.

[0003] In view of this, the present invention is hereby proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device, and medium for converting video animation into drone swarm dance path, which can achieve efficient and accurate conversion from video animation to drone swarm dance path, thereby effectively solving the problems of low efficiency, poor flexibility, and insufficient real-time performance in traditional drone swarm performance dance path generation.

[0005] The objective of this invention is achieved through the following technical solution: A method for resolving video animation into a flight path for a drone swarm includes: Step 1: Obtain the original video corresponding to the image animation to be displayed by the drone swarm; Step 2: Perform video frame interpolation on the acquired raw video to obtain a smooth and fluid video. Step 3: Preprocess the obtained smooth video by mapping the discrete points in each frame of the two-dimensional image to a three-dimensional spatial coordinate system to form a dot matrix distribution map of the drone swarm. Step 4: For the dot matrix distribution map of the drone swarm in two adjacent frames, the position and movement path of each drone in the dot matrix distribution map of the drone swarm in the previous frame and the corresponding drone in the dot matrix distribution map of the drone swarm in the next frame are solved by the nearest neighbor matching algorithm combined with the linear allocation algorithm. The obtained position and movement path of each drone are used as the drone swarm dance path.

[0006] A processing apparatus, comprising: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the method of the present invention.

[0007] A readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the methods described in this invention.

[0008] Compared with existing technologies, the method, equipment, and medium for solving video animation into flight paths for UAV swarm dance provided by this invention have the following advantages: By employing video frame interpolation algorithms to interpolate frames in the video animation, smooth, delicate, and coherent animation effects are generated. Furthermore, a series of preprocessing steps are applied to the AI-interpolated animation, including grayscale conversion, edge detection, and discrete point generation, thereby accurately constructing a dot matrix model of the drone swarm. Finally, an optimization strategy combining the nearest neighbor matching algorithm with the linear assignment problem (unbalanced weighted Hungarian algorithm) is used to achieve efficient and accurate conversion from video animation to drone swarm dance paths, effectively solving the problems of low efficiency, poor flexibility, and insufficient real-time performance in traditional drone swarm performance dance step generation. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating a method for resolving video animation into a drone swarm flight path, as provided in an embodiment of the present invention.

[0011] Figure 2 This is a flowchart of video preprocessing in the method provided in the embodiments of the present invention.

[0012] Figure 3 This is a flowchart illustrating the calculation of the dot matrix distribution map of the UAV swarm in consecutive frames in the method provided by the embodiments of the present invention. Detailed Implementation

[0013] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0014] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0015] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0016] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0017] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0018] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0019] The solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0020] like Figure 1 As shown, this invention provides a method for solving video animation into a drone swarm dance path, including: Step 1: Obtain the original video corresponding to the image animation to be displayed by the drone swarm; Step 2: Perform video frame interpolation on the acquired raw video to obtain a smooth and fluid video. Step 3: Preprocess the obtained smooth video by mapping the discrete points in each frame of the two-dimensional image to a three-dimensional spatial coordinate system to form a dot matrix distribution map of the drone swarm. Step 4: For the dot matrix distribution map of the drone swarm in two adjacent frames, the position and movement path of each drone in the dot matrix distribution map of the drone swarm in the previous frame and the corresponding drone in the dot matrix distribution map of the drone swarm in the next frame are solved by the nearest neighbor matching algorithm combined with the linear allocation algorithm. The obtained position and movement path of each drone are used as the drone swarm dance path.

[0021] Preferably, in step 1 of the above method, the original video corresponding to the image animation to be displayed by the drone swarm is: A 2D animated short film with foreground and background edge densities both below 30 edge lines per 100×100 pixels, an average grayscale contrast greater than 0.5, and a frame rate of 30FPS.

[0022] Preferably, in step 2 of the above method, the acquired original video is subjected to video frame interpolation processing in the following manner to obtain a smooth video: A real-time intermediate stream estimation algorithm is used to perform frame interpolation on the original video to obtain a smooth video. Preferably, the real-time intermediate stream estimation algorithm performs frame interpolation on the original video at a ratio of 8.

[0023] See Figure 2 Preferably, in step 3 of the above method, the obtained smooth video is preprocessed in the following manner to map the discrete points in each frame of the two-dimensional image in the video to a three-dimensional spatial coordinate system to form a point distribution map of the UAV swarm, including: Step 31: Convert each frame of the two-dimensional image of the video from the RGB color space to the grayscale space by calculating the grayscale value of each pixel to obtain the corresponding grayscale image; Step 32: Use the Canny edge detection algorithm to perform edge detection on the grayscale image and identify the lines in the grayscale image; Step 33: Use morphological operations and thinning algorithms to thin each line in the identified grayscale image to obtain clear and coherent single-pixel contour lines. Step 34: Convert all the obtained single-pixel contour lines into a discrete dot matrix of UAVs to obtain the dot matrix distribution map of the UAV swarm.

[0024] Preferably, in step 31 of the above method, the grayscale value of each pixel is calculated using the standard grayscale conversion formula Y=0.299R+0.587G+0.114B, where R, G, and B represent the values ​​of the red, green, and blue color channels, respectively, and Y represents the grayscale value.

[0025] Preferably, in step 34 of the above method, all the obtained single-pixel contour lines are converted into a discrete point matrix of the UAV in the following manner to obtain a point matrix distribution map of the UAV swarm, including: Based on the actual flight speed limitations of drones, sampling points are obtained for the single-pixel contour lines with a fixed safe point spacing. A discrete point matrix of drones corresponding to each single-pixel contour line is generated with each sampling point corresponding to one drone. Each two-dimensional discrete point (x, y) of the discrete point matrix of all drones in the video frame is mapped to a coordinate point (X, Y, Z) in three-dimensional space to form a point matrix distribution map of the drone swarm, where X = x×Δ, Y = y×Δ, Z is a fixed height H, and Δ is the conversion coefficient between pixels and physical distance. That is, Δ is used to map pixel units to physical space units (meters). For example, if the image size is 1024×1024 and the drone's spatial flight range is 1024 meters × 1024 meters, then Δ = 1.

[0026] Preferably, in the above method, the fixed safety point distance refers to the constraint distance within a circular area with a fixed point distance as the radius, where there are no other arbitrary points.

[0027] See Figure 3 Preferably, in step 4 of the above method, for the dot matrix distribution map of the UAV swarm in two adjacent frames, the positions and movement paths of the UAVs in the dot matrix distribution map of the UAV swarm in the previous frame and the UAVs in the dot matrix distribution map of the UAV swarm in the next frame are solved by the nearest neighbor matching algorithm combined with the linear allocation algorithm in the following manner, including: Step 41: For the dot matrix distribution map of the drone swarm in two adjacent frames, the nearest neighbor matching algorithm is used to match the drones in the dot matrix distribution map of the drone swarm in the previous frame with the nearest target points in the dot matrix distribution map of the drone swarm in the next frame. After all the matching is completed, a set of point pairs between the previous and next frames is obtained. Step 42: Find the allocation scheme with the shortest total path from the drone position in the dot matrix distribution map of the drone swarm in the previous frame to the drone position in the dot matrix distribution map of the drone swarm in the next frame through the linear allocation algorithm, and form the optimal matching set with all the obtained allocation schemes. Step 43: The set of point pairs between previous and next frames obtained by the nearest neighbor matching algorithm is intersected with the optimal matching set obtained by the linear allocation algorithm. The matching subset with consistent structure is extracted as the common solution, thereby improving the matching stability and trajectory consistency. Step 44: For the remaining UAV dot matrix that failed to be matched by the common solution, the linear allocation algorithm is used again to solve the matching problem, so as to obtain the position and movement path of each UAV in the dot matrix distribution map of the UAV group in the previous frame and the dot matrix distribution map of the UAV group in the next frame.

[0028] Preferably, in step 4 of the above method, the dot matrix distribution maps of the UAV swarm in two adjacent frames are matched with the nearest target points in the dot matrix distribution map of the UAV swarm in the previous frame using a nearest neighbor matching algorithm, including: The Euclidean distance between corresponding drones in the dot matrix distribution map of the drone swarm in the previous frame and the dot matrix distribution map of the drone swarm in the next frame is measured by the nearest neighbor matching algorithm. The drones in the dot matrix distribution map of the drone swarm in the previous frame are matched with the drones that are closest to the target point in the dot matrix distribution map of the drone swarm in the next frame. The following linear allocation algorithm is used to find the shortest total path allocation scheme from the drone position in the dot matrix distribution map of the drone swarm in the previous frame to the drone position in the dot matrix distribution map of the drone swarm in the next frame, including: The linear allocation algorithm uses the following optimal allocation loss function to find the optimal matching set that minimizes the total path length from the drone position in the previous frame's drone swarm map to the drone position in the next frame's drone swarm map: ; Where C[i,j] represents the path distance between the i-th UAV spatial position in the previous frame dot matrix distribution map and the j-th UAV spatial position in the next frame dot matrix distribution map, and the path distance is preferably a two-dimensional or three-dimensional Euclidean distance between the two points; X is an allocation matrix used to represent the specific path of the UAV moving from the position in the previous frame dot matrix distribution map to the position in the next frame dot matrix distribution map, where if the i-th row is assigned to the j-th column, then X[i,j]=d, and d represents the Euclidean distance cost of the path. C[i,j] is the cost matrix C∈R m×nOne element in the matrix is ​​the cost matrix C, which is a matrix constructed based on the Euclidean distance between the positions of all drones in the dot matrix distribution map of two adjacent frames of the drone swarm. m represents the number of drone positions in the dot matrix distribution map of the previous frame, i.e., the total number of drones in the previous frame, and n represents the number of drone positions in the dot matrix distribution map of the next frame, i.e., the total number of drone target points in the next frame.

[0029] The cost matrix C is consistent with the indices i and j in the allocation matrix X, which correspond to the index number of the UAV position in the previous frame dot matrix distribution map and the index number of the UAV position in the next frame dot matrix distribution map, respectively. The cost matrix C, used as input to linear allocation algorithms (such as the Hungarian algorithm or the Jonker-Volgenant algorithm), represents the movement cost from all UAV positions in the previous frame's dot matrix distribution map to all target positions in the next frame's dot matrix distribution map. C[i,j] = Dis(p i ,q j ); where Dis(p i ,q j ) indicates that the drone started from position p i Move to position q j The required path distance between spatial locations, expressed as a three-dimensional Euclidean distance, is: C[i,j] = sqrt{(x i - x j )^2 +(y i - y j )^2 + (z i - z j )^2}; If the dot matrix is ​​arranged in a two-dimensional plane (i.e., with a constant height), then the path distance is calculated using two-dimensional Euclidean distance: C[i,j]= sqrt{(x i - x j )^2 + (y i - y j )^2}; Where, p i =(x i ,y i ,z i P = {p1, p2, ..., p} is the set of UAV locations extracted from the previous frame's dot matrix distribution map. m A position in}; q j =(x j ,y j ,z j Let Q = {q1, q2, ..., q} be the set of UAV locations in the next frame's dot matrix distribution map. n A position in}; If the allocation matrix X is a square matrix, then the allocation is done in such a way that each row is assigned to exactly one column and each column is assigned to exactly one row. If the allocation matrix X is a rectangle with different numbers of rows and columns, then a linear allocation algorithm (such as the Jonker-Volgenant algorithm) is used to solve it. The allocation scheme with the minimum total movement distance is selected under the constraints of the non-equilibrium state of the number of drones in front and behind and the distance weight between drones. Based on the maximum allowed flight distance of a single frame for the drone, the matching results are limited by a distance threshold. Matching relationships of points that exceed this distance threshold are eliminated, so that the drone prioritizes matching nearby target points and does not match target points that are too far away and exceed the drone's flight distance limit.

[0030] The maximum permissible flight distance d_max for a single frame of a drone is calculated as follows: d_max = v_max × Δt; Where v_max is the maximum flight speed of the drone; Δt = 1 / fps is the inter-frame time interval, and fps is the video frame rate; the point-to-point matching relationship restricted by the above method can ensure that the path is physically reachable.

[0031] This invention also provides a processing apparatus, comprising: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the method of the present invention.

[0032] The present invention further provides a readable storage medium storing a computer program that, when executed by a processor, can implement the method described in the present invention.

[0033] In summary, the method provided by this invention, by converting video animation into drone flight paths, can choreograph drone swarm dance steps based on any animated short film. Using Canny edge detection, discrete point sampling, and optimized matching algorithms, it can generate accurate drone flight paths with greater flexibility, enabling drone performances to adapt to more complex and dynamic animation effects. This improves the flexibility, real-time performance, and accuracy of drone swarm performances, solving the problems of poor flexibility, poor real-time performance, and poor accuracy inherent in traditional drone swarm performances that require professional designers to choreograph fixed flight paths for drone swarm dance steps. The RIFE algorithm is used for video frame interpolation to generate smooth animation effects, and GPU acceleration allows for rapid video frame interpolation and processing, achieving efficient video animation resolution. Steps such as grayscale conversion and Canny edge detection simplify image data, reduce computational complexity, and extract clear contour lines. This paper combines nearest neighbor matching algorithms and linear assignment problems (such as the unbalanced weighted Hungarian algorithm) to achieve precise motion matching for drones in a dance-like process. The nearest neighbor matching algorithm is used for initial matching to determine the optimal position of the drone in the next frame. A derivative algorithm of the Hungarian algorithm is then used to solve the linear assignment problem, optimizing the drone position matching and minimizing the total movement distance. By comparing the results of the two algorithms, the intersection of their consistent structures is selected as the common solution, ensuring the consistency and reliability of the matching and achieving precise motion matching and optimization. By setting a distance threshold, excessively long-distance movements by drones are avoided, ensuring safe flight paths. Drones prioritize matching with nearby target points, avoiding excessively distant target points that exceed flight speed limits. By using coordinate offsets, possible drone movement paths are predicted, and the matching process is optimized. This improves the accuracy of the drone dance steps and enhances the coherence and coordination of the swarm performance. Through precise position matching and optimization, the performance of the drone swarm becomes smoother and more coordinated. By statistically analyzing the actual displacement of points between consecutive frames, an offset pattern set DeltaPool is constructed. Based on historical matching records, an effective offset subset delta in the current animation scene is selected to assist in preliminary position prediction, thereby reducing the matching search space and global cost matrix calculation, and improving matching efficiency and accuracy.

[0034] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the solution provided by the embodiments of the present invention is provided with reference to specific examples.

[0035] Example 1 like Figure 1 As shown, the present invention provides a method for solving video animation into a flight path for a drone swarm, specifically including the following steps: Step 1: First, select a short animated film with a relatively simple foreground and background and high contrast as the original video material. The frame rate of this video material is 30FPS to ensure that the frame interpolation algorithm can more accurately estimate the intermediate frames, thereby improving the continuity and smoothness of the animation and simplifying subsequent processing steps.

[0036] Step 2 employs Real-Time Intermediate Flow Estimation (RIFE) to perform video frame interpolation, generating smooth and detailed animation effects. The RIFE algorithm generates high-quality intermediate frames through optical flow estimation, intermediate frame synthesis, and optimization processing, thus achieving video frame interpolation. A pre-trained model is used to predict the pixel displacement vectors between adjacent frames, obtaining the intermediate flow. Based on the estimated intermediate flow, new intermediate frames are synthesized, positioning them temporally and spatially between adjacent frames. Denoising and sharpening optimization algorithms further enhance the quality of the intermediate frames, making them smoother and more natural. NVIDIA graphics cards are used to load the model for high-performance GPU acceleration, improving video processing speed. When 8x frame interpolation is selected, a sufficiently smooth and satisfactory video is obtained.

[0037] Step 3: This step generates a fixed-interval drone swarm dot matrix through a series of processing steps on the video frames. First, the standard grayscale conversion formula Y=0.299R+0.587G+0.114B is used to calculate the grayscale value of each pixel, converting each video frame from the RGB color space to grayscale. Here, R, G, and B represent the values ​​of the red, green, and blue color channels, respectively, while Y represents the grayscale value. This step simplifies the image data and reduces computational complexity by removing color information, effectively preserving the basic contour information of the image and providing a foundation for subsequent edge detection. Second, the Canny edge detection algorithm is used to process the grayscale image. This includes multiple stages of processing, such as smoothing the image with a Gaussian filter to reduce noise, calculating the edge gradient and direction of pixels using the Sobel operator, scanning the entire image to remove redundant pixels that do not constitute edges, and determining the strength of edges to form the final edge detection result. This process identifies edges in the image. Through morphological operations and thinning algorithms, the lines obtained from Canny edge detection are refined into clear, continuous single-pixel contour lines, providing reliable input for subsequent discrete point generation. Finally, the lines obtained from Canny edge detection are converted into a dot matrix layout for the drone swarm. In this step, based on the actual flight speed limitations of the drones, points are sampled from the refined lines at a fixed point spacing to generate a series of discrete dot matrix images of the drones. This ensures that within a circular area with a fixed point spacing as the radius, there are no other arbitrary points with distance constraints, achieving uniformity of the dot matrix and safe distances between drones. The discrete points in the two-dimensional image are mapped to a coordinate system in three-dimensional space to form the dot matrix distribution of the drone swarm.

[0038] Step 4 involves calculating the drone's movement. By combining the nearest neighbor matching algorithm with a linear assignment problem (unbalanced weighted Hungarian algorithm), precise matching of drone motion between consecutive frames is achieved. This not only improves the accuracy of the drone's movement but also enhances the coherence and coordination of the group performance. First, an initial matching is performed on the drones in consecutive frames to determine the optimal position of each drone in the current frame in the next frame. The nearest neighbor algorithm tracks and analyzes the drone's trajectory by calculating its movement across consecutive video frames. This method matches the drone in the previous frame with the nearest target point in the next frame by measuring the Euclidean distance between the drones in consecutive frames. If the drone's position does not change between consecutive frames, it is considered stationary at that stage. The new position is predicted by simulating possible displacement changes of the drone by defining possible movement offsets at different distance levels, and an attempt is made to match the predicted position with the actual position in the next frame. A set of offsets that are actually applicable to the current drone position data is selected, reducing unnecessary calculations. Secondly, a specific implementation of the linear summation assignment problem applied to UAV dance path planning can be described by a matrix C, where each C[i,j] represents the path matching the UAV spatial position i in the previous frame with the UAV spatial position j in the next frame. The goal is to find an assignment scheme with the shortest total path from the previous frame position to the next frame position. Assuming X is a weighted matrix, if the i-th row is assigned to the j-th column, then X[i,j] = d, where d is the path distance. The loss function for the optimal assignment is: ; In the case where matrix X is a square matrix, it is a classic allocation problem, where each row is assigned to exactly one column and each column is assigned to exactly one row. For a generalized case where the loss matrix is ​​rectangular, with different numbers of rows and columns, the Jonker-Volgenant algorithm, a derivative of the Hungarian algorithm, can be used to solve the imbalance problem of inconsistent drone numbers between frames. During the matching process, distance weights between drones are considered to ensure that the allocation scheme minimizes the total travel distance. Finally, the matching results of the nearest neighbor algorithm and the linear allocation algorithm are compared, and the intersection of their consistent structures is selected as the common solution to ensure consistency and reliability. For the remaining drone points that fail to be matched by the common solution, the linear allocation problem solver is used again for matching. During the matching process, the imbalance of inconsistent drone numbers and the distance weights between drones are used as constraints to ensure that the allocation scheme minimizes the total travel distance. Considering the actual physical limitations of drones, a reasonable distance threshold is set. Drones prioritize matching nearby target points and do not match excessively distant target points exceeding their flight speed limits, avoiding excessively long-distance runs and ensuring the actual flight safety and performance continuity of the drones.

[0039] The method of this invention can accurately calculate the flight path of a drone swarm from video animation, achieving more complex and dynamic performance effects and ensuring the flexibility, real-time performance, and accuracy of the drone swarm during the performance. The method of this invention has at least the following technical advantages:

[0040] (1) Highly efficient image and animation processing: It can quickly convert image and animation into the flight path of the UAV swarm, which improves processing efficiency.

[0041] (2) Precise motion matching: Combining nearest neighbor and linear allocation algorithms, precise matching of UAV motion between consecutive frames is achieved, improving the accuracy of the dance steps.

[0042] (3) Flexible dance steps: The drone swarm dance steps can be choreographed based on any animated short film, which has greater flexibility.

[0043] (4) Safe flight path: Taking into account the physical limitations of the drone, during the process of matching adjacent frame points in the dance trajectory calculation step, a "maximum movable distance threshold" is set to exclude matching pairs that exceed the physical capability range, avoid excessive distance movement, and ensure flight safety and performance continuity.

[0044] Example 2 This embodiment uses a 2D animation clip with a frame rate of 30FPS, a resolution of 720×1280, and a duration of 10 seconds to perform calculations. The complete process of calculating the video animation to the flight path of the UAV swarm is explained as follows: Step 1, Video Frame Interpolation: The Real-Time Intermediate Stream Estimation (RIFE) algorithm is used to interpolate the 2D animation short film to be solved by 8 times, generating a high frame rate video of 240FPS.

[0045] Step 2, preprocessing to generate discrete point matrix: Step 21: Convert the RGB image to a grayscale image using the grayscale conversion formula; Step 22: Apply the Canny algorithm to extract edge contours, and combine it with a thinning algorithm to form single-pixel contour lines; Step 23: Sample along the line at a safe point spacing of 0.2 meters to obtain a discrete point map of the UAV; Step 24: Map to three-dimensional spatial coordinates, with Z being a fixed height.

[0046] Step 3. Assisted Prediction Matching: Step 31: The system counts past inter-frame matching offsets and filters out the effective offset set delta; Step 32: Use the effective offset set delta to predict the possible positions of the points in the previous frame and perform a fast match with the point matrix in the next frame.

[0047] Step 4, Distance Restriction Filtering: Step 41: Based on the maximum flight speed of the UAV v_max and the inter-frame time interval Δt, set the maximum travel distance threshold d_max = v_max × Δt; Step 42: In the cost matrix construction and Hungarian algorithm solution, matching paths with a distance greater than d_max are eliminated.

[0048] Step 5. Linear Assignment Matching and Common Solution Filtering: Step 51: Construct the Euclidean distance cost matrix C for unmatched points and use the Jonker-Volgenant algorithm to perform minimum path allocation; Step 52: Take the intersection of the auxiliary prediction matching result and the linear assignment result as the common solution; Step 53: Apply the allocation algorithm again to the remaining points to complete path matching.

[0049] Step 6. Output path: Export the 3D spatial path points of each UAV in all frames and store them in a structured manner for subsequent simulation or flight control system calls.

[0050] This process fully demonstrates the technical effectiveness of assisted prediction matching, effective offset filtering, distance threshold limitation, and common solution filtering mechanisms in improving matching efficiency and trajectory continuity.

[0051] It is understood that the method of this invention can also be used in 3D animation and virtual reality. It can be applied to 3D animation production to provide realistic simulations of object movement in virtual reality scenes. By capturing motion trajectories in the animation, the motion paths of target objects in the virtual scene can be generated, enhancing the realism of virtual reality.

[0052] In advertising and promotional campaigns, drone swarms can be used to display dynamic advertising patterns or text in the air, attracting viewers' attention and enhancing promotional effectiveness. This technology can provide new possibilities for advertising creativity, enabling more attractive and interactive advertising formats. It can also be widely applied in various fields requiring dynamic visual displays, multi-agent collaboration, and path planning.

[0053] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0054] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for resolving video animation into flight paths for UAV swarm dance, characterized in that, include: Step 1: Obtain the original video corresponding to the image animation to be displayed by the drone swarm; Step 2: Perform video frame interpolation on the acquired raw video to obtain a smooth and fluid video. Step 3: Preprocess the obtained smooth video by mapping the discrete points in each frame of the two-dimensional image to a three-dimensional spatial coordinate system to form a dot matrix distribution map of the drone swarm. Step 4: For the dot matrix distribution map of the drone swarm in two adjacent frames, the position and movement path of each drone in the dot matrix distribution map of the drone swarm in the previous frame and the corresponding drone in the dot matrix distribution map of the drone swarm in the next frame are solved by the nearest neighbor matching algorithm combined with the linear allocation algorithm. The obtained position and movement path of each drone are used as the drone swarm dance path.

2. The method for solving video animation into a drone swarm dance path according to claim 1, characterized in that, In step 1, the original video corresponding to the image animation to be displayed by the drone swarm is obtained as follows: A 2D animated short film with foreground and background edge density both less than 30 edge lines per 100×100 pixels, an average grayscale contrast greater than 0.5, and a frame rate of 30FPS.

3. The method for solving video animation into a drone swarm dance path according to claim 1 or 2, characterized in that, In step 2, the acquired original video is processed by video frame interpolation in the following manner to obtain a smooth video: A real-time intermediate stream estimation algorithm is used to perform frame interpolation on the original video to obtain a smooth and fluid video.

4. The method for solving video animation into a drone swarm dance path according to claim 1 or 2, characterized in that, In step 3, the obtained smooth video is preprocessed in the following manner to map the discrete points in each frame of the two-dimensional image in the video to a three-dimensional spatial coordinate system to form a point distribution map of the UAV swarm, including: Step 31: Convert the two-dimensional image of each frame of the video from the RGB color space to the grayscale space by calculating the grayscale value of each pixel to obtain the corresponding grayscale image; Step 32: Use the Canny edge detection algorithm to perform edge detection on the grayscale image and identify the lines in the grayscale image; Step 33: Use morphological operations and thinning algorithms to thin each line in the identified grayscale image to obtain clear and coherent single-pixel contour lines. Step 34: Convert all the obtained single-pixel contour lines into a discrete dot matrix of UAVs to obtain the dot matrix distribution map of the UAV swarm.

5. The method for solving video animation into a drone swarm dance path according to claim 4, characterized in that, In step 31, the grayscale value of each pixel is calculated using the standard grayscale conversion formula Y=0.299R+0.587G+0.114B, where R, G, and B represent the values ​​of the red, green, and blue color channels, respectively, and Y represents the grayscale value. In step 34, all the obtained single-pixel contour lines are converted into a discrete point matrix of the UAVs in the following manner to obtain a point matrix distribution map of the UAV swarm, including: Based on the actual flight speed limit of the UAV, the obtained single-pixel contour lines are sampled at a fixed safe point distance to obtain sampling points. A UAV discrete point matrix corresponding to each single-pixel contour line is generated with each sampling point corresponding to one UAV. Each two-dimensional discrete point (x, y) of the entire UAV discrete point matrix is ​​mapped to a coordinate point (X, Y, Z) in three-dimensional space to form a point matrix distribution map of the UAV swarm, where X = x×Δ, Y = y×Δ, Z is a fixed height H, and Δ is the conversion coefficient between pixel and physical distance.

6. The method for solving video animation into a drone swarm dance path according to claim 5, characterized in that, The fixed safety distance refers to the constraint distance within a circular area with a fixed distance as its radius, where there are no other arbitrary points.

7. The method for solving video animation into a drone swarm dance path according to claim 1 or 2, characterized in that, In step 4, for the dot matrix distribution maps of the drone swarm in two adjacent frames, the positions and movement paths of the drones in the dot matrix distribution map of the drone swarm in the previous frame and the dot matrix distribution map of the drone swarm in the next frame are solved using the nearest neighbor matching algorithm combined with the linear allocation algorithm, including: Step 41: For the dot matrix distribution map of the drone swarm in two adjacent frames, the nearest neighbor matching algorithm is used to match the drones in the dot matrix distribution map of the drone swarm in the previous frame with the nearest target points in the dot matrix distribution map of the drone swarm in the next frame. After all the matching is completed, a set of point pairs between the previous and next frames is obtained. Step 42: Find the allocation scheme with the shortest total path from the drone position in the dot matrix distribution map of the drone swarm in the previous frame to the drone position in the dot matrix distribution map of the drone swarm in the next frame through the linear allocation algorithm. All the obtained allocation schemes form the optimal matching set. Step 43: The set of point pairs between previous and next frames obtained by the nearest neighbor matching algorithm is intersected with the optimal matching set obtained by the linear allocation algorithm, and the matching subset with consistent structure is extracted as the common solution. Step 44: For the remaining UAV dot matrix that failed to be matched by the common solution, the linear allocation algorithm is used again to solve the matching problem, so as to obtain the position and movement path of each UAV in the dot matrix distribution map of the UAV group in the previous frame and the dot matrix distribution map of the UAV group in the next frame.

8. The method for solving video animation into a drone swarm dance path according to claim 7, characterized in that, In step 4, the dot matrix distribution maps of the drone swarm in two adjacent frames are matched with the nearest target points in the dot matrix distribution map of the drone swarm in the previous frame using a nearest neighbor matching algorithm, including: The Euclidean distance between corresponding drones in the dot matrix distribution map of the drone swarm in the previous frame and the dot matrix distribution map of the drone swarm in the next frame is measured by the nearest neighbor matching algorithm. The drones in the dot matrix distribution map of the drone swarm in the previous frame are matched with the drones that are closest to the target point in the dot matrix distribution map of the drone swarm in the next frame. The following linear allocation algorithm is used to find the shortest total path allocation scheme from the drone position in the dot matrix distribution map of the drone swarm in the previous frame to the drone position in the dot matrix distribution map of the drone swarm in the next frame, including: The linear allocation algorithm uses the following optimal allocation loss function to find the allocation scheme with the shortest total path from the drone position in the previous frame's drone swarm map to the drone position in the next frame's drone swarm map: ； Where m represents the number of drone positions in the previous frame's dot matrix distribution map, i.e., the total number of drones in the previous frame; n represents the number of drone positions in the next frame's dot matrix distribution map, i.e., the total number of drone target points in the next frame; C[i,j] is the cost matrix C∈R m×n One element represents the path matching the spatial position i of a drone in the dot matrix distribution map of the drone swarm in the previous frame with the spatial position j of a drone in the dot matrix distribution map of the drone swarm in the next frame. m represents the number of drone positions in the dot matrix distribution map of the previous frame, i.e., the total number of drones in the previous frame, and n represents the number of drone positions in the dot matrix distribution map of the next frame, i.e., the total number of drone target points in the next frame. The cost matrix C is a matrix constructed based on the Euclidean distance between all drone positions in the dot matrix distribution maps of the drone swarm in two adjacent frames. The cost matrix C serves as the input to the linear allocation algorithm, representing the movement cost from all drone positions in the dot matrix distribution map of the previous frame to all target positions in the dot matrix distribution map of the next frame. C[i,j] = Dis(p i ,q j ); where Dis(p i ,q j ) indicates that the drone started from position p i Move to position q j The required path distance between spatial locations, expressed as a three-dimensional Euclidean distance, is: C[i,j] = sqrt{(x i - x j )^2 + (y i -y j )^2 + (z i - z j )^2}; If the lattice is arranged in a two-dimensional plane with a constant height, then the path distance is calculated using two-dimensional Euclidean distance: C[i,j]= sqrt{(x i - x j )^2 + (y i - y j )^2}; Where, p i =(x i ,y i ,z i P = {p1, p2, ..., p} is the set of UAV locations extracted from the previous frame's dot matrix distribution map. m A position in}; q j =(x j ,y j ,z j Let Q = {q1, q2, ..., q} be the set of UAV locations in the next frame's dot matrix distribution map. n A position in}; X is an allocation matrix used to represent the specific path of a UAV moving from its position in the previous frame dot matrix distribution map to its position in the next frame dot matrix distribution map. If the i-th row is assigned to the j-th column, then X[i,j]=d, where d represents the Euclidean distance cost of the path. The cost matrix C is consistent with the indices i and j in the allocation matrix X, corresponding to the index number of the UAV position in the previous frame dot matrix distribution map and the index number of the UAV position in the next frame dot matrix distribution map, respectively. If the allocation matrix X is a square matrix, then the allocation is done in such a way that each row is assigned to exactly one column and each column is assigned to exactly one row. If the allocation matrix X is a rectangle with different numbers of rows and columns, then the linear allocation algorithm is used to solve the problem. The allocation scheme with the minimum total travel distance is selected, with the constraints of the non-equilibrium state of the number of drones in front and behind being inconsistent and the distance weight between drones. Based on the maximum allowed flight distance of a single frame for the drone, the matching results are limited by a distance threshold. Matching relationships of points that exceed this distance threshold are eliminated, so that the drone prioritizes matching nearby target points and does not match target points that are too far away and exceed the drone's flight distance limit.

9. A processing device, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can perform the method according to any one of claims 1-8.

10. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1-8.

Images