Unmanned aerial vehicle low-frequency message interpolation method, device and equipment and storage medium

Abstract

The application provides a UAV low-frequency message interpolation method and device, equipment and a storage medium, relates to the technical field of geographic information matching, and comprises the following steps: acquiring target route design prior information, combining frame rate information of a UAV video frame to establish a route prior model; receiving a current low-frequency position message sent by the UAV, positioning a target route segment corresponding to the current low-frequency position message by using a preset search space algorithm; based on the current low-frequency position message, the target route segment and the route prior model, interpolating and solving initial position data of each video frame; inputting the initial position data of each video frame, the current low-frequency position message and the target route segment into an environmental factor correction model to correspondingly obtain distance deviation correction amounts of each video frame; and correcting each initial position data by using the distance deviation correction amounts of each video frame to obtain target position data matched with video timing. The application realizes high-precision timing matching of a UAV video picture and position information by using the above scheme.

Description

Technical Field

[0001] This application relates to the field of geographic information matching technology, and in particular to a method, apparatus, device and storage medium for interpolating low-frequency messages from unmanned aerial vehicles (UAVs). Background Technology

[0002] With the development of drone technology, drones can return real-time video footage and geographical location during flight. Combining the video footage with AI target recognition can display the flying target in the video footage and integrate the drone footage into the map space in real time.

[0003] However, since the drone's backhaul status is low-frequency (e.g., 1Hz) and the drone's image is high-frequency (e.g., 25Hz), there is a lack of synchronization in the drone image fusion. How to accurately convert the drone's status to a higher frequency and achieve accurate temporal fusion of the drone's image has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, this application proposes a method, apparatus, device and storage medium for interpolating low-frequency messages from unmanned aerial vehicles (UAVs).

[0005] Firstly, this application provides a method for interpolating low-frequency messages from unmanned aerial vehicles (UAVs), including: Obtain prior information on the target flight path design, and establish a prior flight path model by combining the frame rate information of the UAV video frames. The prior flight path model is used to perform interpolation calculations on the UAV pose data corresponding to the video frames. Receive the current low-frequency pose message sent by the UAV and use a preset retrieval space algorithm to locate the target flight line segment corresponding to the current low-frequency pose message; Based on the current low-frequency pose message, the target flight segment, and the flight prior model, the initial pose data corresponding to each video frame is interpolated and solved. The initial pose data, current low-frequency pose message and target flight line segment corresponding to each video frame are input into the pre-trained environmental factor correction model to obtain the distance deviation correction amount for each video frame. The initial pose data are corrected by adjusting the distance deviation of each video frame to obtain the target pose data that matches the video time sequence.

[0006] In one embodiment, the step of obtaining prior information on the target flight path design and establishing a prior flight path model by combining the frame rate information of the UAV video frames includes: Based on the prior information of the target route design, a route linked list is constructed; Based on the route list, the flight attributes corresponding to each route segment in the route list are determined, wherein the flight attributes include the direction of motion and speed change parameters; Construct a rule for the correspondence between the video frame timing and the flight time of the flight path based on the frame rate information of the UAV video frames; Based on waypoint information, the flight attributes and the time interval correspondence rules, as well as the preset segmented distance-time mapping function and interpolation formula, a priori flight path model is constructed. The waypoint information includes waypoint coordinates and corresponding UAV flight attitude parameters.

[0007] In one embodiment, constructing a route linked list based on the prior information of the target route design includes: Extract all original waypoints from the prior information of the target route design. Each original waypoint includes latitude and longitude, plane coordinates, relative altitude, heading angle, pitch angle, horizontal speed and vertical speed. The Gaussian forward calculation algorithm is used to convert the geographical latitude and longitude of each waypoint into planar coordinates; Traverse all original waypoints in the flight order of the route, compare the relative altitudes of adjacent waypoints, and insert an altitude inflection point waypoint if the altitudes of adjacent waypoints are inconsistent, update the ordered waypoint sequence, and construct a route linked list based on the ordered waypoint sequence.

[0008] In one embodiment, determining the flight attributes corresponding to each route segment in the route list based on the route list includes: Divide two adjacent nodes in the route list into an independent route segment, and traverse the split to obtain all route segments in the list. For each route segment, the relative altitude of the starting waypoint and the relative altitude of the ending waypoint of the route segment are compared, and the movement direction of the route segment is determined in combination with the preset altitude tolerance value. The movement direction includes horizontal movement and vertical movement. Based on the preset horizontal speed and acceleration parameters of waypoints within the route segment, the flight distance and duration corresponding to the acceleration, constant speed, and deceleration segments within the route segment are determined, thus obtaining the variable speed segment parameters of the route segment.

[0009] In one embodiment, the step of constructing a priori route model based on waypoint information, flight attributes, and time interval correspondence rules, as well as a preset segmented distance-time mapping function and interpolation formula, includes: Associate and bind the waypoint information and flight attributes corresponding to each route segment; The time interval correspondence rule is defined as the timing conversion rule from video frame timing to flight route timing; The segmented distance-time mapping function is defined as taking the speed change parameters of the flight segment and the relative time of the flight segment as inputs, and the segmented distance-time mapping function is used as the standard calculation rule for solving the relative flight distance within the flight segment. Configure interpolation formulas that match the direction of motion of the flight segment, which are used to solve for the plane coordinates and relative flight altitude from the relative flight distance, and define the preset heading angle and pitch angle of the reused flight segment as flight attitude parameters.

[0010] In one embodiment, the step of locating the target flight segment corresponding to the current low-frequency pose message using a preset retrieval space algorithm includes: Extract the real-time planar coordinates of the UAV in the current low-frequency pose message, sort all route list nodes in ascending order according to the planar X-axis coordinate, and generate an ordered set of waypoints; A binary search method is used to retrieve the ordered set of waypoints and filter out candidate waypoints whose X coordinate interval covers the real-time coordinates of the UAV. If a unique route segment cannot be matched, the route list is traversed bidirectionally forward and backward based on the candidate waypoints, and the segments are spliced ​​together to form the route segment to be verified. Based on the real-time planar coordinates of the UAV, the X-coordinate range and Y-coordinate range of the flight line segment to be verified are sequentially checked, and the flight line segment whose coordinate range completely covers the current real-time coordinates of the UAV is determined as the target flight line segment.

[0011] In one embodiment, the step of interpolating and solving for the initial pose data corresponding to each video frame based on the current low-frequency pose message, the target flight line segment, and the flight line prior model includes: The time-series transformation rules built into the prior model of the flight path are invoked, and the real flight time of the UAV corresponding to the current low-frequency pose message is used as a dynamic benchmark to deduce the real-time predicted flight path time corresponding to each video frame frame by frame. By combining the speed segment parameters of the target flight segment obtained through real-time matching with the real-time predicted flight time of each video frame, the segmented distance-time mapping function built into the flight prior model is called to calculate the relative flight distance of each video frame within the target flight segment in real time. Based on the real-time movement direction of the target flight line segment, the interpolation formula matched in the prior model of the flight line is called to solve the planar coordinates and relative flight altitude of the corresponding video frame, and the preset heading angle and pitch angle of the target flight line segment are reused as flight attitude parameters to integrate and generate the initial pose data corresponding to each video frame.

[0012] Secondly, this application also provides a low-frequency message interpolation device for unmanned aerial vehicles, comprising: The modeling module is used to obtain prior information on the target flight path design and to establish a prior flight path model by combining the frame rate information of the UAV video frames. The prior flight path model is used to perform interpolation calculations on the UAV pose data corresponding to the video frames. The positioning module is used to receive the current low-frequency pose message sent by the UAV and use a preset retrieval space algorithm to locate the target flight line segment corresponding to the current low-frequency pose message. The interpolation solution module is used to interpolate and solve the initial pose data corresponding to each video frame based on the current low-frequency pose message, the target flight segment and the flight prior model; The correction module is used to input the initial pose data, the current low-frequency pose message and the target flight line segment corresponding to each video frame into the pre-trained environmental factor correction model to obtain the distance deviation correction amount for each video frame. The correction module is used to correct each initial pose data by the distance deviation correction amount of each video frame, so as to obtain the target pose data that matches the video time sequence.

[0013] Thirdly, this application also provides an electronic device, including a processor and a memory; the memory has a computer program stored thereon, wherein the computer program, when executed by the processor, implements the UAV low-frequency message interpolation method as described in the first aspect.

[0014] Fourthly, this application also provides a non-transitory computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the UAV low-frequency message interpolation method as described in the first aspect.

[0015] The UAV low-frequency message interpolation method of this application has the following advantages over related technologies: 1. The UAV low-frequency message interpolation method of this application constructs a flight path prior model, integrates a standardized interpolation calculation framework based on the flight path prior model, uses the real time of the UAV low-frequency pose message as a dynamic benchmark, and completes the interpolation calculation in combination with the retrieved target flight path segment. The initial pose corresponding to each video frame is derived from the discrete low-frequency pose. This method overcomes the data limitations of low-frequency UAV pose backhaul, realizes the upsampling of pose information to the high-frequency frame rate of the video, establishes a continuous temporal correlation between video frames and UAV pose, and alleviates the fusion misalignment problem caused by the temporal mismatch between low-frequency state and high-frequency video frame.

[0016] 2. An environmental factor correction model is used to compensate for the deviation of the initial pose obtained by theoretical interpolation. Based on the distance deviation caused by flight environment disturbances, a correction amount is generated and the pose correction is completed. This overcomes the defect that pure flight path theoretical interpolation does not consider real flight disturbances, improves the accuracy of pose of high-frequency video frames, and finally achieves high-precision temporal matching between UAV video images and pose information, thereby supporting accurate temporal fusion of video. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for interpolating low-frequency messages from unmanned aerial vehicles (UAVs) according to an embodiment of this application. Figure 2 This is a schematic diagram of the process for establishing a priori route model in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a UAV low-frequency message interpolation device in one embodiment of this application. Detailed Implementation

[0019] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] In some embodiments, such as Figure 1 As shown, this application provides a method for interpolating low-frequency messages of unmanned aerial vehicles (UAVs), which includes the following steps S101 to S105.

[0021] S101: Obtain prior information on the target flight path design, and establish a prior flight path model by combining the frame rate information of the UAV video frames. The prior flight path model is used to perform interpolation calculations on the UAV pose data corresponding to the video frames.

[0022] The prior information for the target route design can include global route topology information, waypoint coordinates for each route segment, segmented flight attributes, segment switching constraints, and preset heading angles, pitch angles, and variable speed parameters for each route segment, thus fully recording all static navigation reference parameters pre-set during the route planning phase.

[0023] It is understandable that by building a priori model of the flight route offline based on the inherent static waypoints and flight segment attributes of the flight route design and the video sampling frame rate, and constructing a standardized computing framework for the interpolation operation link, there is no need to repeatedly parse flight route parameters and configure computing logic online in real time. This can provide a unified and standardized computing benchmark for subsequent high-frequency video frame pose extrapolation, support the dynamic extrapolation of each video frame time based on the low-frequency backhaul pose of the UAV and carry out interpolation operations, and provide a reusable computing carrier for upsampling discrete low-frequency pose information to the high-frequency temporal dimension of video.

[0024] S102: Receive the current low-frequency pose message sent by the UAV and use a preset retrieval space algorithm to locate the target flight line segment corresponding to the current low-frequency pose message.

[0025] For example, in the application, an MQTT server can be bound to the drone in the web application. After binding, the drone will continuously transmit its position and attitude at a frequency of 1Hz. After receiving the low-frequency data, the application can call the relevant models and algorithms to complete the high-frequency interpolation of 25Hz, thereby achieving synchronization between the message frequency and the video stream frequency and realizing high-frequency fusion of drone video images.

[0026] It is understandable that the real-time spatial location of the UAV carried in the low-frequency pose message is used as the retrieval input. The spatial retrieval algorithm is used to quickly match the flight segment to which the UAV's location belongs in the flight topology data, accurately lock the static parameters of the flight segment on which the subsequent interpolation calculation is based, avoid the computational overhead caused by global flight path traversal, and at the same time ensure that the flight segment matching result can be adapted to the dynamic flight position of the UAV. This provides an accurate and effective flight segment calculation benchmark for subsequent video frame pose interpolation based on the flight path prior model.

[0027] S103: Based on the current low-frequency pose message, target flight segment, and flight prior model, interpolate to solve the initial pose data corresponding to each video frame.

[0028] It is understandable that the actual flight time of the UAV provided by the low-frequency pose message is used as the dynamic timing reference. The standardized operation rules built into the prior model of the flight path are called, and the timing deduction and pose solution are completed by combining the static parameters of the target flight path segment obtained by retrieval. The discrete low-frequency pose is upsampled to the high-frequency timing of the video frame by interpolation based on the offline pre-built operation framework. This breaks through the data limitations of low-frequency backhaul of UAV pose and finally generates a theoretical initial pose sequence that matches the timing of each video frame.

[0029] S104: Input the initial pose data, current low-frequency pose message and target flight line segment corresponding to each video frame into the pre-trained environmental factor correction model to obtain the distance deviation correction amount for each video frame.

[0030] Among them, the environmental factor correction model adopts a neural network model, and the neural network structure of the environmental factor correction model is defined. : In the formula, x is the input. , For output, , For the first Layer weight values, For the first Layer offset This is the activation function. This represents the fully connected neural network's... The hidden layer outputs a feature vector. The input sample vector is directly assigned to the network; the hidden features of each intermediate layer are obtained by transforming the hidden features of the previous layer with weights, adding biases, and mapping them through the ReLU activation function, finally resulting in the th... The latent features of each layer are used as input to the output layer and are processed through linear operations to obtain the model's predicted values. .

[0031] Define input Specifically .

[0032] In the formula, These represent the current longitude, latitude, and altitude. These are vertical velocity and horizontal velocity, respectively. For electricity, For wind speed, These are the sine and cosine values ​​of the wind direction, respectively. For humidity, The proportion within the current route segment calculated in step (1), For the flight segment type (vertical or horizontal). The prediction time period is (0.04, 0.08, ... 1).

[0033] Define output : In the formula, For horizontal length deviation, This represents the vertical length deviation.

[0034] The loss values ​​for model training are as follows: , In the formula, Horizontal loss value, This represents the loss value in the vertical direction.

[0035] The loss function is defined as follows: In the formula, N is the sample size. , These refer to the sample type; when the sample is a horizontal sample, When the sample is vertical, , , These are the corrected values ​​for the horizontal and vertical directions, respectively. , These are the corrected predicted values ​​for the horizontal and vertical directions, respectively.

[0036] It should be noted that forward propagation, back propagation, and iterative training methods are commonly used. For example, the model training is carried out using forward propagation, back propagation, and gradient descent iterative training in a standard fully connected network. Forward propagation derives the predicted output from the input according to the layer-by-layer mapping relationship. The gradient of each layer parameter is obtained in reverse according to the loss function using the chain rule. The weights and bias parameters are continuously updated iteratively with the help of the gradient descent strategy. The iteration is repeated until the loss index converges, so as to realize the fitting of the model parameters with environmental factor correction.

[0037] It is understandable that by inputting the initial pose data, current low-frequency pose message, and target flight path segment corresponding to each video frame into a pre-trained environmental factor correction model, the distance deviation correction amount for each video frame can be obtained. This method fully combines the initial pose obtained by theoretical interpolation, the low-frequency pose actually transmitted by the UAV, and the reference information of the current flight path segment. It utilizes the pre-trained model to explore the position offset patterns caused by environmental disturbances and navigation deviations, and independently solves the distance deviation correction amount for each video frame. This compensates for the pose system error caused by pure flight path theoretical interpolation not considering real flight conditions, and provides a quantitative correction basis for subsequent accurate correction of the initial pose and reducing the deviation between the theoretical trajectory and the actual flight trajectory.

[0038] S105: Correct each initial pose data by the distance deviation correction of each video frame to obtain the target pose data matching the video time sequence.

[0039] It is understandable that the distance deviation correction of each video frame is introduced into a preset mapping function between distance deviation and coordinates, elevation, and attitude to complete multi-dimensional deviation conversion. Based on the converted correction term, the initial pose data generated by theoretical interpolation is compensated and corrected. Finally, a high-frequency coordinate, elevation, and attitude target pose sequence that is strictly aligned with the video acquisition time sequence (e.g., 25Hz) and fused with the influence of real flight disturbances can be output, thereby realizing the conversion of the UAV state to high frequency and enabling accurate fusion of UAV images in time sequence.

[0040] The aforementioned UAV low-frequency message interpolation method constructs a priori flight path model and integrates a standardized interpolation framework based on this model. Using the actual time of the UAV's low-frequency pose message as a dynamic benchmark, it performs interpolation calculations by combining the retrieved target flight path segments. The initial pose corresponding to each video frame is derived from the discrete low-frequency pose. This overcomes the data limitations of low-frequency UAV pose transmission, enabling upsampling of pose information to the high-frequency frame rate of the video. It establishes a continuous temporal correlation between video frames and UAV pose, mitigating the fusion misalignment problem caused by temporal mismatch between low-frequency states and high-frequency video frames. An environmental factor correction model is used to compensate for the deviation of the theoretically interpolated initial pose. Based on the distance deviation caused by flight environment disturbances, a correction amount is generated and pose correction is completed. This overcomes the deficiency of pure flight path theoretical interpolation, which does not consider real flight disturbances, improving the accuracy of high-frequency video frame pose. Ultimately, it achieves high-precision temporal matching between UAV video footage and pose information, thus supporting accurate video temporal fusion.

[0041] In some embodiments, such as Figure 2 As shown, in step S101, prior information on the target flight path design is obtained, and a prior model of the flight path is established by combining the frame rate information of the UAV video frames, including the following steps S201 to S204.

[0042] S201: Construct a route linked list based on prior information designed for the target route.

[0043] It is understandable that the prior information for the design of the target route includes the order and information of waypoints. Adjacent waypoints can be used to construct routes. Therefore, by using the prior information for the design of the target route, a chain-structured data carrier that can fully express the route topology and the order of waypoints can be built, namely, the route linked list.

[0044] S202: Determine the flight attributes corresponding to each route segment in the route list based on the route list. The flight attributes include the direction of motion and the parameters of the speed change segment.

[0045] The direction of motion represents the course of the flight segment and can include both horizontal and vertical directions. The parameters for the variable speed segment represent the speed variation pattern of the flight segment and can be divided into acceleration, constant speed, and deceleration segments.

[0046] It is understandable that, based on the structured topology of the established route list, the entire route can be broken down and its attributes analyzed segment by segment. Based on the spatial distribution of adjacent waypoints in the list and the preset parameters of the route planning, the complete flight attribute information corresponding to each route segment can be uniformly extracted and defined.

[0047] S203: Construct a rule for the correspondence between the video frame timing and the flight time of the flight path based on the frame rate information of the UAV video frames.

[0048] In the application, by segmenting and decoupling the flight path chain list, the flight path attributes such as speed range, pitch and heading constraints of each flight path segment are divided and extracted. Then, by combining the video acquisition frame rate of the UAV's onboard camera to establish a unified time sequence alignment criterion, the time conversion relationship between the sampling time of each video frame and the theoretical flight time of the flight path can be clearly defined.

[0049] S204: A priori flight path model is constructed based on waypoint information, flight attributes, and time interval correspondence rules, as well as a preset segmented distance-time mapping function and interpolation formula. Waypoint information includes waypoint coordinates and corresponding UAV flight attitude parameters.

[0050] It is understandable that in the model construction process, adjacent waypoints arranged in an ordered manner within the flight path list are used as interpolation boundary nodes. The waypoint coordinates are defined as the reference spatial positions at both ends of the flight segment, and the attitude parameters corresponding to the waypoints are used as the attitude references for the start and end of the flight segment. The continuous change constraints of position and attitude within the flight segment are limited by the flight attributes of each flight segment. The spatial coordinates of waypoints, the corresponding attitude parameters, the flight constraints of each flight segment, and the temporal matching rules are integrated. The computational logic is encapsulated by combining the predefined segmented distance-time quantization relationship and the pose interpolation calculation formula. A standardized flight path prior model is generated offline. It can convert the flight time corresponding to the video frame into the travel distance within the flight segment by combining the segmented distance-time mapping function. Then, the interpolation formula is used to continuously solve the spatial coordinates and attitude values ​​corresponding to any distance position on the flight path along the flight path with waypoint coordinates and attitude parameters as boundary references. This realizes the continuous derivation of the position and attitude of any time within the flight segment from the waypoint reference information. It provides a unified and reusable computational framework for dynamically deducing the theoretical coordinates, elevation, and attitude corresponding to each video sampling time from the low-frequency backhaul position of the UAV during online operation.

[0051] In some embodiments, constructing a route linked list based on prior information of the target route design includes: extracting all original waypoints from the prior information of the target route design; converting the geographical latitude and longitude of each waypoint into planar coordinates using a Gaussian forward algorithm; traversing all original waypoints in the flight order of the route, comparing the relative altitudes of adjacent waypoints, inserting an altitude inflection point waypoint if the altitudes of adjacent waypoints are inconsistent, updating the ordered waypoint sequence, and constructing a route linked list based on the ordered waypoint sequence. Each original waypoint includes latitude and longitude, planar coordinates, relative altitude, heading angle, pitch angle, horizontal speed, and vertical speed.

[0052] In the application, the spatial coordinates, attitude, horizontal velocity, and vertical velocity of the takeoff, flight point, and landing points are mainly collected. When the altitude of two waypoints changes, an altitude-lifting waypoint needs to be inserted. The UAV will first lift (or descend) to this waypoint in the vertical direction before flying to the next waypoint. After collecting an ordered list of waypoints, a preliminary mapping relationship between a certain spatial coordinate and the prediction frame is constructed.

[0053] Since drones typically use geographic coordinate systems, the results need to be converted to planar coordinates before calculation, and then converted back to geographic coordinates. This can be accomplished by directly sampling Gaussian forward and inverse calculation algorithms. In the formula, x and y are planar coordinates, and lon and lat are geographic latitude and longitude. It is a Gaussian positive calculation function. This is the Gaussian inverse function.

[0054] Define the original waypoint as: In the formula, w is the set of original waypoints, including the departure point, waypoints, and return point. This represents the state information of the i-th node. Latitude and longitude For the Gaussian forward calculation, the plane coordinates are... For relative flight altitude, For heading angle and pitch angle, These are vertical velocity and horizontal velocity, respectively.

[0055] Traverse the original set of waypoints and insert altitude inflection points: In the formula, This is an insertion function that iterates to the i-th point. If the elevation of the i-th point is inconsistent with that of the next point, a node is inserted; otherwise, no node is inserted.

[0056] Finally, iterate through the set of waypoints to construct the route linked list: ; In the formula, For the i-th node of the route, Let i be the next node after the i-th node. It is the previous node of the i-th node.

[0057] In some embodiments, determining the flight attributes corresponding to each route segment in the route list based on the route list includes: dividing two adjacent nodes in the route list into an independent route segment, and traversing the list to obtain all route segments; for each route segment, comparing the relative altitude of the starting waypoint with the relative altitude of the ending waypoint, and determining the motion direction of the route segment based on a preset altitude tolerance value; and determining the flight distance and duration corresponding to the acceleration segment, constant speed segment, and deceleration segment within the route segment based on preset horizontal speed and acceleration parameters of waypoints within the route segment, thus obtaining the variable speed segment parameters of the route segment. The motion direction includes horizontal motion and vertical motion.

[0058] In the application, the current location coordinates are: In the formula, The current position's planar coordinates, This refers to the current location's latitude and longitude.

[0059] To determine the direction of movement, the difference between the relative altitude and the end of the flight segment should be less than the tolerance value. , is horizontal motion; otherwise, is vertical motion. In the formula, This represents the relative height of the current location. Let k be the altitude of the endpoint of the flight path segment. For high tolerance.

[0060] When moving horizontally, calculate the distance traveled by the current point along the route segment: In the formula, The distance is in the horizontal direction. Let x be the current point's x-coordinate. Let x be the x-coordinate of the starting point of the flight path. The x-coordinate of the destination of the route. This refers to the length of the flight segment.

[0061] When moving in a vertical direction, calculate the distance traveled by the current point along the route segment: In the formula, The distance is in the vertical direction. The current height. The altitude at the start of the flight segment. The altitude at the end of the flight path. This refers to the length of the flight segment.

[0062] The time and distance of the acceleration, constant speed, and deceleration phases of the flight path: In the formula, , , These are the times for the acceleration phase, the constant speed phase, and the deceleration phase, respectively. , , These represent the distances of the acceleration phase, the constant speed phase, and the deceleration phase, respectively.

[0063] In some embodiments, a priori model of the flight route is constructed based on waypoint information, flight attributes, and time interval correspondence rules, as well as a preset segmented distance-time mapping function and interpolation formula. This includes: associating and binding waypoint information and flight attributes corresponding to each flight route segment; defining the time interval correspondence rule as a time sequence conversion rule from video frame timing to flight route timing; defining the input of the segmented distance-time mapping function as the variable speed segment parameters of the flight route segment and the relative timing of the flight route, and using the segmented distance-time mapping function as the standard calculation rule for solving the relative flight distance within the flight route segment; configuring an interpolation formula that matches the motion direction of the flight route segment, used to solve the planar coordinates and relative flight altitude from the relative flight distance, and defining the step of reusing the preset heading angle and pitch angle of the flight route segment as flight attitude parameters.

[0064] The following explanation uses a video frame rate of 25 frames. Based on the previous embodiment, a time-distance mapping relationship can be constructed: In the formula, t is the starting time of the relative flight segment. This represents the relative distance at a certain moment; the meanings of other variables are the same as in the previous step.

[0065] Calculate the current time from the current coordinates: In the formula, d is the relative distance to the current position calculated in the previous step. This refers to the current moment.

[0066] For the position of the next 25 frames, the frame interval Δt = 0.04 seconds, for j = 0, 1, 2, ... 24: In the formula, For the (j+1)th frame in the future, This represents the relative distance in the (j+1)th frame in the future.

[0067] If the movement is horizontal, the coordinates are: , , In the formula, , , , , The coordinates, altitude, yaw angle, and pitch angle of the (j+1)th frame. , , The coordinates and altitude of the starting point of the flight segment. , These are the yaw and pitch angles set for the waypoints, respectively.

[0068] If the movement is in the vertical direction, the coordinates are: , , In this formula, the meanings of the variables are the same as in the previous formula.

[0069] It is understandable that establishing a one-to-one correspondence between flight segment entities and flight attributes such as coordinates, attitude parameters, and speed constraints at both ends of the flight route linked list segmentation is based on the segmentation of the flight route linked list. This clarifies the data ownership boundaries and solidifies the conversion logic of mapping video frame sampling time sequence to theoretical flight time of the flight route into a time sequence conversion rule, standardizing the unified conversion standard of time sequence information. At the same time, the input variable type of the segment distance-time mapping function is set, and a quantitative calculation benchmark from time sequence to flight mileage is established based on the flight segment speed variation characteristics and relative time parameters. Then, according to the flight direction and elevation change characteristics of each flight segment, the corresponding spatial interpolation calculation formula is adapted. Based on the plane coordinates and relative flight altitude corresponding to each position on the continuous derivation path of the relative flight distance within the flight segment output by the mapping function, the preset heading angle and pitch angle of the flight segment are directly used as the attitude benchmark parameters of the corresponding position. This completely builds a hierarchical derivation link from flight time to spatial coordinates, elevation, and flight attitude, which can provide standardized and segmented offline calculation constraints for quickly solving the theoretical initial pose corresponding to each video frame in the subsequent online stage.

[0070] In some embodiments, a preset retrieval space algorithm is used to locate the target flight line segment corresponding to the current low-frequency pose message, including: extracting the real-time planar coordinates of the UAV within the current low-frequency pose message; sorting all flight line list nodes in ascending order according to the planar X-axis coordinates to generate an ordered waypoint set; using a binary search method to search the ordered waypoint set and filter out candidate waypoints whose X-coordinate range covers the real-time coordinates of the UAV; if a unique flight line segment cannot be matched, traversing the flight line list bidirectionally forward and backward based on the candidate waypoints and splicing them to form a flight line segment to be verified; and sequentially verifying the X-coordinate range and Y-coordinate range of the flight line segment to be verified based on the real-time planar coordinates of the UAV, and determining the flight line segment whose coordinate range completely covers the current real-time coordinates of the UAV as the target flight line segment.

[0071] In the application, waypoints are first sorted in the x-direction: In the formula, G is the set sorted in the X direction. This is the Nth node.

[0072] Then, the starting point is located using a binary search method: In the formula, Let X be the X coordinate of the intermediate node. Current X coordinate.

[0073] Next, the destination waypoint is found based on the route list. The current point should be within a route segment, requiring comparison. If the binary search does not meet the conditions, the search continues westward and eastward respectively until the route segment is located. End of route segment: The current x-coordinate is within the flight path range: The current y-coordinate is within the flight path range: It is understandable that by parsing the current low-frequency pose message, the current planar position information of the UAV can be obtained. Then, all waypoints in the flight path list are rearranged in ascending order according to the plane X coordinate to construct an ordered set. Binary search is used to quickly narrow down the coordinate matching range and initially screen candidate waypoints that can surround the real-time position of the UAV in the X dimension. For scenarios where matching in a single X dimension cannot lock a unique flight segment, multiple sets of flight segments to be verified are obtained by bidirectionally traversing the original flight path list based on candidate waypoints. Then, integrity verification is carried out by combining the plane X and Y bidirectional coordinate boundaries, and finally the target flight path segment currently occupied by the UAV is located. This effectively avoids the problem of multiple flight segments confusion caused by single coordinate axis retrieval, improves the retrieval efficiency and positioning accuracy of flight segment matching in the online stage, and provides an accurate flight segment boundary basis for subsequent high-frequency pose interpolation calculation based on the flight path prior model.

[0074] In some embodiments, the initial pose data corresponding to each video frame is solved by interpolation based on the current low-frequency pose message, the target flight line segment, and the flight line prior model. This includes: calling the time-series transformation rules built into the flight line prior model, using the actual flight time of the UAV corresponding to the current low-frequency pose message as a dynamic benchmark, and deducing the real-time predicted flight line time corresponding to each video frame frame by frame; combining the variable speed segment parameters of the target flight line segment obtained by real-time matching with the real-time predicted flight line time of each video frame, calling the segmented distance-time mapping function built into the flight line prior model, and calculating the relative flight distance of each video frame within the target flight line segment in real time; and, based on the real-time motion direction of the target flight line segment, calling the interpolation formula matched in the flight line prior model to solve for the planar coordinates and relative flight altitude of the corresponding video frame, and reusing the preset heading angle and pitch angle of the target flight line segment as flight attitude parameters, and integrating to generate the initial pose data corresponding to each video frame.

[0075] The process of generating the initial pose data for each video frame online can be found in the description of the aforementioned embodiments.

[0076] Based on this, during the online operation phase, the real-time planar coordinate information in the UAV's low-frequency pose message is first extracted. All nodes in the flight path list are sorted in ascending order according to the planar X-axis coordinate to generate an ordered set of waypoints. A binary search method is used to quickly retrieve and filter candidate waypoints whose X-coordinate intervals cover the UAV's real-time coordinates. For scenarios where a single X-axis search cannot match a unique flight path segment, the flight path list is bidirectionally traversed based on the candidate waypoints and spliced ​​to generate flight path segments to be verified. The X and Y coordinate coverage of each flight path segment to be verified is further checked in conjunction with the UAV's real-time planar coordinates. Finally, the flight path segment that completely matches the UAV's real-time position is determined as the current target flight path segment.

[0077] After accurately locking onto the target flight path segment, the time-series transformation rules built into the flight path prior model are invoked. Using the actual flight time of the UAV corresponding to the current low-frequency pose message as the dynamic time-series benchmark, the real-time predicted flight path time corresponding to each high-frequency video frame is calculated frame by frame. Combining the speed change parameters of the target flight path segment with the real-time predicted flight path time of each video frame, the segmented distance-time mapping function built into the model is invoked to calculate the relative flight distance of each video frame within the target flight path segment in real time. Then, based on the real-time motion direction of the target flight path segment, the matching interpolation formula is invoked to solve for the plane coordinates and relative flight altitude corresponding to each video frame. The heading angle and pitch angle preset by the flight path segment are reused as flight attitude parameters, and the initial pose data corresponding to all video frames are integrated to generate the data.

[0078] Subsequently, the initial pose data of each video frame, the UAV's low-frequency MQTT pose message data, and the matched target flight line information are input into the pre-trained environmental factor correction model. By fully combining real flight conditions and flight line reference information, the model can be used to mine the flight deviation patterns caused by environmental disturbances. The model solves the precise distance deviation correction amount frame by frame. Then, through the preset mapping function between the distance deviation amount and coordinates, elevation, and attitude, the frame-by-frame distance deviation correction amount is converted into pose correction amounts in various dimensions. The initial pose data obtained by theoretical interpolation is then subjected to comprehensive deviation compensation and correction. Finally, high-frequency precise coordinates, elevation, and attitude target pose data that perfectly match the 25Hz video acquisition timing and fit the real flight state of the UAV are obtained.

[0079] In some embodiments, please refer to Figure 3 The present application provides a low-frequency message interpolation device 30 for unmanned aerial vehicles, comprising: a modeling module 31, a positioning module 32, an interpolation solving module 33, a correction module 34, and a correction module 35.

[0080] Modeling module 31 is used to obtain prior information on the target flight path design and to establish a prior flight path model by combining the frame rate information of the UAV video frame. The prior flight path model is used to realize the interpolation calculation of the UAV pose data corresponding to the video frame. The positioning module 32 is used to receive the current low-frequency pose message sent by the UAV and use a preset retrieval space algorithm to locate the target flight line segment corresponding to the current low-frequency pose message; Interpolation module 33 is used to interpolate and solve the initial pose data corresponding to each video frame based on the current low-frequency pose message, target flight segment and flight prior model; The correction module 34 is used to input the initial pose data, current low-frequency pose message and target line segment corresponding to each video frame into the pre-trained environmental factor correction model to obtain the distance deviation correction amount of each video frame. The correction module 35 is used to correct each initial pose data by the distance deviation correction amount of each video frame to obtain target pose data that matches the video time sequence.

[0081] It should be noted that the UAV low-frequency message interpolation device 30 provided in this application embodiment and the UAV low-frequency message interpolation method provided in this application embodiment are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned UAV low-frequency message interpolation method, and the repeated parts will not be described again.

[0082] In some embodiments, an electronic device provided in this application includes a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the above-described UAV low-frequency message interpolation method.

[0083] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0084] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0085] This application also provides a non-transitory computer storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned UAV low-frequency message interpolation method. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0086] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0088] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application, even if such combinations or combinations are not explicitly described in this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for interpolating low-frequency messages from unmanned aerial vehicles (UAVs), characterized in that, include: Obtain prior information on the target flight path design, and establish a prior flight path model by combining the frame rate information of the UAV video frames. The prior flight path model is used to perform interpolation calculations on the UAV pose data corresponding to the video frames. Receive the current low-frequency pose message sent by the UAV and use a preset retrieval space algorithm to locate the target flight line segment corresponding to the current low-frequency pose message; Based on the current low-frequency pose message, the target flight segment, and the flight prior model, the initial pose data corresponding to each video frame is interpolated and solved. The initial pose data, current low-frequency pose message and target flight line segment corresponding to each video frame are input into the pre-trained environmental factor correction model to obtain the distance deviation correction amount for each video frame. The initial pose data are corrected by adjusting the distance deviation of each video frame to obtain the target pose data that matches the video time sequence.

2. The UAV low-frequency message interpolation method as described in claim 1, characterized in that, The process of obtaining prior information on the target flight path design and establishing a prior flight path model by combining the frame rate information of the UAV video frames includes: Based on the prior information of the target route design, a route linked list is constructed; Based on the route list, the flight attributes corresponding to each route segment in the route list are determined, wherein the flight attributes include the direction of motion and speed change parameters; Construct a rule for the correspondence between the video frame timing and the flight time of the flight path based on the frame rate information of the UAV video frames; Based on waypoint information, the flight attributes and the time interval correspondence rules, as well as the preset segmented distance-time mapping function and interpolation formula, a priori flight path model is constructed. The waypoint information includes waypoint coordinates and corresponding UAV flight attitude parameters.

3. The UAV low-frequency message interpolation method as described in claim 2, characterized in that, The construction of the route linked list based on the prior information of the target route design includes: Extract all original waypoints from the prior information of the target route design. Each original waypoint includes latitude and longitude, plane coordinates, relative altitude, heading angle, pitch angle, horizontal speed and vertical speed. The Gaussian forward calculation algorithm is used to convert the geographical latitude and longitude of each waypoint into planar coordinates; Traverse all original waypoints in the flight order of the route, compare the relative altitudes of adjacent waypoints, and insert an altitude inflection point waypoint if the altitudes of adjacent waypoints are inconsistent, update the ordered waypoint sequence, and construct a route linked list based on the ordered waypoint sequence.

4. The UAV low-frequency message interpolation method as described in claim 2, characterized in that, The step of determining the flight attributes corresponding to each route segment in the route list based on the route list includes: Divide two adjacent nodes in the route list into an independent route segment, and traverse the split to obtain all route segments in the list. For each route segment, the relative altitude of the starting waypoint and the relative altitude of the ending waypoint of the route segment are compared, and the movement direction of the route segment is determined in combination with the preset altitude tolerance value. The movement direction includes horizontal movement and vertical movement. Based on the preset horizontal speed and acceleration parameters of waypoints within the route segment, the flight distance and duration corresponding to the acceleration, constant speed, and deceleration segments within the route segment are determined, thus obtaining the variable speed segment parameters of the route segment.

5. The UAV low-frequency message interpolation method as described in claim 2, characterized in that, The construction of the route prior model based on waypoint information, flight attributes, and time interval correspondence rules, as well as a preset segmented distance-time mapping function and interpolation formula, includes: Associate and bind the waypoint information and flight attributes corresponding to each route segment; The time interval correspondence rule is defined as the timing conversion rule from video frame timing to flight route timing; The segmented distance-time mapping function is defined as taking the speed change parameters of the flight segment and the relative time of the flight segment as inputs, and the segmented distance-time mapping function is used as the standard calculation rule for solving the relative flight distance within the flight segment. Configure interpolation formulas that match the direction of motion of the flight segment, which are used to solve for the plane coordinates and relative flight altitude from the relative flight distance, and define the preset heading angle and pitch angle of the reused flight segment as flight attitude parameters.

6. The UAV low-frequency message interpolation method as described in claim 2, characterized in that, The step of locating the target flight segment corresponding to the current low-frequency pose message using a preset retrieval space algorithm includes: Extract the real-time planar coordinates of the UAV in the current low-frequency pose message, sort all route list nodes in ascending order according to the planar X-axis coordinate, and generate an ordered set of waypoints; A binary search method is used to retrieve the ordered set of waypoints and filter out candidate waypoints whose X coordinate interval covers the real-time coordinates of the UAV. If a unique route segment cannot be matched, the route list is traversed bidirectionally forward and backward based on the candidate waypoints, and the segments are spliced ​​together to form the route segment to be verified. Based on the real-time planar coordinates of the UAV, the X-coordinate range and Y-coordinate range of the flight line segment to be verified are sequentially checked, and the flight line segment whose coordinate range completely covers the current real-time coordinates of the UAV is determined as the target flight line segment.

7. The UAV low-frequency message interpolation method according to any one of claims 1 to 6, characterized in that, The process of interpolating and solving for the initial pose data corresponding to each video frame based on the current low-frequency pose message, the target flight line segment, and the flight line prior model includes: The time-series transformation rules built into the prior model of the flight path are invoked, and the real flight time of the UAV corresponding to the current low-frequency pose message is used as a dynamic benchmark to deduce the real-time predicted flight path time corresponding to each video frame frame by frame. By combining the speed segment parameters of the target flight segment obtained through real-time matching with the real-time predicted flight time of each video frame, the segmented distance-time mapping function built into the flight prior model is called to calculate the relative flight distance of each video frame within the target flight segment in real time. Based on the real-time movement direction of the target flight line segment, the interpolation formula matched in the prior model of the flight line is called to solve the planar coordinates and relative flight altitude of the corresponding video frame, and the preset heading angle and pitch angle of the target flight line segment are reused as flight attitude parameters to integrate and generate the initial pose data corresponding to each video frame.

8. A low-frequency message interpolation device for unmanned aerial vehicles (UAVs), characterized in that, include: The modeling module is used to obtain prior information on the target flight path design and to establish a prior flight path model by combining the frame rate information of the UAV video frames. The prior flight path model is used to perform interpolation calculations on the UAV pose data corresponding to the video frames. The positioning module is used to receive the current low-frequency pose message sent by the UAV and use a preset retrieval space algorithm to locate the target flight line segment corresponding to the current low-frequency pose message. The interpolation solution module is used to interpolate and solve the initial pose data corresponding to each video frame based on the current low-frequency pose message, the target flight segment and the flight prior model; The correction module is used to input the initial pose data, the current low-frequency pose message and the target flight line segment corresponding to each video frame into the pre-trained environmental factor correction model to obtain the distance deviation correction amount for each video frame. The correction module is used to correct each initial pose data by the distance deviation correction amount of each video frame, so as to obtain the target pose data that matches the video time sequence.

9. An electronic device, characterized in that, It includes a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the UAV low-frequency message interpolation method as described in any one of claims 1 to 7.

10. A non-transitory computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the UAV low-frequency message interpolation method as described in any one of claims 1 to 7.

Images