An unmanned aerial vehicle video geographic information superimposition method based on double-channel time delay alignment

Abstract

The application discloses a kind of unmanned plane video geographic information superimposition methods based on double channel time delay alignment, comprising the following steps: based on the difference between the ground receiving monotonous time of telemetry message and the time difference of corresponding equipment generation, the observable network time delay of this telemetry message is calculated;Based on the physical fact that video frame stream and telemetry message stream pass through the same uplink transmission, the time delay variation of two, using the observable network time delay, adaptively estimate video channel time delay;With the pose estimation, the pipeline obtained by projection is superimposed and rendered to the video frame in hierarchical electronic fence, and the nearest distance from the projection point directly below unmanned plane to the center line of pipeline is used to determine the electronic fence level where it is located, to make picture classification prompt or alarm according to it.The application gives the precision-real-time compromise that can be engineered.

Description

Technical Field

[0001] This invention relates to the field of video GIS technology, and in particular to a method for overlaying UAV video geographic information based on dual-channel time delay alignment. Background Technology

[0002] Drones equipped with gimbal cameras fly along pipelines and transmit video in real time to the ground command center via 4G / 5G / satellite links. This technology has been widely used for the daily inspection of above-ground pipelines and long-distance pipelines. To enable inspection personnel to directly see the location of pipelines, the boundaries of protected areas, and the distance of potential hazards from pipelines on the video feed, the industry is exploring the real-time overlay of pipeline geographic information system (GIS) vector data onto the transmitted video.

[0003] However, to truly implement this "real-time overlay of video and geographic information" in an engineering environment where "real-time backhaul via public network links and telemetry and video are transmitted through two separate channels," existing technical solutions still have the following unavoidable shortcomings: 1. In the drone backhaul link, the video stream carrying the image and the telemetry stream carrying the pose are two independent channels with different end-to-end latency levels. The end-to-end latency of the video channel is usually greater than that of the telemetry channel. If the ground end simply "projects the video frame received at this moment using the telemetry pose received at this moment", the projected pipelines and fences will deviate from their actual positions in the image due to the latency difference between the two channels.

[0004] 2. The timestamp of telemetry messages is often marked as "the time of arrival at the middleware or aggregation" rather than "the time when the aircraft generates data". There is a time-varying lag between the two due to network jitter. If this "time of arrival" is directly used as the pose time axis and aligned with the video, even if the video delay is accurately estimated, it will lead to a superimposed systematic offset. Moreover, due to this time-varying lag, it cannot be compensated by a fixed offset. Timestamp caliber normalization and adaptive estimation must be performed first.

[0005] 3. When transmitting back via the public network, the telemetry pose reporting frequency is often reduced to save bandwidth. The telemetry reporting frequency is lower than the video frame rate and differs by one to two orders of magnitude. If the UAV performs maneuvers such as turning, acceleration, or deceleration within the sampling interval, linear interpolation is only performed using two adjacent low-frequency poses, which will result in severe distortion within the interval.

[0006] 4. To avoid latency issues, some existing solutions perform the overlay locally on the airborne end, or rely on a tight combination of real-time dynamic positioning (RTK) and inertial navigation system (INS) to provide high-frequency, high-precision pose before overlay. When estimating video channel latency, existing methods either rely on prior truth values ​​(which are not available in engineering) or on the cross-correlation between image optical flow and pose velocity. However, this approach is unstable due to extremely low signal-to-noise ratio in images with weak textures such as smooth return, water surfaces, and vegetation. Even if alignment is performed, latency estimation, pose interpolation, rendering buffer duration, and real-time / accuracy trade-offs are not integrated into a single solution.

[0007] 5. The GIS coordinate systems for pipelines come from various sources. If they are not unified before overlay, the hundreds of meters of offset between the Mars coordinate system GCJ-02 and the 1984 World Geodetic Coordinate System WGS84 will cause the pipeline to be completely shifted out of the picture. Moreover, most existing solutions only perform vector outlining and do not achieve an integrated geometric approach that includes multi-coordinate system generation, hierarchical buffer zone generation, projection, and real-time hierarchical determination based on the closest distance from the UAV projection point to the pipeline. Summary of the Invention

[0008] The purpose of this invention is to provide a method for overlaying UAV video geographic information based on dual-channel time delay alignment. This invention can reduce video geographic information overlay errors under real-world conditions such as weak public network links, low-frequency telemetry, and inaccurate timestamp standards, using an alignment structure independent of image texture and prior truth values, and provides an engineering-tunable trade-off between accuracy and real-time performance.

[0009] The technical solution of this invention: A method for overlaying UAV video geographic information based on dual-channel time delay alignment, comprising the following steps: Step 1: Access the video frame stream via the first channel and record the ground reception monotonic time for each frame; access the telemetry message stream carrying the pose via the second channel, record the ground reception monotonic time for each telemetry message, and normalize its timestamp to the device generation time when the UAV actually generates the corresponding pose; use the normalized pose as the sequence element and maintain the pose timeline buffer in ascending order of device generation time; calculate the observable network latency of the telemetry message based on the difference between the ground reception monotonic time of the telemetry message and the corresponding device generation time. Step 2: Based on the physical fact that the video frame stream and the telemetry message stream are transmitted through the same uplink and their delays are related, the video channel delay is adaptively estimated using the observable network delay; the video channel delay is subtracted from the ground reception monotonic moment of the video frame to obtain the UAV generation time corresponding to the frame; the UAV generation time is interpolated in the pose timeline buffer to obtain the pose estimate of the video frame. Step 3: Unify the pipeline centerline and the hierarchical electronic fence to the target geodetic coordinate system, generate hierarchical buffer zones on the local metric plane, project back latitude and longitude, and project the geographic feature points as pixel coordinates on the video frame. Step 4: Based on the pose estimation, overlay the projected pipeline and the graded electronic fence onto the video frame, and determine the grade of the electronic fence based on the shortest distance from the projection point directly below the drone to the center line of the pipeline, so as to provide a graded prompt or alarm for the screen.

[0010] The above-mentioned UAV video geographic information overlay method based on dual-channel delay alignment, in step 1, the telemetry message stream is carried by a unified pose envelope containing a source type field. The source type includes one or more of the following: UAV model subtitles, airborne software development kits, cloud interfaces, or airport backend logs. The timestamp normalization in step 1 and the delay estimation in step 2 are both performed based on the device generation time and pose field in the unified pose envelope. Changes in the telemetry source do not change the alignment and interpolation process.

[0011] In the aforementioned UAV video geographic information overlay method based on dual-channel delay alignment, step 2, which adaptively estimates the video channel delay using an exponential moving average, is updated online recursively using the following formula: ; In the formula, For telemetry self-time delay smoothing; The time-delay sliding average learning rate; For observable network latency; The adaptive estimation of video channel latency is calculated as follows: ; In the formula, An adaptive estimate of the video channel latency; A fixed offset between the video pipeline and the telemetry pipeline; The fixed bias and the time-delay sliding average learning rate are determined before deployment by recalibrating the target backhaul link using on-site calibration flight samples combined with known ground control points. Based on the product of the estimated standard deviation of the video channel delay and the UAV ground speed, and the product of the UAV maneuvering acceleration and the square of the telemetry sampling interval, the superposition position error boundary is estimated, and a decision is made to increase the telemetry reporting frequency or increase the rendering buffer delay. When the number of poses in the pose timeline buffer is insufficient to support interpolation, a preheating flag is output and the initial delay value is temporarily used as the video channel delay.

[0012] The aforementioned method for overlaying UAV video geographic information based on dual-channel delay alignment further includes, in step 2, setting an adjustable rendering buffer delay, wherein the video frame is sent for display or encoded output at a time determined by the following formula: ; In the formula, This is the actual displayed time; To buffer rendering latency; For the monotonic moments of ground reception of video frames; By adjusting The size of the image frame is adjusted to delay the display of video frames while waiting for telemetry data to arrive on both sides of the frame at the corresponding time, thus striking a balance between real-time performance and overlay accuracy.

[0013] The aforementioned UAV video geographic information overlay method based on dual-channel time delay alignment, the interpolation in the pose timeline buffer in step 2 includes: when the telemetry message contains a velocity component and the telemetry reporting frequency is lower than a preset threshold, cubic Hermite interpolation with adjacent point velocity constraints is used; otherwise, linear interpolation is used; for the yaw angle component, 0° / 360° cross-boundary interpolation is processed according to the minimum arc length method.

[0014] The aforementioned method for overlaying UAV video geographic information based on dual-channel time delay alignment includes the following steps in step 3: when the declared coordinate system is a coordinate system with a second offset, it is converted to the target geodetic coordinate system by numerical iterative inverse transformation; when the pipeline centerline does not declare a coordinate system, it refuses to overlay and generates an alarm.

[0015] The aforementioned method for overlaying UAV video geographic information based on dual-channel time delay alignment, the generation method of the hierarchical buffer zone in step 3 is as follows: project the unified and normalized points of the pipeline centerline onto the local East-North-Tian metric plane, generate multi-level flat-headed buffer polygons with increasing radius on the pipeline centerline on the metric plane, and then project the vertices of each level of flat-headed buffer polygons back to latitude and longitude coordinates to obtain hierarchical fence rings that correspond one-to-one with each level of electronic fence.

[0016] The aforementioned UAV video geographic information overlay method based on dual-channel delay alignment further includes, in step 4, bypassing critical boundary crossing or hazard alarms to a telemetry direct reading path that does not depend on the video channel delay. This allows for independent hierarchical judgment using high-frequency telemetry pose, and the generation of hierarchical alarms when a preset threshold is reached. The hierarchical judgment accuracy of the telemetry direct reading path is governed by the telemetry's own positioning accuracy and the interpolation error under the telemetry sampling interval, excluding the product of the video channel delay estimation deviation and the UAV ground speed. Therefore, the positional accuracy of the hierarchical alarm is superior to the positioning accuracy of the overlaid rendered image.

[0017] The aforementioned UAV video geographic information overlay method based on dual-channel latency alignment automatically increases the rendering buffer latency and / or outputs a precision degradation prompt in the image when the jitter of the observable network latency exceeds a preset threshold, while maintaining the hierarchical judgment and alarm output of the telemetry direct reading path without interruption.

[0018] In the aforementioned UAV video geographic information overlay method based on dual-channel time delay alignment, step 2 further estimates video time delay candidates by cross-correlation of the visual motion amplitude sequence and telemetry velocity amplitude sequence when the video image texture is sufficient. These candidates are then fused into the adaptive estimation of the video channel time delay after outlier elimination and moving average. When the texture is insufficient, this cross-correlation is not relied upon.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. Based on the physical fact that "video stream and telemetry stream share the same uplink and their latency changes are strongly correlated", this invention uses the exponential moving average of network latency that can be accurately measured by telemetry itself, plus a fixed bias, as an adaptive proxy for video channel latency. It does not depend on image texture or prior truth value and is equally robust on images with weak textures such as smooth return, water surface, and vegetation, effectively reducing superposition error.

[0020] 2. This invention normalizes the telemetry "arrival / aggregation time" to the "aircraft data generation time" by normalizing the timestamp caliber, thereby eliminating the systematic lag introduced by the timestamp caliber and enabling the entire real-time alignment path to be established. By introducing a tunable rendering buffer delay, the screen is delayed to wait for the corresponding telemetry arrival, thus providing an engineering-tunable trade-off between "real-time performance" and "overlay accuracy". The low-frequency pose is completed by linear / velocity-constrained Hermite interpolation, and the alignment error boundary is provided to guide "frequency increase or buffer addition", making the tuning based on evidence rather than trial and error.

[0021] 3. This invention achieves integrated overlay and real-time hierarchical determination of first- and third-level electronic fences in a multi-coordinate system; through unified normalization processing, it solves the problem of pipelines shifting out of the screen due to coordinate system offset between GCJ-02 and WGS84; with the help of a flat-head buffer zone generated by the local metric plane, it ensures that the width of each level of fence strictly corresponds to the actual ground metric distance, rather than being approximated by latitude and longitude; at the same time, it determines in real time which level (L1 / L2 / L3) the pipeline falls into based on the shortest distance from the projection point directly below the drone to the center line of the pipeline, and provides real-time prompts on the screen with hierarchical color matching and head-up display, realizing "seeing the pipeline, seeing the protected area, and seeing the boundary crossing".

[0022] 4. This invention bypasses critical boundary crossing / hazard alarms to a telemetry direct reading path that does not rely on video alignment. Its accuracy is determined only by the telemetry's own positioning and does not include video latency, so it is superior to image overlay, allowing "visualization using alignment path and alarm using direct reading path" to each have their own advantages. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This is a schematic diagram illustrating the strong correlation between dual-channel latency and video latency estimation. Figure 3 This is a diagram illustrating the standardization of timestamp calibers; Figure 4 This is a schematic diagram of pose timeline buffering and reverse lookup interpolation; Figure 5 This is a schematic diagram illustrating the trade-off between real-time performance and overlay accuracy in rendering buffers; Figure 6 It is a geometrical diagram of the projection of geographical elements; Figure 7 This is a schematic diagram of the generation of buffer zones for level one and level three electronic fences in a multi-coordinate system; Figure 8 This is a schematic diagram of real-time classification and determination of a three-level electronic fence; Figure 9 This is a diagram illustrating real-time overlay rendering of video frames and a HUD. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0025] Example 1: This example provides a method for overlaying UAV video geographic information based on dual-channel time-delay alignment. This method achieves robust real-time overlay display and hierarchical alarms even under weak texture conditions through dual-channel time-delay adaptive alignment and hierarchical projection rendering, which is particularly suitable for long-distance inspection scenarios such as long-distance pipelines. Specifically, the method includes the following steps: Step 1: Access the video frame stream via the first channel and record the ground reception monotonic time for each frame; access the telemetry message stream carrying the pose via the second channel, record the ground reception monotonic time for each telemetry message, and normalize its timestamp to the device generation time when the UAV actually generates the corresponding pose; use the normalized pose as the sequence element and maintain the pose timeline buffer in ascending order of device generation time; calculate the observable network latency of the telemetry message based on the difference between the ground reception monotonic time of the telemetry message and the corresponding device generation time. In this step, the video frames and telemetry messages generated by the UAV are accessed through the UAV's backhaul link. The UAV's backhaul link adopts a dual-channel approach. The video frame stream is accessed in the first channel. After encoding, streaming, forwarding, and decoding, the video frame stream reaches the ground server frame by frame. The telemetry message stream carrying the pose data is accessed in the second channel. After passing through airborne, airport aggregation, cloud, and message middleware, the telemetry message stream reaches the ground server. The pose data includes the device generation time, latitude, longitude, absolute altitude, relative altitude, gimbal yaw / pitch / roll, and east and north velocity components.

[0026] Furthermore, after receiving the video frame stream and telemetry message stream, the ground server performs the following processing steps: 1.1 Establish video frame arrival callback and telemetry message arrival callback, and record the monotonic reception time of each to the ground. When recording, always use the monotonic clock to avoid wall clock backtracking contaminating the delay measurement. 1.2 In the telemetry callback, perform field validity validation on the telemetry message. For example, latitude and longitude must fall within the specified range. Furthermore, the time of the telemetry message must be parsable; if the result of the field validity check is invalid, the telemetry message is determined to be an invalid telemetry message and discarded. 1.3 If the telemetry message already carries the device generation time, it is used directly; otherwise, the "arrival / aggregation time" of the telemetry message is converted to the device generation time according to the preset conversion rules (e.g., the fixed offset configured by the ground server, or the offset value dynamically filled in by the link-side handshake protocol). And record the observable network latency of the telemetry message. Specifically, let the monotonic reception time of a certain telemetry message reaching the ground be... The time of generation of the equipment it carries and after caliber standardization is Then the observable network latency of this telemetry channel on this telemetry message is: ; In the formula, For observable network latency; The monotonic reception time of the telemetry message reaching the ground; The time when the device generates the signal.

[0027] The time complexity for normalizing the timestamp of each telemetry message and calculating the observable network latency is O(n). .

[0028] Figure 3This diagram illustrates the process of normalizing the timestamp of a telemetry message from "arrival time" to "device generation time." The upper time axis in the diagram represents the time the aircraft generates data, while the lower time axis represents the time the message arrives at the middleware. The diagonal lines in the diagram connect the generation and reception times of the same telemetry message, with the line length representing... This normalization process eliminates the systematic lag caused by network jitter and middleware processing, laying the foundation for accurate latency alignment in the future.

[0029] 1.4. Insert the normalized pose into the pose timeline buffer, maintaining ascending order and an upper bound on its capacity. The pose timeline buffer can be represented as: ; In the formula, Buffering the pose timeline; Indicates that the latest version is retained. Data items; For the normalized pose record; This is the upper bound of the capacity; It should be noted that the pose timeline buffer is a line that is... Ascending order, upper limit of capacity is An ordered sequence, and because the pose is according to Since the pose timeline buffer is inserted in ascending order rather than in arrival order, it naturally has the ability to tolerate out-of-order arrivals of telemetry streams; new poses are appended to the end of the sequence if the capacity is exceeded. Then discard the oldest pose. The frame rate and reporting frequency can be adjusted; the default is... , Next contract coverage minute, Next contract coverage Second; The time complexity of inserting the normalized poses into the pose timeline buffer in ascending order of device generation time is... (Using binary search positioning), if messages arrive in roughly the correct order, then... .

[0030] 1.5. In the video callback, record the monotonic moments of the ground reception of the video frames. With the frame payload, the expression is: ; In the formula, For video frames; This means that for each arriving video frame, it is recorded as a tuple; For the monotonic moments of ground reception of video frames; For frame payload.

[0031] Record and The time complexity is .

[0032] Step 2: Based on the physical fact that the video frame stream and the telemetry message stream are transmitted through the same uplink and their delays are related, the video channel delay is adaptively estimated using the observable network delay; the video channel delay is subtracted from the ground reception monotonic moment of the video frame to obtain the UAV generation time corresponding to the frame; the UAV generation time is interpolated in the pose timeline buffer to obtain the pose estimate of the video frame. The specific process for this step is as follows: 2.1 When a telemetry message arrives, update the online self-time delay sliding average of that telemetry path. The expression is: ; In the formula, For telemetry self-time delay smoothing, and satisfying the first condition ; The time-delay sliding average learning rate is 0.2 by default.

[0033] Online updates of the telemetry self-time delay sliding average The time complexity is .

[0034] 2.2 When a video frame arrives, if the pose timeline buffer is insufficient to support interpolation (e.g., fewer than 4 poses in the buffer), a "warm-up flag" is generated and the initial delay is temporarily applied. ,default Otherwise, obtain the video latency estimate. The expression is: ; In the formula, For adaptive estimation of video channel latency, subscript It means Video; The video pipeline is fixedly offset relative to the telemetry pipeline by default. ; Figure 2 This graph illustrates the trend of end-to-end latency of the video and telemetry channels over time. It includes three curves: one representing the precisely measurable latency of the telemetry channel. Its fluctuations are quite drastic; a smooth curve is obtained by applying an exponential moving average to the telemetry delay. ; and the latency of the video channel located at the top, which has a larger overall latency. The core idea of ​​this diagram is to leverage the physical fact that video and telemetry data are transmitted in the same uplink and their latency variations are strongly correlated, by adding a "fixed bias" to the "exponential moving average of telemetry latency". "To estimate the latency of the video channel."

[0035] 2.3 Optionally, as a supplement to the above-described steps for calculating video delay estimation (steps 2.1 to 2.2), the following can be added: To improve overlay accuracy, fine-tuning is performed. Specifically, when the image texture is sufficient, a sequence of video frames within a sliding time window is selected, and its optical flow is calculated to obtain the visual motion amplitude sequence. Simultaneously, a velocity amplitude sequence from telemetry messages within the same time window is selected. Both are resampled to a fixed-time grid, mean-free, and then cross-correlation is performed. The time shift that best matches the two sequences is selected as a candidate for video delay fine-tuning. The expression is: ; In the formula, The optimal time shift (i.e., a candidate for video latency refinement) is used to maximize the cross-correlation value between the two sequences. The time-shifting independent variable slides within the search interval; This represents the index of discrete time points after resampling to a fixed-time raster. This is the lower limit of the cross-correlation time-shift search interval (default 0 seconds); This is the upper limit of the cross-correlation time-shift search interval (default 8 seconds); Visual motion amplitude sequence at grid points The value after removing the mean; Shift the velocity amplitude sequence of telemetry messages Then at the grid point The value after removing the mean; get Subsequently, an exponential moving average mechanism, similar to the telemetry self-delay moving average in step 2.1, was used for time series smoothing. A median outlier removal strategy was employed to filter out abnormal peaks, ultimately yielding... The refinement value.

[0036] The complexity of recalculating the optical flow cross-correlation refinement each time is: ,in The number of grid points within the time window has extremely low overhead after being amortized over the recalculation period.

[0037] 2.4, according to and The difference is used to determine the drone time corresponding to the video frame. The expression is: ; In the formula, This refers to the drone's time, used as a timeline query point for pose, where the subscript... This indicates a query. calculate The time complexity is .

[0038] The following formula determines the actual display time of the video frame. : ; In the formula, This is the actual displayed time; To reduce rendering buffer latency, the default... The larger the value, the faster it can wait for telemetry from both sides, and the more accurate the superposition, but the worse the real-time performance.

[0039] It should be noted that the time-delay sliding average learning rate With rendering buffer latency Calibration can be performed as follows: Under the target link (e.g., high-quality 4G, ordinary 4G, satellite), perform a parameter scan on a backhaul sample containing both maneuvering and stationary modes, under "real-time constraints (e.g., The operating point that minimizes the average superposition error is selected below the upper limit. Figure 4 This demonstrates the use of pose timeline buffering and reverse lookup interpolation. Figure 5 The actual display time is determined using a tunable rendering buffer, as shown below: exist Figure 4 In the diagram, multiple known pose points are marked on the horizontal line, sorted by the device's generation time. When the pose at a specific time is needed, the two adjacent known pose points are located, and then an interpolation algorithm is used to calculate the pose. Precise pose at any given moment.

[0040] exist Figure 5 In the graph, the horizontal axis represents rendering buffer latency, and the vertical axis represents error / latency. As rendering buffer latency increases, the video display delay also increases linearly, leading to a deterioration in real-time performance (shown by the dashed line). However, at the same time, there is more time to wait for telemetry data to arrive, resulting in more accurate pose interpolation and thus reducing stacking errors (shown by the solid line). This graph visually demonstrates that an engineering-acceptable balance between "real-time performance" and "stacking accuracy" can be found by adjusting the rendering buffer latency.

[0041] 2.5. Press in the pose timeline buffer Binary positioning of adjacent two poses ( , ),as well as( , The video frame's pose is estimated by interpolation, where the interpolation factor is expressed as: ; In the formula, The interpolation factor; In the pose timeline buffer. The device generation time of the adjacent pose on the left; In the pose timeline buffer. The device generation time of the adjacent pose on the right; Take the smallest positive number To prevent the denominator from being zero; This indicates taking the maximum value; For the yaw angle, since 0 / 360 crosses the boundary, it must be interpolated using the minimum arc, and the expression is: ; ; In the formula, This is the yaw angle difference (processed in minimum arcs to handle 0 / 360 degree crossovers). The yaw angle is the angle of the adjacent pose on the right. The yaw angle is the angle of the pose adjacent to the left. This is the interpolated yaw angle; For latitude, longitude, altitude, speed, and other components, interpolation is performed using the following expression: ; In the formula, This represents the component value to be estimated, i.e., the query time. The interpolation result for that component; This represents the same component value of adjacent poses on the left. This indicates the same component value of the adjacent pose on the right. in accordance with The time complexity of binary search for locating adjacent poses in the pose timeline buffer is O(n). Interpolation calculation itself involves constant number of component operations; It should be noted that during interpolation, if If the pose timeline is outside the buffer, it is clamped to the boundary and the extrapolation gap is recorded. If the gap exceeds the threshold, the head-up display (HUD) will indicate "pose extrapolation, accuracy degraded".

[0042] 2.6 As a preferred option, when the telemetry message contains a velocity component and the reporting frequency is low, the position component of the telemetry message is interpolated using cubic Hermite interpolation, i.e., using the position of two adjacent points... With speed (Converted from East / North velocity components) to calculate the interval length The interpolation expression is: ; In the formula, This is the position vector obtained after cubic Hermite interpolation; The position vector of the left adjacent pose; The position vector of the adjacent pose on the right; The time interval between two adjacent poses, i.e. ; The velocity vector of the adjacent pose on the right; The velocity vector is the one adjacent to the left pose.

[0043] 2.7. Preferably, time offset self-calibration is performed periodically or offline for verification. With caliber normalization, the expression is: ; In the formula, The optimal time offset that minimizes the sum of the distances between the telemetry pose and the reference pose; Indicates the time shift of the candidate to be searched; For the first in the reference pose sequence A timestamp; For high-frequency reference pose; The telemetry interpolated pose after time translation; The time point numbering for the reference pose sequence; The scan half-width is self-calibrated for time offset (default 3 seconds); Let the distance between two points be a function of the geodetic distance. Taking the Haversine formula, the expression is: ; In the formula, Represents the Earth's radius, taking ; The latitude of the first point; The latitude of the second point; The distance between two points on Earth; The longitude of the first point; The longitude of the second point; It should be noted that, The calibration can be performed as follows: During a field calibration flight, prepare a set of high-frequency pose reference data that is aligned frame by frame with the video (e.g., a frame-by-frame SRT format subtitle file exported from DJI Pilot, approximately 30Hz), and use this high-frequency pose reference data to solve for... The result That is Initial value; Next, the verification is performed using the known ground control points (GCPs) in the image, where the GCP coordinates are determined by RTK (accuracy). (According to actual measurements; during verification, GCP will be applied according to the current...) Project the image onto the corresponding video frame, calculate the residual between the projected point and the actual GCP pixel position, and select the value that minimizes the residual. .

[0044] The time offset self-calibration is an offline calculation of the number of scan points multiplied by the number of reference points, and is not in the real-time processing path.

[0045] 2.8. As a preferred option, when the fluctuation amplitude of the telemetry self-delay sliding average... (Right now The estimated standard deviation is so large that it approximates the displacement within a single sampling period (e.g., When ), the time delay estimation bias term This has become the dominant factor in the superposition error, and relying solely on time delay alignment is no longer sufficient to guarantee superposition accuracy, as shown in the following expression: ; ; In the formula, This represents the error in the ground superposition position; To improve the accuracy of telemetry positioning; for Estimate the standard deviation; This is the current ground speed of the drone; This serves as the interpolation error bound. For motor acceleration; This refers to the telemetry sampling interval; At this time, the rendering buffer latency will be automatically increased. Alternatively, the HUD may be downgraded to display a message indicating "network congestion, reduced overlay accuracy," and critical alarms may be switched to the telemetry direct reading path.

[0046] The computation time complexity of each of the above error terms is 1. .

[0047] Step 3: Unify the pipeline centerline and the hierarchical electronic fence to the target geodetic coordinate system, generate hierarchical buffer zones on the local metric plane, project back latitude and longitude, and project the geographic feature points as pixel coordinates on the video frame. The specific process for this step is as follows: 3.1. The generation of the first and third level electronic fence buffer zones of the multi-coordinate system is achieved through one-time preprocessing.

[0048] Furthermore, the pipeline KML / KMZ (Keyhole markup language and its compressed format) files are collected and normalized to WGS84 according to their declared coordinate system to obtain the pipeline centerline point list, the expression of which is: ; ; In the formula, Longitude in the WGS84 coordinate system; Longitude in the GCJ-02 coordinate system; Latitude in the GCJ-02 coordinate system; Latitude in WGS84 coordinate system; Indicates the inverse transform from GCJ-02 to WGS84; subscript Indicates the WGS84 coordinate system; subscript Indicates the GCJ-02 coordinate system; This represents the positive transform from WGS84 to GCJ-02; After obtaining the pipeline centerline point sequence, the coordinates are transformed to the metric plane using the preset representative point of the jurisdiction as the origin of the northeast-central coordinate system (ENU). The expression is as follows: ; ; ; In the formula, This represents the eastward offset of the target point relative to the origin. The longitude of the target point; The longitude of the ENU origin (camera position); The equatorial radius of the Earth's ellipsoid is taken as 6,378,137 meters. The latitude of the ENU origin (camera position); This represents the northward offset of the target point relative to the origin. The latitude of the target point; This represents the offset of the target point relative to the origin in the celestial direction. The absolute altitude of the target point; This represents the absolute altitude of the ENU origin (camera position); Generate in metric plane / / A three-level flat-top buffer polygon is generated and its latitude and longitude are returned. The buffer is a hierarchical fence ring, and the expression is: ; In the formula, For the first Graded and tiered fence rings; List the centerline points of the pipeline; For the first Level buffer radius, with example values ​​of 5, 10, and 15 (unit: meters); Number the fence level; This indicates that the latitude and longitude pipeline point list is converted into ENU plane coordinates; This indicates the generation of a flat-topped buffer polygon on the ENU metric plane; This indicates that the ENU plane coordinates are inversely mapped back to latitude and longitude; Figure 7 The process of generating tiered fence rings is demonstrated, specifically as follows: pipeline centerline data from different sources (such as GCJ-02 or WGS84) are uniformly converted to the target geodetic coordinate system (such as WGS84); the pipeline centerline after unifying the coordinate system is projected onto the local East-North-Sky (ENU) metric plane, and multiple flat-topped buffer polygons with increasing radii (e.g., 5 meters, 10 meters, 15 meters) are generated on this plane; the vertices of these buffer polygons generated on the metric plane are then converted back to latitude and longitude coordinates to form tiered fence rings that correspond one-to-one with each level of electronic fence.

[0049] The time complexity of one-time preprocessing is .

[0050] 3.2. Construct the camera intrinsic parameters using image resolution and horizontal field of view (prioritizing intrinsic parameters after calibration), expressed as: ; ; In the formula, Focal length value converted to pixel units for the equivalent physical focal length of the camera; The width of the video frame; This is the focal length scaling adjustment factor, default value. ; This is the horizontal field of view, the default value. ; The horizontal focal length is the camera intrinsic parameter matrix. The focal length in the vertical direction of the camera intrinsic parameter matrix; The x-coordinate of the camera principal point (the intersection of the optical axis and the image plane); The ordinate of the camera principal point (the intersection of the optical axis and the image plane); The height of the video frame; 3.3. Take the aligned pose frame by frame and construct the rotation matrix from the world coordinate system to the camera coordinate system. The expression is: ; ; In the formula, This is the rotation matrix from the camera coordinate system to the world (ENU) coordinate system; For the yaw rotation about the world's axis; For camera neutral base; For pitch and rotation about the camera's right axis; For roll rotation around the front axis of the camera; Indicates matrix transpose; This is the rotation matrix from the world coordinate system to the camera coordinate system; 3.4. Project the pipeline centerline points and the vertices of each level of fence rings into pixels in batches and take the visibility mask. The expression is: ; ; ; ; In the formula, In camera coordinate system coordinate; In camera coordinate system coordinate; In camera coordinate system coordinate; Indicates matrix transpose; The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. For visibility; Figure 6 This demonstrates the process of projecting geographic features onto drone video footage. The diagram includes a local metric coordinate system (East-North-Sky, ENU) with the drone as the origin, and the coordinates of a geographic point. First, the coordinates of the geographic point are transformed into this local metric coordinate system, and then a rotation matrix from the world coordinate system to the camera coordinate system is used. By combining the camera's intrinsic parameters, the pixel coordinates of the geographic point on the two-dimensional image plane are finally calculated. .

[0051] 3.5 As a preferred option, for large-scale pipeline KML files, the computational load would be large if all pipeline points were projected at once. Therefore, in order to improve efficiency, a local subset that falls into the intersection area and its vicinity is extracted from the complete pipeline centerline point list based on the ground range covered by the camera's field of view in the current video frame (i.e., the intersection area of ​​the view cone and the ground surface). Then, the projection calculation is performed on the local subset.

[0052] Comparing steps 3.4 and 3.5, it can be seen that step 3.4 performs batch projection on all pipeline centerline points and the vertices of each level of fence rings by default. The coordinate transformation (matrix multiplication) and pinhole projection (division) of each point are constant operations. Therefore, the time complexity of each frame of projection is: ; When the pipeline data is large, the view frustum clipping in step 3.5 can first be performed by extracting a local subset falling near the image from the complete pipeline point set based on the ground area covered by the current camera view. This reduces the time complexity of the actual projection calculation to: ; This effectively reduces the amount of computation per frame while maintaining projection accuracy.

[0053] Step 4: Based on the pose estimation, overlay the projected pipeline and the graded electronic fence onto the video frame, and determine the grade of the electronic fence based on the shortest distance from the projection point directly below the drone to the center line of the pipeline, so as to provide a graded prompt or alarm for the screen.

[0054] The specific process for this step is as follows: 4.1. Let the ground projection point of the current pose of the UAV be... (Taking camera latitude and longitude), the pipeline centerline is an ordered point sequence { After projecting each segment onto the local ENU metric plane, calculate the perpendicular from the point to the foot of the line segment (taking the distance from the endpoints outside the endpoints) and take the global minimum to obtain the shortest distance from the projection point directly below the UAV to the centerline of the pipeline. The expression is: ; In the formula, The shortest distance from the ground projection point directly below the drone to the centerline of the pipeline; This is the ground projection point of the UAV's current pose; For the pipeline centerline One point; For the pipeline centerline One point; The index number of the pipeline centerline point column; This represents the shortest distance from a point to a line segment; The time complexity of the above nearest distance calculation is: ; 4.2, according to Determining the classification of electronic fences The expression is: ; In the formula, Classify electronic fences; This indicates a Level 1 electronic fence; Indicates a level 2 electronic fence; This indicates a level 3 electronic fence; This indicates that it is outside the level three electronic fence; Figure 8 This demonstrates how to classify electronic fences based on the location of a drone. Specifically, the image shows three levels of electronic fences generated around the centerline of a pipeline. , , First, calculate the projection point directly below the drone. Shortest distance to the centerline of the pipeline and according to The area within which the drone lands is used to determine the zone level of the area it is currently in, for example, as shown in the image. It is 8 meters, which meets the requirements. Therefore, it is determined to be (Orange) area.

[0055] 4.3, From the lowest level (outermost) ) to the highest level (the innermost) The fence polygon is filled with a graded color from the outside in, then alpha blended, and the center line of the pipeline is stroked. A visibility mask is used to draw only the visible pixels. The expression is: ; In the formula, This represents the pixel values ​​of the superimposed image; Opacity (default) ); Represents the pixel values ​​of the original video frame; Color grading; The time complexity of outlining is ; 4.4 Overlay display on the current video frame , , HUD information such as gap and warm-up / degradation prompts, and complete video frames that have completed overlay rendering are displayed in order. The output is either displayed or encoded, where the time complexity of overlaying HUD information is O(n log n). .

[0056] Figure 9 This is a schematic of the final overlay rendering effect. The pipeline centerline and three levels of electronic fences (represented by different colored areas) are overlaid in real-time on the video footage captured by the drone. Simultaneously, a head-up display (HUD) is integrated in the upper left corner of the screen to present important real-time information, including... (orange), , And the gap is normal.

[0057] 4.5 As a preferred option, critical boundary crossing or hazard alarms are bypassed to a telemetry direct reading path that does not rely on video channel latency, and the problem is solved independently using high-frequency telemetry pose. And determine , When the threshold is exceeded, a graded alarm is generated, and as can be seen from the following formula, its accuracy is not affected by video alignment: ; In the formula, For the accuracy error of the telemetry direct reading path; To improve the accuracy of telemetry positioning; For motor acceleration; This is the telemetry sampling interval.

[0058] In the telemetry direct reading path, the time complexity of the nearest distance and classification determination is: .

[0059] The method in this embodiment, through the synergy of dual-channel time-delay adaptive alignment, pose timeline buffer interpolation, and tunable rendering buffer delay, controls the video geographic information overlay error within an acceptable engineering range without relying on image texture and prior truth. At the same time, by combining multi-coordinate system, hierarchical electronic fence projection rendering, and telemetry direct reading alarm bypass, it achieves robust real-time visualization and hierarchical early warning under real-world conditions such as weak texture, low-frequency telemetry, and time-varying delay, providing an engineering-deployable precision-real-time trade-off solution for UAV inspection of long-distance pipelines.

[0060] Example 2: This example is based on the method of Example 1, and further illustrates the present invention in conjunction with a specific application scenario.

[0061] I. Scenario Background: In a real-world scenario of drone inspection of a long-distance ground pipeline, the drone transmits 1080p video and telemetry pose data in real time via a high-quality 4G link. The telemetry pose data is aggregated at the airport and uploaded to the cloud, then processed by a message middleware. Forward.

[0062] II. Example parameters for this embodiment: The following parameter values ​​are merely examples, and those skilled in the art can adjust them according to the field link without departing from the essence of the present invention.

[0063] Latency Sliding Average Learning Rate Fixed offset delay seconds; rendering buffer Seconds; pose interpolation is taken as linear, and the motion segment is cut Hermite; camera intrinsic parameters are given by calibration results. (After checkerboard calibration and correction); fence color scheme red / orange / Green, Opacity The pipeline KML declares the coordinate system as WGS84, and a three-level electronic fence with a radius along the pipeline is set up. .

[0064] III. Specific steps: Step 1: Connect the video frame stream and telemetry stream separately: Record the monotonic moments of ground reception for each frame. Each telemetry record can observe network latency. (Actual measurement approximately) This section is a contract. The estimated standard deviation of jitter is approximately After being normalized to the device generation time, the pose timeline buffer is inserted. Step 2: For each telemetry message, update the telemetry self-delay sliding average. Seconds; when a certain video frame arrives, it is taken Find the drone's time frame corresponding to this frame. The frame is delayed. Showing until Telemetry messages arrive from both sides; press in the pose timeline buffer. The adjacent poses are located by binary search, and linear interpolation is performed (cubic Hermite interpolation with velocity constraints is used in the turning maneuver segment) to obtain the aligned pose of the frame.

[0065] After review, the average superposition error under this configuration is approximately If telemetry is reduced to Then rise to about ;exist For a segment, the cubic Hermite interpolation with velocity constraints relative to the linear interpolation for a stationary segment can be approximately... Further down to approximately Due to the inherent loss of motion in low-frequency velocity sampling and the limited Hermite gain during the maneuvering phase, it is necessary to apply the error bound formula ( , About time (When the motor is more intense, it is magnified many times over) This can be cured by increasing the frequency.

[0066] Step 3: Normalize the coordinate system of the pipeline KML. First, transform it to WGS84 according to its declared coordinate system. Then, generate the WGS84 pipeline centerline on the local ENU metric plane. Three-level flat-head buffer, return latitude and longitude buffer; It should be noted that, as a verification of the correctness of the coordinate system, the following results were obtained by comparing WGS84 and GCJ-02 dual rendering: Under WGS84, the pipeline and the three-level buffer band fall normally in the center of the image and fit perfectly throughout; under GCJ-02, the entire segment is translated. Since the screen is running, the coordinate system is configured as WGS84.

[0067] Frame by frame, the gimbal angle in the aligned pose is used ( (Top-down view, including yaw angle and fine-tuning), constructing the rotation matrix from the world coordinate system to the camera coordinate system. ,based on Project the pipeline points and the vertices of the third-level fence ring into pixels and take their visibility.

[0068] Step 4, Self Green to Fill the three-level buffer with red in sequence and blend with alpha at 0.3 opacity, then stroke the center line of the pipeline, and only draw the visible pixels; Find the shortest distance from the point directly below the drone's projection to the pipeline. And determine the level of the electronic fence, when When, satisfy Therefore, the classification is as follows: (Orange), at this time the HUD displays the following content in real time: = , , The gap is normal, and a video frame that has been overlaid and rendered is generated. The overlaid and rendered video frame is then delayed. Send display / encoding output; During the image overlay process, tiered alarms are generated in parallel. Specifically, critical alarms are bypassed to the telemetry direct reading path, utilizing... Telemetry pose is determined independently when the drone enters from directly below. ( When the alarm is triggered, a graded alarm is generated for "suspected entry into the core protection zone of the pipeline", along with a telemetry record timestamp and latitude and longitude tracing. The accuracy of this graded alarm is determined solely by the telemetry positioning itself (within meters) and does not include video latency, thus it is superior to image overlay.

[0069] IV. Comparative Benefits: In the entire inspection task covered by this embodiment, if the simple approach of "directly superimposing according to arrival time" is adopted, the end-to-end delay of the video channel is approximately Within seconds, the projected center line of the pipeline will deviate from its actual position by more than 20 meters from the electronic fence, and the warning area on the screen will not match the actual pipeline at all, thus losing its reference value.

[0070] After introducing the dual-channel latency correlation alignment strategy proposed in this invention, the video latency is estimated by telemetry self-latency moving average plus fixed bias adaptive estimation, and the rendering buffer waits for telemetry arrival from both sides and low-frequency pose interpolation to complete the image. The superposition error is stably controlled at about 3 meters, and the superimposed pipelines and the three-level protection zone can fit the real picture. At the same time, the critical boundary crossing and hidden danger alarms do not rely on this visualization path, but are classified and judged by high-frequency pose through an independent telemetry direct reading path. Its accuracy is determined only by the telemetry's own positioning accuracy and sampling interpolation error, without the video latency estimation deviation term. Therefore, the alarm accuracy is better than the image superposition and can be kept within the meter level. It realizes the division of labor and complementarity of "visualization using alignment path and alarm using direct reading path", which not only ensures the intuitiveness of the picture, but also maintains the accuracy of the warning.

[0071] The method in this embodiment demonstrates the entire process of an above-ground pipeline inspection in a specific application scenario, from dual-channel access, timestamp caliber normalization, dual-channel delay adaptive alignment, low-frequency pose interpolation completion, multi-coordinate system first and third-level fence projection, real-time overlay rendering to hierarchical alarm.

[0072] In summary, under the realistic conditions of weak public network links, low-frequency telemetry, and inaccurate timestamp standards, this invention effectively reduces the video geographic information overlay error by using an alignment structure that does not rely on image texture and prior truth values, and provides an engineering-tunable trade-off between accuracy and real-time performance.

[0073] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for overlaying UAV video geographic information based on dual-channel time delay alignment, characterized in that, Includes the following steps: Step 1: Access the video frame stream through the first channel and record the monotonic moments of ground reception for each frame; The telemetry message stream carrying the pose is accessed through the second channel. The ground reception monotonic time of each telemetry message is recorded, and its timestamp is normalized to the device generation time when the UAV actually generates the corresponding pose. The pose timeline buffer is maintained in ascending order of device generation time using the normalized pose as the sequence element. The observable network latency of the telemetry message is calculated based on the difference between the ground reception monotonic time of the telemetry message and the corresponding device generation time. Step 2: Based on the physical fact that the video frame stream and the telemetry message stream are transmitted through the same uplink and their delays are related, the video channel delay is adaptively estimated using the observable network delay. The video channel delay is subtracted from the ground reception monotonic moment of the video frame to obtain the UAV generation time corresponding to the frame; the UAV generation time is interpolated in the pose timeline buffer to obtain the pose estimate of the video frame. Step 3: Unify the pipeline centerline and the hierarchical electronic fence to the target geodetic coordinate system, generate hierarchical buffer zones on the local metric plane, project back latitude and longitude, and project the geographic feature points as pixel coordinates on the video frame. Step 4: Based on the pose estimation, overlay the projected pipeline and the graded electronic fence onto the video frame, and determine the grade of the electronic fence based on the shortest distance from the projection point directly below the drone to the center line of the pipeline, so as to provide a graded prompt or alarm for the screen.

2. The method according to claim 1, characterized in that, The telemetry message stream in step 1 is carried in a unified pose envelope containing a source type field. The source type includes one or more of the following: UAV model subtitles, airborne software development kits, cloud interfaces, or airport backend logs. The timestamp normalization in step 1 and the latency estimation in step 2 are both performed based on the device generation time and pose field in the unified pose envelope. Changes in the telemetry source do not change the alignment and interpolation process.

3. The method according to claim 1, characterized in that, The adaptive estimation of video channel delay in step 2 uses an exponential moving average, which is updated online recursively using the following formula: ； In the formula, For telemetry self-time delay smoothing; The time-delay sliding average learning rate; For observable network latency; The adaptive estimation of video channel latency is calculated as follows: ； In the formula, An adaptive estimate of the video channel latency; A fixed offset between the video pipeline and the telemetry pipeline; The fixed bias and the time-delay sliding average learning rate are determined before deployment by recalibrating the target backhaul link using on-site calibration flight samples combined with known ground control points. Based on the product of the estimated standard deviation of the video channel delay and the UAV ground speed, and the product of the UAV maneuvering acceleration and the square of the telemetry sampling interval, the superposition position error boundary is estimated, and a decision is made to increase the telemetry reporting frequency or increase the rendering buffer delay. When the number of poses in the pose timeline buffer is insufficient to support interpolation, a preheating flag is output and the initial delay value is temporarily used as the video channel delay.

4. The method according to claim 3, characterized in that, Step 2 also includes setting an adjustable rendering buffer delay, wherein the video frame is sent for display or encoded output at a time determined by the following formula: ； In the formula, This is the actual displayed time; To buffer rendering latency; For the monotonic moments of ground reception of video frames; By adjusting The size of the image frame is adjusted to delay the display of video frames while waiting for telemetry data to arrive on both sides of the frame at the corresponding time, thus striking a balance between real-time performance and overlay accuracy.

5. The method according to claim 1, characterized in that, The interpolation in the pose timeline buffer described in step 2 includes: when the telemetry message contains a velocity component and the telemetry reporting frequency is lower than a preset threshold, cubic Hermite interpolation with adjacent point velocity constraints is used; otherwise, linear interpolation is used. For the yaw angle component, 0° / 360° cross-boundary interpolation is handled in the minimum arc length manner.

6. The method according to claim 1, characterized in that, The unification mentioned in step 3 includes: when the declared coordinate system is a coordinate system with a second offset, converting it to the target geodetic coordinate system by numerical iterative inverse transformation; when the coordinate system of the pipeline centerline is not declared, rejecting superposition and generating an alarm.

7. The method according to claim 6, characterized in that, The generation method of the hierarchical buffer zone in step 3 is as follows: Project the unified and normalized points of the pipeline centerline onto the local East-North-Tian metric plane, generate multi-level flat-head buffer polygons with increasing radii on the pipeline centerline on this metric plane, and then project the vertices of each level of flat-head buffer polygons back to latitude and longitude coordinates to obtain hierarchical fence rings that correspond one-to-one with each level of electronic fence.

8. The method according to claim 1, characterized in that, Step 4 also includes bypassing critical boundary crossing or hidden danger alarms to a telemetry direct reading path that does not rely on the video channel latency. The hierarchical judgment is performed independently using high-frequency telemetry pose, and a hierarchical alarm is generated when a preset threshold is reached. The hierarchical judgment accuracy of the telemetry direct reading path is dominated by the telemetry's own positioning accuracy and the interpolation error under the telemetry sampling interval, and does not include the product of the video channel latency estimation deviation and the UAV ground speed. Therefore, the positional accuracy of the hierarchical alarm is better than the positioning accuracy of the superimposed rendering image.

9. The method according to claim 1 or 8, characterized in that, When the jitter of the observable network latency exceeds a preset threshold, the rendering buffer latency is automatically increased and / or a precision degradation prompt is output on the screen, while the hierarchical determination and alarm output of the telemetry direct reading path are kept uninterrupted.

10. The method according to claim 1, characterized in that, In step 2, when the video image has sufficient texture, the cross-correlation between the visual motion amplitude sequence and the telemetry velocity amplitude sequence is used to estimate video delay candidates, which are then fused into the adaptive estimation of the video channel delay after outlier elimination and moving average; when the texture is insufficient, this cross-correlation is not relied upon.

Images