Precise drawing fusion device and method based on real-time video of unmanned aerial vehicle

Abstract

The invention relates to an accurate drawing fusion device and method based on an unmanned aerial vehicle real-time video, and the device comprises a data acquisition module which is used for synchronously collecting an unmanned aerial vehicle load video stream, camera information and unmanned aerial vehicle positioning information; the positioning auxiliary module is used for performing feature matching on the real-time image of the unmanned aerial vehicle load video stream and a pre-stored map image, and generating final positioning information in combination with the unmanned aerial vehicle positioning information; the coordinate transformation and calibration module is used for mapping physical space information of a drawing to a video pixel space according to the final positioning information and the camera shooting information and performing dynamic calibration of a mapping relation to obtain drawing information after coordinate transformation and calibration; and the graphic rendering module is used for carrying out real-time superposition rendering on the drawing information subjected to coordinate transformation and calibration and the load video stream to obtain a rendered fusion video. Compared with the prior art, the method has the advantages of high data synchronism, accurate positioning, high fusion precision and the like.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) technology, and relates to a UAV image fusion method, and more particularly to a device and method for accurate fusion of drawings based on real-time UAV video. Background Technology

[0002] With the rapid development of drone technology, its application in various industries is becoming increasingly widespread, especially in the field of inspection, such as inspection of track protection zones, pipelines, and power lines. In these inspection applications, integrating real-time video captured by drones with relevant design drawings (such as CAD drawings) can provide operators with more intuitive and accurate information, assisting them in making business decisions. For example, patent application CN120510541A discloses an aerial data processing method for urban road modeling. This method uses a drone equipped with camera equipment to capture aerial video of the target road area according to a preset flight path. Based on road engineering design drawings, a 3D modeling software is used to construct an urban road BIM model including the main road structure, bridge structure, and ancillary facilities. The aerial video, BIM model, and camera motion trajectory data are imported into a 3D rendering platform. Based on the camera motion trajectory data, a real-time rendering engine is driven to perform real-time rendering operations, dynamically compositing the registered BIM model with the aerial video background frame by frame.

[0003] However, several problems exist in the current process of fusing UAV video and flight control monitoring data. Firstly, the asynchrony between UAV video and flight control monitoring data is quite common. Due to factors such as the operating frequency of the video acquisition module and the flight control monitoring data acquisition module, as well as data transmission delays, during UAV flight, the spatial location information corresponding to the video images deviates. For example, in the scenario of inspecting a railway protection zone, when the UAV flies to a certain location, the scene displayed in the video differs from the actual location indicated by the flight control monitoring data, leading to errors in the drawing information generated after the fusion of video and flight control data. Secondly, due to inaccurate spatial location, users cannot intuitively obtain effective business judgment criteria from the fused information. During road construction zone inspections, spatial location deviations in the video images cause the construction zone drawing information superimposed on the video to mismatch with the actual video image, making it difficult for users to determine whether the road construction zone meets the requirements. Similarly, in railway protection zone inspections, errors in the drawing information caused by the asynchrony between video and data lead to inaccurate judgments regarding whether personnel have intruded into the railway protection zone, potentially causing misjudgments of the actual situation within the protection zone and posing potential risks to railway safety.

[0004] Furthermore, most existing fusion technologies do not adequately consider situations in complex environments, such as when positioning information is weak or absent, like with BeiDou. Poor positioning signals severely impact the accuracy of UAV positioning information, consequently affecting the fusion of video and drawings. Moreover, the lack of an effective dynamic calibration mechanism when mapping CAD drawing information to video pixel space prevents the fused information from accurately adjusting to real-time changes in the UAV's flight attitude and position, significantly reducing the practicality of the fused information.

[0005] In summary, existing UAV video and drawing fusion technologies have significant shortcomings in terms of data synchronization, spatial positioning accuracy, and adaptability to complex environments, which severely restrict their application effectiveness and efficiency in business areas such as inspection and patrol, and urgently need to be improved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art by providing a device and method for precise fusion of drawings based on real-time video from UAVs, which features high data synchronization, accurate positioning, and high fusion precision.

[0007] The objective of this invention can be achieved through the following technical solutions: A device for precise fusion of drawings based on real-time video from drones, comprising: The data acquisition module is used to simultaneously collect video streams, camera information, and UAV positioning information from the UAV payload; The positioning assistance module is used to perform feature matching between the real-time image of the UAV payload video stream and the pre-stored map image, and generate the final positioning information by combining the UAV positioning information. The coordinate transformation and calibration module is used to map the physical space information of the drawing to the video pixel space based on the final positioning information and the camera information, and to perform dynamic calibration of the mapping relationship to obtain the drawing information after coordinate transformation and calibration. The graphics rendering module is used to overlay and render the coordinate-transformed and calibrated drawing information with the load video stream in real time to obtain a rendered and fused video.

[0008] Furthermore, the data acquisition module uses a high-precision clock synchronization circuit to synchronously acquire the UAV payload video stream, camera information, and UAV positioning information.

[0009] Furthermore, the positioning assistance module includes: The feature extraction unit is used to perform feature detection on the real-time image, obtain the current features, and generate the corresponding first feature descriptor; and to perform feature detection on the map image, obtain the map features, and generate the corresponding second feature descriptor. The feature matching engine is used to perform feature matching based on the current feature, the map feature, and the corresponding first feature descriptor and second feature descriptor to obtain matching point pairs; The positioning calculation unit is used to perform motion estimation and pose calculation based on the matching point pairs to generate final positioning information.

[0010] Furthermore, the positioning calculation unit is also used to evaluate the accuracy of the generated final positioning information. If the accuracy does not meet the standard, a repositioning mechanism is used to update the final positioning information.

[0011] Furthermore, the coordinate transformation and calibration module includes: The coordinate transformation unit is used to determine the transformation relationship between the drawing coordinate system and the video pixel coordinate system based on the final positioning information and camera information, and to map the physical space information of the drawing to the video pixel space based on the transformation relationship. The dynamic calibration engine responds to changes in UAV flight parameters or feature objects, generates error compensation parameters based on the changes, adjusts the mapping relationship between the physical space information of the drawing and the pixel space of the video, and feeds the error compensation parameters back to the positioning assistance module.

[0012] Furthermore, the UAV flight parameters include at least one or more of the following: flight attitude data and flight altitude data.

[0013] Furthermore, computer vision technology is used to identify and track the feature objects in order to obtain changes in the feature objects.

[0014] Furthermore, the error includes at least one or more of the following: time synchronization error, IMU zero bias error, lens distortion error, and mechanical vibration error.

[0015] Furthermore, the dynamic calibration engine is also used to dynamically calibrate the mapping relationship based on the rendering feedback signal from the graphics rendering module.

[0016] Furthermore, in the graphics rendering module, the corresponding rendering priority and transparency are set according to the layers and attributes of the drawing information.

[0017] This invention also provides a method for accurate fusion of drawings based on real-time video from unmanned aerial vehicles, comprising the following steps: Simultaneously collect video streams, camera information, and drone positioning information from the drone payload; Feature matching is performed between the real-time images of the UAV payload video stream and the pre-stored map images, and the final positioning information is generated by combining the UAV positioning information. Based on the final positioning information and camera information, the physical space information of the drawing is mapped to the video pixel space and the mapping relationship is dynamically calibrated to obtain the coordinate transformation and calibrated drawing information. The coordinate-transformed and calibrated drawing information is overlaid and rendered in real time with the real-time video stream to obtain a rendered and blended video.

[0018] The present invention also provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the drawing precision fusion method based on real-time video from a UAV as described above.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. The data acquisition module of this invention has a real-time synchronous acquisition mechanism for multi-source data, ensuring the synchronous collection of payload video stream, camera information and UAV positioning information, providing a reliable foundation for accurate fusion.

[0020] 2. The present invention is equipped with a positioning assistance module, which is used to perform feature matching between the real-time image of the UAV payload video stream and the pre-stored map image to generate positioning information. It uses image and map image feature matching technology to achieve assisted positioning in complex environments and ensures the accuracy of UAV positioning when the satellite positioning signal is poor.

[0021] 3. The coordinate transformation and calibration module of the present invention is based on a dynamic calibration method using coordinate transformation and computer vision technology. It can adjust the mapping of CAD drawings in the video pixel space in real time according to the UAV flight status, ensuring the accuracy of the fused information.

[0022] 4. The graphics rendering module of the present invention can efficiently realize the real-time overlay of drawing information and real-time video stream, and quickly and clearly integrate the calibrated CAD drawing information and video stream to present to the user.

[0023] Based on the aforementioned significant beneficial effects, this invention can demonstrate unique advantages in multiple inspection business scenarios: In terms of inspecting the track protection zone, this device enables a direct and accurate assessment of the situation within the zone. Previously, the lack of synchronization between video and flight control monitoring data made it difficult to accurately determine whether construction, foreign objects, or personnel intrusions existed within the protection zone. Now, by accurately fusing drone video with relevant map information in real time, operators can clearly see the overlaid map of the track protection zone on the video feed, directly observing any anomalies within the zone. For example, during actual inspections, when personnel or construction equipment enter the protection zone, based on the accurate fused information, operators can detect and respond immediately, greatly improving the safety of the track protection zone, reducing safety risks caused by misjudgments or omissions, and ensuring the normal operation of rail transit.

[0024] The device plays a crucial role in pipeline inspection and fault location. During industrial pipeline inspections, by automatically overlaying pipeline routes from CAD drawings and pressure sensor location information onto real-time video footage, workers can intuitively understand the overall pipeline layout and the location of key components. Furthermore, when leaks or corrosion areas are detected, the device can mark them in real time, thanks to precise coordinate transformation and calibration technology, ensuring that the marked locations accurately correspond to the actual video footage. Compared to traditional manual inspection and analysis methods, this significantly improves inspection efficiency and fault location accuracy. For example, in long-distance oil pipeline inspections, previously requiring substantial manpower and time to locate potential leaks, this device allows for rapid location of problem areas, enabling timely repair measures and reducing economic losses and environmental hazards caused by pipeline leaks.

[0025] This device also performs exceptionally well in line inspection and fault location scenarios. When using drones to photograph transmission lines, it accurately overlays CAD drawings of the transmission lines onto the video stream, automatically identifying issues such as tower misalignment and abnormal cable sag, while also marking deviations from the design drawings. This eliminates the need for inspection personnel to rely on complex measurements and comparisons to determine line condition; they can quickly identify problems simply by observing the fused video feed. For example, when inspecting transmission lines in mountainous or other terrain-challenged areas, traditional methods struggle to accurately capture the actual condition of towers and cables. This device provides intuitive visualizations, helping inspection personnel promptly identify potential safety hazards, enabling early maintenance and repairs, and ensuring the stability and reliability of power supply.

[0026] Overall, the device and method of the present invention solve key problems such as the asynchrony between UAV video and flight control monitoring data, and achieve high-precision drawing fusion. This provides efficient and accurate judgment basis for various inspection operations such as tracks, pipelines, and lines, and has significant value in improving inspection efficiency, reducing safety risks, and minimizing economic losses. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a detailed diagram of the data acquisition module in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the feature extraction process in an embodiment of the present invention. Figure 4 This is a flowchart of the feature matching process in an embodiment of the present invention; Figure 5 This is a flowchart of the positioning calculation in an embodiment of the present invention; Figure 6This is a flowchart of the relocation mechanism in an embodiment of the present invention; Figure 7 This is a diagram showing the coordinate system definition relationship in an embodiment of the present invention; Figure 8 This is a schematic diagram of the coordinate transformation process in an embodiment of the present invention; Figure 9 This is a schematic diagram of the adaptive calibration strategy in an embodiment of the present invention; Figure 10 This is a schematic diagram of closed-loop feedback in an embodiment of the present invention; Figure 11 This is a schematic diagram of the graphics rendering process that takes into account the information display priority in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0029] Example 1 This embodiment provides a device for precise fusion of drawings based on real-time video from a drone, such as... Figure 1 As shown, the system includes a data acquisition module 1, a positioning assistance module 2, a coordinate transformation and calibration module 3, and a graphics rendering module 4. The data acquisition module 1 synchronously collects raw data streams, data from multiple sensors, and image streams and sends them to the positioning assistance module 2. The positioning assistance module 2 performs assisted positioning based on the collected information and outputs position and attitude data in the relative coordinate system. The coordinate transformation and calibration module 3 generates calibrated coordinate data in the world coordinate system and transmits it to the graphics rendering module 4. It also feeds back the error compensation parameters from the calibration to the positioning assistance module 2. At the same time, the graphics rendering module 4 generates a rendering feedback signal and feeds it back to the coordinate transformation and calibration module 3.

[0030] The data acquisition module 1 is used to synchronously collect UAV payload video stream, camera information, and UAV positioning information; the positioning assistance module 2 is used to perform feature matching between the real-time image of the UAV payload video stream and the pre-stored map image, and generate the final positioning information by combining the UAV positioning information; the coordinate transformation and calibration module 3 is used to map the physical space information of the drawing to the video pixel space according to the final positioning information and camera information, and perform dynamic calibration of the mapping relationship; the graphics rendering module 4 is used to overlay and render the drawing information after coordinate transformation and calibration with the real-time video stream in real time.

[0031] The specific descriptions of each of the above modules are as follows.

[0032] 1. Data Acquisition Module Input: External sensors (IMU / GPS / camera, etc.) Output: Raw sensor data, image stream Flow direction: → Positioning auxiliary module In this embodiment, the data acquisition module is responsible for acquiring the video stream and camera information of the payload in real time, while simultaneously acquiring the UAV's positioning information. On the hardware side, a high-performance image sensor is used to acquire the video stream, ensuring that the video clarity and frame rate meet the requirements of practical applications. For example, an image sensor with a resolution of 4K and a frame rate of 60fps is selected, which can clearly capture detailed information in the inspection scene. For the acquisition of camera information, parameters such as the camera's focal length and angle of view are included, which are crucial for subsequent image analysis and coordinate transformation. Simultaneously, by integrating a high-precision satellite positioning module (such as BeiDou, GPS, etc.), the UAV's position information is acquired in real time, with positioning accuracy reaching the centimeter level. Furthermore, to ensure the synchronization of data acquisition, a high-precision clock synchronization circuit is set up inside the module. The clock signal triggers the video acquisition and positioning information acquisition operations, ensuring that the two are precisely aligned in time.

[0033] like Figure 2 As shown, the high-precision clock synchronization circuit is connected to the image sensor, satellite positioning module, and data aggregation unit. The high-precision clock synchronization circuit provides a 10MHz reference clock and 1PPS (pulses per second) synchronization pulse to ensure the time accuracy of the entire system, and simultaneously provides PTPv2 (Precision Time Protocol version 2) as a time reference to the data aggregation unit. The image sensor acquires image data and outputs it via MPI CSI-2 image streams, each with a nanosecond-level timestamp. The satellite positioning module obtains UTC time and GGA-format location data via the NMEA-0183 protocol and outputs RS-232-format location data, each also with a nanosecond-level timestamp. The data aggregation unit receives the image stream from the image sensor and the location data from the satellite positioning module, both with high-precision timestamps, performs time alignment and data fusion processing, and outputs the aggregated high-precision time-synchronized data for subsequent processing or storage.

[0034] The specific technical parameters of each component of the data acquisition module in this embodiment are as follows: 1) High-precision clock synchronization circuit Core components: TCXO temperature-controlled crystal oscillator (±0.1ppm) + FPGA timer Function: Generate a 10MHz master clock reference signal Distribute 1PPS (pulse per second) synchronization signal Supports PTPv2 (IEEE 1588) precision clock protocol connect: → Image sensor (provides exposure synchronization trigger) → Satellite module (time alignment) ← Receive GPS / BeiDou UTC time calibration 2) Image sensor Type: Global Shutter CMOS Key features: Supports externally triggered exposure MIPI CSI-2 high-speed interface (1.5Gbps / lane) On-chip HDR processing Data flow: ← Receive clock synchronization signal (accurate to ±50ns) → Output 1080p@60fps image stream with timestamps 3) Satellite positioning module Configuration: Multi-band GNSS receiver (GPS L1 / L5 + Beidou B1 / B2) Output data: NMEA-0183 Protocol Format GGA (Location / Altitude) + RMC (Time / Velocity) data frames Original pseudorange observations Synchronization mechanism: ← Receive 1PPS for local clock calibration → Output UTC time for system clock discipline 4) Data aggregation unit Core processor: ARM Cortex-M7 + Time-Sensitive Networking (TSN) switch Key features: Add nanosecond-level precise timestamps (IEEE 1588) Packet alignment (image + location + time triple) Implementing the LLDP delay compensation algorithm Output interface: → WiFi 802.11n / ac protocol → CAN FD auxiliary channel → 2.4GHz and 5.8GHz band radio signals, MAVLink and UART protocols 2. Positioning Auxiliary Module Processing: GPS / sensor fusion / feature point extraction Output: Device position and attitude data (relative coordinate system) Flow direction: → Coordinate transformation and calibration module Feedback received: ← Error compensation parameters (from calibration module) The positioning assistance module pre-stores detailed map image data and employs advanced image feature extraction algorithms, such as Scale Invariant Feature Transform (SIFT) or Accelerated Robust Feature Transform (SURF), to accurately calculate the UAV's position on the map. This serves as auxiliary positioning, effectively compensating for positioning losses when satellite positioning signals are weak. This module plays a crucial role when BeiDou or other positioning information is weak or absent.

[0035] In this embodiment, the positioning assistance module includes a feature extraction unit, a feature matching engine, and a positioning calculation unit. The feature extraction unit is used to perform feature detection on the real-time image to obtain current features and generate a corresponding first feature descriptor; and to perform feature detection on the map image to obtain map features and generate a corresponding second feature descriptor. The feature matching engine is used to perform feature matching based on the current features, the map features, and the corresponding first and second feature descriptors to obtain matching point pairs. The positioning calculation unit is used to perform motion estimation and pose calculation based on the matching point pairs to generate final positioning information.

[0036] 1) Feature extraction unit like Figure 3 The diagram shows the workflow of the feature extraction unit in this embodiment. After acquiring the input image, the unit performs preprocessing and feature detection to construct a scale space for the real-time image and map image captured by the UAV. It detects extreme points in the image at different scales as feature points and calculates the orientation and feature descriptors of these feature points.

[0037] In this embodiment, preprocessing includes grayscale conversion, histogram equalization, distortion correction, and Gaussian pyramid construction. Grayscale conversion reduces data dimensionality and computational complexity while preserving key structural and texture information; histogram equalization improves visual effects and recognizability; distortion correction restores the true shape and spatial relationships of objects, enhancing image geometric accuracy; and the application of Gaussian pyramids supports feature analysis at different resolutions, enhancing the algorithm's robustness to scale changes and achieving scale invariance.

[0038] In this embodiment, the feature detection steps include FAST corner detection, ORB (Oriented FAST and Rotated BRIEF) feature extraction, and edge feature extraction. The FAST corner detection algorithm can quickly identify corner regions in an image; the ORB feature extraction method further processes the detected corners. ORB not only inherits the high efficiency of FAST but also introduces orientation information and rotation invariance, enabling the generation of binary feature descriptors with strong discriminative power; edge detection operators such as Canny and Sobel are used to capture contours and structural boundaries in the image, enhancing the ability to express object shapes and topological relationships. These steps together constitute a multi-layered, complementary feature detection process, ensuring extraction efficiency while improving the stability and adaptability of feature extraction under different scales and viewpoints.

[0039] In this embodiment, in order to ensure the real-time performance of feature extraction, the following parameters are limited for the feature extraction unit: FAST detector response time <5ms; binary descriptor (BRISK / FREAK) computation is reduced by 80%.

[0040] 2) Feature Matching Engine like Figure 4 The feature matching process shown in this embodiment is as follows: the feature matching engine receives map features and current features, and obtains the final matching point pair through KNN search, bidirectional matching, distance ratio test, preliminary matching, RANSAC filtering, EPnP constraint, and refined matching.

[0041] Specifically, firstly, a KNN search (such as a Kd-tree nearest neighbor search) is performed on the current feature and map features to find the K most similar candidate matches in the map features for each current feature in the high-dimensional feature space, achieving rapid filtering. Next, bidirectional matching is performed, simultaneously verifying the consistency from the current feature to the map feature and from the map feature to the current feature, eliminating mismatches to eliminate ambiguity in one-sided matching. Then, a distance ratio test is conducted to compare the descriptor distance ratio between the best and second-best matches, filtering out unstable matching pairs with insufficient discriminative power to obtain preliminary matching results. Based on this, a combined optimization approach using RANSAC filtering (random sampling consensus algorithm) and EPnP constraints (Efficient Perspective-n-Point) is further employed to exclude matches that do not conform to projection constraints, improving the geometric consistency of the matching set. In addition, nonlinear optimization and other strategies are used to refine the matching results and perform motion consistency checks, further improving the matching accuracy and stability in dynamic scenes, thus outputting reliable final matching point pairs.

[0042] 3) Positioning Calculation Unit like Figure 5The diagram shows the workflow of the positioning calculation unit in this embodiment. The positioning calculation unit performs motion estimation and pose solving based on the matching point pairs to achieve 6-DOF output, including position coordinates and attitude information (such as attitude quaternions).

[0043] In this embodiment, the motion estimation process includes IMU pre-integration and Kalman filtering.

[0044] In this embodiment, the location solution process includes PnP problem solving, LM optimization, and ICP alignment.

[0045] Preferably, the positioning calculation unit also implements a repositioning mechanism, which returns to the feature extraction stage for adjustment based on the reason for positioning failure in order to achieve repositioning. For example... Figure 6 As shown, the corresponding relocation mechanism process is executed for different reasons for location failure: a) Insufficient features: Expand the search radius, increase the pyramid levels, and re-perform feature matching and localization; b) Dynamic occlusion: Enable urgent frames, global descriptor matching, and re-perform feature matching and localization; c) Illumination changes: Switch to SIFT descriptors and re-perform feature matching and localization.

[0046] 3. Coordinate Transformation and Calibration Module Key operations: Coordinate system transformation (device coordinate system → world coordinate system) Nonlinear error correction Timestamp synchronization Output: Calibrated, precise coordinates (world coordinate system) Flow path: → Graphics rendering module Feedback mechanism: ← Rendering feedback signal (used for dynamic calibration) → Error compensation parameters (to positioning module) The coordinate transformation and calibration module is responsible for mapping the physical spatial information of the CAD drawing to the video pixel space and performing dynamic calibration. First, based on the camera parameters and the UAV's position and attitude information, the transformation relationship between the CAD drawing coordinate system and the video pixel coordinate system is determined. This transformation is achieved through a series of rotation, translation, and scaling transformation matrices. In practice, the parameters of the transformation matrices are calculated using trigonometric functions and spatial geometry principles. Simultaneously, to achieve dynamic calibration, the module monitors the UAV's flight attitude sensor data (such as accelerometer and gyroscope data) and altitude sensor data in real time. When these data change, the calibration process is immediately triggered. For example, the rotation angle of the CAD drawing in the video is adjusted according to changes in flight attitude, and the scaling ratio of the CAD drawing is adjusted according to changes in altitude. Furthermore, computer vision technology is used to identify and track feature objects in the video. Different feature objects need to be identified and tracked in different scenarios; for example, in a track inspection scenario, track markers and other feature objects need to be identified. The mapping position of the CAD drawing is further calibrated based on changes in the position of these feature objects in the video, ensuring that the CAD drawing information and the video image always match precisely.

[0047] In this embodiment, the coordinate transformation and calibration module includes a coordinate transformation unit and a dynamic calibration engine. The coordinate transformation unit is used to determine the transformation relationship between the drawing coordinate system and the video pixel coordinate system based on the final positioning information and camera information, and to map the physical space information of the drawing to the video pixel space based on the transformation relationship. The dynamic calibration engine responds when the UAV flight parameters or feature objects change, and is used to generate error compensation parameters according to the changes, adjust the mapping relationship between the physical space information of the drawing and the video pixel space, and feed the error compensation parameters back to the positioning assistance module.

[0048] 1) Coordinate transformation unit like Figure 7 The diagram shown illustrates the coordinate system definition relationship used in this embodiment. The coordinate transformation unit transforms the coordinates from the body coordinate system to the world coordinate system by calculating the transformation matrix and performing a homogeneous transformation.

[0049] In this embodiment, the coordinate transformation process is as follows: Figure 8As shown, the process includes steps such as homogeneous transformation, height compensation, and Earth curvature correction. Homogeneous transformation establishes rigid body motion relationships between coordinate systems, using a 4×4 transformation matrix to simultaneously express rotation and translation, mapping points or vectors from one coordinate system to another, providing a unified geometric framework for subsequent processing. Height compensation corrects errors introduced by elevation differences, such as converting ellipsoid-based geodetic height to level surface-based orthographic height (altitude), or unifying different elevation benchmarks to improve vertical accuracy. Since the Earth is not flat, in large-scale or high-precision spatial measurements, the influence of Earth curvature and the reference ellipsoid model must be considered. Appropriate Earth models (such as EGM96 or WGS84) and algorithms are used to correct the horizontal and elevation coordinates for Earth curvature, reducing distortions and errors caused by the approximation of a plane on the surface. These steps systematically solve the geometric, elevation, and spherical effects problems in three-dimensional spatial coordinate transformation, jointly ensuring the accuracy and reliability of the final coordinate output, laying a solid data foundation for high-altitude applications.

[0050] 2) Dynamic calibration engine In this embodiment, the dynamic calibration engine is activated when the flight attitude sensor data (such as accelerometer and gyroscope data) and altitude sensor data of the UAV change. Its working process includes steps such as error source analysis and visual reprojection error calibration.

[0051] Error source analysis involves errors such as time synchronization error, IMU bias error, lens distortion error, and mechanical vibration error. Different calibration methods are used for different errors. For example, time synchronization error is calibrated using the PTP protocol, IMU bias is estimated using Kalman filtering, lens distortion is calibrated using a polynomial model, and mechanical vibration is suppressed through spectral analysis.

[0052] In this embodiment, the dynamic calibration engine uses flight parameters to assist in the calibration of coordinate data, and also performs adaptive calibration based on the flight status. Flight parameter-assisted calibration is shown in Table 1. The adaptive calibration strategy is as follows: Figure 9 As shown.

[0053] Table 1 In this embodiment, the output of the dynamic calibration engine includes the calibrated position (WGS84 latitude and longitude, accuracy 0.00001°), the calibrated attitude information (quaternion, accuracy 0.01°), the position covariance (3×3 matrix, centimeter level), and the timestamp (UTC nanosecond time).

[0054] In this embodiment, a feedback mechanism exists between the dynamic calibration engine, the positioning assistance module, and the graphics rendering model, such as... Figure 10As shown, the dynamic calibration engine performs dynamic calibration based on the pose data from the positioning assistance module and the rendering residuals during hovering fed back from the graphics rendering model. It then feeds back the obtained error compensation parameters (such as time synchronization error, IMU zero-bias error, lens distortion error, and / or mechanical vibration error) to the positioning assistance module. Simultaneously, the dynamic calibration engine sends IMU zero-bias calibration commands to the UAV, which then confirms whether the calibration is complete. Specifically, the dynamic calibration engine can employ existing rendering residual algorithms to achieve dynamic calibration.

[0055] 4. Graphics rendering module deal with: 3D engine rendering (OpenGL / Vulkan) Virtual and real fusion and superposition Projection transformation Output: Final visualization results (monitor / AR device) Feedback: → Calibration signal (to coordinate transformation and calibration module) The graphics rendering module employs advanced graphics rendering engines, such as OpenGL or DirectX, to overlay and render the coordinate-transformed and calibrated CAD drawing information onto the real-time video stream. During rendering, different rendering priorities and opacities are set according to different layers and attributes of the CAD drawing information. For example, important annotation information is given a higher rendering priority and opacity to ensure its clear visibility in the video. Simultaneously, to improve rendering efficiency, techniques such as texture mapping and occlusion culling are used to reduce unnecessary graphics calculations and rendering, quickly generating the merged video image, which is then output to display or storage devices for convenient real-time viewing or subsequent analysis.

[0056] In this embodiment, the graphics rendering process that considers information display priority is as follows: Figure 11 As shown, it specifically includes: Receive new rendering frames, perform conditional checks, and determine the rendering content and display method accordingly: Condition 1: Are there any alarm messages? Yes: The alarm layer will be rendered. This layer has the highest priority and will be forced to be displayed on top, covering all other interface elements; No: Proceed to the next condition.

[0057] Condition 2: Is there a navigation command? Yes: The navigation layer will be rendered, which will cover the underlying CAD drawing; No: The status layer will be rendered, which will not completely cover the underlying content, but will only cover some areas while preserving the visibility of the CAD drawing.

[0058] Underlying Background: In all cases, the bottom layer is always the underlying video stream, which is continuously displayed as the background image. The aforementioned alarm layer, navigation layer, or status layer are all overlaid on this video stream.

[0059] In this embodiment, the key rendering technologies employed by the graphics rendering module include CAD-video alignment technology, mixed reality rendering, and dynamic LOD (Level of Detail) control. The mixed reality rendering process is shown in Table 2. Dynamic LOD control adjusts rendering based on object distance and flight speed. For example, when the object distance is less than 5m, full-detail model rendering is used; when the distance is between 5-20m, a simplified outline rendering mode is used; and when the distance is greater than 20m, symbolic representation is used. If the flight speed is less than 2m / s, high-precision rendering is used; otherwise, a simplified rendering mode is used.

[0060] Table 2 The graphics rendering module also outputs rendering residuals to the coordinate transformation and calibration module for calibration. Rendering residuals refer to the difference between pre-calculated and real-time rendering values ​​used to optimize the final output effect.

[0061] Example 2 This embodiment provides a method for accurate fusion of drawings based on real-time video from a UAV, including the following steps: synchronously acquiring UAV payload video stream, camera information, and UAV positioning information; performing feature matching between the real-time image of the UAV payload video stream and a pre-stored map image, and generating final positioning information by combining the UAV positioning information; mapping the physical spatial information of the drawings to the video pixel space based on the final positioning information and camera information, and performing dynamic calibration of the mapping relationship; and overlaying and rendering the coordinate-transformed and calibrated drawing information with the real-time video stream in real time.

[0062] The rest is the same as in Example 1.

[0063] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A device for precise fusion of drawings based on real-time video from unmanned aerial vehicles (UAVs), characterized in that, include: The data acquisition module is used to simultaneously collect video streams, camera information, and UAV positioning information from the UAV payload; The positioning assistance module is used to perform feature matching between the real-time image of the UAV payload video stream and the pre-stored map image, and generate the final positioning information by combining the UAV positioning information. The coordinate transformation and calibration module is used to map the physical space information of the drawing to the video pixel space based on the final positioning information and the camera information, and to perform dynamic calibration of the mapping relationship to obtain the drawing information after coordinate transformation and calibration. The graphics rendering module is used to overlay and render the coordinate-transformed and calibrated drawing information with the load video stream in real time to obtain a rendered and fused video.

2. The drawing precision fusion device based on UAV real-time video according to claim 1, characterized in that, The data acquisition module uses a high-precision clock synchronization circuit to synchronously collect video streams, camera information, and UAV positioning information from the UAV payload.

3. The drawing precision fusion device based on UAV real-time video according to claim 1, characterized in that, The positioning assistance module includes: The feature extraction unit is used to perform feature detection on the real-time image, obtain the current features, and generate the corresponding first feature descriptor; and to perform feature detection on the map image, obtain the map features, and generate the corresponding second feature descriptor. The feature matching engine is used to perform feature matching based on the current feature, the map feature, and the corresponding first feature descriptor and second feature descriptor to obtain matching point pairs; The positioning calculation unit is used to perform motion estimation and pose calculation based on the matching point pairs to generate final positioning information.

4. The drawing precision fusion device based on UAV real-time video according to claim 3, characterized in that, The positioning calculation unit is also used to evaluate the accuracy of the generated final positioning information. If the accuracy does not meet the standard, a repositioning mechanism is used to update the final positioning information.

5. The drawing precision fusion device based on UAV real-time video according to claim 1, characterized in that, The coordinate transformation and calibration module includes: The coordinate transformation unit is used to determine the transformation relationship between the drawing coordinate system and the video pixel coordinate system based on the final positioning information and camera information, and to map the physical space information of the drawing to the video pixel space based on the transformation relationship. The dynamic calibration engine responds to changes in UAV flight parameters or feature objects, generates error compensation parameters based on the changes, adjusts the mapping relationship between the physical space information of the drawing and the pixel space of the video, and feeds the error compensation parameters back to the positioning assistance module.

6. The drawing precision fusion device based on UAV real-time video according to claim 5, characterized in that, The UAV flight parameters include at least one or more of the following: flight attitude data and flight altitude data.

7. The drawing precision fusion device based on UAV real-time video according to claim 5, characterized in that, Computer vision technology is used to identify and track feature objects in order to obtain changes in the feature objects.

8. The drawing precision fusion device based on UAV real-time video according to claim 5, characterized in that, The error includes at least one or more of the following: time synchronization error, IMU zero bias error, lens distortion error, and mechanical vibration error.

9. The drawing precision fusion device based on UAV real-time video according to claim 5, characterized in that, The dynamic calibration engine is also used to dynamically calibrate the mapping relationship based on the rendering feedback signal from the graphics rendering module.

10. The drawing precision fusion device based on UAV real-time video according to claim 1, characterized in that, In the graphics rendering module, the corresponding rendering priority and transparency are set according to the layers and attributes of the drawing information.

11. A method for accurate fusion of drawings based on real-time video from unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Simultaneously collect video streams, camera information, and drone positioning information from the drone payload; Feature matching is performed between the real-time images of the UAV payload video stream and the pre-stored map images, and the final positioning information is generated by combining the UAV positioning information. Based on the final positioning information and camera information, the physical space information of the drawing is mapped to the video pixel space and the mapping relationship is dynamically calibrated to obtain the coordinate transformation and calibrated drawing information. The coordinate-transformed and calibrated drawing information is overlaid and rendered in real time with the real-time video stream to obtain a rendered and blended video.

12. A computer-readable storage medium, characterized in that, Includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the drawing precision fusion method based on real-time video from a drone as described in claim 11.

Images