Unmanned aerial vehicle inspection route mapping method based on 3D Gaussian point cloud

Abstract

The invention provides an unmanned aerial vehicle routing inspection route mapping method based on 3D Gaussian point cloud. The method comprises the following steps: planning a routing inspection route of an unmanned aerial vehicle; the inspection route of the unmanned aerial vehicle is mapped and bound to a 3DGS digital twinborn model of the transformer substation; acquiring a spatial position and an airborne video stream of the unmanned aerial vehicle in the inspection process according to the inspection route, and transmitting the spatial position information of the unmanned aerial vehicle and the airborne video stream of the unmanned aerial vehicle in real time; and receiving the spatial position information of the unmanned aerial vehicle and the airborne video stream of the unmanned aerial vehicle, and superposing and rendering the airborne video stream and the spatial position of the unmanned aerial vehicle into the 3DGS digital twin model of the transformer substation. According to the invention, the inspection condition can be fed back in real time, the integration efficiency of the airborne video stream and the three-dimensional model of the unmanned aerial vehicle is improved, delay and errors are reduced, and accurate linkage of the visual angle of the unmanned aerial vehicle and the digital model is realized.

Description

A Method for Mapping UAV Inspection Routes Based on 3D Gaussian Point Clouds Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection technology, and in particular to a UAV inspection route mapping method based on 3D Gaussian point clouds. Background Technology

[0002] With the continuous increase in my country's electricity transmission demand and the ongoing expansion of the power system, the safety and stability of the power system have become increasingly prominent issues. This places higher demands on the operational reliability of power equipment in substations, making inspection a crucial link in ensuring the safe and stable operation of power equipment. With the development of drone technology and deep learning, traditional manual inspection methods are gradually being replaced by integrated and intelligent drone inspections, which are increasingly being applied to substations to monitor them and ensure their operation.

[0003] Currently, when using drones for substation inspections, the onboard video streams collected by the drones typically require offline analysis after the flight, making real-time feedback on the inspection status impossible. Furthermore, the integration of the drone's onboard video streams with 3D models often suffers from significant delays and errors, hindering accurate linkage and preventing maintenance personnel from accurately assessing the status of power equipment. Therefore, a drone inspection flight path mapping method is urgently needed to address these technical issues. Summary of the Invention

[0004] The purpose of this invention is to provide a method for mapping UAV inspection routes based on 3D Gaussian point clouds, which can provide real-time feedback on inspection status, improve the efficiency of integrating UAV onboard video streams with 3D models, reduce latency and errors, and achieve precise linkage between the UAV perspective and the digital model, enabling more accurate real-time linkage between the UAV perspective and the digital model.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for mapping UAV inspection routes based on 3D Gaussian point clouds, comprising: planning the inspection route of the UAV; mapping and binding the inspection route of the UAV to the 3DGS digital twin model of the substation; acquiring the spatial position and onboard video stream of the UAV during the inspection process according to the inspection route in real time, and transmitting the UAV spatial position information and UAV onboard video stream in real time; receiving the UAV spatial position information and UAV onboard video stream, and overlaying and rendering the onboard video stream combined with the UAV spatial position onto the 3DGS digital twin model of the substation.

[0006] Preferably, mapping and binding the UAV inspection route to the substation's 3DGS digital twin model includes: standardizing and regulating the UAV inspection route, performing coordinate system consistency analysis and judgment, and obtaining the analysis and judgment results; performing coordinate system transformation on the inspection route based on the analysis and judgment results to obtain the inspection route in the target coordinate system; using a point cloud matching algorithm to match the waypoints in the inspection route in the target coordinate system with the substation's 3DGS digital twin model, and aligning based on the matching results; obtaining the inspection parameters of the waypoints in the inspection route in the target coordinate system, and binding the data to the corresponding points in the substation's 3DGS digital twin model based on the inspection parameters according to the matching results.

[0007] Preferably, real-time transmission of UAV spatial location information includes: acquiring the communication method of the data receiving module; performing data feature analysis on the UAV spatial location information to determine the data features of the UAV spatial location information; performing communication matching based on the communication information of the data receiving module and the data features of the UAV spatial location information; and transmitting the UAV spatial location information to the data receiving module based on the communication matching result.

[0008] Preferably, real-time transmission of the UAV-borne video stream includes: configuring the transmission protocol for the UAV-borne video stream and determining the target transmission protocol for the UAV-borne video stream; establishing a data stream communication connection between the transmission module and the data receiving module according to the target transmission protocol of the UAV-borne video stream; and transmitting the UAV-borne video stream to the data receiving module in real time based on the data stream communication connection.

[0009] Preferably, the airborne video stream is overlaid and rendered in the 3DGS digital twin model of the substation, combining the airborne video stream with the spatial position of the drone. This includes: establishing a drone inspection model for drone inspection; driving the drone inspection model based on real-time data of the airborne video stream and the drone's spatial position to obtain a real-time view of drone inspection; obtaining pose data information based on the drone's spatial position in the 3DGS digital twin model of the substation; spatially embedding the pose data information into the real-time view of drone inspection to obtain an inspection monitoring view; and displaying the inspection monitoring view in real time.

[0010] Preferably, the drone inspection model is driven by real-time data of the airborne video stream and the drone's spatial position, including: performing a corresponding match between the airborne video stream and the drone's spatial position to obtain a first matching result; determining the current video acquisition information and the drone's current spatial position based on the first matching result; optimizing the current video acquisition information to obtain optimized current video acquisition information; and updating the drone inspection model based on the optimized current video acquisition information and the drone's current spatial position to generate a real-time drone inspection view.

[0011] Preferably, the spatial embedding of the real-time UAV inspection view based on the overlay rendering position and pose data information includes: constructing the UAV's view frustum by combining the pose data information with the field of view and focal length; determining the projection range of the UAV video in the 3DGS rendering space based on the view frustum; and fusing the rendering results of the UAV inspection real-time view with the 3DGS digital twin model based on the projection range to obtain the inspection monitoring view.

[0012] Preferably, before displaying the inspection monitoring view in real time, a fusion effect test is performed on the inspection monitoring view, including: analyzing the image parameters of the inspection monitoring view and determining whether the inspection monitoring view needs color adjustment, and obtaining the test analysis results; adjusting the color of the inspection monitoring view according to the test analysis results; and performing light and shadow rendering on the color-adjusted inspection monitoring view to obtain the optimized inspection monitoring view.

[0013] Preferably, when displaying the inspection monitoring view in real time, the drone's trajectory is also marked in the inspection monitoring view according to its spatial position.

[0014] Preferably, the inspection and collection information is also labeled in the flight track, including: performing a correlation analysis of UAV inspection waypoints for power equipment in the substation based on the flight track to determine the associated UAV inspection waypoints for the power equipment; retrieving the airborne video stream collection information obtained by the UAV based on the corresponding cruise point during the inspection of the UAV according to the associated UAV inspection waypoints to obtain the target inspection and collection information of the power equipment; performing an overall inspection analysis of the power equipment based on the target inspection and collection information of the power equipment to determine the overall inspection and labeling data of the power equipment; determining the labeling position of the power equipment in the flight track, and labeling the overall inspection and labeling data of the power equipment at the labeling position.

[0015] This invention provides real-time feedback on inspection status and effectively improves the efficiency of integrating the onboard video stream of the UAV with the 3D model, reducing latency and errors, and enabling more accurate real-time linkage between the UAV's perspective and the digital model. By using a UAV inspection flight path mapping method based on 3D Gaussian point clouds to integrate UAV inspection with the 3DGS digital twin model, it not only presents the onboard video stream obtained by the UAV in a timely manner and provides real-time feedback on the inspection status, but also achieves real-time linkage between the UAV's perspective and the digital model. This allows maintenance personnel to intuitively monitor the UAV's real-time flight status and observe the specific position of its captured images within the overall model in a realistic 3D digital environment, facilitating the analysis and judgment of the power equipment's status. It also provides holographic and spatial monitoring and decision support for unmanned autonomous inspections in complex substation environments.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the application.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the present invention and form part of the specification. They are used together with the embodiments of the present invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 is a schematic diagram of the steps of the UAV inspection route mapping method of the present invention; Figure 2 is a schematic diagram of step two in the UAV inspection route mapping method of the present invention; Figure 3 is a schematic diagram of step four in the UAV inspection route mapping method of the present invention. Detailed Implementation

[0019] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] As shown in Figure 1, this embodiment of the invention provides a method for mapping UAV inspection routes based on 3D Gaussian point clouds, including: Step 1, planning the inspection route of the UAV.

[0021] The inspection route of the UAV is the autonomous inspection flight of the UAV, which includes waypoints, inspection altitude, flight speed, and shooting parameters. When planning the inspection route of the UAV, the waypoints are analyzed in combination with the layout of the substation and the inspection requirements. The shooting parameters are determined based on the waypoints and the UAV's own parameter information. It is also determined whether the UAV will collide with the power equipment in the substation. Based on the judgment result, the waypoints are adjusted and optimized to obtain the optimized waypoints. Then, the inspection route is determined based on the optimized waypoints. At the same time, the inspection altitude and flight speed are determined for the optimized waypoints to obtain the inspection route of the UAV.

[0022] Step 2: Map and bind the inspection route of the drone to the 3DGS digital twin model of the substation.

[0023] When mapping and binding the inspection flight path of the UAV to the 3DGS digital twin model of the substation, a mapping relationship is established between the UAV inspection flight path and the 3DGS digital twin model of the substation. The mapping and binding of the UAV inspection flight path to the 3DGS digital twin model of the substation is then performed according to this mapping relationship. The 3DGS digital twin model is a 3D Gaussian point cloud model, which refers to a digital twin model of the substation built based on 3D Gaussian point cloud technology, used to achieve high-precision mapping of the UAV inspection flight path.

[0024] Step 3: Acquire the spatial position and onboard video stream of the UAV during the inspection process along the inspection route in real time, and transmit the UAV spatial position information and onboard video stream in real time.

[0025] The drone conducts inspections within the substation according to a planned inspection route, acquiring its spatial position and onboard video stream in real time during the inspection. The drone comprises an RTK module, a data acquisition module, a transmission module, and a control module. The control module guides the drone along the inspection route, while the RTK module acquires the drone's location information in real time, providing its spatial position. The data acquisition module captures video data during the inspection, generating the onboard video stream. The transmission module then transmits both the drone's spatial position information and the onboard video stream in real time. The RTK module utilizes a high-precision satellite positioning system, combined with GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit) sensors, to calculate the drone's precise position in real time. The data acquisition module, typically an image acquisition device such as a camera mounted on the drone, encodes and compresses the video data acquired during the inspection to produce the onboard video stream. The transmission module performs real-time transmission of UAV spatial location information and UAV onboard video stream, including real-time transmission of human-machine spatial location information and real-time transmission of UAV onboard video stream.

[0026] Step 4: Receive the UAV's spatial location information and onboard video stream, and overlay and render the onboard video stream and the UAV's spatial location onto the substation's 3DGS digital twin model.

[0027] The server that overlays and renders the airborne video stream and the drone's spatial location onto the 3DGS digital twin model of the substation is equipped with a data receiving module. This module receives the real-time transmitted drone spatial location information and the drone's airborne video stream, and then overlays and renders the airborne video stream and the drone's spatial location onto the corresponding position in the 3DGS digital twin model of the substation. This allows the 3DGS digital twin model of the substation to simultaneously present a holographic monitoring view of the drone inspection, achieving real-time linkage between the drone's perspective and the digital model.

[0028] The above-mentioned method of integrating UAV inspection with 3DGS digital twin model by mapping UAV inspection routes based on 3D Gaussian point clouds not only enables timely presentation of onboard video streams obtained by UAVs and real-time feedback on inspection status, but also achieves real-time linkage between the UAV's perspective and the digital model. This allows maintenance personnel to intuitively monitor the real-time flight status of UAVs and observe the specific position of their captured images in the overall model within a realistic 3D digital environment, facilitating the analysis and judgment of the status of power equipment. It also provides holographic and spatial monitoring and decision support for unmanned autonomous inspection in complex substation environments.

[0029] Furthermore, the RTK module in the drone enables centimeter-level high-precision positioning, improving the accuracy of the drone's location information. Simultaneously, the acquisition module collects data on the substation's condition during inspections, providing a clear picture of the substation's real-time status. Encoding and compression of the video data effectively reduces data volume, facilitating transmission and improving efficiency. The transmission module can simultaneously transmit the drone's spatial location information and onboard video stream in real time, allowing for timely feedback during inspections and improving the timeliness of feedback. This enables maintenance personnel to promptly assess the condition of power equipment. It also facilitates the integration of the drone's onboard video stream with the 3DGS digital twin model, allowing maintenance personnel to accurately pinpoint anomalies by analyzing the information in the onboard video stream and considering the overall substation situation, providing greater convenience for maintenance staff.

[0030] In one embodiment of the present invention, as shown in Figure 2, the inspection route of the UAV is mapped and bound to the 3DGS digital twin model of the substation, including: A1, standardizing and regulating the inspection route of the UAV, and performing coordinate system consistency analysis and judgment to obtain the analysis and judgment results.

[0031] When standardizing the inspection routes of drones, it is determined whether the format of the drone inspection routes is a standard format. If it is, no format conversion is needed; otherwise, the drone inspection routes are converted to a standard format. Here, a standard format is a unified data format for easy information interpretation, such as KML or CSV. When performing coordinate system consistency analysis, it is analyzed whether the coordinate system used by the inspection routes is the same as the coordinate system used by the 3DGS digital twin model of the substation, thus obtaining the analysis and judgment results.

[0032] A2. Based on the analysis and judgment results, perform coordinate system transformation on the inspection route to obtain the inspection route in the target coordinate system.

[0033] Specifically, when the analysis and judgment result is that the coordinate system used by the inspection route is the same as the coordinate system used by the 3DGS digital twin model of the substation, there is no need to perform coordinate system transformation for the inspection route. When the analysis and judgment result is that the coordinate system used by the inspection route is different from the coordinate system used by the 3DGS digital twin model of the substation, coordinate system transformation is performed for the inspection route to convert the data information in the inspection route to the coordinate system used by the 3DGS digital twin model of the substation, thereby obtaining the inspection route in the target coordinate system.

[0034] A3. A point cloud matching algorithm is used to match waypoints in the inspection route under the target coordinate system in the 3DGS digital twin model of the substation, and alignment is performed based on the matching results.

[0035] The point cloud matching algorithm can be ICP-Iterative Closest Point or other matching algorithms.

[0036] A4. Obtain the inspection parameters of the waypoints in the inspection route under the target coordinate system, and bind the data to the corresponding points in the 3DGS digital twin model of the substation according to the inspection parameters based on the matching results.

[0037] Among them, inspection parameters refer to data information such as inspection altitude, flight speed, and shooting parameters in the inspection route. When binding data boards to corresponding points in the 3DGS digital twin model of the substation based on the matching results, the inspection parameters are bound to the corresponding spatial coordinates or Gaussian point attributes in the 3DGS digital twin model of the substation, forming a data association mapping between the inspection route and the 3DGS model.

[0038] The standardization and specification of UAV inspection routes, as described above, allows for better interpretation of inspection routes determined through different methods, facilitating the extraction of required information from these routes. Furthermore, by analyzing whether the coordinate system used in the inspection route is the same as that used in the 3DGS digital twin model of the substation, matching and binding can be performed based on information within the same coordinate system. This ensures accurate alignment between the inspection route and the 3DGS digital twin model of the substation, avoiding discrepancies caused by inconsistencies in coordinate systems. Moreover, by binding data to corresponding points in the 3DGS digital twin model of the substation based on the matching results and inspection parameters, these points possess the data attributes of waypoints within the inspection route. This ensures that the binding information of waypoints is consistent with the geometry and topology of the model, achieving high-precision spatial binding between the inspection route and the 3DGS digital twin model.

[0039] In one embodiment of the present invention, real-time transmission of UAV spatial location information includes: acquiring the communication method of the data receiving module.

[0040] The data receiving module refers to the module used to receive the spatial location information of the drone when the airborne video stream is combined with the spatial location of the drone and rendered onto the 3DGS digital twin model of the substation. Its communication methods include: 4G / 5G network communication, Wi-Fi communication, dedicated radio frequency communication, etc.

[0041] Data feature analysis is performed on the spatial location information of UAVs to determine the data characteristics of UAV spatial location information.

[0042] When performing data feature analysis on the location information of drones, it is necessary to clarify the composition of the location information, determine whether the location information is pure text information, pure digital information, or a mixture of text and digital information, and thus obtain the data features of the location information.

[0043] Communication matching is performed based on the communication information from the data receiving module and the data characteristics of the UAV's spatial location information.

[0044] When performing communication matching based on the communication method of the data receiving module and the data characteristics of the location information, firstly, the communication method of the transmission module for transmitting the UAV's spatial location information is determined based on the communication method of the data receiving module, thus determining the target communication method. Then, the communication protocol is configured based on the target communication method and the data characteristics of the UAV's spatial location information, thereby determining the target transmission protocol.

[0045] Based on the communication matching results, the UAV's spatial location information is transmitted to the data receiving module.

[0046] Specifically, when transmitting the UAV spatial location information to the data receiving module based on the communication matching result, the UAV spatial location information is processed according to the target transmission protocol, and the processed UAV spatial location information is transmitted to the data receiving module in real time according to the target communication method.

[0047] The above-mentioned implementation enables real-time transmission of UAV spatial location information, providing timely feedback on the UAV's position during inspections. Furthermore, the transmission of UAV spatial location information takes into full account the communication methods of the data receiving module, ensuring that the communication methods of the transmission and receiving modules are identical. This guarantees the effectiveness of data transmission and avoids any abnormalities in UAV spatial location information transmission that could affect real-time overlay rendering in the 3DGS digital twin model of the substation. Moreover, by performing data feature analysis on the UAV spatial location information and combining this data feature with communication matching, the integrity and security of the UAV spatial location information during transmission are guaranteed, ensuring the accuracy of the UAV spatial location information in the data receiving module.

[0048] In one embodiment of the present invention, real-time transmission of an onboard video stream from a drone includes: configuring a transmission protocol for the onboard video stream from the drone and determining the target transmission protocol for the onboard video stream from the drone.

[0049] When configuring the transmission protocol for drone-borne video streams, real-time transmission is typically achieved using streaming media technology, such as the RTMP protocol, or other protocols capable of data stream transmission.

[0050] Establish a data stream communication connection between the transmission module and the data receiving module based on the target transmission protocol of the UAV's onboard video stream.

[0051] The transmission module is the transmission module within the UAV. A low-latency data stream communication connection is established between the transmission module and the data receiving module according to the target transmission protocol of the UAV's onboard video stream. A lightweight handshake mechanism is employed to complete protocol negotiation and channel verification during the transmission initialization phase, ensuring connection reliability while avoiding additional latency introduced by multiple round-trip communications. During video stream transmission, a timestamp-based frame order management and priority scheduling strategy is used to further guarantee real-time performance and smoothness.

[0052] The UAV's onboard video stream is transmitted to the data receiving module in real time via a data stream communication connection.

[0053] Specifically, when transmitting the UAV onboard video stream to the data receiving module in real time based on the data stream communication connection, the UAV onboard video stream is transmitted to the target data receiving server in the data receiving module in real time according to the data stream communication connection. Moreover, when transmitting the UAV onboard video stream in real time, the UAV onboard video stream is encapsulated into messages to be transmitted frame by frame according to the target transmission protocol. The messages to be transmitted are weighted and queued to form a queue to be transmitted, and then the messages to be transmitted are transmitted sequentially according to the queue to be transmitted.

[0054] The aforementioned implementation of real-time transmission of UAV-borne video streams enables timely display of these streams within the substation's 3DGS digital twin model. This facilitates real-time feedback on inspection progress, allowing maintenance personnel to promptly analyze and assess the status of power equipment, thus simplifying substation management and maintenance. Furthermore, by configuring the transmission protocol for the UAV-borne video streams, the transmission module ensures efficient and accurate transmission, guaranteeing the security of the video streams.

[0055] In one embodiment of the present invention, as shown in Figure 3, the airborne video stream is combined with the spatial position of the UAV and overlaid and rendered into the 3DGS digital twin model of the substation, including: B1, establishing a UAV inspection model for UAV inspection.

[0056] Among them, the UAV inspection model is a full-element static scene model. When establishing the UAV inspection model, a visual image model is established based on the information such as the perspective collected by the UAV during the inspection process according to the inspection route, thereby obtaining the UAV inspection model.

[0057] B2. Drive the UAV inspection model based on the real-time data of the airborne video stream and the UAV's spatial location to obtain a real-time view of the UAV inspection.

[0058] The airborne video stream and the spatial position of the drone change in real time during the inspection process of the drone following the inspection route. Therefore, the drone inspection model is driven to form a real-time updated digital twin based on the real-time data of the airborne video stream and the spatial position of the drone, thus determining the real-time view of the drone inspection.

[0059] B3. Obtain pose data information based on the spatial location of the UAV in the 3DGS digital twin model of the substation.

[0060] The pose data includes position information and attitude information. When acquiring pose data based on the UAV's spatial location in the 3DGS digital twin model of the substation, the UAV's spatial location is optimized and then analyzed to obtain its position and attitude information. Here, optimization typically refers to signal optimization, including filtering and noise reduction, to ensure the accuracy of the pose information.

[0061] B4. Spatially embed the real-time view of the UAV inspection by combining the pose data information to obtain the inspection monitoring view.

[0062] In particular, when spatially embedding the real-time view of UAV inspection by combining pose data information, the holographic monitoring view is dynamically superimposed and rendered onto the corresponding surface position in the 3DGS digital twin model of the substation, thereby obtaining the inspection monitoring view.

[0063] B5. Real-time display of the inspection monitoring view.

[0064] When displaying the inspection monitoring view in real time, the inspection monitoring view is visualized through display devices, including VR / AR devices, large-screen displays, etc.

[0065] The above-described system enables real-time linkage between the drone's perspective and the digital model, allowing maintenance personnel to intuitively monitor the drone's real-time flight status and observe the specific position of its captured images within the overall model in a realistic 3D digital environment. This facilitates the analysis and judgment of the power equipment's status and provides holographic and spatial monitoring and decision support for unmanned autonomous inspections in complex substation environments. Furthermore, by establishing a drone inspection model based on the airborne video stream and the drone's spatial position, the system facilitates the acquisition of drone inspection views, enabling rapid generation of drone inspection views when the airborne video stream and drone's spatial position are available in real time, thus improving the efficiency of determining real-time drone inspection views. By spatially embedding the drone inspection real-time view with pose data, the images obtained from the drone inspection are directly overlaid on the substation's 3DGS digital twin model, achieving a WYSIWYG holographic monitoring view that presents the drone inspection footage more intuitively.

[0066] In one embodiment of the present invention, a drone inspection model is driven based on real-time data of the airborne video stream and the spatial position of the drone, including: performing a corresponding match between the airborne video stream and the spatial position of the drone to obtain a first matching result.

[0067] Specifically, when matching the spatial locations of airborne video streams and drones, the matching is performed based on timestamps. The timestamps of the airborne video streams and the timestamps of the drones' spatial locations are analyzed, and the airborne video streams and drones with the same timestamp are matched together to determine the first matching result.

[0068] Based on the airborne video stream and the drone's spatial location, the current video acquisition information and the drone's current spatial location are determined according to the first matching result.

[0069] Specifically, when determining the current video acquisition information and the current spatial location of the drone, the current timestamp is combined with the first matching result to use the corresponding airborne video stream and the spatial location of the drone as the current video acquisition information and the current spatial location of the drone.

[0070] The current video capture information is optimized to obtain the optimized current video capture information.

[0071] The optimization process includes image preprocessing and image correction.

[0072] The drone inspection model is updated based on the optimized current video capture information and the drone's current spatial location to generate a real-time view of the drone inspection.

[0073] Specifically, when updating the drone inspection model based on the optimized current video acquisition information and the drone's current spatial location, the display overview is determined according to the drone's current spatial location, and then filled in according to the optimized current video acquisition information, thereby updating the drone inspection model and obtaining a real-time drone inspection view based on the optimized current video acquisition information and the drone's current spatial location.

[0074] The aforementioned drone inspection model can efficiently obtain a real-time view of drone inspection based on the airborne video stream and the drone's spatial position. Furthermore, by matching the airborne video stream and the drone's spatial position, it ensures the consistency of timestamps, avoids data corruption, and guarantees the accuracy of the real-time drone inspection view. In addition, by optimizing the current video acquisition information, it improves the quality of the current video acquisition information, eliminates distortion defects caused by changes in the field of view, and thus improves the quality of the real-time drone inspection view, enabling better feedback on the inspection situation.

[0075] In one embodiment of the present invention, spatial embedding of a real-time view of a UAV inspection is performed by combining pose data information, including: constructing a view frustum of the UAV by combining pose data information with field of view and focal length.

[0076] In constructing the UAV's view frustum by combining pose data with field of view and focal length, the near clipping plane, far clipping plane, left clipping plane, right clipping plane, top clipping plane, and bottom clipping plane are analyzed and calculated based on the pose data, field of view, and focal length, respectively, resulting in six planar equations for the view frustum. The UAV's view frustum is then constructed based on these six planar equations. A spatial index is then used to quickly retrieve a subset of 3D Gaussian point clouds located within the view frustum, preparing for subsequent projection and fusion. Here, the spatial index can be, for example, a KD-Tree or an octree.

[0077] The projection range of the drone video in the 3DGS rendering space is determined based on the view frustum.

[0078] Specifically, when determining the projection range of the UAV video in the 3DGS rendering space based on the view frustum, continuous geometric and visual information that can be used for fusion is generated for the discrete point cloud to achieve the generation of projection range and depth information. The specific steps are as follows: Gaussian points obtained by spatial indexing of the view frustum are projected onto the UAV imaging plane. Based on the spatial position, covariance and opacity of each Gaussian point, corresponding depth buffers and color buffers are generated through forward rendering or fast rasterization technology. Thus, the projection range of the UAV video in the 3DGS rendering space is determined according to the depth buffers and color buffers.

[0079] The real-time view of the drone inspection is fused with the rendering results of the 3DGS digital twin model based on the projection range to obtain the inspection monitoring view.

[0080] In the process of fusing the real-time view of the drone inspection with the rendering results of the 3DGS digital twin model based on the projection range, the real-time view of the drone inspection is projected and rendered according to the projection range of the 3DGS digital twin model. Depth testing or transparency blending is used to resolve occlusion and eliminate abruptness. The real-time video frames of the drone are used as textures and aligned with the generated 3DGS color buffer. Depth testing is performed. For visual boundary areas, an adaptive transparency blending algorithm is used to dynamically adjust the blending weights according to the Gaussian point density and the edge gradient of the video frame to smooth the visual transition between discrete point clouds and continuous video images, thereby achieving transparency blending. Finally, the rendered and blended image is output to obtain the inspection and monitoring view.

[0081] When spatially embedding the real-time view of UAV inspection based on the superimposed rendering position and pose data information, it can avoid the inability to directly fuse due to the lack of continuous surfaces, ensuring the rendering fusion of the real-time view of UAV inspection and the 3DGS digital twin model, and guaranteeing the presentation of the inspection monitoring view. Moreover, through efficient spatial indexing, frustum point cloud filtering, and depth-color buffer generation and adaptive blending strategy for discrete data characteristics, it achieves accurate, natural, and low-latency fusion of real-time video streams while maintaining the original rendering efficiency and visual fidelity of 3DGS, improving the acquisition efficiency of the inspection monitoring view, and enabling Shudie to obtain the inspection monitoring view efficiently and accurately.

[0082] In one embodiment of the present invention, before the real-time display of the inspection monitoring view, a fusion effect test is performed on the inspection monitoring view, including: analyzing the image parameters of the inspection monitoring view and determining whether the inspection monitoring view needs color adjustment, and obtaining the test analysis results.

[0083] The image parameters include hue, saturation, and brightness. To determine whether color adjustment is needed in the inspection monitoring view, the gradient of image parameters between the real-time UAV inspection view and the 3DGS digital twin model of the substation at the projection rendering position is analyzed. Based on this gradient, the need for color adjustment is determined, thus obtaining the verification analysis results.

[0084] The colors of the inspection monitoring view were adjusted based on the test and analysis results.

[0085] Specifically, when the inspection and analysis results indicate that the inspection and monitoring view needs color adjustment, the color adjustment range is determined for the inspection and monitoring view, and then the image parameters are adjusted within the color adjustment range to improve the color effect of the image, thus obtaining the color-adjusted inspection and monitoring view.

[0086] The inspection monitoring view with adjusted colors is rendered with light and shadow to obtain an optimized inspection monitoring view.

[0087] In the process of rendering light and shadow on the color-adjusted inspection and monitoring view, the light and shadow are rendered by simulating real lighting effects to enhance the three-dimensionality and realism of the image, resulting in an optimized inspection and monitoring view.

[0088] The above-mentioned verification of the fusion effect of the inspection and monitoring view ensures that the inspection and monitoring view can have a better visual effect when displayed in real time. By analyzing the image parameters of the inspection and monitoring view, the color difference between the real-time view of the UAV inspection and the 3DGS digital twin model of the substation is clarified, avoiding visual conflict caused by combining two images with large color differences. When the color of the inspection and monitoring view is consistent, it can provide a better visual experience for operation and maintenance personnel. Moreover, by applying light and shadow rendering to the inspection and monitoring view after color adjustment, the three-dimensionality and realism of the image are enhanced, improving the presentation effect of the inspection and monitoring view.

[0089] In one embodiment of the present invention, when displaying the inspection monitoring view in real time, the drone's trajectory is also marked in the inspection monitoring view according to its spatial position.

[0090] Specifically, when marking the drone's trajectory in the inspection monitoring view based on its spatial location, the drone's inspection position is determined in the substation's 3DGS digital twin model according to the drone's spatial location. The drone's inspection position is then marked in real time to determine the drone's inspection waypoints. The drone's inspection trajectory is then determined for the drone's inspection waypoints and displayed in real time in the inspection monitoring view.

[0091] The aforementioned method of marking the drone's flight path in the inspection monitoring view based on its spatial location allows for a direct and intuitive display of the drone's actual inspection flight within the substation. This enables the optimization of the drone's inspection route based on the actual flight path, thereby improving the monitoring effectiveness of drone inspections. Furthermore, by storing and recording inspection information based on the onboard video stream at the drone's inspection waypoints, an autonomous inspection record is created, facilitating data retrieval and verification.

[0092] In one embodiment of the present invention, inspection and collection information is also marked in the flight track, including: performing a correlation analysis of UAV inspection waypoints for power equipment in the substation in combination with the flight track, and determining the associated UAV inspection waypoints for the power equipment.

[0093] Specifically, when performing drone inspection waypoint association analysis on power equipment in substations in conjunction with flight tracks, drone inspection waypoint association analysis is performed on each power equipment in the substation in conjunction with flight tracks to determine the drone inspection waypoints that can be collected by the drone during the inspection process according to the inspection route, and thus these drone inspection waypoints are used as associated drone inspection waypoints for the power equipment.

[0094] By retrieving the airborne video stream information collected by the drone during its inspection along the inspection route from the associated drone inspection waypoints, the target inspection information of the power equipment can be obtained.

[0095] Among them, the target inspection information for power equipment includes airborne video stream information corresponding to each associated UAV inspection waypoint.

[0096] Based on the target inspection information collected from the power equipment, a comprehensive inspection analysis of the power equipment is conducted to determine the overall inspection labeling data for the power equipment.

[0097] Specifically, when conducting overall inspection analysis of power equipment based on the target inspection information collected from the power equipment, the target information of the power equipment is combined with the target information of the power equipment to extract the target information of the power equipment. Then, the target information of the power equipment is combined to obtain the overall inspection information of the power equipment. The perspective analysis of the power equipment is performed, and the perspective that maximizes the display of the overview of the power equipment is taken as the optimal perspective. The perspective of the power equipment inspection information is adjusted according to the optimal perspective to obtain the overall inspection annotation data of the power equipment.

[0098] The location of the power equipment is determined in the flight track, and the overall inspection data of the power equipment is marked at the marked location.

[0099] Specifically, when determining the location of power equipment in the flight path, the location of the power equipment is analyzed in conjunction with the flight path. The nearest UAV inspection waypoint to the power equipment is determined in the flight path, and this point is used as the location of the power equipment. In this way, the overall inspection and labeling data of the power equipment is labeled at the location of the power equipment.

[0100] The aforementioned annotation of inspection and data collection information in the flight path is based on power equipment. This allows maintenance personnel to directly obtain the overall inspection and annotation data of the power equipment for analysis and judgment when analyzing and judging the substation based on the inspection and monitoring view. This provides convenience for maintenance personnel. Furthermore, by combining the power equipment in the substation with the flight path for drone inspection waypoint correlation analysis, all drone inspection waypoints that can collect data on the power equipment during the inspection process are locked, ensuring the comprehensiveness of the target inspection and data collection of the power equipment. This, in turn, ensures the comprehensiveness of the overall inspection and annotation data of the power equipment and avoids omissions. In addition, by determining the annotation position of the power equipment in the flight path and annotating the overall inspection and annotation data of the power equipment, the presentation content of the flight path is enriched, allowing the inspection and monitoring view to contain more data information. Moreover, by determining the drone inspection waypoint closest to the power equipment in the flight path, the overall inspection and annotation data of the power equipment is marked near the power equipment, improving the orderliness of the flight path annotation and avoiding an overly chaotic inspection and monitoring view that would affect the analysis and judgment of the power equipment by maintenance personnel. This, in turn, enables better monitoring of the substation and ensures the operation of the substation.

[0101] Those skilled in the art should understand that the terms "first" and "second" in this invention merely refer to different application stages.

[0102] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0103] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for mapping unmanned aerial vehicle (UAV) inspection routes based on 3D Gaussian point clouds, characterized in that, include: Plan the inspection routes for drones; Map and bind the inspection routes of drones to the 3DGS digital twin model of the substation; The system acquires the spatial location and onboard video stream of the drone during the inspection process along the inspection route, and transmits the drone's spatial location information and onboard video stream in real time; it receives the drone's spatial location information and onboard video stream, and overlays and renders the onboard video stream with the drone's spatial location onto the 3DGS digital twin model of the substation.

2. The UAV inspection route mapping method according to claim 1, characterized in that, Mapping and binding the inspection routes of UAVs to the 3DGS digital twin model of the substation includes: standardizing and regulating the inspection routes of UAVs, performing coordinate system consistency analysis and judgment, and obtaining the analysis and judgment results; performing coordinate system transformation on the inspection routes based on the analysis and judgment results to obtain the inspection routes in the target coordinate system; using a point cloud matching algorithm to match the waypoints in the inspection routes in the target coordinate system with the 3DGS digital twin model of the substation, and aligning them based on the matching results; obtaining the inspection parameters of the waypoints in the inspection routes in the target coordinate system, and binding the data to the corresponding points in the 3DGS digital twin model of the substation based on the inspection parameters according to the matching results.

3. The UAV inspection route mapping method according to claim 1, characterized in that, Real-time transmission of UAV spatial location information includes: acquiring the communication method of the data receiving module; performing data feature analysis on the UAV spatial location information to determine the data characteristics of the UAV spatial location information; performing communication matching based on the communication information of the data receiving module and the data characteristics of the UAV spatial location information; and transmitting the UAV spatial location information to the data receiving module based on the communication matching results.

4. The UAV inspection route mapping method according to claim 3, characterized in that, Real-time transmission of UAV-borne video streams includes: configuring the transmission protocol for the UAV-borne video stream and determining the target transmission protocol for the UAV-borne video stream; establishing a data stream communication connection between the transmission module and the data receiving module based on the target transmission protocol of the UAV-borne video stream; and transmitting the UAV-borne video stream to the data receiving module in real time based on the data stream communication connection.

5. The UAV inspection route mapping method according to claim 1, characterized in that, The process involves overlaying and rendering airborne video streams combined with the spatial position of drones into a 3DGS digital twin model of the substation. This includes: establishing a drone inspection model for drone inspections; driving the drone inspection model based on real-time data from the airborne video stream and the drone's spatial position to obtain a real-time view of the drone inspection; acquiring pose data information from the drone's spatial position within the 3DGS digital twin model of the substation; spatially embedding the pose data information into the real-time view of the drone inspection to obtain an inspection monitoring view; and displaying the inspection monitoring view in real time.

6. The UAV inspection route mapping method according to claim 5, characterized in that, The drone inspection model is driven by real-time data from the airborne video stream and the drone's spatial location, including: matching the airborne video stream and the drone's spatial location to obtain a first matching result; determining the current video acquisition information and the drone's current spatial location based on the first matching result; optimizing the current video acquisition information to obtain optimized current video acquisition information; and updating the drone inspection model based on the optimized current video acquisition information and the drone's current spatial location to generate a real-time drone inspection view.

7. The UAV inspection route mapping method according to claim 5, characterized in that, Spatially embedding the real-time view of UAV inspection by combining pose data information includes: constructing the UAV's view frustum by combining pose data information with field of view and focal length; determining the projection range of the UAV video in the 3DGS rendering space based on the view frustum; and fusing the real-time view of UAV inspection with the rendering result of the 3DGS digital twin model based on the projection range to obtain the inspection monitoring view.

8. The UAV inspection route mapping method according to claim 5, characterized in that, Before displaying the inspection monitoring view in real time, the fusion effect of the inspection monitoring view is tested, including: analyzing the image parameters of the inspection monitoring view and determining whether the inspection monitoring view needs color adjustment, and obtaining the test analysis results; adjusting the color of the inspection monitoring view according to the test analysis results; and performing light and shadow rendering on the color-adjusted inspection monitoring view to obtain the optimized inspection monitoring view.

9. The UAV inspection route mapping method according to claim 5, characterized in that, When displaying the inspection monitoring view in real time, the drone's trajectory is also marked in the inspection monitoring view based on its spatial position.

10. The UAV inspection route mapping method according to claim 9, characterized in that, The process also includes labeling inspection data collected during flight paths, including: performing a correlation analysis of UAV inspection waypoints for power equipment in substations based on flight paths to determine associated UAV inspection waypoints for power equipment; retrieving airborne video stream data collected by UAVs during inspections along inspection routes based on corresponding cruise points to obtain target inspection data for power equipment; performing overall inspection analysis of power equipment based on the target inspection data to determine overall inspection labeling data for power equipment; determining the labeling location of power equipment in the flight path and labeling the overall inspection data for power equipment at the labeling location.

Images