Unmanned aerial vehicle-oriented intelligent window inspection method based on multi-modal perception
By combining multimodal perception and multi-target PID controller, the problems of insufficient perception and limited autonomous navigation capability in UAV window inspection are solved, realizing efficient and safe fully automatic window inspection, generating structured inspection reports, and reducing system costs and reliance on professional operators.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN UNIV OF TECH
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-19
AI Technical Summary
Existing drone window inspection technology suffers from insufficient perception capabilities, limited autonomous navigation capabilities, and a single control strategy, making it difficult to achieve efficient and safe fully automated inspections, especially posing safety hazards in high-rise buildings.
A multimodal perception method is adopted, which combines RGB visual information, LiDAR information and nine-point structured depth sampling to construct an egocentric semantic map. Combined with a multi-target PID controller, the UAV can achieve autonomous navigation and window recognition. Priority control strategy ensures the safety and accuracy of inspection.
It enables efficient and safe fully automated window inspection by drones in high-rise buildings, reducing reliance on professional operators, lowering system deployment and maintenance costs, improving inspection efficiency and safety, and generating structured inspection reports.
Smart Images
Figure CN122064098A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous navigation and intelligent inspection technology for unmanned aerial vehicles (UAVs), specifically to an intelligent window inspection method based on multimodal perception for UAVs. Background Technology
[0002] Window inspection is an important part of building safety maintenance and regular inspections, especially in modern urban environments with an increasing number of high-rise buildings. Traditional window inspections mainly rely on manual operation, which carries extremely high safety risks and is inefficient.
[0003] With the rapid development of drone technology, the field of building inspection has ushered in an opportunity for technological innovation. Drones, with their advantages of flexibility, unique perspective, and operational safety, are gradually being applied to areas such as building exterior inspection, bridge inspection, and power line inspection. In window inspection, existing technologies mainly employ the following two approaches:
[0004] The first type is an automated inspection system based on pre-programmed flight paths. This type of system utilizes RTK (Real-Time Kinematic) technology to achieve centimeter-level positioning accuracy. Ground station software plans a three-dimensional flight path, and the UAV automatically executes the inspection task according to the preset path. However, this method lacks environmental adaptability and real-time target recognition capabilities in practical applications, making it difficult to guarantee the safety and accuracy of the mission.
[0005] The second type is a vision-guided intelligent inspection method. This type of method typically uses SLAM (Simultaneous Localization and Mapping) technology to achieve autonomous localization and navigation for the drone, combined with computer vision algorithms to identify window targets. However, this method has significant shortcomings in practical applications: visual SLAM is prone to tracking loss on simple building surfaces; deep learning algorithms are sensitive to lighting conditions, and their detection performance drops significantly in complex environments such as backlighting and strong reflections; the system lacks depth perception capabilities, making it difficult to accurately control the safe distance between the drone and the wall, posing a collision risk.
[0006] These technological bottlenecks severely restrict the large-scale application and commercialization of drone window inspection technology. Especially against the backdrop of the rapid increase in the number of high-rise buildings and increasingly stringent safety regulations, developing a complete technical solution that integrates multimodal perception, intelligent decision-making, and adaptive control has significant theoretical and practical value. This invention, against this technological backdrop, addresses the shortcomings of existing technologies by proposing a drone-based intelligent window inspection method based on multimodal perception. Summary of the Invention
[0007] The purpose of this invention is to provide an intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs), which solves the technical problems of insufficient perception capability, limited autonomous navigation capability, and single control strategy in the prior art, and realizes efficient, safe and fully automatic window inspection of high-rise buildings.
[0008] The objective of this invention is achieved through the following technical solution:
[0009] A smart window inspection method based on multimodal perception for drones includes the following steps:
[0010] S1. Receive the preset task parameters of the inspection task, including the target number of floors, floor height interval, and wall target distance, and obtain the structured task features through the parameter parsing module;
[0011] S2. Capture the RGB view of the drone's current location, identify and locate window objects using the YOLO object detection network, and obtain environmental decoupling knowledge;
[0012] S3. Simultaneously capture the depth view of the current location, combine the task features in S1 and the RGB view in S2, and obtain an egocentric semantic depth map through nine-point structured depth sampling.
[0013] S4. Combining the task features in S1 and the egocentric semantic map obtained in S3, generate a sequence of navigable waypoints leading to each floor based on the floor state machine;
[0014] S5. Combining the environmental decoupling knowledge obtained in S2 and the navigable waypoints obtained in S4, the multi-objective PID controller predicts the multi-dimensional navigation actions that the UAV should perform.
[0015] S6. Repeat the above steps until the drone completes the inspection of all floors or reaches the preset altitude.
[0016] The next step, S1, uses preset data thresholds to set the aircraft's inspection distance, floor height interval, and flight accuracy.
[0017] The specific workflow for window recognition using the YOLO object detection network in the further step S2 is as follows:
[0018] First, a pre-trained window detection model is loaded, and forward inference is performed on the acquired RGB image. Median filtering and historical smoothing are used to eliminate noise interference, and detection results containing bounding box coordinates, detection confidence and window category are obtained.
[0019] Then, based on the detection results, a window selection strategy based on the distance to the image center is established. By calculating and evaluating the distance between each detection window and the image center, the window closest to the image center is selected as the main tracking target, and its bounding box center coordinates are calculated.
[0020] Next, continuous frame tracking is performed on the detected windows. The continuity of the windows is evaluated based on the intersection-over-union ratio (IOU) of the bounding boxes and the positional similarity. The positional similarity (pos_similarity) and size similarity (size_similarity) are calculated together to form a comprehensive similarity score (overall_similarity). The calculation method is as follows:
[0021] (1)
[0022] Finally, the selected window information is combined with other perception data in the environment to form multimodal structured environment decoupling knowledge, which includes window position coordinates, bounding box size, detection confidence, corresponding depth value and timestamp, providing multi-dimensional perception input for subsequent navigation action prediction.
[0023] The specific process for obtaining the egocentric semantic map in the further step S3 is as follows:
[0024] First, a depth image from a depth camera is received, and nine fixed sampling positions are pre-defined on the image: top left, middle left, bottom left, top middle, center, bottom middle, top right, middle right, and bottom right. These nine positions are evenly distributed in different regions of the image. Then, the local depth value is extracted for each sampling point, and median filtering and historical smoothing are performed to obtain a stable depth estimate.
[0025] Next, at each sampling location, depth data within a 5×5 pixel range around it is extracted. Based on the depth distribution of the nine points, the wall edge and building top positions are detected in real time. The wall edge detection is achieved by comparing the relationship between the depth of the left and right point sets and the threshold, and the building top detection is achieved by comparing the relationship between the depth of the top point set and the threshold.
[0026] Finally, the depth information, edge detection results, and window detection results in the RGB view are fused to construct an egocentric semantic map that includes obstacle distances, passable areas, and key target locations.
[0027] The specific process for generating navigable waypoints in the further step S4 is as follows:
[0028] First, the formula for calculating the target flight altitude based on the current floor number is:
[0029] (2)
[0030] Where Initial_height is the initial flight height, Current_floor is the current floor number, and Floor_height_interval is the floor height interval;
[0031] Then, the horizontal movement strategy is determined based on the current scanning direction: when the direction is "left", the waypoint generation strategy is to maintain a horizontal speed to the left, and when the direction is "right", the waypoint generation strategy is to maintain a horizontal speed to the right.
[0032] Next, the waypoint sequence is dynamically adjusted based on the wall edge detection results in the semantic map, and a stop waypoint is inserted when a wall edge is detected;
[0033] Finally, taking into account the window alignment requirements, an alignment waypoint is inserted when a window is detected to align the center of the drone image with the center of the window.
[0034] The further step S5, which involves predicting navigation actions and employing a priority control strategy in the multi-objective PID controller, is as follows:
[0035] First, four independent PID controllers are established, responsible for altitude control, wall distance control, yaw control, and window alignment control, respectively. Altitude control has the highest priority to ensure the safety of UAV flight. Wall distance control has the second highest priority to maintain a safe distance from the wall. Yaw control and window alignment control work together under safe conditions, with the former maintaining a stable heading and the latter achieving precise alignment. When multiple control objectives conflict, the higher priority control requirement is given priority.
[0036] Then, the error signals for each dimension are calculated: height error is the difference between the target height and the actual height, wall distance error is the difference between the target distance and the minimum depth, yaw error is the difference between the initial yaw and the current yaw, and window alignment error is the pixel difference between the window center and the image center.
[0037] Next, each PID controller calculates the control output based on the corresponding error and performs output limiting processing;
[0038] Finally, the four control outputs are combined into a final navigation action command, which is then sent to the drone for execution.
[0039] The further conditional judgments in step S6 include:
[0040] If the depth of the upper point set is detected to be greater than the set threshold or the preset maximum number of floors has been inspected, it is determined that the highest floor has been reached and the inspection is completed.
[0041] If a wall edge is detected during a single-layer scan, a floor switch and direction reversal are triggered, and the above steps are continued to perform the inspection task.
[0042] The system uptime exceeded the safety limit, therefore the inspection is considered a failure.
[0043] Further intelligent window inspection methods for drones based on multimodal perception also include data recording and transmission modules. Each time a window is detected and aligned, window information is automatically recorded, including timestamp, floor, location coordinates, orientation, and alignment status; window images with annotation information, including bounding boxes and status information, are saved; and an inspection statistics report is generated, including the total number of windows detected and their distribution across floors.
[0044] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-described intelligent window inspection method based on multimodal perception for unmanned aerial vehicles.
[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described intelligent window inspection method for UAVs based on multimodal perception.
[0046] Compared with existing technologies, the beneficial effects of the technical solution of this invention are as follows: Unlike existing window inspection technologies that heavily rely on the experience of professional operators, the intelligent window inspection method based on multimodal perception for UAVs provided by this invention achieves a high degree of autonomous operation. The system can automatically complete the entire process from takeoff, inspection to landing by preset task parameters, reducing the need for manual intervention to below 95%, and greatly reducing the dependence on professional operators and the technical threshold.
[0047] Unlike existing drone inspection technologies that generally rely on external positioning systems such as RTK and GPS, the method of this invention does not depend on external positioning signals. Instead, it achieves precise relative navigation by fusing RGB visual information, LiDAR information, and nine-point structured depth sampling to construct an egocentric semantic map. This avoids the problem of weakened or even failed external positioning signals in urban canyon environments and significantly reduces system deployment costs and maintenance complexity.
[0048] The core algorithm of this invention has low hardware requirements and can be easily integrated into mainstream consumer-grade drone platforms. It only requires a standard RGB camera and depth sensor to function, significantly reducing equipment procurement and maintenance costs. Its modular design allows the technology to be seamlessly embedded into existing building inspection processes, improving its applicability and accessibility.
[0049] Through innovative priority control strategies and multi-layered safety protection mechanisms, this invention significantly improves operational safety while ensuring inspection efficiency. Wall distance control accuracy reaches ±0.3 meters, and height control accuracy reaches ±0.1 meters. Combined with real-time depth threshold warnings and emergency hovering mechanisms, the collision risk is reduced to less than 10% of traditional methods, providing technical assurance for the safe and reliable application of drone window inspection.
[0050] Unlike the scattered and unstructured inspection data generated by existing methods, this invention generates standardized inspection reports containing multi-dimensional information such as timestamps, location coordinates, and window status through automated data recording and structured storage. Attached Figure Description
[0051] Figure 1 This is a flowchart illustrating the method of the present invention.
[0052] Figure 2 This is a schematic diagram of the overall framework structure of the method of the present invention. Detailed Implementation
[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.
[0054] The technical solution of this invention has been described in detail with reference to the accompanying drawings and specific embodiments, aiming to demonstrate its basic concept and operation process. The schematic diagrams provided in the drawings are for illustrative purposes only and do not represent actual product dimensions or limit the scope of the patent. Where there is no conflict, features in the embodiments can be combined with each other to explore more implementation possibilities. Those skilled in the art should understand that, in order to more clearly convey the concept of this invention, certain components may be omitted, enlarged, or reduced in the drawings, and well-known structures and their descriptions may be omitted.
[0055] Please refer to the attached diagram for details. Figure 1 and Figure 2 This embodiment relates to a smart window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs). The detailed steps are as follows:
[0056] S1. Receive preset task parameters for multi-level window inspection and extract task features such as floor height interval, scanning direction, and target distance through the parameter parsing module. The task parameters for this step are preset before each inspection begins and include key parameters such as the target floor number, floor height interval, and target distance to the wall. In this embodiment, the parameter parsing module uses a structured data parsing method to convert the raw parameters into structured task features that the system can understand, providing a basis for decision-making in subsequent navigation and control.
[0057] S2. Capture RGB images during the inspection process, and use the YOLO object detection network to identify and locate window objects, obtaining environmental decoupling knowledge. The RGB view in this step is captured in real time by the drone during the inspection, reflecting the environmental information of the drone's current location. The specific workflow of window recognition is as follows:
[0058] 201. A pre-trained YOLO window detection model is used to perform forward inference on the acquired RGB image. Median filtering and history smoothing are employed to eliminate noise interference, and detection results including bounding box coordinates, detection confidence, and window category are obtained. The YOLO model used in this embodiment has been specially trained and can accurately identify various types of architectural windows.
[0059] 202. A window selection strategy based on image center distance is established based on the detection results. By calculating and evaluating the Euclidean distance between each detected window and the image center, the window closest to the image center is preferentially selected as the primary tracking target, and its bounding box center coordinates are calculated. This strategy ensures that the UAV can focus on the most relevant window object.
[0060] 203. Perform continuous frame tracking on the detected windows, and evaluate the continuity of the windows based on the intersection-over-union ratio (IOU) of the bounding boxes and positional similarity. Combine the positional similarity (pos_similarity) and size similarity (size_similarity) to form a comprehensive similarity score (overall_similarity), calculated as follows:
[0061] (3)
[0062] This similarity calculation mechanism ensures the stability and accuracy of window tracking.
[0063] 204. The selected window information is combined with other perceptual data in the environment to form multimodal structured decoupled environmental knowledge, including window position coordinates, bounding box size, detection confidence, corresponding depth value, and timestamp. This knowledge representation provides multi-dimensional perceptual input for subsequent navigation action prediction.
[0064] S3. Simultaneously capture the depth image, combine it with RGB visual information and task features, and obtain an egocentric multi-point depth semantic map through structured depth sampling. In this step, the depth view and RGB view are captured synchronously, and they have a temporal and spatial correspondence. The specific process for obtaining the egocentric semantic map is as follows:
[0065] 301. Receive depth images from a depth camera and pre-define nine fixed sampling positions on the image: top left, middle left, bottom left, top center, center, bottom center, top right, middle right, and bottom right. These nine positions are evenly distributed across different areas of the image. These sampling points cover key areas of the field of view and can comprehensively reflect environmental structural information.
[0066] 302. Extract the local depth value for each sampling point, perform median filtering and historical smoothing to obtain a stable depth estimate. At each sampling location, extract the depth data within a 5×5 pixel range around it, use median filtering to eliminate outliers, and perform smoothing using a moving average of historical data.
[0067] 303. Based on the depth distribution of nine points, the system detects the positions of wall edges and building tops in real time. Wall edge detection is achieved by comparing the depth values of the left and right point sets with a threshold: when the depth values of the three left points (top left, middle left, and bottom left) are all greater than a preset threshold, it is determined that a left wall edge has been detected; the same applies to the right side. Building top detection is achieved by comparing the depth values of the top point set (top left, top middle, and top right) with a threshold.
[0068] 304. By fusing depth information, edge detection results, and window detection results from the RGB view, an egocentric semantic map is constructed, containing obstacle distances, passable areas, and key target locations. This map contains not only geometric information but also semantic information, providing comprehensive environmental awareness for intelligent navigation.
[0069] S4. Combining semantic maps and task features, generate a sequence of navigable waypoints leading to each floor based on a multi-level state machine. Specifically, the process for generating navigable waypoints is as follows:
[0070] 401. Calculate the target's flight altitude based on the current floor number, using the following formula: (4) Where Initial_height is the initial flight altitude, Current_floor is the current floor number, and Floor_height_interval is the floor height interval.
[0071] 402. Determine the horizontal movement strategy based on the current scanning direction: when the direction is "left", the waypoint generation strategy is to maintain a horizontal speed to the left; when the direction is "right", the waypoint generation strategy is to maintain a horizontal speed to the right. This strategy ensures that the UAV can perform a systematic scan along the wall.
[0072] 403. Dynamically adjust waypoint sequence based on wall edge detection results in the semantic map. When a wall edge is detected, a stop waypoint is inserted, causing the UAV to stop scanning in the current direction and prepare to switch floors or directions. This dynamic adjustment mechanism ensures the integrity and safety of the inspection.
[0073] 404. To meet window alignment requirements, an alignment waypoint is inserted when a window is detected, aligning the center of the UAV image with the center of the window. The alignment waypoint is generated based on the pixel deviation between the window center and the image center, and precise alignment is achieved through PID control.
[0074] S5. Utilizing navigable waypoints and environmental decoupling knowledge, a multi-objective PID controller predicts the current navigation actions of the UAV. The specific process is as follows:
[0075] 501. Establish four independent PID controllers, responsible for altitude control, wall distance control, yaw control, and window alignment control, respectively. Altitude control has the highest priority to ensure the UAV's flight safety; wall distance control has the second highest priority to maintain a safe distance from the wall; yaw control and window alignment control work together under safe conditions, with the former maintaining a stable heading and the latter achieving precise alignment. When multiple control objectives conflict, the higher-priority control requirement is prioritized.
[0076] 502. Calculate the error signals for each dimension: height error is the difference between the target height and the actual height; wall distance error is the difference between the target distance and the minimum depth; yaw error is the difference between the initial yaw and the current yaw; and window alignment error is the pixel difference between the window center and the image center. These error signals serve as inputs to the PID controller.
[0077] 503. Each PID controller calculates the control output based on the corresponding error and performs output limiting processing. The limiting range is set according to the UAV's dynamic characteristics and safety requirements to ensure that the control commands are within the safe range.
[0078] 504. Combine the four control outputs into a final navigation action command and send it to the UAV for execution. The combined command includes linear velocity (vx, vy, vz) and angular velocity (az), achieving six degrees of freedom control for the UAV.
[0079] S6. Repeat the above steps until all floors have been scanned or the preset floor height has been reached. Conditional checks include:
[0080] 601. If the depth of the upper point set is detected to be greater than the set threshold or the preset maximum number of floors has been inspected, it is determined that the highest floor has been reached and the current inspection is completed.
[0081] 602. If a wall edge is detected during a single-layer scan, a floor switch is triggered and the inspection direction is reversed. Continue the above steps to perform the inspection task.
[0082] 603. If the system runtime exceeds the safety limit or a serious anomaly is detected, the inspection is considered a failure, and the safe return procedure is initiated.
[0083] This embodiment also includes a data recording and feedback module. Each time a window is detected and aligned, the window information is automatically recorded, including timestamp, floor, location coordinates, direction, and alignment status; the window image with annotation information, including bounding box and status information, is saved; and an inspection statistics report is generated, including the total number of windows detected and their distribution on each floor.
[0084] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps in the UAV-based intelligent window inspection method described in the above embodiments. The electronic device specifically includes the following:
[0085] Processor, memory, communications interface, and bus;
[0086] The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between UAV flight control systems, ground control stations, and remote monitoring centers.
[0087] The processor is used to call the computer program in memory. When the processor executes the computer program, it implements all the steps in the UAV-based intelligent window inspection method in the above embodiments.
[0088] Preferably, embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the above-described intelligent window inspection method for UAVs based on multimodal perception. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the above-described intelligent window inspection method for UAVs based on multimodal perception.
[0089] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0090] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment).
[0091] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0094] It should be stated that the positional relationships and detailed descriptions shown in the accompanying drawings are intended to provide illustrative examples of the technical solutions to facilitate understanding of the core concepts and operating principles of the invention. These examples are not intended to limit the scope of patent protection, but rather serve as auxiliary means of explaining the technical solutions. Within the framework of this invention, those skilled in the art can, based on existing technical knowledge and the guidance of this invention, make appropriate modifications, equivalent substitutions, or innovations to the technical solutions through logical analysis, reasoning, and experimentation to adapt to different application scenarios and needs. These changes and adjustments, as long as they do not depart from the basic principles and spirit of the invention, should be considered within the scope of protection defined by the claims. The detailed description of this embodiment illustrates multiple aspects of the technical solutions, but they do not exclude other possible implementations. Those skilled in the art can make various specific modifications based on the teachings of this invention without departing from the spirit of the invention and the scope of protection of the claims, including but not limited to adjustments in hardware configuration, software implementation, system architecture, and operating procedures. These modifications, aimed at optimizing the performance, efficiency, or reliability of the technical solutions, or expanding their application scope, all fall within the scope of protection of this invention.
[0095] Furthermore, the terminology and expressions used in this invention are not intended to provide a restrictive definition of the technical solutions, but rather to offer clarity and consistency in the description. The use of terminology in the description is intended to facilitate understanding and description of the technical content and should not be considered as a limitation on the scope of patent protection. Therefore, any reasonable interpretation and application of the terminology, as long as it conforms to the purpose of the invention and the substance of the claims, should be included within the scope of protection of this invention.
[0096] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.
Claims
1. A smart window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: S1. Receive the preset task parameters of the inspection task, including the target number of floors, floor height interval, and wall target distance, and obtain the structured task features through the parameter parsing module; S2. Capture the RGB view of the drone's current location, identify and locate window objects using the YOLO object detection network, and obtain environmental decoupling knowledge; S3. Simultaneously capture the depth view of the current location, combine the task features and the RGB view, and obtain an egocentric semantic depth map through nine-point structured depth sampling; S4. Combining the task characteristics and the egocentric semantic map, generate a sequence of navigable waypoints leading to each floor based on the floor state machine; S5. By combining the environmental decoupling knowledge and the navigable waypoint sequence, a multi-objective PID controller is used to predict the multi-dimensional navigation actions that the UAV should perform. S6. Repeat the above steps until the drone completes the inspection of all floors or reaches the preset altitude.
2. The intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Step S1 uses preset data thresholds to set the aircraft's inspection distance, floor height, and flight accuracy.
3. The intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The specific workflow for window recognition using the YOLO object detection network in step S2 is as follows: S21. Load the pre-trained window detection model, perform forward inference on the acquired RGB image, use median filtering and history smoothing to eliminate noise interference, and obtain the detection results including bounding box coordinates, detection confidence and window category. S22. Based on the detection results, establish a window selection strategy based on the distance to the image center. By calculating and evaluating the distance between each detection window and the image center, select the window closest to the image center as the tracking target and calculate its bounding box center coordinates. S23. Perform continuous frame tracking on the detected windows, evaluate the continuity of the windows based on the intersection-union ratio (IOU) of the bounding boxes and the positional similarity, and comprehensively calculate the positional similarity (pos_similarity) and size similarity (size_similarity) to form a comprehensive similarity score (overall_similarity). The calculation method is as follows: (1)； S24. Combine the selected window information with the perception data in the environment to form a multimodal structured environment decoupling knowledge that includes window position coordinates, bounding box size, detection confidence, corresponding depth value and timestamp, providing multi-dimensional perception input for subsequent navigation action prediction.
4. The intelligent window inspection method based on multimodal perception for UAVs according to claim 1, characterized in that, The specific process for obtaining the egocentric semantic map in step S3 is as follows: S31. Receive a depth image from a depth camera and pre-set nine fixed sampling positions on the image: top left, middle left, bottom left, top middle, center, bottom middle, top right, middle right, and bottom right. These nine positions are evenly distributed in different areas of the image. Then, extract the local depth value for each sampling point, perform median filtering and historical smoothing to obtain a stable depth estimate. S32. At each sampling location, extract the depth data within a 5×5 pixel range around it. Based on the depth distribution of the nine points, detect the wall edge and the top of the building in real time. The wall edge detection is achieved by comparing the relationship between the depth of the left and right point sets and the threshold, and the top of the building detection is achieved by comparing the relationship between the depth of the top point set and the threshold. S33. Integrate depth information, edge detection results, and window detection results in the RGB view to construct an egocentric semantic map containing obstacle distances, passable areas, and key target locations.
5. The intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The specific process for generating navigable waypoints in step S4 is as follows: S41. The formula for calculating the target's flight altitude based on the current floor number is: (2)； Where Initial_height is the initial flight height, Current_floor is the current floor number, and Floor_height_interval is the floor height interval; S42. Determine the horizontal movement strategy based on the current scanning direction: when the direction is "left", the waypoint generation strategy is to maintain a horizontal speed to the left; when the direction is "right", the waypoint generation strategy is to maintain a horizontal speed to the right. S43. Dynamically adjust the waypoint sequence based on the wall edge detection results in the semantic map, and insert a stop waypoint when a wall edge is detected; S44. Based on the window alignment requirements, insert alignment waypoints when a window is detected to align the center of the drone image with the center of the window.
6. The intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The specific process of predicting navigation actions in step S5 and the priority control strategy adopted by the multi-objective PID controller are as follows: S51. Establish four independent PID controllers, which are responsible for altitude control, wall distance control, yaw control, and window alignment control, respectively. Altitude control has the highest priority, wall distance control has the second highest priority, and yaw control and window alignment control work together under safe conditions. When multiple control objectives conflict, the higher priority control requirement is given priority. S52. Calculate the error signals for each dimension: height error is the difference between the target height and the actual height, wall distance error is the difference between the target distance and the minimum depth, yaw error is the difference between the initial yaw and the current yaw, and window alignment error is the pixel difference between the window center and the image center. S53. Each PID controller calculates the control output based on the corresponding error and performs output limiting processing. S54. Combine the four control outputs into the final navigation action command and send it to the UAV for execution.
7. The intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The conditional judgments in step S6 include: If the depth of the upper point set is detected to be greater than the set threshold, or if the preset maximum number of floors has been inspected, it is determined that the highest floor has been reached and the current inspection is completed. If a wall edge is detected during a single-layer scan, a floor switch and direction reversal are triggered, and the above steps are continued to perform the inspection task. The system uptime exceeded the safety limit, therefore the inspection is considered a failure.
8. The intelligent window inspection method based on multimodal perception for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The method also includes data recording and feedback steps: after each window is detected and aligned, the window information is automatically recorded, including timestamp, floor, location coordinates, direction and alignment status; window images with annotation information are saved, including bounding boxes and status information; and an inspection statistics report is generated, which includes the total number of windows detected and their distribution on each floor.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the intelligent window inspection method based on multimodal perception for unmanned aerial vehicles as described in any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the intelligent window inspection method based on multimodal perception for unmanned aerial vehicles as described in any one of claims 1 to 8.