Curtain wall cleaning quality detection method and system based on unmanned aerial vehicle multi-mode perception

Abstract

The invention relates to the technical field of cleaning unmanned aerial vehicle cleaning methods, in particular to a curtain wall cleaning quality detection method and system based on unmanned aerial vehicle multi-mode sensing. A visual camera and a millimeter wave radar are adopted to synchronously collect images and distance data of a curtain wall cleaning unit, after the collected multi-source data are preprocessed, texture features of the images and relative height features of the radar are extracted, the two types of features are fused and then input into a classifier, and common plane and three-dimensional protruding stain types are distinguished according to the relative height features. And the pollution degrees of light pollution and heavy pollution are judged according to the texture features, and an accurate stain identification result is output. According to the scheme, the technical problems that multi-source data is not comprehensively considered in stain identification of an existing curtain wall cleaning unmanned aerial vehicle, the existing curtain wall cleaning unmanned aerial vehicle depends on a single visual sensor and is easily interfered by the environment, and standardized feature fusion and quantitative judgment rules do not exist, so that the stain identification precision is low, and the type and the pollution degree are not accurately judged are solved. And an accurate decision basis is provided for subsequent cleaning operation.

Description

Technical Field

[0001] This invention relates to the technical field of cleaning methods using cleaning drones, specifically a method and system for detecting the quality of curtain wall cleaning based on multimodal perception by drones. Background Technology

[0002] With the continuous growth in the number of high-rise buildings in modern cities, the cleanliness of curtain walls, as a core component of building facades, directly affects the building's aesthetics and lifespan. However, the height and large area of ​​high-rise building curtain walls create complex cleaning environments, making traditional manual cleaning methods insufficient to meet the demands for efficient and safe operations. Therefore, developing automated and intelligent curtain wall cleaning technologies has become a core direction for the industry's development. Drone technology, with its advantages of flexibility, mobility, and wide operating range, is gradually being applied to the curtain wall cleaning field, driving the industry's transformation from manual reliance to automation.

[0003] Current cleaning drones still rely on a single visual sensor for stain recognition. This method is highly susceptible to environmental factors such as changes in lighting and reflections from curtain wall materials. It fails to effectively distinguish between ordinary planar stains and stubborn, three-dimensional raised stains, and lacks a unified quantitative standard for determining the degree of contamination. This results in low stain recognition accuracy and significant issues with missed or false positives. Even when some products attempt to combine data from multiple sensors, they haven't broken free from the traditional single-data analysis approach. They haven't established a scientific feature fusion system and professional classification mechanism for visual and distance characteristics. The synergistic value of multi-source data is not realized, making it difficult to output accurate and reliable stain recognition results. Consequently, they cannot provide targeted parameter support for subsequent cleaning operations, directly impacting the effectiveness and efficiency of the spraying process.

[0004] In the cleaning quality inspection process, most cleaning drones lack a systematic closed-loop quality inspection mechanism. The cleaning effect relies solely on manual visual judgment, which is not only highly subjective and has low accuracy, but also cannot provide a standardized and quantitative assessment of the cleaning quality. Even if some products are equipped with basic inspection functions, they only use a single image comparison index for judgment, without fully considering key influencing factors such as the regional alignment accuracy of images before and after cleaning and interference from prohibited areas of the curtain wall. This can easily lead to distorted inspection results and fail to truly reflect the actual cleaning effect.

[0005] This shows that current cleaning drones still need further improvement in terms of stain recognition and cleaning quality inspection. Summary of the Invention

[0006] To address the technical problems of low stain identification accuracy and inaccurate determination of stain type and severity in curtain wall cleaning due to the lack of comprehensive consideration of multi-source data, this invention provides a curtain wall stain identification method based on UAV multimodal perception. Simultaneously, to address the technical problem of the inability to quantitatively assess cleaning quality due to the strong subjectivity of manual judgment, this invention provides a curtain wall cleaning quality inspection method based on multimodal recognition. Based on the above two methods, this invention also provides a fully automated curtain wall cleaning UAV system applying both methods.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for identifying stains on curtain walls based on UAV multimodal perception includes the following identification steps: A1. Visual images of the curtain wall surface and real-time distance sequences between the drone and the surface are simultaneously acquired by a visual camera and millimeter-wave radar mounted on a cleaning drone. A2. Perform Gaussian filtering on the visual image to remove noise and crop out the ROI region corresponding to the cleaned area; use the Grubbs criterion to remove outliers in the real-time distance sequence; A3. Extract the LBP texture feature vector from the visual image processed in step A2, and calculate the relative height feature using the real-time distance sequence processed in step A2. When the average gray-level difference of the LBP texture feature vector is less than the set contamination threshold, it is considered light contamination; otherwise, it is considered heavy contamination. When the relative height feature is greater than the distance threshold, it is considered three-dimensional raised contamination; otherwise, it is considered planar contamination. A4. The LBP texture feature vector and the relative height feature vector are concatenated and fused to form a fused feature vector, which is then input into the classifier to identify the type and degree of stains on the curtain wall.

[0008] As a further improvement to the above scheme: the pollution threshold is 8, and the distance threshold is 3mm.

[0009] As a further improvement to the above scheme, the specific steps for removing outliers using the Grubbs criterion are as follows: Calculate the mean and standard deviation of real-time distances in the real-time distance sequence; Based on the mean and standard deviation of the real-time distance, calculate the Grubbs statistic for each real-time distance in the real-time distance sequence; Real-time distances with Grubbs statistics exceeding a preset threshold are identified as outliers and removed.

[0010] As a further improvement to the above scheme: the real-time distances that are not removed are considered valid real-time distances, and for the vacant positions resulting from the removal of outliers, the average of the adjacent valid real-time distances is used to fill them.

[0011] As a further improvement to the above scheme: relative height feature h = -d0, where, The real-time mean distance of the real-time distance sequence after removing outliers is d0, which is the preset hovering distance of the cleaning drone.

[0012] A method for inspecting the cleaning quality of curtain walls based on multimodal recognition includes the following inspection steps: B1. Before carrying out the cleaning operation, acquire a reference image of the target cleaning unit, and use the above-mentioned method for identifying the type of stains and the degree of contamination in the target cleaning unit based on UAV multimodal perception. B2. Based on the identification results of step B1, perform cleaning operations on the target cleaning unit using a cleaning drone; B3. After the cleaning operation is completed, acquire cleaning images of the target cleaning unit using a vision camera; align the cleaning images with the reference images within the same ROI area; B4. Based on the aligned image, calculate its structural similarity (SSIM) value and the mean grayscale difference. When the SSIM value is greater than or equal to the SSIM threshold and the mean grayscale difference is less than the grayscale difference threshold, the cleaning quality is deemed to meet the standard.

[0013] As a further improvement to the above scheme: the SSIM threshold is 0.92 and the grayscale difference threshold is 12.

[0014] As a further improvement to the above solution: after the cleaning operation is completed, the stain residue rate is calculated: the gray values ​​of the cleaned image and the reference image are compared pixel by pixel, and pixels with a difference exceeding the preset residue judgment threshold are judged as residual pixels; the ratio of the total number of residual pixels to the total number of pixels is calculated, and this ratio is the stain residue rate.

[0015] As a further improvement to the above scheme: when the difference between the gray value of the current pixel in the cleaned image and the gray value of the corresponding pixel in the reference image is ≥6, it is determined to be a residual pixel; otherwise, it is determined to be a clean pixel.

[0016] A fully automated curtain wall cleaning drone system includes: a cleaning drone body, a vision camera, a millimeter-wave radar, a cleaning device, and a processing and control system; Both the visual camera and the millimeter-wave radar are installed at the front of the cleaning drone to simultaneously acquire visual images of the curtain wall surface and the real-time distance sequence between the drone and the curtain wall surface. The cleaning device is installed under the cleaning drone body and includes a high-pressure water spray device and a steering mechanism. The high-pressure water spray device is connected to the ground water tank through a tethered water pipe, and the steering mechanism is driven by a servo electric cylinder to adjust the water spray direction. The processing and control system is integrated inside the cleaning drone itself and is configured as follows: The above-described method for identifying curtain wall stains based on UAV multimodal perception is implemented by fusing image data collected by a visual camera with distance data collected by millimeter-wave radar to identify the type and degree of stains on the curtain wall. Based on the stain identification results, control commands are generated to adjust the water spray pressure, spray mode, and flight speed of the cleaning drone of the cleaning device, and to control the cleaning device to perform cleaning operations on the target cleaning unit. The above-described curtain wall cleaning quality inspection method based on multimodal recognition is implemented by collecting images after cleaning and comparing them with reference images to calculate the structural similarity (SSIM) value, the mean grayscale difference, and the stain residue rate, thereby determining whether the cleaning quality meets the standards.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. Visual cameras capture images of the curtain wall, while millimeter-wave radar simultaneously acquires distance data. Texture features from the images and relative height features from the radar are extracted. These two types of features are then fused and classified using a classifier. The system clearly distinguishes stain types based on relative height features and determines the degree of contamination based on texture features. Through comprehensive analysis of multi-source data and quantitative judgment rules, the limitations of single-data identification are overcome, enabling accurate determination of stain types and contamination levels. This solves the problem of recognition bias caused by not comprehensively considering multi-source data.

[0018] 2. After the cleaning unit is completed, the post-cleaning image is acquired and precisely aligned with the baseline image of the same area before cleaning. The image similarity is quantitatively analyzed using the SSIM algorithm, and the stain residue rate is simultaneously calculated. These two quantitative indicators are used as the basis for judging the cleaning quality, replacing the subjective judgment of manual visual inspection. A standardized quantitative evaluation system is established to achieve objective and quantifiable assessment of cleaning quality, solving the core problems of the lack of unified standards and the inability to quantify manual judgment. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall structure of the cleaning drone in this invention.

[0020] Figure 2 This is a schematic diagram of the water spraying device in this invention.

[0021] Figure 3 This is a schematic diagram of the steering mechanism structure in this invention.

[0022] Figure 4 This is a flowchart of the stain identification method in this invention.

[0023] Figure 5 This is a flowchart of the cleaning quality detection method in this invention.

[0024] In the diagram: 10. Cleaning drone body; 11. Landing gear; 12. Clamping device; 20. Battery; 30. Support platform; 40. Water spraying device; 41. Fixed bracket; 42. Water spray gun; 43. Duckbill nozzle; 44. Electromagnetic regulating valve; 45. Steering wheel; 46. Mooring water pipe; 50. Steering mechanism; 51. Driven gear; 52. Driven rack; 53. Servo electric cylinder; 54. Rack slide rail; 55. Gear mounting shaft; 56. Mounting platform. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figures 1-5 This invention focuses on a fully automated curtain wall cleaning drone system, detailing its structural composition, assembly and debugging process, fully automated cleaning operation steps, and effect verification. The system comprises a cleaning drone body 10, a perception module consisting of a visual camera and millimeter-wave radar, a cleaning device consisting of a high-pressure water spray device 40 and a servo electric cylinder 53 driving steering mechanism 50, and a processing and control system integrating core algorithms. The core relies on a multimodal perception curtain wall stain recognition method (accurately identifying planar / three-dimensional raised stains and light / heavy pollution levels through simultaneous data acquisition, preprocessing, feature extraction and fusion, and SVM classification) and a cleaning quality detection method based on the recognition results (determining cleaning compliance through image alignment, SSIM value, average grayscale difference, and stain residue rate; triggering secondary cleaning if standards are not met). This achieves full automation from job preparation, stain recognition, adaptive cleaning to quality detection and final maintenance. Actual testing shows that the system has high stain recognition accuracy, significantly improved cleaning efficiency compared to manual cleaning, meets cleaning quality standards, and has low safety risks, effectively meeting the high-efficiency, precise, and safe requirements for high-rise building curtain wall cleaning.

[0027] I. Cleaning drones

[0028] 1. Clean the drone body

[0029] like Figures 1-3As shown, the cleaning drone body 10 of this invention adopts a six-rotor redundant power structure, equipped with six brushless motors. This not only provides ample lift thrust but also features fault redundancy protection—if any one propeller fails, the remaining motor speeds can be adjusted in real time through an internal algorithm, ensuring stable flight of the drone until a safe landing, significantly improving operational safety. The arms are made of carbon fiber and feature a foldable design, balancing lightweight design and portability for convenient storage and transportation.

[0030] Two steel tube landing gears 11 are symmetrically installed on the underside of the fuselage and fixed to the fuselage via quick-release connectors, allowing for flexible disassembly and replacement and further optimizing storage convenience. Four rubber buffers are fitted to the bottom of the landing gear 11 to effectively cushion the impact and vibration during drone landing, protecting the fuselage and components. Eight aluminum alloy clamping pieces 12 are fixed to the landing gear 11; these are lightweight yet strong, secured in preset positions with screws, and are specifically used to fix the power unit and cleaning device, ensuring strong assembly stability.

[0031] The airframe integrates a processing and control system, with its core being an embedded industrial computer with a main frequency of no less than 1.5 GHz. It features a built-in Linux operating system and dedicated image processing algorithms, ensuring high-efficiency computational response. The system achieves precise communication with the flight controller via a CAN bus and establishes data interaction with the millimeter-wave radar, servo cylinder 53, and proportional solenoid valve via an RS485 bus or Ethernet, ensuring stable and reliable command transmission.

[0032] The front of the unit is equipped with a millimeter-wave radar and a vision camera, forming a multimodal perception unit: the radar antenna emits vertically towards the curtain wall surface, with a ranging range of 0.2 to 5 meters, a ranging accuracy better than ±3 cm, and a sampling frequency of 50 Hz, which can stably acquire real-time distance data; the vision camera has a field of view of about 90°, a resolution of 1920×1080, a frame rate of 30 fps, strong backlight adaptability, and can clearly capture images of stains on the curtain wall surface, providing a high-quality data source for subsequent recognition algorithms.

[0033] The power unit consists of two 6S batteries 20 and a carbon fiber support platform 30. The support platform 30 has multiple weight-reduction grooves, which further reduce weight while ensuring structural strength. The support platform 30 is fixedly connected to the clamping parts 12 on the landing gear 11 by screws, which is convenient to install and disassemble and highly interchangeable, and can quickly complete the replacement and maintenance of the batteries 20.

[0034] 2. Cleaning device

[0035] The water spraying device 40 consists of a fixed bracket 41, a water gun 42, a duckbill nozzle 43, an electromagnetic regulating valve 44, and a steering wheel 45. The fixed bracket 41 is manufactured using sheet metal bending technology, combining ease of processing and cost advantages. The water gun 42 is assembled with the steering wheel 45 via the fixed bracket 41, and the two are fastened together with hexagonal screws, ensuring strong assembly stability. The cleaning device is connected to a ground water tank via a tethered water pipe 46, which eliminates the weight burden of carrying a water tank on the drone body and enables continuous water supply, ensuring long-term continuous operation of the drone. The electromagnetic regulating valve 44 is electrically connected to the internal processing and control system of the drone body, which can precisely control the water flow and pressure to adapt to different stain cleaning needs. The duckbill nozzle 43 is designed to form a fan-shaped spray surface, improving the cleaning coverage per unit area.

[0036] The steering mechanism 50 consists of a driven gear 51, a driving rack 52, a servo cylinder 53, a rack and pinion slide rail 54, a gear mounting shaft 55, and a mounting platform 56. The mounting platform 56 is made of aluminum alloy, which is lightweight and structurally reliable. Multiple through holes are machined on the platform to reduce weight and facilitate the positioning and installation of each component. The gear mounting shaft 55 has evenly distributed threaded holes machined on its largest end face and is fixed to the mounting platform 56 with screws. The driven gear 51 and the gear mounting shaft 55 use a clearance fit, and the contact surface is lubricated to ensure smooth rotation without jamming. Axial limiting is achieved through a shaft end retaining ring to prevent displacement. The driving rack 52 and the rack and pinion slide rail 54 use a dovetail fit design, providing precise guidance and strong stability. The contact surface is also lubricated to effectively reduce transmission wear and improve sliding efficiency. The rack and pinion slide rail 54 is fastened to the mounting platform 56 with hexagonal socket head cap screws. The servo electric cylinder 53 is fixed to the mounting platform 56 by an internal hex screw. Its output end is threadedly connected to the active rack 52 and is electrically linked to the processing and control system. It can adjust the angle of the water spray device 40 in real time according to the stain recognition result. It can drive the device to swing left and right to achieve dynamic rinsing, or fix the angle to spray stubborn stains at a fixed point.

[0037] 3. Cleaning preparation

[0038] After the components are assembled, the power-on debugging phase begins to ensure that the functions of each system work together and the parameters meet the standards accurately, laying the foundation for subsequent fully automated cleaning operations. The specific process is as follows: (1) Power-on debugging Basic parameter calibration: Power on the drone and prioritize RTK positioning system calibration to ensure hovering accuracy ≤8cm and position deviation ≤3cm within 1 minute, ensuring flight trajectory control accuracy; then test the ranging accuracy of the millimeter-wave radar installed at the front of the drone, verifying it at key distance points of 0.5m, 1m, and 2m, with ranging error ≤±3cm; simultaneously test the imaging quality of the visual camera, focusing on verifying image effects under strong backlight conditions, ensuring clear images without overexposure and accurate capture of dirt and details.

[0039] Algorithm and Control Testing: Comprehensive verification of core algorithms and control functions to ensure adaptability to actual operational scenarios: Input standard image libraries and radar distance data covering dust, oil stains, and three-dimensional raised stains to verify that the classification accuracy of the multimodal fusion recognition algorithm is ≥95%, and that there are no missed or false judgments of three-dimensional raised stains, ensuring the accuracy of stain recognition; Simulate sudden obstacle scenarios such as exposed pipes and temporary billboards to test the rationality of the unit division and obstacle avoidance response speed of the dynamic path planning algorithm, requiring an obstacle avoidance response time ≤200ms and a path regression accuracy ≤1cm after obstacle avoidance; Calibrate the proportional solenoid valve opening error ≤5% and the servo electric cylinder 53 swing angle accuracy ±0.5° to verify that the parameter matching response speed of the cleaning parameter adaptive algorithm is ≤50ms, ensuring that the cleaning action and parameter adjustment are accurately synchronized; Further simulate complex environments such as rain and fog (radar sampling frequency automatically switches to 100Hz) and sensor single-point failure scenarios to verify that the algorithm can automatically start the anti-interference mode and emergency switching logic to ensure the safe operation of the equipment and fully meet the reliability requirements of engineering operations.

[0040] (2) Deployment of media and equipment

[0041] After all debugging is successful, start the media preparation and equipment deployment work before the operation: add clean water (water hardness ≤100mg / L) to the ground water tank, add neutral detergent at a ratio of 1:50-1:80 (pH value controlled in the range of 6-8), and stir thoroughly until uniformly mixed; deploy the ground station and tethered water supply system, establish a data communication link between the ground station and the UAV processing and control system to ensure real-time and stable command transmission and data feedback; preset the cleaning quality standard (stain residue rate ≤5%) and emergency return trigger conditions in advance, and complete the entire process preparation before the operation.

[0042] II. Cleaning Operation

[0043] The fully automatic curtain wall cleaning drone and its system of the present invention cover the following core steps in sequence during the fully automatic cleaning operation process: take-off and path planning, flight attitude and safe distance maintenance, stain identification and cleaning parameter matching, dynamic spraying operation, cleaning quality inspection, final rinsing and return to base, equipment recovery and subsequent maintenance.

[0044] 1. Takeoff and Path Planning

[0045] The operator initiates the operation procedure through the ground station. The system then conducts a comprehensive self-check of the power unit, sensing system, processing and control system, and cleaning device. After all components have smooth communication links and no fault feedback, the operator determines the optimal hovering distance between the UAV and the curtain wall within a 3-5 meter range, taking into account environmental conditions such as wind direction and altitude. This distance must be compatible with the optimal ranging accuracy of the millimeter-wave radar and the effective field of view of the vision camera, while avoiding areas with unstable airflow such as strong winds and high altitudes. After determination, the relevant parameters are entered through the ground station.

[0046] After receiving the parameters, the system drives the six propellers to start synchronously, enabling the drone to take off smoothly. Relying on RTK real-time dynamic differential positioning technology, the drone can accurately capture its relative attitude to the curtain wall. After flying to the top of the curtain wall at a set distance, it maintains a stable hovering state parallel to the curtain wall. Its hovering accuracy can reach ±8 cm, and its position deviation is ≤3 cm within 1 minute. RTK technology works in tandem with the power system to achieve fuselage force balance. The specific logic is as follows: First, gravity balance: RTK provides real-time feedback on vertical deviations, dynamically adjusting the speed ratio of the six propellers to generate total lift equal to and opposite to the fuselage's weight, maintaining vertical attitude stability. Second, flushing and recoil force cancellation: The magnitude of recoil force is calculated in advance according to preset parameters (maximum 1.2MPa water jet pressure corresponds to 8N recoil force). RTK predicts the fuselage offset trend and increases the basic propeller speed by 10% to accurately cancel the impact of recoil force. Third, lateral force suppression: Relying on RTK positioning data combined with 50Hz high-frequency ranging data from millimeter-wave radar, the speed difference between the propellers on both sides of the fuselage (±5 rpm) is corrected in real time. An adjustment strategy of increasing speed on the windward side and decreasing speed on the leeward side is adopted, coupled with a symmetrical weight distribution design (center of gravity deviation ≤2cm), ensuring stable horizontal attitude without offset.

[0047] After the drone hovers stably atop the curtain wall, its visual camera automatically captures a panoramic image of the wall. The operator then manually marks and outlines the boundaries of the overall cleaning area via a ground station. Following this, the area is manually divided into several independent cleaning units according to a pre-set 2m x 2m standard, with a clear cleaning sequence from top to bottom and from left to right. The flight path is then preset via ground station software. Simultaneously, the preprocessing module of the multimodal data fusion and recognition algorithm is activated. By extracting regional contours, grayscale, and texture features from the images, it automatically identifies and marks prohibited areas to mitigate operational risks. Prohibited areas are non-functional areas and vulnerable parts of the curtain wall that do not require cleaning or are prohibited from being sprayed. These specifically include the area within 5 cm of the curtain wall edge, window frames and sealing strips, glass sealant joints, exposed pipe interfaces, decorative moldings, and other ancillary structures.

[0048] The system combines RTK positioning data to calibrate the precise coordinates of forbidden areas, generating a set of forbidden areas and integrating it into the flight path planning. During operation, the UAV automatically avoids forbidden areas, maintaining a safe distance of ≥10 cm. The spraying device automatically stops spraying when it approaches a forbidden area within 5 cm, and simultaneously compensates for the edge cleaning range by adjusting the path overlap rate of adjacent cleaning units to avoid cleaning blind spots. The final planned flight path is synchronously transmitted to the flight controller, providing reliable support for subsequent precise operations.

[0049] 2. Flight and Distance Maintenance

[0050] After flight path planning is completed, the drone flies smoothly and at a low speed along the preset route, with the flight speed consistently controlled at 0.3-0.5 m / s. This speed ensures both the uniformity of cleaning coverage and allows sufficient response time for subsequent stain identification and dynamic adjustment of cleaning parameters. During flight, the drone continuously feeds back its own position information using RTK real-time dynamic differential positioning technology, while simultaneously using 50Hz high-frequency real-time ranging data from millimeter-wave radar. Through dual data verification, closed-loop distance control is achieved, ensuring that the distance between the drone and the curtain wall always closely matches the operator's preset value, with a fluctuation error not exceeding ±0.05m.

[0051] If the millimeter-wave radar detects a sudden obstacle (such as an exposed pipe or temporary billboard) with a difference distance greater than 0.5m from the preset safety distance during flight (the obstacle detection signal is immediately transmitted to the processing and control system via the CAN bus), the system quickly encapsulates core data such as the obstacle's precise location and distance deviation, and then feeds it back to the ground station in real time via the wireless data transmission bus. After the operator manually confirms the detour path based on the obstacle information displayed on the ground station, the UAV precisely adjusts its flight attitude to complete obstacle avoidance, with a detour response time of ≤200ms. After avoiding the obstacle, the UAV immediately returns to the original preset route to continue the cleaning operation, with a return accuracy of ≤1cm, ensuring that no cleaning area is missed and the route is continuous and uninterrupted.

[0052] 3. Stain identification

[0053] like Figure 4 As shown, the curtain wall stain recognition of this invention relies on multimodal perception and data fusion technology. It simultaneously collects curtain wall images and distance data through a visual camera and millimeter-wave radar. After preprocessing such as Gaussian filtering for noise reduction, ROI cropping, and Grubbs criterion outlier removal, it accurately extracts 128-dimensional LBP texture features and 1-dimensional relative height features. These are then concatenated to form a fused feature vector, which is then input into an SVM classifier (parameters C=5, gamma=0.2). Finally, it achieves accurate determination of stain type (ordinary flat / three-dimensional raised) and degree of contamination (light / heavy).

[0054] (1) Data synchronous acquisition

[0055] After the drone smoothly enters the preset 2m×2m cleaning unit, it immediately starts multimodal perception data acquisition: the visual camera (1280×720 resolution, 20fps frame rate) mounted on the front of the drone collects visual images inside the cleaning unit, and the 24GHz millimeter-wave radar simultaneously collects the real-time distance sequence between the drone and the curtain wall at a sampling frequency of 50Hz. The two achieve precise timestamp alignment through a hardware synchronization trigger mechanism, with an alignment error of ≤80ms, providing a highly consistent and accurate data source support for subsequent curtain wall stain identification methods.

[0056] The output data specifically includes: a visual image pixel matrix (1280×720×1, grayscale image) and a real-time distance sequence (each group contains 100 consecutive sampling points, denoted as d1~d2). 100 This provides standardized input for subsequent data preprocessing.

[0057] (2) Data preprocessing

[0058] 1) Visual data preprocessing (Gaussian filtering and ROI cropping)

[0059] Gaussian filtering for noise reduction (kernel radius 3, 3×3 neighborhood): For each target pixel (x0, y0) in the visual image, take its 8 surrounding neighboring pixels, calculate the weight of each neighboring pixel according to the Gaussian function, and use the weighted sum as the new gray value of the target pixel.

[0060] The formula for neighboring pixel weights is: ; in, =3 (fixed kernel size), x and y are the coordinates of the neighboring pixels relative to the target pixel (values ​​{-1, 0, 1}), e≈2.718 (natural constant), x² and y² are the squares of the coordinates.

[0061] Formula for the new grayscale value of the target pixel: ; The gray values ​​of the nine neighboring pixels are multiplied by their corresponding weights to obtain a new gray value for the target pixel (x0, y0). .

[0062] ROI cropping: Based on manually annotated clean area boundaries, crop the effective area to 1200×800 pixels and remove irrelevant background.

[0063] 2) Real-time distance data preprocessing (Grubbs criterion for noise reduction)

[0064] According to Grubbs' criterion, a single group of 100 distance sampling points (d1~d2) were sampled. 100The noise reduction process is as follows: First, the mean and standard deviation of the distance sequence are calculated. Then, the Grubbs statistic of each sampling point is calculated point by point. Sampling points with a statistic greater than a preset threshold are identified as outliers and removed, while retaining the valid distance data. For the missing positions after the outlier removal, the mean of the two adjacent valid distance data is used to fill them, ensuring the integrity and continuity of the radar distance data sequence and providing a reliable data foundation for subsequent relative height feature extraction.

[0065] Mean of distance sequence :

[0066] The standard deviation s of the distance sequence: ; In the formula, d k This represents the k-th real-time distance; Grubbs statistic: ; Judgment rule: α=0.1, find G from the critical value table. 临界 ≈2.57, if the Grubbs statistic G of the k-th real-time distance is... k If the value is greater than 2.57, it is considered an outlier and removed.

[0067] The missing position data = (previous valid distance data + next valid distance data) / 2; when the missing position is filled with data, the data becomes valid distance data.

[0068] G 临界 It is derived from the Grubbs criterion critical value table, which is a statistical comparison table. The horizontal axis represents the significance level α, and the vertical axis represents the sample size n. In this scenario, α=0.1 and the sample size n=100, the table can be consulted to obtain the value. It is used to unify the criteria for judging outliers and avoid subjective bias.

[0069] (3) Feature extraction

[0070] 1) Visual LBP texture features (128 dimensions)

[0071] A 3×3 neighborhood coding mechanism is adopted, comparing the gray values ​​of 8 neighboring pixels with the center pixel pixel pixel pixel pixel by pixel: when the gray value of a neighboring pixel pixel is greater than or equal to the gray value of the center pixel pixel, it is recorded as 1; when the gray value of a neighboring pixel pixel is less than the gray value of the center pixel pixel, it is recorded as 0. An 8-bit binary number is generated and converted into a decimal value in the range of 0~255. The preprocessed ROI image is divided into 128 sub-blocks, and the coding histogram of each sub-block is calculated. Finally, a 128-dimensional LBP texture feature vector is formed, which provides the core basis for the determination of the degree of pollution.

[0072] 2) Radar relative altitude characteristics (1D)

[0073] According to the calculation formula h= -d0 extracts, where, The effective mean of the radar range sequence after noise reduction using the Grubbs criterion is d0, which is the hovering distance preset by the operator. Combined with a distance threshold of 3mm, when h>3mm, it is determined to be a three-dimensional raised stain feature; when h≤3mm, it is determined to be a normal planar stain feature, thus achieving accurate differentiation of the three-dimensional morphology of the stain.

[0074] (4) Classification decision

[0075] The 128-dimensional LBP texture feature vector and the 1-dimensional relative height feature are concatenated according to their dimensions to form a 129-dimensional fused feature vector. No dimensionality reduction processing is required. The vector is directly input into the basic SVM classifier (with preset fixed parameters: C=5, gamma=0.2). The classifier achieves accurate judgment through collaborative analysis of the fused features.

[0076] Stain type: Ordinary flat stain / 3D raised stain (based on relative height characteristics and a 3mm threshold).

[0077] Contamination level: Light contamination / Heavy contamination (based on the average gray-level difference corresponding to LBP texture features, determined by a contamination threshold of 8: average gray-level difference < 8 indicates light contamination, average gray-level difference ≥ 8 indicates heavy contamination).

[0078] 4. Dynamic spray washing operation

[0079] After the multimodal fusion recognition algorithm accurately determines the type and degree of stains, it feeds back the results to the ground station in real time via a wireless data transmission bus. The operator, considering on-site operating conditions (such as airflow stability and curtain wall material characteristics) and the control logic of the cleaning device, manually adjusts the spraying and flight parameters. The ground station then sends control commands to the UAV's processing and control system. The processing and control system then transmits these commands precisely to the corresponding actuators, such as the proportional solenoid valve and servo cylinder 53, via the CAN bus, enabling targeted cleaning. (1) Normal light pollution condition: The operator manually adjusts the opening of the electromagnetic regulating valve 44 to 60%, corresponding to a water spray pressure of 0.8MPa; sets the drone flight speed to 0.5m / s; controls the servo electric cylinder 53 to drive the water spray device 40 to swing back and forth within a range of ±25°, with a swing frequency of 1 time / second. Through the combination of the fan-shaped water spray surface and the swing action, the cleaning unit can achieve full coverage spraying without dead angles.

[0080] (2) Three-dimensional protrusion / heavy pollution condition: The operator manually adjusts the opening of the proportional electromagnetic regulating valve 44 to 100%, and the corresponding water spray pressure is increased to 1.2MPa; the drone flight speed is reduced to 0.2m / s to extend the spraying time; the water spray device 40 swing function is turned off (swing frequency 0 times / second), and the water spray angle is locked to perform 4 seconds of fixed-point high-pressure spraying on stubborn stains to ensure that the stains are completely removed; after the fixed-point spraying is completed, the operator manually restores the normal swing spraying mode and continues to clean the remaining areas in the unit.

[0081] During the spraying process, in order to maintain the preset hovering distance within the 3-5m range specified in claim 10 and ensure the stability of the spraying posture and the uniformity of the cleaning effect, the millimeter-wave radar collects the measured distance data between the machine body and the curtain wall in real time at a frequency of 50Hz. The processing and control system simultaneously activates the PID closed-loop correction algorithm to automatically correct the machine body position. The specific correction steps are as follows: Step 1: Real-time data acquisition and deviation calculation The millimeter-wave radar acquires the measured distance d at a frequency of 50 Hz (Δt = 0.02 s / time). 实 Simultaneously calculate the deviation e(t): e(t) = d0 - d 实 ; The direction of the fuselage offset is determined by the sign of the deviation (positive means deviation, negative means approach).

[0082] Step 2: Three-stage correction calculation (K) p =5.0 (proportionality coefficient), K i =0.1 (integral coefficient), K d =0.5 (differential coefficient)

[0083] Proportional link ( ): It responds quickly to deviations and counteracts the jetting recoil force.

[0084] Points system ( ): Accumulate small deviations to eliminate slight fluctuations caused by airflow disturbances and avoid fuselage shaking.

[0085] Differential element ( ): Anticipate deviation trends, suppress overshoot, and prevent deviation from expanding.

[0086] Step 3: Speed ​​Adjustment and Position Correction

[0087] The total speed regulation is obtained by superimposing the results of the three stages. The results are obtained by superimposing the results of the proportional, integral, and differential operations.

[0088] ; in, This represents the change over time.

[0089] The system adjusts the speed of the six propellers based on the total speed adjustment: when the fuselage deviates from the curtain wall, the speed of the corresponding propeller is increased to close the distance; when the fuselage is close to the curtain wall, the speed of the corresponding propeller is decreased to increase the distance. At the same time, the system adjusts the attitude by coordinating with the speed difference between the propellers on both sides of the fuselage to ensure that the distance between the fuselage and the curtain wall is stable within the preset value of ±0.05m, perfectly meeting the precision requirements of dynamic spraying.

[0090] 5. Cleaning quality inspection

[0091] like Figure 5 As shown, after each cleaning unit completes the spraying operation, the drone hovers in the unit area for 1-2 seconds (to allow time for image acquisition and data transmission), and the vision camera acquires the cleaning image of the unit. Then, the cleaning image is aligned with the reference image obtained before the operation (i.e., the image before cleaning) within the same ROI area. The alignment method adopts a dual collaboration of RTK positioning data and image feature point matching to ensure that the alignment accuracy is ≤2 pixels. Before alignment, the forbidden area described in claim 11 is automatically masked, and only the effective comparison area (ROI area excluding the forbidden area) is retained.

[0092] Based on the aligned image, the system executes a cleaning quality inspection process: on the one hand, it calculates the structural similarity SSIM value and the mean grayscale difference using the SSIM algorithm, which serve as the core quality judgment indicators; on the other hand, it compares the grayscale values ​​of the cleaned image with the reference image pixel by pixel, counts the number of residual pixels, and calculates the stain residue rate. The specific process is as follows: Step 1: Compare and locate the region Region shape and source: The comparison region directly reuses the 2m×2m square cleaning unit divided above, corresponding to the cropped 1200×800 pixel ROI region (completely consistent with the ROI region in the visual preprocessing stage).

[0093] Region alignment method: Based on RTK positioning data (error ≤ 8cm) and image feature points, double alignment is used to ensure that the comparison areas of the images before and after cleaning are completely overlapped: Taking the upper left corner pixel of the cleaning unit as the reference point, combined with the body position deviation fed back by radar ranging, the pixel coordinates of the image after cleaning are finely adjusted to compensate for the region misalignment caused by the slight offset of the UAV, with an alignment accuracy of ≤ 2 pixels.

[0094] Exclusion of prohibited areas: Before comparison, the marked prohibited areas (windows, pipes, etc.) are automatically blocked, and only the effective cleaned areas (ROI areas excluding prohibited areas) are compared to avoid non-cleaned areas from interfering with the test results.

[0095] Step 2: SSIM Dual-Indicator Comparison

[0096] The system uses two metrics: Structural Similarity SSIM (SMS) value and Mean Gray-Level Difference. These two metrics work together to determine the image's structural integrity and the difference in gray-level brightness, making it suitable for stain detection needs in curtain wall cleaning scenarios. Data extraction: Extract the grayscale value matrix of corresponding pixels from the effective regions before and after cleaning (image X) and after cleaning (image Y), and calculate the core parameters respectively. Gray-scale mean: μ x (Image X grayscale average value), μ y (Average grayscale value of image Y); ; ; In the formula, M×N is the effective area pixel size (in this invention, M=1200 and N=800). Let X be the grayscale value of the pixel in the i-th row and j-th column of the image. This represents the grayscale value of the pixel at the corresponding position Y in the image.

[0097] Gray standard deviation: σ x (Image X grayscale dispersion), σ y (The degree of grayscale dispersion in the image Y); ; ; Covariance: σ xy (The correlation between the gray values ​​of image X and Y reflects structural consistency.) ; SSIM value calculation: Substitute into the following formula to quantify the structural similarity between two images. The closer the value is to 1, the more consistent the structure of the cleaned image with the clean state (baseline before cleaning) and the less dirt residue there is.

[0098] ; In the formula, C1=(0.01×255)² and C2=(0.03×255)² are both fixed constants to avoid the denominator being 0 and to adapt to 8-bit grayscale images.

[0099] Gray-scale difference mean calculation: quantifies the difference in brightness between two images. The gray-scale value of the stained area usually deviates significantly from that of the clean area. The smaller the difference, the more thorough the cleaning.

[0100] ; Result determination: Based on the combined judgment of the two index thresholds, if SSIM≥0.92 and the average grayscale difference<12, it is determined that the cleaning meets the standard; if either index is not met, it is determined that the cleaning does not meet the standard, and a second spray cleaning is triggered.

[0101] Step 3: Calculation method for stain residue rate

[0102] The residual rate is calculated based on grayscale difference and binarization. Set grayscale difference threshold: Based on the previous classification of pollution level (light pollution grayscale difference < 8, heavy pollution ≥ 8), set the residue judgment threshold T=6 (taking into account both sensitivity and anti-interference).

[0103] Image binarization: The effective comparison area of ​​the cleaned image is binarized: when the difference between the gray value of a single pixel and the gray value of the corresponding pixel before cleaning is ≥T, it is determined as a "residual pixel" (denoted as 1); when the difference is <T, it is determined as a "clean pixel" (denoted as 0), and a binarized image is generated.

[0104] Statistical calculation: Stain residue rate = Total number of residual pixels / Total number of pixels in the effective contrast area × 100%; When the residual rate is ≤8%, the SSIM dual indicators are verified simultaneously. If the standard is met, the next cleaning unit is entered. When the residual rate is >8%, it is directly judged as not meeting the standard, triggering a second spray cleaning. A maximum of 2 spray cleanings are performed. If the standard is still not met, the location is recorded and a manual review is prompted.

[0105] 6. Final washing and return to port

[0106] After all cleaning units meet the standards, the system switches to clean water rinsing mode: the electromagnetic regulating valve 44 is adjusted to 50% opening, and the water spray pressure is reduced to 0.4MPa to avoid high pressure damage to the curtain wall. The drone reverses its original route, driving the water spray device 40 to swing and rinse, with the rinsing time set at 0.5 seconds per square meter to remove residual detergent.

[0107] After rinsing, a vision camera captures panoramic images, recording only the number of units that meet the cleaning standards and the locations that do not, generating a simplified quality report which is then transmitted to the ground station for archiving. Subsequently, using RTK positioning, a return flight path is planned, and the aircraft flies smoothly to the take-off and landing point, landing via the rubber buffers on its bottom to complete the operation.

[0108] 7. Equipment recycling and maintenance

[0109] After the drone lands, the operator shuts off the power and cleaning systems and disconnects the power supply from battery 20. ① Pipeline maintenance: Disassemble the water pipe 46, drain the residual water inside through the drain valve, rinse the pipeline with clean water and let it dry, check the unobstructed flow of the nozzle 43, and clear it if necessary.

[0110] ② Sensor maintenance: Wipe the radar antenna and camera lens with a lint-free cloth to remove surface dirt, and check that the lens is free of scratches and the radar is not obstructed.

[0111] ③ Power and control maintenance: Check the wear of the four propellers and replace any aging or damaged parts; check the battery charge level and charge it promptly if the remaining charge is less than 30%.

[0112] ④ Regular maintenance: Disassemble the steering mechanism 50 times per month, check the gear and rack transmission status, clean impurities as needed, no additional lubrication medium is required.

[0113] After maintenance is completed, record the parts replaced and the fault conditions, update the operation data through the ground station, optimize the parameter matching accuracy, and facilitate subsequent traceability.

[0114] In actual operation, the operator presets the scanning trajectory of the drone along the curtain wall via the ground station. The drone then performs cleaning operations layer by layer from the bottom of the curtain wall upwards according to the trajectory: First, the vision camera collects information on the stains on the wall in real time. After the processing and control system receives the data, it activates the water spray device 40. The washing mixture in the ground water tank is delivered to the duckbill nozzle 43 through the tethered water pipe 46 and sprayed out. At the same time, the servo cylinder 53 receives instructions from the control system and drives the active rack 52 to slide back and forth in a linear motion, which in turn drives the driven gear 51 to rotate back and forth within a set angle, ultimately realizing the dynamic oscillation and rinsing of the water spray device 40. To avoid blind spots and ensure full coverage, a 20% overlap rate is set between adjacent oscillation areas. If the vision camera detects stubborn stains, the control system immediately sends a signal to lock the position of the servo cylinder 53, and the water spray device 40 maintains a fixed angle to continuously rinse the stubborn stains with high pressure. After cleaning a single layer, the drone rises to the next layer and repeats the above process, with a 20% overlap rate also set between adjacent cleaning areas to ensure no cleaning is missed. After cleaning all the walls with the detergent mixture, switch the floor water tank to clean water and perform a thorough rinse following the same procedure to completely remove any residual detergent and complete the overall cleaning.

[0115] Actual testing has verified that, in a cleaning experiment of the exterior wall of a 20-story glass curtain wall building, this device reduces the operation time by more than 40% compared to traditional manual suspended platform cleaning. Furthermore, through multimodal perception and dynamic path control throughout the operation, the drone always maintains a safe distance from the curtain wall, with no collisions or scratches occurring, and no obvious watermarks or detergent residue. The cleaning quality and safety are significantly better than traditional methods.

[0116] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying curtain wall stains based on UAV multimodal perception, characterized in that, The identification steps include the following: A1. Visual images of the curtain wall surface and real-time distance sequences between the drone and the surface are simultaneously acquired by a visual camera and millimeter-wave radar mounted on a cleaning drone. A2. Perform Gaussian filtering on the visual image to remove noise and crop out the ROI region corresponding to the cleaned area; use the Grubbs criterion to remove outliers in the real-time distance sequence; A3. Extract the LBP texture feature vector from the visual image processed in step A2, and calculate the relative height feature using the real-time distance sequence processed in step A2. When the average gray-level difference of the LBP texture feature vector is less than the set contamination threshold, it is considered light contamination; otherwise, it is considered heavy contamination. When the relative height feature is greater than the distance threshold, it is considered three-dimensional raised contamination; otherwise, it is considered planar contamination. A4. The LBP texture feature vector and the relative height feature vector are concatenated and fused to form a fused feature vector, which is then input into the classifier to identify the type and degree of stains on the curtain wall.

2. The method for identifying curtain wall stain types based on UAV multimodal perception according to claim 1, characterized in that, The contamination threshold is 8, and the distance threshold is 3mm.

3. The method for identifying curtain wall stain types based on UAV multimodal perception according to claim 2, characterized in that, The specific steps for removing outliers using the Grubbs criterion are as follows: Calculate the mean and standard deviation of real-time distances in the real-time distance sequence; Based on the mean and standard deviation of the real-time distance, calculate the Grubbs statistic for each real-time distance in the real-time distance sequence; Real-time distances with Grubbs statistics exceeding a preset threshold are identified as outliers and removed.

4. The method for identifying curtain wall stain types based on UAV multimodal perception according to claim 3, characterized in that, The real-time distances that are not removed are considered valid real-time distances. For the gaps created after outliers are removed, the average of the adjacent valid real-time distances is used to fill them.

5. The method for identifying curtain wall stain types based on UAV multimodal perception according to claim 1, characterized in that, Relative height feature h= -d0, where, The real-time mean distance of the real-time distance sequence after removing outliers is d0, which is the preset hovering distance of the cleaning drone.

6. A method for detecting the quality of curtain wall cleaning based on multimodal recognition, characterized in that, The following testing steps are included: B1. Before carrying out the cleaning operation, a reference image of the target cleaning unit is obtained, and the type and degree of stains in the target cleaning unit are identified using a curtain wall stain type identification method based on UAV multimodal perception as described in any one of claims 1-5. B2. Based on the identification results of step B1, a cleaning drone is used to perform a cleaning operation on the target cleaning unit. B3. After the cleaning operation is completed, the cleaning images of the target cleaning unit are acquired through a vision camera; Align the cleaned image with the reference image within the same ROI region; B4. Based on the aligned image, calculate its structural similarity (SSIM) value and the mean grayscale difference. When the SSIM value is greater than or equal to the SSIM threshold and the mean grayscale difference is less than the grayscale difference threshold, the cleaning quality is deemed to meet the standard.

7. The method for detecting the quality of curtain wall cleaning based on multimodal recognition according to claim 6, characterized in that, The SSIM threshold is 0.92, and the grayscale difference threshold is 12.

8. The method for detecting the quality of curtain wall cleaning based on multimodal recognition according to claim 6, characterized in that, After the cleaning operation is completed, the stain residue rate is calculated: the gray values ​​of the cleaned image and the reference image are compared pixel by pixel, and pixels with a difference exceeding the preset residue judgment threshold are judged as residual pixels; The ratio of the total number of residual pixels to the total number of pixels is the stain residue rate.

9. The method for detecting the quality of curtain wall cleaning based on multimodal recognition according to claim 8, characterized in that, If the difference between the grayscale value of the current pixel in the cleaned image and the grayscale value of the corresponding pixel in the reference image is ≥6, it is determined to be a residual pixel; otherwise, it is determined to be a clean pixel.

10. A fully automated curtain wall cleaning drone system, characterized in that, include: Cleaning of the drone body, visual camera, millimeter-wave radar, cleaning device, and processing and control system; Both the visual camera and the millimeter-wave radar are installed at the front of the cleaning drone to simultaneously acquire visual images of the curtain wall surface and the real-time distance sequence between the drone and the curtain wall surface. The cleaning device is installed under the cleaning drone body and includes a high-pressure water spray device and a steering mechanism. The high-pressure water spray device is connected to the ground water tank through a tethered water pipe, and the steering mechanism is driven by a servo electric cylinder to adjust the water spray direction. The processing and control system is integrated inside the cleaning drone itself and is configured as follows: The curtain wall stain identification method based on UAV multimodal perception according to any one of claims 1-5 identifies the type and degree of stains on the curtain wall by fusing image data collected by a visual camera with distance data collected by millimeter-wave radar. Based on the stain identification results, control commands are generated to adjust the water spray pressure, spray mode, and flight speed of the cleaning drone of the cleaning device, and to control the cleaning device to perform cleaning operations on the target cleaning unit. The curtain wall cleaning quality detection method based on multimodal recognition according to any one of claims 6-9 involves acquiring images after cleaning and comparing them with a reference image to calculate the structural similarity SSIM value, the mean grayscale difference, and the stain residue rate, thereby determining whether the cleaning quality meets the standards.

Images