Method, device and robotic cleaning system for cleaning a surface of a building

Abstract

The application provides a building surface cleaning method, device and robot cleaning system, and belongs to the technical field of building surface cleaning. The method is applied to a robot cleaning system, the robot cleaning system comprises at least two unmanned aerial vehicles, the at least two unmanned aerial vehicles comprise a spraying machine and a cleaning machine, and the method comprises the following steps: controlling the spraying machine to pre-wash a building surface based on a first planning path; controlling the cleaning machine to scrub the building surface based on a second planning path; the second planning path and the first planning path are obtained based on a same preset initial planning path; and controlling the spraying machine to finally wash the building surface. Through the pre-washing, scrubbing and final washing processes of the building surface by the at least two unmanned aerial vehicles, manual cleaning operation is not needed, and completely autonomous cleaning of building surfaces with various structures is achieved.

Description

Technical Field

[0001] This application relates to the technical field of building surface cleaning, and more particularly to a method, apparatus and robotic cleaning system for cleaning building surfaces. Background Technology

[0002] The autonomy of robots has always been a significant challenge in the control systems industry, a challenge particularly evident in practical applications, such as building exterior cleaning. The exterior walls of high-rise buildings often serve as platforms for advertising and public relations, thus exhibiting diverse and unique structural features. As building renovation becomes an effective strategy for extending building lifespan, the cost of building surface maintenance is steadily increasing.

[0003] Currently, most building exterior cleaning methods, such as those using ropes, cable cars, and winch systems, rely on manual operation at height, which can endanger the safety of the operators. Furthermore, while some cleaning equipment can clean building surfaces, operators are still required to control the entire cleaning process to complete the task; autonomous cleaning is not yet possible. Summary of the Invention

[0004] In view of this, the present application provides a method, apparatus and robotic cleaning system for cleaning building surfaces, in order to solve the problem that cleaning building surfaces in the prior art requires manual operation and cannot achieve autonomous cleaning.

[0005] The first aspect of this application provides a method for cleaning building surfaces, applied to a robotic cleaning system. The robotic cleaning system includes at least two drones, each drone comprising a sprayer and a cleaning machine. The method includes: The sprayer is controlled to pre-wash the building surface based on a first planned path; The cleaning machine is controlled to scrub the building surface based on a second planned path; the second planned path and the first planned path are obtained based on the same preset initial planned path; Control the sprayer to give the building surface a final rinse.

[0006] In one possible implementation, the drone is equipped with a sensor assembly, which includes at least two of the following: lidar, inertial measurement unit, ultra-wideband antenna, global navigation satellite system, and camera sensor. The first or second planning path is used as the target planning path and is obtained as follows: Acquire at least two types of measurement data obtained through the sensor assembly; Data fusion processing is performed based on at least two types of measurement data to determine the pose state estimation information of the UAV; the pose state estimation information is an estimation of the position information and attitude information. Based on the pose state estimation information, the initial planned path is adjusted to obtain the target planned path.

[0007] In one possible implementation, data fusion processing is performed based on at least two types of measurement data to determine pose state estimation information, including: Using a keyframe alignment method, frequency alignment is performed on at least two types of measurement data to obtain at least two types of measurement data to be processed. Construct a factor graph based on at least two types of measurement data to be processed; Based on the factor graph, pose state estimation information is determined.

[0008] In one possible implementation, a factor graph is constructed based on at least two types of measurement data to be processed, including: Based on the measurement data to be processed obtained from the Global Navigation Satellite System, the double-difference pseudorange factor is determined; the double-difference pseudorange factor is an observation model used to calculate the pseudorange measurement from the receiver to the satellite at a specific time. Based on the measurement data to be processed obtained from the Global Navigation Satellite System, the Doppler observation function is determined as the Doppler observation factor; the Doppler observation factor is used to correct the double-difference pseudorange factor. Based on the measurement data obtained from the inertial measurement unit, the inertial factor is determined; the inertial factor is used to adjust the changes in the observed values ​​of acceleration and angular velocity. Based on the measurement data obtained from the camera sensor, a visual factor is determined; the visual factor is used to adjust the pose state information of the camera sensor. Based on the measurement data to be processed obtained from lidar measurements, lidar factors are determined; lidar factors are used to adjust the pose state information of the lidar. Based on the measurement data to be processed obtained from the ultra-wideband antenna, the ultra-wideband factor is determined; the ultra-wideband factor is used to perform positioning compensation on the pose state information of the UAV. Using the Schur complement method, during the frequency alignment process of at least two types of measurement data, the constraints of keyframes of the measurement data are marginalized to determine the marginalization factor; the marginalization factor is used to marginalize the adjustment of the factor map. The factor map is determined based on the dual-difference pseudorange factor, Doppler observation factor, inertial factor, visual factor, lidar factor, ultra-wideband factor, and marginalization factor.

[0009] In one possible implementation, the lidar factor includes a first lidar factor and a second lidar factor; Based on the measurement data to be processed obtained from lidar measurements, lidar factors are determined, including: Based on the measurement data to be processed obtained by lidar measurement and the lidar's preset scan-to-map matching model, a first lidar factor is determined; the first lidar factor is used to adjust the lidar's pose state information based on the local point cloud map. Based on the measurement data to be processed obtained from lidar measurement and the lidar's preset scan-to-multi-scan model, a second lidar factor is determined; the second lidar factor is used to adjust the lidar's pose state information through the relative constraints between adjacent keyframes.

[0010] In one possible implementation, the initial planned path is adjusted based on pose state estimation information to obtain the target planned path, including: Based on pose state estimation information and a pre-set target reinforcement learning model, the initial planned path is adjusted to obtain the target planned path; The target reinforcement learning model is obtained by training the initial reinforcement learning model using preset constraints, preset multi-index reward functions, and UAV flight sample data.

[0011] In one possible implementation, the initial reinforcement learning model is constructed as follows: An initial reinforcement learning model is constructed based on the model state, UAV action information, UAV transfer probability, reward information, and discount factor. The model state includes the drone's current pose state information, the proportion of cleaned grids in the current area, and the drone's remaining energy. The drone action information includes the drone's historical path. The drone transfer probability is used to represent the probability of the drone's next displacement based on the current pose state information and historical path. The reward information is used to comprehensively evaluate the drone's cleaning efficiency, energy consumption, obstacle avoidance performance, and task completion. The discount factor is used to measure the importance of future rewards relative to immediate rewards.

[0012] In one possible implementation, constraints include: the drone maintaining a safe distance from any obstacle or building surface, and energy limits through the energy consumption function of each drone action; The multi-metric reward function includes: cleaning efficiency reward, energy penalty, collision penalty, and task completion reward; Cleaning efficiency reward is used to represent the reward for the incremental improvement in mesh clearing in each displacement step, energy penalty is used to punish the energy consumed in the previous action, collision penalty is used to punish the drone for colliding with obstacles or building surfaces, and mission completion reward is used to reward the degree to which the drone completes the mission.

[0013] In one possible implementation, the cleaning machine is controlled to scrub the building surface based on a second planned path, including any of the following: In response to the first position information sent by the sprayer, when it is determined that the sprayer has moved to the predetermined position, the cleaning machine is controlled to scrub the building surface based on the second planned path; In response to the sprayer starting pre-rinsing time reaching the preset time, the cleaning machine is controlled to scrub the building surface based on the second planned path; In response to the start command for the cleaning machine, control the cleaning machine to scrub the building surface based on the second planned path.

[0014] In one possible implementation, the method also includes: Acquire information on the drone's battery level, water tank level, and pose status; If at least one of the drone's battery level, water tank water level, and pose status information meets the corresponding return conditions, a return command is sent to the drone so that the drone can return based on the return command.

[0015] A second aspect of this application provides a cleaning device for building surfaces, applied to a robotic cleaning system. The robotic cleaning system includes at least two drones, each drone including a sprayer and a cleaning machine. The device includes: The first control module is used to control the sprayer to pre-wash the building surface based on the first planned path; The second control module is used to control the cleaning machine to scrub the building surface based on the second planned path; the second planned path and the first planned path are obtained based on the same preset initial planned path; The third control module is used to control the sprayer to perform a final rinse on the building surface.

[0016] A third aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the first aspect.

[0017] A fourth aspect of this application provides a robotic cleaning system, comprising: at least two drones and a terminal device of the third aspect, wherein the at least two drones include a sprayer and a cleaner.

[0018] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method of the first aspect.

[0019] Compared with the prior art, the embodiments of this application have at least the following technical effects: The building surface cleaning method of the first aspect of this application can be applied to a robotic cleaning system. The robotic cleaning system includes at least two drones, each comprising a sprayer and a cleaning machine. The building surface cleaning method involves controlling the sprayer to pre-rinse the building surface based on a first planned path; then, controlling the cleaning machine to scrub the building surface based on a second planned path, where the second and first planned paths are derived from the same preset initial planned path; finally, controlling the sprayer to perform a final rinse. Therefore, this application embodiment, through the pre-rinsing, scrubbing, and final rinsing of the building surface using at least two drones, eliminates the need for manual cleaning, achieving fully autonomous cleaning of building surfaces of various structures.

[0020] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a robotic cleaning system provided in an embodiment of this application; Figure 2 This is a flowchart of a method for cleaning building surfaces provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a method for determining a target planning path according to an embodiment of this application; Figure 4 This is a flowchart illustrating a method for obtaining a target reinforcement learning model, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of the structure of a building surface cleaning device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] In the description of this application, unless otherwise stated, the " / " used in this specification and appended claims indicates that the related objects are in an "or" relationship. For example, A / B can mean A or B. The "and / or" in this application merely describes the relationship between the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c. Here, a, b, and c can be single or multiple.

[0027] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0028] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] The technical solution of this application and how it solves the above-mentioned technical problems are described in detail below with specific embodiments. It should be noted that the following embodiments can be referenced, borrowed, or combined with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be described again.

[0031] See Figure 1 As shown, this application provides a schematic diagram of the structure of a robot cleaning system 10. Figure 1 As shown, the robotic cleaning system 10 includes at least two drones and a terminal device 13 according to the present application embodiment. The at least two drones include a sprayer 11 and a cleaner 12.

[0032] The sprayer 11 is used to pre-wash the building surface based on a first planned path; the cleaning machine 12 is used to scrub the building surface based on a second planned path. The sprayer 11 is also used to perform a final rinse on the building surface. The second planned path and the first planned path are derived from the same preset initial planned path, and the sprayer 11 and the cleaning machine 12 can perform pre-washing and scrubbing on the building surface based on the same initial planned path.

[0033] Terminal device 1 is communicatively connected to sprayer 11 and cleaner 12 respectively. Terminal device 13 can control the working process of sprayer 11 and cleaner 12, and can also control sprayer 11 and cleaner 12 in case of abnormality. It can also control the collaborative division of labor process of sprayer 11 and cleaner 12.

[0034] Optionally, the sprayer 11 and the cleaner 12 can also communicate with each other. For example, the sprayer 11 and the cleaner 12 can exchange position information to avoid collisions.

[0035] The pre-rinse and final rinse can be completed by two different sprayers 11. If the cleaning machine 12 has a water spraying function, it can also be used as a sprayer 11. Both the sprayer 11 and the cleaning machine 12 are equipped with water tanks. The sprayer 11 is equipped with spraying components for spraying water, and the cleaning machine 12 is equipped with cleaning components for cleaning walls or glass surfaces.

[0036] See Figure 2 As shown, this application provides a flowchart of a method for cleaning building surfaces. Figure 2 As shown, the robot cleaning system 10 includes at least two drones, each drone including a sprayer 11 and a cleaning machine 12. The cleaning method for building surfaces includes steps S201 to S203.

[0037] S201, Control the sprayer 11 to pre-wash the building surface based on the first planned path.

[0038] Optionally, the sprayer 11 pre-washes the building surface to facilitate subsequent scrubbing by the cleaning machine 12. The pre-washing process of the sprayer 11 can be controlled by the terminal equipment 1 or by the controller of the sprayer 11 itself.

[0039] S202, the cleaning machine 12 is controlled to scrub the building surface based on the second planned path. The second planned path and the first planned path are obtained based on the same preset initial planned path.

[0040] Optionally, the cleaning machine 12 and the sprayer 11 can move along the same preset initial planned path. During the movement, the initial planned path can be adjusted accordingly based on information about obstacles and the cleaning status to obtain the first and second planned paths for actual movement, which can improve cleaning efficiency and ensure cleaning coverage.

[0041] In practical applications, the initial planned path can be determined by dividing the building surface into multiple grids and using an ultra-wideband approach. The initial planned path can resemble the shape of a pulse wave.

[0042] Optionally, the pre-rinsing process of the cleaning machine 12 on the building surface can be controlled by the terminal device 1 or by the controller of the cleaning machine 12 itself.

[0043] S203, control the sprayer 11 to perform final rinsing of the building surface.

[0044] Optionally, the sprayer 11 can perform a final rinse on the building surface based on a third planned path, which can also be derived from the same preset initial planned path. In practical applications, the sprayer 11 can also adjust the planned path based on the cleanliness of the building surface to ensure that the building surface is clean.

[0045] The building surface cleaning method of this application embodiment can be applied to a robotic cleaning system 10. The robotic cleaning system 10 includes at least two drones, each including a sprayer 11 and a cleaning machine 12. The building surface cleaning method can control the sprayer 11 to pre-rinse the building surface based on a first planned path; then, control the cleaning machine 12 to scrub the building surface based on a second planned path; the second planned path and the first planned path are obtained based on the same preset initial planned path; and finally, control the sprayer 11 to perform a final rinse on the building surface. This application embodiment achieves fully autonomous cleaning of building surfaces of various structures by using at least two drones for pre-rinsing, scrubbing, and final rinsing of the building surface, eliminating the need for manual cleaning.

[0046] In some embodiments, the drone is equipped with a sensor assembly, which includes at least two of the following: lidar, inertial measurement unit, ultra-wideband antenna, global navigation satellite system, and camera sensor.

[0047] The first or second planning path is used as the target planning path and is obtained as follows: Acquire at least two types of measurement data obtained through the sensor assembly; Data fusion processing is performed based on at least two types of measurement data to determine the pose state estimation information of the UAV; the pose state estimation information is an estimation of the position information and attitude information. Based on the pose state estimation information, the initial planned path is adjusted to obtain the target planned path.

[0048] The path planning principles of the sprayer 11 and the cleaner 12 in this application embodiment are the same. They both obtain pose state estimation information through the measurement data of the sensor components, and then obtain the target planned path.

[0049] The embodiments of this application can use at least two types of measurement data for data fusion processing to determine the pose state estimation information of the UAV, which serves as the real-time pose information of the UAV.

[0050] In practical applications, embodiments of this application can employ an onboard navigation UAV with a multi-sensor fusion positioning system. The multi-sensor fusion positioning system includes a composite positioning module responsible for processing raw data from various sensors, a decision module for transmitting action commands to the flight control system, and a motion module that controls the UAV's dynamics through a series of driver programs integrated into the UAV frame.

[0051] Furthermore, embodiments of this application can employ a sophisticated multi-sensor SLAM module, integrating LiDAR, an Inertial Measurement Unit (IMU), Ultra Wide Band (UWB), Global Navigation Satellite System (GNSS), and a camera sensor (Visual Inertial Odometry (VIO)). Through this advanced framework, these diverse inputs are harmoniously combined to generate a dynamic 3D map of the operational environment. This capability facilitates precise navigation and obstacle detection, ensuring optimal performance of the UAV in both GNSS-supported and GNSS-free environments.

[0052] In some embodiments, data fusion processing is performed based on at least two types of measurement data to determine pose state estimation information, including: Using a keyframe alignment method, frequency alignment is performed on at least two types of measurement data to obtain at least two types of measurement data to be processed. Construct a factor graph based on at least two types of measurement data to be processed; Based on the factor graph, pose state estimation information is determined.

[0053] This application embodiment can also utilize a 3D map to establish a preliminary initial planned path, taking into account distance constraints and environmental characteristics. The initial planned path will be adjusted in real time based on the input from the sensor components to cope with sudden obstacles or changes in the clean environment.

[0054] See Figure 3 As shown, this application provides a flowchart of a method for determining a target planning path. Figure 3 As shown, the method for determining the target planning path includes steps S301 to S305.

[0055] S301. Acquire at least two types of measurement data measured by the sensor assembly.

[0056] S302. Using a keyframe alignment method, frequency alignment is performed on at least two types of measurement data to obtain at least two types of measurement data to be processed.

[0057] S303. Construct a factor graph based on at least two types of measurement data to be processed.

[0058] S304. Based on the factor graph, determine the pose state estimation information.

[0059] S305. Based on the pose state estimation information, adjust the initial planned path to obtain the target planned path.

[0060] This application embodiment takes a sensor component including a lidar, an inertial measurement unit, an ultra-wideband antenna, a global navigation satellite system, and a camera sensor as an example to construct a factor graph.

[0061] In some embodiments, a factor graph is constructed based on at least two types of measurement data to be processed, including steps (1) to (8).

[0062] (1) Based on the measurement data to be processed obtained from the Global Navigation Satellite System, determine the double-difference pseudorange factor; the double-difference pseudorange factor is an observation model used to calculate the pseudorange measurement from the receiver to the satellite at a specific time.

[0063] Optionally, the double-difference pseudorange factor is an observation model used to calculate the pseudorange measurement from the receiver to the satellite at a specific time. It considers geometric distance, ionospheric delay, tropospheric delay, and various error sources, including non-line-of-sight (NLOS) or multipath reception, receiver and antenna-related noise errors. The double-difference pseudorange factor plays a crucial role in accurately estimating the receiver position in a GNSS system. The pseudorange observation function is:

[0064] in, Indicates receiver At any moment From satellite The received pseudorange measurement. The actual geometric distance is determined by... Given. Ionospheric delay and tropospheric delay are respectively given by and This indicates that the remaining errors, including those caused by non-line-of-sight or multipath reception, receiver and antenna-related noise errors, are... express.

[0065] To prevent serious errors introduced by the system due to clock synchronization problems or atmospheric delays, this application employs double-difference (DD) technology, which involves using a receiver (user receiver) to... Between the satellite and the reference station, and between the satellite and the reference station. ) and main satellite ( Two single differences are performed between the two points. The satellite with the highest elevation angle at the corresponding time is selected as the primary satellite to reduce the impact of multipath and non-line-of-sight reception.

[0066] (2) Based on the measurement data to be processed obtained from the Global Navigation Satellite System, the Doppler observation function is determined as the Doppler observation factor; the Doppler observation factor is used to correct the double difference pseudorange factor.

[0067] Optionally, the Doppler observation function is:

[0068] in, This represents the error term with noise. Represents the speed of light. It is the carrier wavelength of the signal. The receiver velocity in the geocentric-fixed coordinate system changes through the state in the East-North-Up coordinate system. get. It is a line-of-sight unit vector pointing towards the satellite, indicating the direction from the receiver to the satellite.

[0069] (3) Determine the inertial factor based on the measurement data to be processed obtained by the inertial measurement unit; the inertial factor is used to adjust the changes in the observed values ​​of acceleration and angular velocity.

[0070] Optionally, the inertial measurement unit (IMU) provides observations of acceleration and angular velocity, which are affected by bias and additive noise. Its higher measurement frequency compared to other sensing and positioning sensors such as lidar, cameras, and GNSS has spurred the development of pre-integration methods, which are widely used in efficient sensor fusion. This method pre-integrates multiple raw measurements to establish keyframes. To its adjacent next frame Relative pose constraints.

[0071] (4) Based on the measurement data to be processed obtained by the camera sensor, determine the visual factor; the visual factor is used to adjust the pose state information of the camera sensor.

[0072] Optionally, preprocessing and initialization are crucial before constructing visual factors. Whenever a new image is acquired, the KLT sparse optical flow algorithm is used to track existing features while detecting new corner features to keep the number of features in each frame to a minimum. The detector ensures uniform feature distribution by setting the minimum pixel distance between neighboring feature points. Two-dimensional features are first distortion-corrected, then projected onto a unit sphere after outlier removal. Outlier removal employs RANSAC combined with a fundamental matrix model. After preprocessing, vision-only structure-of-motion (SfM) is initiated to jointly estimate scale, motion, and structure. Subsequently, initialization is performed by aligning the IMU trajectory with the SfM results to recover scale, velocity, gravity, and IMU bias. Visual factors are essentially reprojection errors, i.e., the difference between theoretical values ​​projected onto the image plane and actual observations.

[0073] (5) Based on the measurement data to be processed obtained by lidar measurement, determine the lidar factor; the lidar factor is used to adjust the pose state information of lidar.

[0074] In some embodiments, the lidar factor includes a first lidar factor and a second lidar factor. Based on the measurement data to be processed obtained from lidar measurements, lidar factors are determined, including: Based on the measurement data to be processed obtained from lidar measurement and the lidar's preset scan-to-map matching model, a first lidar factor is determined; the first lidar factor is used to adjust the lidar's pose state information based on the local point cloud map.

[0075] Based on the measurement data to be processed obtained from lidar measurement and the lidar's preset scan-to-multi-scan model, a second lidar factor is determined; the second lidar factor is used to adjust the lidar's pose state information through the relative constraints between adjacent keyframes.

[0076] For the first lidar factor, this application embodiment evaluates odometry performance using edge features and surface features respectively, finding that surface features offer better robustness and higher accuracy. Therefore, surface points are used in the implementation, excluding edge points. To select surface feature points, local regions are extracted from the keyframe point cloud, their local distribution is evaluated, and points satisfying the planar distribution are selected. Then, a nearest neighbor search method is used to find the correspondence between the source point cloud and the target point cloud (from the keyframe and the local map). Finally, based on the keyframe... Feature points in Corresponding feature points in the local point cloud map , Calculate the point cloud matching residual from point to plane, representing the planar patch in the local point cloud map. : (3)

[0077] Among them, the plane points in the local sensor coordinate system are formed by It means that, through Transform it from the LiDAR coordinate system to the IMU coordinate system, and then through Transform from the body coordinate system to the ENU coordinate system.

[0078] For the second lidar factor, since the local point cloud map is fixed, lidar constraints based on the scanned map can essentially be considered absolute constraints without global positioning information. This approach has problems in large-scale integration because it cannot guarantee drift-free alignment between local and global coordinates. To address this issue, this application proposes using multi-scan lidar factors for tight coupling to maintain global consistency through absolute constraints from GNSS, per keyframe. its adjacent Keyframes are correlated to provide high-precision scan-to-scan plane constraints, using the same planar model and involving the states of both the target and source keyframes. To enhance the robustness of the system, this application also utilizes relative attitude constraints.

[0079] The multi-scan lidar factor proposed in this application provides only relative constraints, thereby achieving consistent integration of global GNSS and local lidar. Therefore, this application achieves smoother local trajectories and better robustness to outliers.

[0080] (6) Based on the measurement data to be processed obtained by the ultra-wideband antenna, determine the ultra-wideband factor; the ultra-wideband factor is used to perform positioning compensation for the pose state information of the UAV.

[0081] Optionally, the utilization of the ultra-wideband factor, combined with line-of-sight (LOS) measurements filtered through non-line-of-sight (NLOS) identification and laser SLAM 6-DOF attitude estimation, provides accurate positioning compensation for tightly coupled systems when all other NLOS measurements are excluded. It is important to note that the positioning accuracy of the integrated system is affected not only by the accuracy of the UWB measurements but also by the spatial arrangement of the UWB anchor points. If the geometry of the remaining LOS anchor points is poor, relying solely on LOS measurements may unintentionally introduce significant positioning errors.

[0082] UWB Mobile Tag With the LOS anchor points The actual distance between It can be represented as:

[0083] Ignoring components of order two and above after Taylor series expansion, we get:

[0084] The error covariance matrix Q and HDOP values ​​are listed below:

[0085] (7) Using the Schul complement method, during the frequency alignment of at least two types of measurement data, the constraints of the key frames of the measurement data are marginalized to determine the marginalization factor; the marginalization factor is used to marginalize the adjustment of the factor map.

[0086] To reduce computational complexity while preserving the impact of previous information constraints, this application employs marginalization during the first-stage sliding window optimization process. This involves progressively marginalizing the constraints of previous keyframes using the Schur complement method. After marginalizing the constraints of older keyframes, new prior factors are added to the updated window.

[0087] (8) Determine the factor map based on the double-difference pseudorange factor, Doppler observation factor, inertial factor, visual factor, lidar factor, ultra-wideband factor and marginalization factor.

[0088] The sensor data processing in this embodiment mainly includes three steps: data preprocessing, factor graph construction, and sliding window-based factor graph optimization fusion. The system inputs include UWB measurements, GNSS measurements (raw pseudorange and Doppler measurements), LiDAR measurements, camera measurements, and IMU measurements (initial sensor data). To efficiently handle the computational load, a keyframe mode is used for frequency alignment. The raw IMU measurements are converted into relative constraints using a pre-integration method, and the initial pose of the keyframes is propagated.

[0089] Furthermore, this application embodiment constructs a factor graph, including double-difference (DD) pseudorange factors, Doppler factors, lidar factors, and visual factors. After preprocessing, the pose state is collectively optimized using Factor Graph Optimization (FGO). The second-stage batch optimization is performed at specific intervals. In batch fusion, a large factor graph containing constraints similar to GNSS and IMU is maintained (by continuously optimizing the factor graph and saving trajectory history). Moreover, the lidar factors are designed as a scan-to-multi-scan scheme to avoid local minima and ensure global consistency.

[0090] The factor map in this embodiment is obtained by fusing the factor map in the first stage with the factor map in the second stage. The optimized factor map in the two stages omits the marginalization factors of the first stage fusion based on the sliding window. The main difference in the first stage lies in the formulation of the lidar factors. The first stage uses a short sliding window for efficient estimation. Within a short time window, GNSS measurements are limited and susceptible to outliers. LiDAR measurements, on the other hand, can maintain high accuracy within a short time. Therefore, a lidar factor based on a scan-to-map scheme is adopted, where the local map is constructed from previous keyframe states as high-quality prior information. In this way, GNSS measurements can assist laser inertial odometry (LIO) in achieving reliable positioning and velocity estimation in highly dynamic environments. At the same time, visual inertial odometry (VIO) provides map texture and color based on the map information constructed by LIO.

[0091] On the other hand, in this embodiment, each lidar keyframe is associated with 12 adjacent keyframes. For each pair of frames, only 25 planar features are randomly selected to reduce computational load and balance the weights among different factors. Furthermore, the second-stage batch optimization iterates three times for outlier detection and elimination based on the residual. The factor graph in the second stage must significantly enhance the constraint redundancy of the state while maintaining global consistency, thereby achieving accurate outlier detection and elimination. In short, the second-stage batch optimization can utilize sufficient and healthy GNSS measurements over a long period, combined with strong relative attitude constraints provided by lidar and IMU, ultimately achieving globally consistent and locally robust optimization. The fusion in the second stage is performed in another thread, and the global state is updated once optimization is complete.

[0092] In some embodiments, the initial planned path is adjusted based on pose state estimation information to obtain the target planned path, including: Based on pose state estimation information and a pre-set target reinforcement learning model, the initial planned path is adjusted to obtain the target planned path; The target reinforcement learning model is obtained by training the initial reinforcement learning model using preset constraints, preset multi-index reward functions, and UAV flight sample data.

[0093] See Figure 4 As shown, this application embodiment provides a flowchart of a method for obtaining a target reinforcement learning model. Figure 4 As shown, the method for obtaining the target reinforcement learning model includes steps S401 to S402.

[0094] S401. Based on the model state, UAV action information, UAV transfer probability, reward information, and discount factor, construct an initial reinforcement learning model.

[0095] Specifically, the initial reinforcement learning model is constructed as follows: An initial reinforcement learning model is constructed based on the model state, drone action information, drone transfer probability, reward information, and discount factor.

[0096] The model state includes the drone's current pose state information, the proportion of cleaned grids in the current area, and the drone's remaining energy. The drone action information includes the drone's historical path. The drone transfer probability is used to represent the probability of the drone's next displacement based on the current pose state information and historical path. The reward information is used to comprehensively evaluate the drone's cleaning efficiency, energy consumption, obstacle avoidance performance, and task completion. The discount factor is used to measure the importance of future rewards relative to immediate rewards.

[0097] S402. Using preset constraints, preset multi-index reward functions, and UAV flight sample data, the initial reinforcement learning model is trained to obtain the target reinforcement learning model.

[0098] The path planning task in this embodiment can be defined as a Markov decision process with multiple key objectives: cleaning efficiency to maximize the removal of surface mesh, energy saving to minimize system energy consumption, obstacle avoidance to prevent the UAV from colliding with obstacles or building structures, and task completion to ensure that the UAV trajectory covers key areas within battery or time constraints. As these objectives are transformed into different constraints, the mathematical model of the system is also established.

[0099] set up Let be a state space, where one state is a state space. express: Drone location:

[0100] Cleanliness status: This indicates the amount of grid space cleared in each building area. Because the target cleaning area is divided into... Discrete units It can be described as a vector ,in, In the region The percentage of the grid that has been cleared; Remaining energy: Used to track battery limits in the current state. Therefore, it is described by the following function:

[0101] In some embodiments, the constraints include: maintaining a safe distance between the drone and any obstacle or building surface, and an energy limit through the energy consumption function of each drone action; The multi-metric reward function includes: cleaning efficiency reward, energy penalty, collision penalty, and task completion reward; Cleaning efficiency reward is used to represent the reward for the incremental improvement in mesh clearing in each displacement step, energy penalty is used to punish the energy consumed in the previous action, collision penalty is used to punish the drone for colliding with obstacles or building surfaces, and mission completion reward is used to reward the degree to which the drone completes the mission.

[0102] Furthermore, this application provides a method for obtaining a target reinforcement learning model.

[0103] 1. Constraints Several constraints need to be followed to optimize the drone trajectory. The constraints are defined as follows: (1) Collision avoidance: The drone must not violate the safe distance from any obstacle or building surface. Therefore, the corresponding expression is:

[0104] When collision constraints are violated, the reward function will be significantly penalized.

[0105] (2) Energy / battery limitations:

[0106] in, This is the energy consumption function for each action of the drone system. Mesh clearing / cleaning efficiency: The cleaning target is focused on clearing the mesh. If... Indicates in the region The proportion of cleared grid cells in the time frame... Overall cleaning efficiency It can be represented as:

[0107] A completely clean building means .

[0108] 2. Modeling of Multi-Indicator Reinforcement Learning Models This application models an initial reinforcement learning model MDP. The model state can be represented as The actions taken by the drones were Drone transfer probability Describes the transition to a position given the current pose state information and the historical path. The probability. Reward. The discount factor reflects the cost of the coverage process and actions. .

[0109] 3. Multi-indicator reward function To combine cleaning efficiency, energy saving, obstacle avoidance, and task completion, this application can combine multiple items into a single scalar reward at each time step: (12) in, It's a cleaning efficiency reward. It's an energy penalty. It's a collision penalty. It represents the task completion rate.

[0110] (1) Cleaning efficiency reward As mentioned above, cleaning efficiency bonus It is an incremental improvement in mesh clearing at each step. It can be represented as Cleaning (t). This is a weight used to measure the degree of importance this application places on improving cleaning efficiency. Therefore, Always greater than 0.

[0111] (2) Energy penalty Energy Punishment It's the energy consumed in the last operation. It can be represented as It is a weight that encourages energy conservation, therefore It is always positive. Because subtraction is performed here, the more energy used, the lower the reward.

[0112] (3) Collision penalty Collision Penalty The penalty value is usually 0 when no collision occurs, but it is larger if the drone collides with or gets too close to an obstacle. It is designed to severely penalize collisions, always being greater than 0.

[0113] (4) Task completion rate Task completion rate Is the drone completely responsible for the cleaning target (e.g.) (or a large positive reward when an intermediate sub-goal is achieved.) The incentive for completing the task in this application is controlled to remain positive at all times.

[0114] Since this application aims to maximize the expected discounted return of the drone, the objective can be described as follows:

[0115] in, and It's a strategy involving drones.

[0116] Furthermore, embodiments of this application also provide a mathematical optimization of the Bellman framework.

[0117] (1) Value-based approach If the action space is discrete (e.g., moving within a grid), this application can define an action-value function: (14) optimal Satisfies the Bellman optimality equation:

[0118] It can be approximated by a neural network (DQN). Then select:

[0119] (2. Policy Gradient Method) If the action is continuous (e.g., 3D velocity commands), this application uses a policy gradient or actor-critic approach. This application can define and optimize parameterized policies. (e.g., neural networks):

[0120] Use gradient-based updates:

[0121] in It is an advantage estimate.

[0122] Furthermore, embodiments of this application also provide a reinforcement learning process, including the following steps: (a) Simulation environment: The environment is initialized by modeling the building structure, defining potential obstacles, and configuring the UAV dynamics using physical methods, including a mesh map representing the cleaning target. .

[0123] (b) Initialization: The UAV starts from its initial position. Initially, the battery is fully charged. .

[0124] (c) State and Action Space: The system state is defined as follows: ,action By strategy Decide.

[0125] (d) State transition: The drone moves to the next location. , The update reflects the removal of the mesh after cleaning, and the energy consumption is calculated as follows: Consider the energy consumption during operation.

[0126] (e) Reward function: The reward at each time step is defined in equation (12), where It's a cleaning efficiency reward. It's an energy penalty. It's a collision penalty. This is a reward for completing the mission. The collision penalty is set relatively high, but the mission reward increases significantly as the cleaning coverage approaches 100%.

[0127] (f) Termination conditions: The training round is terminated upon the occurrence of a collision, safety violation, energy depletion, or the achievement of the expected cleaning efficiency (e.g., 99%).

[0128] (g) Policy optimization: Based on the action space, perform function update or policy gradient steps. Train over multiple rounds to iteratively optimize the policy.

[0129] In some embodiments, the cleaning machine 12 is controlled to scrub the building surface based on a second planned path, including any of the following: In response to the first position information sent by the sprayer 11, when it is determined that the sprayer 11 has moved to the predetermined position, the cleaning machine 12 is controlled to scrub the building surface based on the second planned path; In response to the sprayer 11 starting the pre-rinse time reaching the preset time, the cleaning machine 12 is controlled to scrub the building surface based on the second planned path; In response to the start command for the cleaning machine 12, the cleaning machine 12 is controlled to scrub the building surface based on the second planned path.

[0130] In this embodiment, the sprayer 11 can be moved to a predetermined position, the sprayer 11 can be started to pre-rinse for a preset time, and the cleaning machine 12 can be started manually. After the sprayer 11 has been pre-rinsing for a period of time, the cleaning machine 12 can be controlled to perform wiping, so as to ensure the distance between the sprayer 11 and the cleaning machine 12, avoid collision and improve work efficiency.

[0131] It is conceivable that using the sprayer 11 for final rinsing can also achieve the same distance from the cleaning machine 12 in the above manner, thus avoiding collisions while improving work efficiency.

[0132] In some embodiments, the method further includes: acquiring the drone's battery power information, water tank water storage information, and pose status information; If at least one of the drone's battery level, water tank water level, and pose status information meets the corresponding return conditions, a return command is sent to the drone so that the drone can return based on the return command.

[0133] This application embodiment can detect the drone's battery level, water tank level, and pose status in real time. When the drone's battery level is lower than a preset level, the water tank level is lower than a preset level, and the pose status information indicates that the drone has collided or deviated from the planned path, thus meeting the return conditions, a return command is sent to the drone so that the drone can return based on the return command.

[0134] In practical applications, after a drone returns, another drone can be deployed to continue the work.

[0135] This application embodiment can dynamically allocate tasks, such as pre-rinsing, scrubbing, or final rinsing, based on the drone's capabilities and real-time status through a distributed control system. This application embodiment can achieve task-area-based pre-allocation, dividing the surface into multiple areas, and then guiding paired drones to complete cleaning tasks within the assigned areas. This application embodiment achieves seamless paired drone coordination, further optimizing the collaborative cleaning problem of paired drones in building cleaning by combining synchronous operation, collision avoidance protocols, and dynamic workload balancing. To improve cleaning efficiency and safety, these considerations can be mathematically modeled and incorporated into the problem, as follows: (1) Synchronization constraints make and Indicates drones to (sprayer) and (Cleaning machine) reaches the section Time:

[0136] in, It is the delay required for the cleaning drone to follow the sprayer after the surface is wetted. It depends on the surface drying time and may vary depending on environmental conditions (such as wind, temperature, exterior wall material, surface angle, etc.).

[0137] Due to the collaborative working mode between drone pairs, it is necessary to minimize the time lag penalty function:

[0138] To ensure close synchronization between paired drones.

[0139] (2) Collision avoidance protocol The default collision model is constructed as follows: To prevent drones from colliding during operation, the distance between any two drones must always exceed a predefined safety threshold. For all drones and ,make sure:

[0140] To prevent collisions within a drone swarm. Furthermore, for all drones... and ,make sure:

[0141] To prevent collisions within the drone swarm. Furthermore, the enhanced cost function is shown below:

[0142]

[0143] in, It is an indicator function that activates a penalty when a drone gets too close.

[0144] (3) Dynamic workload balancing: Dynamic workload balancing ensures that drones with lower energy or higher workloads will transfer tasks to other drones in the swarm. Workload allocation Defined as a drone The workload depends on the number of surface segments assigned to it. .

[0145] in, It is a binary variable representing the drone. Was it assigned to a fragment? Minimize workload imbalance among drones to leverage constraints:

[0146] in, This is the average workload. However, tasks are being redistributed to drones with higher remaining energy. To prevent operator overload:

[0147] in, It is a weighting balance between energy and workload considerations.

[0148] (4) Overall objective function: The overall optimization function integrates these considerations:

[0149]

[0150]

[0151] in, It is the weighting coefficient. It's about the cleaning efficiency of drones. It is the communication load of the drone. This is the available bandwidth. The particle swarm optimization algorithm is used to solve it. and To minimize .

[0152] This application's embodiments achieve high reliability, safety, and efficiency by combining synchronized operation, collision avoidance, and workload balancing, thus addressing significant challenges in collaborative cleaning tasks. The collaborative cleaning framework ensures effective task completion, improves operational efficiency, and meets stringent safety standards in urban environments.

[0153] The drones in this embodiment achieve collaborative cleaning in the following ways: (1) pairing drone roles (such as pre-rinsing, scrubbing, and final rinsing); (2) making a return-to-home decision based on battery or resource limitations; and (3) a task reallocation mechanism to dynamically balance the workload. Meanwhile, this embodiment considers load and battery power limitations in the decision-making process, allowing the drones to autonomously return to replace batteries or replenish water.

[0154] The primary objective of this application is to automate manual labor to minimize accident risks while improving the efficiency and autonomy of cleaning tasks. However, existing robotic solutions, such as remotely controlled drone cleaning systems, often face challenges in terms of autonomy or effectiveness, especially when dealing with buildings with edges or protrusions. Therefore, the industry urgently needs to create a universal autonomous system.

[0155] Against this backdrop, embodiments of this application integrate multi-sensor fusion technology to ensure precise system positioning. Furthermore, a cost-effective coverage path planning algorithm is deployed on paired drones to improve the efficiency and energy utilization of building cleaning. In addition, a carefully designed reinforcement learning-based composite UAV (Unmanned Aerial Vehicle) cleaning mode aims to optimize exterior wall cleaning results and reduce energy consumption. With a specially designed UAV docking station, paired drones can seamlessly complete the cleaning task of an entire building in autonomous mode.

[0156] Furthermore, this application embodiment employs a collaborative cleaning mode, enabling multiple drones to coordinate tasks in a distributed manner. Through intelligent task allocation and collision avoidance protocols, these drones can achieve wider coverage in a shorter time, improving efficiency and safety. Therefore, the safety-assured drone-based building exterior wall cleaning method proposed in this application embodiment provides a promising solution for regular maintenance in dense urban environments, significantly reducing manual operation and occupational hazards.

[0157] This application provides an innovative solution that combines advanced security mechanisms, precise control algorithms, and environmentally friendly cleaning technologies. This advancement aims to achieve a harmonious balance of adaptability, efficiency, and safety in exterior wall maintenance within dense urban environments.

[0158] See Figure 5 As shown in the diagram, this application provides a structural schematic of a building surface cleaning device 50. The building surface cleaning device 50 is applied to a robotic cleaning system 10, which includes at least two drones, each drone comprising a sprayer 11 and a cleaning machine 12. Figure 5 As shown, the building surface cleaning device 50 includes: a first control module 501, a second control module 502, and a third control module 503.

[0159] The first control module 501 is used to control the sprayer 11 to pre-wash the building surface based on the first planned path; The second control module 502 is used to control the cleaning machine 12 to scrub the building surface based on the second planned path; the second planned path and the first planned path are obtained based on the same preset initial planned path; The third control module 503 is used to control the sprayer 11 to perform a final rinse on the building surface.

[0160] Optionally, the drones are equipped with sensor components, including at least two of the following: lidar, inertial measurement unit, ultra-wideband antenna, global navigation satellite system, and camera sensor; Optionally, the building surface cleaning device 50 includes a path planning module, which is used to acquire at least two types of measurement data measured by sensor components; perform data fusion processing based on the at least two types of measurement data to determine the pose state estimation information of the UAV; the pose state estimation information is an estimation information of position information and attitude information; and adjust the initial planned path based on the pose state estimation information to obtain the target planned path.

[0161] Optionally, the path planning module is used to perform frequency alignment on at least two types of measurement data using a keyframe alignment method to obtain at least two types of measurement data to be processed; construct a factor graph based on the at least two types of measurement data to be processed; and determine pose state estimation information based on the factor graph.

[0162] Optionally, the path planning module is used to determine the double-difference pseudorange factor based on the measurement data to be processed obtained from the Global Navigation Satellite System; the double-difference pseudorange factor is an observation model used to calculate the pseudorange measurement from the receiver to the satellite at a specific time. Based on the measurement data to be processed obtained from the Global Navigation Satellite System, the Doppler observation function is determined as the Doppler observation factor; the Doppler observation factor is used to correct the double-difference pseudorange factor. Based on the measurement data obtained from the inertial measurement unit, the inertial factor is determined; the inertial factor is used to adjust the changes in the observed values ​​of acceleration and angular velocity. Based on the measurement data obtained from the camera sensor, a visual factor is determined; the visual factor is used to adjust the pose state information of the camera sensor. Based on the measurement data to be processed obtained from lidar measurements, lidar factors are determined; lidar factors are used to adjust the pose state information of the lidar. Based on the measurement data to be processed obtained from the ultra-wideband antenna, the ultra-wideband factor is determined; the ultra-wideband factor is used to perform positioning compensation on the pose state information of the UAV. Using the Schur complement method, during the frequency alignment process of at least two types of measurement data, the constraints of keyframes of the measurement data are marginalized to determine the marginalization factor; the marginalization factor is used to marginalize the adjustment of the factor map. The factor map is determined based on the dual-difference pseudorange factor, Doppler observation factor, inertial factor, visual factor, lidar factor, ultra-wideband factor, and marginalization factor.

[0163] Optionally, the lidar factor includes a first lidar factor and a second lidar factor; Optionally, the path planning module is used to determine a first lidar factor based on the measurement data to be processed obtained by lidar measurement and the lidar's preset scan-to-map matching model; the first lidar factor is used to adjust the pose state information of the lidar based on the local point cloud map; Based on the measurement data to be processed obtained from lidar measurement and the lidar's preset scan-to-multi-scan model, a second lidar factor is determined; the second lidar factor is used to adjust the lidar's pose state information through the relative constraints between adjacent keyframes.

[0164] Optionally, the path planning module is used to adjust the preset initial planning path based on pose state estimation information and a preset target reinforcement learning model to obtain the target planning path; The target reinforcement learning model is obtained by training the initial reinforcement learning model using preset constraints, preset multi-index reward functions, and UAV flight sample data.

[0165] Optionally, the initial reinforcement learning model is constructed as follows: An initial reinforcement learning model is constructed based on the model state, UAV action information, UAV transfer probability, reward information, and discount factor. The model state includes the drone's current pose state information, the proportion of cleaned grids in the current area, and the drone's remaining energy. The drone action information includes the drone's historical path. The drone transfer probability is used to represent the probability of the drone's next displacement based on the current pose state information and historical path. The reward information is used to comprehensively evaluate the drone's cleaning efficiency, energy consumption, obstacle avoidance performance, and task completion. The discount factor is used to measure the importance of future rewards relative to immediate rewards.

[0166] Optionally, the constraints include: maintaining a safe distance between the drone and any obstacle or building surface, and an energy limit through the energy consumption function of each drone action; The multi-metric reward function includes: cleaning efficiency reward, energy penalty, collision penalty, and task completion reward; Cleaning efficiency reward is used to represent the reward for the incremental improvement in mesh clearing in each displacement step, energy penalty is used to punish the energy consumed in the previous action, collision penalty is used to punish the drone for colliding with obstacles or building surfaces, and mission completion reward is used to reward the degree to which the drone completes the mission.

[0167] Optionally, the second control module 502 is used to implement any of the following: In response to the first position information sent by the sprayer, when it is determined that the sprayer has moved to the predetermined position, the cleaning machine is controlled to scrub the building surface based on the second planned path; In response to the sprayer starting pre-rinsing time reaching the preset time, the cleaning machine is controlled to scrub the building surface based on the second planned path; In response to the start command for the cleaning machine, control the cleaning machine to scrub the building surface based on the second planned path.

[0168] Optionally, the building surface cleaning device 50 further includes: a fourth control module, which is used to acquire the drone's power information, water tank storage information, and pose status information; if at least one of the drone's power information, water tank storage information, and pose status information meets the corresponding return conditions, the module sends a return command to the drone so that the drone returns based on the return command.

[0169] In applications, the modules in the building surface cleaning device 50 can be software program modules, or they can be implemented through different logic circuits integrated in a processor, or they can be implemented through multiple distributed processors.

[0170] The building surface cleaning device 50 of this application embodiment can execute the method provided in this application embodiment. The implementation principle is similar. The actions performed by each module in the building surface cleaning device 50 of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the building surface cleaning device 50, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0171] See Figure 6 As shown, this application provides a schematic diagram of the structure of a terminal device 13. Figure 6 As shown, the terminal device 13 of this application embodiment includes: a memory 132, a processor 131, and a computer program 133 stored in the memory 132 and executable on the processor 131. When the processor 131 executes the computer program, it implements the steps of the methods of the various embodiments of this application.

[0172] Terminal device 13 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 13 may include, but is not limited to, a processor 131 and a memory 132. Those skilled in the art will understand that terminal device 13 may also include more or fewer components, or combinations of certain components, or different components, such as input / output devices, network access devices, etc.

[0173] The processor 131 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0174] In some embodiments, memory 132 may be an internal storage unit, such as a hard disk or RAM. Memory 132 may be a removable / non-removable, volatile / non-volatile computer system storage medium; for example, memory 132 may be a non-volatile memory used for reading and writing non-volatile magnetic media. In other embodiments, memory 132 may also be an external storage device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on terminal device 13. Memory 132 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 132 may also be used to temporarily store data that has been output or will be output.

[0175] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0176] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0177] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0178] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0179] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc. The storage medium can also include combinations of the above types of memory.

[0180] This application provides a computer program product that, when run on a processor, enables the processor to execute the steps described in the various method embodiments above.

[0181] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0182] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for cleaning building surfaces, characterized in that, Applied to a robotic cleaning system, the robotic cleaning system comprising at least two drones, the at least two drones comprising a sprayer and a cleaning machine, the method comprising: The sprayer is controlled to pre-wash the building surface based on a first planned path; The cleaning machine is controlled to scrub the building surface based on a second planned path; the second planned path and the first planned path are obtained based on the same preset initial planned path; Control the sprayer to perform a final rinse on the surface of the building.

2. The method for cleaning building surfaces according to claim 1, characterized in that, The drones are all equipped with sensor components, which include at least two of the following: lidar, inertial measurement unit, ultra-wideband antenna, global navigation satellite system, and camera sensor. The first or second planned path is obtained as the target planned path in the following manner: Acquire at least two types of measurement data measured by the sensor assembly; Based on data fusion processing of at least two types of measurement data, the pose state estimation information of the UAV is determined; the pose state estimation information is an estimation of position information and attitude information. Based on the pose state estimation information, the initial planned path is adjusted to obtain the target planned path.

3. The method for cleaning building surfaces according to claim 2, characterized in that, The process of performing data fusion based on the at least two types of measurement data to determine pose state estimation information includes: Using a keyframe alignment method, at least two types of measurement data are frequency aligned to obtain at least two types of measurement data to be processed. Construct a factor graph based on at least two types of the measurement data to be processed; Based on the factor graph, pose state estimation information is determined.

4. The method for cleaning building surfaces according to claim 3, characterized in that, The construction of a factor graph based on at least two types of measurement data to be processed includes: Based on the measurement data to be processed obtained from the Global Navigation Satellite System, the double-difference pseudorange factor is determined; the double-difference pseudorange factor is an observation model used to calculate the pseudorange measurement from the receiver to the satellite at a specific time. Based on the measurement data to be processed obtained from the global navigation satellite system, the Doppler observation function is determined as the Doppler observation factor; the Doppler observation factor is used to correct the double-difference pseudorange factor. Based on the measurement data to be processed obtained by the inertial measurement unit, the inertial factor is determined; the inertial factor is used to adjust the changes in the observed values ​​of acceleration and angular velocity. Based on the measurement data to be processed obtained by the camera sensor, a visual factor is determined; the visual factor is used to adjust the pose state information of the camera sensor. Based on the measurement data to be processed obtained from the lidar measurement, lidar factors are determined; the lidar factors are used to adjust the pose state information of the lidar. Based on the measurement data to be processed obtained from the ultra-wideband antenna, the ultra-wideband factor is determined; the ultra-wideband factor is used to perform positioning compensation on the pose state information of the UAV. Using the Schur complement method, during the frequency alignment process of at least two types of measurement data, the constraints of keyframes of the measurement data are marginalized to determine a marginalization factor; the marginalization factor is used to marginalize the adjustment of the factor map. A factor map is determined based on the dual-difference pseudorange factor, the Doppler observation factor, the inertial factor, the visual factor, the lidar factor, the ultra-wideband factor, and the marginalization factor.

5. The method for cleaning building surfaces according to claim 4, characterized in that, The lidar factor includes a first lidar factor and a second lidar factor. The process of determining lidar factors based on the measurement data obtained from the lidar measurements includes: Based on the measurement data to be processed obtained by the lidar and the lidar's preset scan-to-map matching model, a first lidar factor is determined; the first lidar factor is used to adjust the pose state information of the lidar based on the local point cloud map. Based on the measurement data to be processed obtained by the lidar and the lidar's preset scan-to-multi-scan model, a second lidar factor is determined; the second lidar factor is used to adjust the pose state information of the lidar through the relative constraints between adjacent keyframes.

6. The method for cleaning building surfaces according to claim 2, characterized in that, The step of adjusting the initial planned path based on the pose state estimation information to obtain the target planned path includes: Based on the pose state estimation information and the preset target reinforcement learning model, the initial planned path is adjusted to obtain the target planned path; The target reinforcement learning model is obtained by training an initial reinforcement learning model using preset constraints, a preset multi-index reward function, and UAV flight sample data.

7. The method for cleaning building surfaces according to claim 6, characterized in that, The initial reinforcement learning model is constructed in the following manner: An initial reinforcement learning model is constructed based on the model state, UAV action information, UAV transfer probability, reward information, and discount factor. The model state includes the drone's current pose state information, the proportion of cleaned grids in the current area, and the drone's remaining energy. The drone action information includes the drone's historical path. The drone transfer probability is used to represent the probability of the drone's next displacement based on the current pose state information and the historical path. The reward information is used to comprehensively evaluate the drone's cleaning efficiency, energy consumption, obstacle avoidance performance, and task completion. The discount factor is used to measure the importance of future rewards relative to immediate rewards.

8. The method for cleaning building surfaces according to claim 6, characterized in that, The constraints include: the drone maintaining a safe distance from any obstacle or the surface of the building, and an energy limit based on the energy consumption function of each action of the drone; The multi-metric reward function includes: cleaning efficiency reward, energy penalty, collision penalty, and task completion reward; The cleaning efficiency reward is used to represent the reward for the incremental improvement in mesh clearing in each step of displacement; the energy penalty is used to penalize the energy consumed in the previous action; the collision penalty is used to penalize the drone for colliding with the obstacle or the building surface; and the task completion reward is used to reward the degree to which the drone completes the task.

9. The method for cleaning building surfaces according to any one of claims 1-8, characterized in that, The control of the cleaning machine to scrub the building surface based on a second planned path includes any one of the following: In response to the first location information sent by the sprayer, when it is determined that the sprayer has moved to a predetermined position, the cleaning machine is controlled to scrub the building surface based on a second planned path; In response to the sprayer starting pre-rinsing for a preset time, the cleaning machine is controlled to scrub the building surface based on a second planned path; In response to a start command for the cleaning machine, the cleaning machine is controlled to scrub the building surface based on a second planned path.

10. The method for cleaning building surfaces according to any one of claims 1-8, characterized in that, The method further includes: Obtain the drone's battery level, water tank water level, and pose status information; If at least one of the drone's battery power information, water tank water storage information, and pose status information meets the corresponding return conditions, a return command is sent to the drone so that the drone returns based on the return command.

11. A cleaning device for building surfaces, characterized in that, Applied to a robotic cleaning system, the robotic cleaning system comprising at least two drones, the at least two drones comprising a sprayer and a cleaning machine, the device comprising: The first control module is used to control the sprayer to pre-wash the building surface based on the first planned path; The second control module is used to control the cleaning machine to scrub the building surface based on a second planned path; the second planned path and the first planned path are obtained based on the same preset initial planned path; The third control module is used to control the sprayer to perform a final rinse on the building surface.

12. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 10.

13. A robotic cleaning system, characterized in that, include: At least two drones and the terminal device as described in claim 12, wherein the at least two drones include a sprayer and a cleaner.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 10.

Images