Control methods, devices and systems for cleaning systems

Abstract

This application provides a control method, apparatus, and system for a cleaning system, relating to the field of robot vision positioning and control. The cleaning system includes a cleaning device, a base station, and a robotic arm. The cleaning device is equipped with a first vision sensor, the robotic arm's end effector is equipped with a second vision sensor, and the base station's accommodating cavity wall has at least two image markers with known physical spacing. The method includes: controlling the cleaning device to move to a target area near the base station based on the first vision sensor; acquiring a first image containing the image markers based on the second vision sensor during the robotic arm's insertion into the cleaning tank; controlling the cleaning component to move to a target position within the cleaning tank based on the first image; and sending a cleaning command to the base station. This application utilizes the vision sensors at the cleaning device and the robotic arm's end effector, as well as the image markers on the base station's accommodating cavity wall, for hierarchical positioning, achieving precise alignment between the cleaning component and the cleaning tank, thus improving the cleaning success rate.

Description

Technical Field

[0001] This application relates to the field of robot vision positioning and control, and in particular to a control method, apparatus and system for a cleaning system. Background Technology

[0002] With the development of intelligent cleaning technology, robotic vacuum cleaners are usually equipped with robotic arms, which use cleaning components (such as a mop tray) at the end of the robotic arm to clean narrow crevices, baseboards, and other areas. In addition, some base stations have automatic cleaning functions that can extend the cleaning component into the cleaning tank for cleaning, but how to ensure that the cleaning component extends accurately into the cleaning tank has become an urgent problem to be solved.

[0003] In existing solutions, cleaning equipment typically first uses lidar or infrared sensors to roughly locate the base station, then controls the robotic arm to extend, and relies on a monocular camera at the end of the robotic arm to identify the outline of the base station or the edge of the cleaning tank, estimate the relative position of the cleaning component and the cleaning tank, and adjust the posture of the robotic arm accordingly to send the cleaning component into the cleaning tank.

[0004] However, this positioning method, which relies on lidar or infrared sensors in conjunction with a monocular camera, suffers from insufficient positioning accuracy. Summary of the Invention

[0005] This application provides a control method, apparatus, and system for a cleaning system to solve the technical problems of insufficient positioning accuracy and difficulty in accurately aligning cleaning components with the cleaning tank in the prior art.

[0006] In a first aspect, embodiments of this application provide a control method for a cleaning system. The cleaning system includes a cleaning device and a base station. The cleaning device is equipped with a robotic arm, and a cleaning component is disposed at the end of the robotic arm. The base station has a receiving cavity, and a cleaning tank for receiving the cleaning component is disposed on the bottom surface of the receiving cavity. At least two image markers with known physical spacing are disposed on the wall surface of the receiving cavity. The method includes:

[0007] In response to a request to clean the cleaning equipment, the cleaning equipment is controlled to move to a target area near the base station based on the first vision sensor on the cleaning equipment.

[0008] Control the extension of the robotic arm, and during the process of the robotic arm extending into the cleaning tank, acquire a first image based on a second vision sensor on the end of the robotic arm. The first image includes some or all image markers.

[0009] Based on the first image, control the cleaning component to move to the target position within the cleaning tank;

[0010] A cleaning command is sent to the base station to trigger the cleaning task for the cleaning components.

[0011] In this embodiment, in response to a request to clean the cleaning components, the first visual sensor on the cleaning equipment observes image markers on the wall of the base station receiving cavity, guiding the cleaning equipment to move to the target area near the base station. During the process of the robotic arm extending into the cleaning tank, the second visual sensor at the end of the robotic arm collects the same set of image markers at close range, calculates the pose between the end of the robotic arm and the cleaning tank, and finally sends a cleaning command to the base station to trigger the cleaning task.

[0012] In this manner, the cleaning equipment and the robotic arm end effector share the same set of image markers with known physical spacing on the wall of the base station's receiving cavity as a reference system. The cleaning equipment performs coarse positioning (i.e., the cleaning equipment guides itself to move to the target area near the base station using a first visual sensor), while the robotic arm performs fine positioning (i.e., the robotic arm observes the image markers at close range during insertion into the cleaning tank using a second visual sensor, calculates the pose, and controls the cleaning component to move to the target position within the cleaning tank), thus effectively connecting coarse and fine positioning. Compared to existing technologies where the cleaning equipment's docking and the robotic arm's insertion are independent and prone to accumulating positioning errors, this embodiment effectively compensates for cleaning equipment docking deviations and visual recognition errors, avoiding the problem of cleaning components failing to be properly inserted into the cleaning tank due to inaccurate positioning. This improves the alignment accuracy between the cleaning component and the cleaning tank and increases the cleaning success rate.

[0013] In one possible embodiment, based on the first image, controlling the cleaning component to move to a target position within the cleaning tank includes:

[0014] Obtain the corner pixel positions of identifiable image markers in the first image;

[0015] If the number of identifiable image markers is less than a preset threshold, the predicted corner pixel positions of unidentified image markers are obtained based on the corner pixel positions of the identified image markers and the physical coordinates of the corners of each image marker.

[0016] Based on the identified corner pixel positions of the image markers, the predicted corner pixel positions, and the physical coordinates of the corners of each image marker, the pose of the robotic arm relative to the cleaning tank is determined.

[0017] The cleaning device is moved to the target position based on the user's posture.

[0018] In this implementation, when the number of identifiable image markers in the first image acquired by the robotic arm's end effector is insufficient, the system calculates the predicted corner pixel positions of the unidentified image markers based on the corner pixel positions of the identified image markers and the known physical coordinates of each corner. Then, the identified corner pixel positions are merged with the predicted corner pixel positions to calculate the robotic arm's pose relative to the cleaning tank, and the cleaning component is moved to the target position based on this pose. Thus, even if some image markers on the base station's cavity wall are obscured or cannot be fully identified due to the shooting angle, the system can still use the corner information of the identified markers combined with known physical coordinate relationships to fill in the missing corner information, obtaining a sufficient number of corners for pose calculation. Compared to existing technologies that directly abandon positioning due to incomplete marker recognition or rely on retry mechanisms, this embodiment can maintain stable positioning capabilities even when markers are partially missing, avoiding cleaning component delivery failures due to partial occlusion, and improving the system's positioning reliability and cleaning success rate in complex environments.

[0019] In one possible embodiment, triggering a cleaning task for the cleaning component includes:

[0020] Control the robotic arm to rotate, causing the cleaning components to rotate within the cleaning tank.

[0021] In this method, the cleaning component is driven to rotate actively by a robotic arm, which causes relative motion between the cleaning component and the inner wall of the cleaning tank, creating a frictional scrubbing effect. This effectively removes stains and dirt from the surface of the cleaning component, improving cleaning cleanliness and efficiency.

[0022] In one possible embodiment, the cleaning system also includes an accessory compartment independent of the base station, which is used to store cleaning parts; after the cleaning task is completed, the robotic arm is controlled to remove the cleaning parts from the cleaning tank and then move the cleaning parts back to the accessory compartment.

[0023] In this implementation, after the cleaning task is completed, the robotic arm removes the cleaned parts from the cleaning tank and returns them to the parts compartment. This allows the cleaned parts to automatically return to their original positions after cleaning, without manual intervention, making it easy for the robotic arm to grab and use them again in the next cleaning task. This forms a complete automated cycle from removal and use to return for cleaning and storage, improving the automation level and ease of use of the cleaning system.

[0024] In one possible embodiment, when the cleaned part needs to be moved back to the parts compartment after cleaning, the process of controlling the robotic arm to move the cleaned part back to the parts compartment includes:

[0025] First, the cleaning equipment is controlled to move to a target area near the accessory compartment based on the first vision sensor on the cleaning equipment. Specifically, the first vision sensor on the cleaning equipment acquires an image containing the accessory compartment, identifies image markers set on the accessory compartment, and extracts the corner pixel positions of the image markers in the image. Since there are at least two image markers with known physical spacing on the accessory compartment, and the physical coordinates of the corner points of each image marker are known in advance, the relative pose between the cleaning equipment and the accessory compartment is calculated based on the correspondence between the corner pixel positions and the corner physical coordinates. Optionally, an additional image marker can be set below the accessory compartment, with its position lower than the other image markers, to facilitate priority recognition by the cleaning equipment at a greater distance. Further, the first vision sensor is a binocular vision sensor. The binocular vision sensor acquires binocular point cloud information of the accessory compartment and its surrounding area. Combining the corner pixel positions, corner physical coordinates, and binocular point cloud information, the relative position between the cleaning equipment and the accessory compartment is determined, and the cleaning equipment is controlled to move to the target area near the accessory compartment.

[0026] Then, a second image is acquired via a second vision sensor at the end effector of the robotic arm. This second image includes some or all of the image markers set on the parts compartment. The corner pixel positions of the image markers in the second image are extracted, and combined with the known physical coordinates of the corner points of each image marker, the pose of the robotic arm relative to the parts compartment is calculated. The robotic arm is then controlled to move to the target pose, which is perpendicular to the face of the robotic arm facing the parts compartment, so that the robotic arm places the cleaning parts into the parts compartment in an orientation perpendicular to the front of the parts compartment.

[0027] In this implementation, the cleaning device uses a first vision sensor to acquire the corner pixel positions of the image markers on the front of the parts compartment. Combining this with the known physical coordinates of the corner points, it calculates its relative pose with the parts compartment and moves to the target area near the parts compartment. Optionally, an additional image marker can be placed below the parts compartment to facilitate priority identification by the cleaning device at a greater distance, achieving rapid coarse positioning. The first vision sensor can also be a binocular vision sensor, using binocular point cloud information combined with corner data for joint calculation to further improve positioning accuracy. The robotic arm's end effector uses a second vision sensor to acquire the corner pixel positions of the image markers on the front of the parts compartment. Combining this with the known physical coordinates of the corner points, it calculates the robotic arm's pose relative to the parts compartment and places the cleaning parts into the parts compartment in an orientation perpendicular to the front of the compartment.

[0028] Through the coordinated operation of these two levels of positioning, the cleaning equipment is responsible for bringing itself to the vicinity of the parts compartment, while the robotic arm is responsible for precisely placing the cleaning parts into the preset positions within the parts compartment. Even if there is a deviation in the positioning of the cleaning equipment, the robotic arm can compensate and correct it through its own visual observation, ensuring that the cleaning parts are accurately placed and avoiding placement failures or collisions with the parts compartment. This achieves automatic return of the cleaning parts after cleaning, making it convenient for the robotic arm to pick them up and use them again in the next cleaning task.

[0029] In one possible embodiment, during the placement of the cleaning component into the accessory compartment, when the number of identifiable image markers in the second image is less than a preset threshold, the predicted corner pixel positions of the unidentified image markers are obtained based on the corner pixel positions of the identified image markers and the physical coordinates of each corner pixel. Based on the corner pixel positions of the identified image markers, the predicted corner pixel positions, and the physical coordinates of each corner pixel, the pose of the robotic arm relative to the accessory compartment is determined. The robotic arm is then controlled to move to the target pose based on this pose.

[0030] In this implementation, when the number of identifiable image markers in the second image is insufficient, the system uses the correspondence between the corner pixel positions of the identified image markers and their known physical coordinates to calculate the predicted corner pixel positions of the unidentified image markers. The identification and prediction results are then used together for pose calculation. Since the physical coordinates of the corner points of each image marker are known in advance, and the relative positional relationships between the markers are fixed, the calculation results have high reliability. Thus, even if some markers are occluded, the system can still obtain a sufficient number of corner points for pose calculation, avoiding positioning failures due to missing markers. This ensures that the cleaning parts can still be accurately placed into the accessory compartment even when occluded.

[0031] In one possible embodiment, the first vision sensor is a binocular vision sensor; controlling the cleaning equipment to move to a target area near the base station based on the first vision sensor on the cleaning equipment includes:

[0032] A third image is acquired using a binocular vision sensor, and the third image includes image markers;

[0033] Based on the corner pixel positions of the image markers in the third image and the physical coordinates of the corner points of each image marker, the first pose of the cleaning device relative to the base station is determined.

[0034] Based on the first pose, the cleaning equipment is controlled to move to the target area near the base station.

[0035] In this implementation, a binocular vision sensor acquires a third image containing image markers on the wall of the base station's accommodating cavity. The corner pixel positions of the image markers in the third image are extracted, and combined with the known physical coordinates of the corner points of each image marker, the first pose of the cleaning device relative to the base station is calculated. Based on this, the cleaning device is controlled to move to the target area near the base station. The binocular vision sensor can obtain depth information through parallax calculation of the left and right images, offering higher accuracy in distance measurement and pose estimation compared to a monocular camera. This improves the accuracy of calculating the relative pose between the cleaning device and the base station, providing a more reliable starting position for the subsequent insertion of the robotic arm into the cleaning tank.

[0036] In one possible embodiment, if the first pose does not meet the preset conditions before the cleaning equipment moves to the target area near the base station, the following steps are repeated until the first pose meets the preset conditions:

[0037] A third image is acquired using a binocular vision sensor;

[0038] The updated first pose of the cleaning equipment relative to the base station is determined based on the image markers in the third image;

[0039] The cleaning equipment movement is controlled based on the updated first pose.

[0040] The preset conditions include that the distance between the cleaning equipment and the base station is greater than a preset distance threshold, and the deviation between the orientation of the cleaning equipment and the orientation of the base station is less than a preset angle threshold.

[0041] In this implementation, before the cleaning equipment moves to the target area near the base station, it is first determined whether the first pose meets preset conditions. These preset conditions include that the distance between the cleaning equipment and the base station is greater than a preset distance threshold, and the deviation between the orientation of the cleaning equipment and the orientation of the base station is less than a preset angle threshold. If the first pose does not meet the preset conditions, the steps of acquiring a third image through a binocular vision sensor, determining the updated first pose of the cleaning equipment relative to the base station based on the image markers in the third image, and controlling the movement of the cleaning equipment based on the updated first pose are repeated until the first pose meets the preset conditions.

[0042] Through multiple iterative adjustments, the cleaning equipment can gradually approach the target area near the base station, eliminating the cumulative errors generated during a single positioning and movement. Even if the initial docking position is significantly off, after multiple adjustments, the cleaning equipment can still reach the target area with the required position and orientation, providing precise starting conditions for the subsequent insertion of the robotic arm into the cleaning tank.

[0043] In one possible embodiment, the following steps are repeated a preset number of times before the cleaning component is moved to the target position within the cleaning tank:

[0044] The first image is acquired using a second visual sensor;

[0045] Based on the corner pixel positions of the image markers in the first image, the pose of the robotic arm relative to the cleaning tank is redefined;

[0046] The cleaning component is moved according to the redefined pose to gradually approach the target position.

[0047] In this implementation, before the cleaning component moves to its target position within the cleaning tank, the steps of acquiring the first image, re-determining the robotic arm's pose relative to the cleaning tank, and controlling the movement of the cleaning component are repeated a preset number of times. Each repetition updates and corrects the position and orientation of the cleaning component based on the latest acquired image information, gradually reducing the deviation from the target position. Through multiple iterative adjustments, the robotic arm can overcome potential visual recognition or motion control errors in a single positioning attempt, enabling the cleaning component to reach its target position within the cleaning tank with higher precision. This avoids the cleaning component failing to be properly delivered into the cleaning tank or interfering with the edge of the cleaning tank due to inaccurate positioning in a single instance.

[0048] In one possible embodiment, image markers are disposed on multiple walls of the base station housing cavity, including at least two of a front wall, side walls, and a top surface.

[0049] In this implementation, image markers are placed on multiple walls of the base station housing cavity, such as at least two of the front wall, side walls, and top surface. When the robotic arm extends into the base station housing cavity from different angles, or when a marker on one wall is obscured by the robotic arm itself, the markers on other walls can still be observed by the second visual sensor, increasing the number and distribution range of identifiable markers. This improves the robotic arm's adaptability to different postures and occlusion conditions during extension, avoids positioning failures caused by the occlusion of a single wall marker, and enhances the robustness of positioning.

[0050] In one possible embodiment, the second vision sensor is mounted on the end of the robotic arm; when acquiring the first image through the second vision sensor, if the number of identifiable image markers in the first image is less than a preset threshold, the position of the robotic arm is adjusted according to the corner pixel position of the identified image markers, and the first image is reacquired through the second vision sensor.

[0051] In this implementation, the second vision sensor is mounted sideways at the end of the robotic arm. Due to limitations in its mounting position and field of view, it may only be able to observe a portion of the image markers during the initial capture. When the number of recognizable image markers is insufficient, the system adjusts the position of the robotic arm based on the corner pixel positions of the already identified image markers, changing the shooting angle and field of view of the second vision sensor, and then reacquires images for identifying the image markers. By actively adjusting the position of the robotic arm to improve observation conditions, the system can obtain images containing more image markers, thereby improving the accuracy and reliability of pose calculation and avoiding positioning failures due to poor initial field of view.

[0052] Secondly, embodiments of this application provide a control device for a cleaning system. The cleaning system includes a cleaning device and a base station. The cleaning device is equipped with a robotic arm, and a cleaning component is provided at the end of the robotic arm. The base station has a receiving cavity, and a cleaning tank for receiving the cleaning component is provided on the bottom surface of the receiving cavity. At least two image markers with known physical spacing are provided on the wall surface of the receiving cavity.

[0053] The control device includes a control module for performing the method described in any of the first aspects above.

[0054] Thirdly, this application provides a cleaning system, which includes a cleaning device and a base station. The cleaning device is equipped with a robotic arm, and a cleaning component is provided at the end of the robotic arm. The base station has a receiving cavity, and a cleaning tank for receiving the cleaning component is provided on the bottom surface of the receiving cavity. At least two image markers with known physical spacing are provided on the wall surface of the receiving cavity.

[0055] The cleaning system also includes a control module for performing the methods described in any of the first aspects above.

[0056] In summary, the first vision sensor on the cleaning equipment guides it to the target area near the base station for coarse positioning. Then, the second vision sensor at the end of the robotic arm acquires an image containing image markers during insertion into the cleaning tank. This image, combined with image markers on the wall of the base station's housing with known physical distances, enables fine positioning, effectively connecting the coarse and fine positioning. This improves the positioning accuracy of the robotic arm's end effector relative to the cleaning tank, reduces accumulated alignment deviations, and enhances the stability and reliability of the cleaning components entering the cleaning tank and triggering the cleaning task, even under conditions of limited field of view, varying lighting, or partial obstruction within the base station's housing. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0058] Figure 1This is a schematic diagram of the hardware architecture of a cleaning system provided in an embodiment of this application;

[0059] Figure 2 This is a schematic diagram of the internal structure of a base station accommodating cavity provided in an embodiment of this application;

[0060] Figure 3 A flowchart illustrating a control method for a cleaning system provided in an embodiment of this application;

[0061] Figure 4 This is a schematic diagram of the structure of a control device for a cleaning system provided in an embodiment of this application.

[0062] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0064] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] In the embodiments of this application, the use of terms such as "first" and "second" is to distinguish between identical or similar items that have essentially the same function and effect. For example, "first electronic device" and "second electronic device" are merely used to distinguish different electronic devices and do not limit their order of execution. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0066] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0067] Currently, cleaning equipment with robotic arms (such as robotic vacuum cleaners) is usually equipped with automatic cleaning functions. When it is necessary to clean the cleaning parts (such as the mop tray) at the end of the robotic arm, the cleaning equipment needs to accurately deliver the cleaning parts into the cleaning tank of the base station.

[0068] In existing solutions, cleaning equipment typically first uses LiDAR or infrared sensors to roughly locate the base station, obtaining its approximate direction and distance, guiding the cleaning equipment to move in front of the base station. After rough positioning, the cleaning equipment extends its robotic arm, using a monocular camera at the end of the arm to capture an image of the base station. Image recognition algorithms are then used to identify the base station's outline or the edge of the cleaning tank, estimating the relative position between the cleaning component and the tank. Finally, based on the estimation results, the robotic arm's posture is adjusted, and it is controlled to deliver the cleaning component into the cleaning tank.

[0069] However, the above solution has the following problems in practical applications.

[0070] First, the positioning accuracy of lidar or infrared sensors is limited, providing only approximate location information for the base station, which is insufficient to meet the requirement of precise alignment between the cleaning components and the washing tank. When there is a deviation in the docking of the cleaning equipment, this deviation will be directly transmitted to the subsequent robotic arm positioning process, causing the end-effector alignment error to accumulate gradually. During use, users may find that the cleaning components cannot smoothly enter the washing tank, requiring repeated adjustments or re-docking, affecting the user experience.

[0071] Secondly, the field of view of the monocular camera at the end of the robotic arm is easily limited by the cavity structure after it extends into the base station's housing. The identification of the base station outline or the edge of the cleaning tank is easily affected by changes in lighting and partial obstruction, resulting in a large error in estimating the relative position between the cleaning component and the cleaning tank. When the cleaning component is not accurately placed, not only can the cleaning effect not be guaranteed, but in severe cases, it may also cause interference between the robotic arm and the wall of the base station's housing, damaging the equipment.

[0072] Furthermore, when the cleaning tank is located inside the base station's containment cavity, the robotic arm may further obstruct the effective field of view during its extension, resulting in incomplete visual positioning information. Due to the lack of sufficient visual feature points, the system struggles to continuously and stably obtain the relative pose between the robotic arm's end effector and the cleaning tank, leading to positioning interruptions or alignment failures, further reducing the cleaning success rate.

[0073] Therefore, how to improve the positioning accuracy and operational reliability of the robotic arm end relative to the cleaning tank in complex environments and under obstructed conditions has become an urgent problem to be solved for this type of cleaning system.

[0074] To address the aforementioned issues, this application provides a control method for a cleaning system. The core concept is as follows: A first vision sensor on the cleaning equipment guides the equipment to a target area near a base station, achieving coarse positioning; during the insertion of the robotic arm into the cleaning tank, a second vision sensor at the end of the robotic arm acquires image marker information with known physical spacing on the wall of the base station's accommodating cavity, achieving fine positioning; when the number of identifiable image markers is insufficient, the predicted corner pixel positions of unidentified image markers are calculated based on the corner pixel positions of the identified image markers and the known physical coordinates of each image marker's corner points; the identified corner points and predicted corner points are used together for pose calculation, maintaining positioning accuracy even when image markers are partially occluded; finally, a cleaning command is sent to the base station to trigger the cleaning task.

[0075] Through the above concept, the embodiments of this application can, when image markers on the wall of the base station cavity are partially obscured, use the corner point information of the identified image markers to fill in the corner pixel positions of the missing image markers, ensuring the continuity of pose calculation. Compared with the prior art, which directly abandons positioning or relies on retry mechanisms due to incomplete marker recognition, the embodiments of this application can effectively compensate for the docking deviation of the cleaning equipment and visual recognition errors. Even under the above-mentioned obscuration or docking deviation conditions, it can still maintain the precise alignment of the cleaning component and the cleaning tank, improving the cleaning success rate and the stability of system operation.

[0076] The control method for the cleaning system provided in this application can be applied to intelligent cleaning devices such as robotic vacuum cleaners, and is particularly suitable for application scenarios equipped with robotic arms and where the base station has an automatic cleaning function for cleaning components. In such scenarios, the cleaning system mainly consists of cleaning equipment (such as a robotic vacuum cleaner), a robotic arm installed on the cleaning equipment, and a fixed base station. When the cleaning equipment performs floor cleaning tasks, the cleaning components (such as a mop tray) at the end of the robotic arm easily accumulate dust and stains, requiring regular cleaning to maintain the cleaning effect. To achieve automatic cleaning, the cleaning equipment can autonomously move to the vicinity of the base station and use the robotic arm to deliver the cleaning components into the cleaning tank of the base station to complete the cleaning operation, thereby forming a cyclical working mode of cleaning, washing, and re-cleaning.

[0077] The following is combined with Figure 1 The cleaning system architecture provided in the embodiments of this application will be described. Figure 1 This is a schematic diagram of the hardware architecture of a cleaning system provided in an embodiment of this application.

[0078] like Figure 1As shown, the cleaning system 100 includes a base station 200, a cleaning device 300, and a robotic arm 400. The cleaning device 300 is a movable floor cleaning unit (e.g., a robotic vacuum cleaner), on which the robotic arm 400 is mounted. The robotic arm 400 can extend or retract relative to the cleaning device 300. The base station 200 is fixedly placed and has a receiving cavity 210 inside. The bottom surface of the receiving cavity 210 is arranged with a washing tank 230 for receiving cleaning components.

[0079] The cleaning device 300 is equipped with a first vision sensor 310, which can be a binocular camera, to acquire image marks on the wall of the base station receiving cavity 210 during the movement of the cleaning device 300, and to guide the cleaning device 300 to a target area near the base station 200. The end of the robotic arm 400 is equipped with a second vision sensor 410, which can be a monocular camera, to acquire image marks on the wall of the base station receiving cavity 210 during the extension of the robotic arm 400 into the receiving cavity 210, and to guide the cleaning component to a designated position within the cleaning tank 230.

[0080] At least two image markers with known physical spacing are disposed on the wall surface of the receiving cavity 210 of the base station 200 (220a and 220b schematically shown in the figure). As an example, the number of image markers can be three, respectively arranged on at least two of the front wall, side wall and top surface of the receiving cavity. The physical spacing between each image marker is obtained in advance through calibration and stored in the memory of the cleaning device 300 as a geometric constraint for the pose calculation of the visual sensor.

[0081] It should be noted that, Figure 1 This is only used to illustrate the relative positions of the components and does not constitute a limitation on specific dimensions or proportions. The specific types of the first visual sensor 310 and the second visual sensor 410, the number of image markers, and their layout can all be adjusted according to actual application requirements.

[0082] The following combination Figure 2 The internal structure of the base station accommodating cavity provided in the embodiments of this application will be further described. Figure 2 This is a schematic diagram of the internal structure of a base station accommodating cavity provided in an embodiment of this application.

[0083] like Figure 2As shown, the base station 200 has a receiving cavity 210, and the bottom surface of the receiving cavity 210 is provided with a cleaning tank (not shown in the figure) for receiving cleaning parts. Image markers, 220a, 220b, and 220c, are respectively provided on the front wall (i.e., the wall facing the direction of the robotic arm extension), left side wall, and right side wall of the receiving cavity 210. Image marker 220a is located on the left side wall of the receiving cavity, image marker 220b is located on the front wall of the receiving cavity, and image marker 220c is located on the right side wall of the receiving cavity. The physical spacing between the three image markers is pre-measured and known, and is used to assist the second visual sensor in pose calculation.

[0084] It should be noted that, Figure 2 This diagram illustrates the distribution of image markers across multiple walls of the receiving cavity and does not limit the specific number, size, or shape of the image markers. In practical applications, image markers can be placed on at least two of the front, side, and top walls of the receiving cavity, or on more walls or in other layouts as needed. The specific type and physical spacing of the image markers can be determined based on actual requirements.

[0085] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0086] Figure 3 This is a flowchart illustrating a control method for a cleaning system provided in an embodiment of this application. This embodiment is based on... Figure 1 The cleaning system architecture shown includes the following steps:

[0087] S301, In response to a request to clean the cleaning device, control the cleaning device to move to a target area near the base station based on a first vision sensor on the cleaning device.

[0088] The cleaning request can originate from an automatic cleaning command triggered by the cleaning equipment based on the cumulative usage time and dirt detection results of the cleaning component, or it can originate from a manual cleaning command triggered by the user. A first visual sensor is mounted on the cleaning equipment body to capture images of the environment in front of the cleaning equipment to identify the location of the base station. The base station is a fixed docking and cleaning structure with a receiving cavity. A cleaning tank is provided on the bottom surface of the receiving cavity, and at least two image markers with known physical spacing are provided on the walls of the receiving cavity.

[0089] When the cleaning equipment needs to clean the cleaning component at the end of the robotic arm, the controller initiates the cleaning task. A first vision sensor (e.g., a binocular camera) acquires images of the environment in front of the cleaning equipment, identifying image markers on the walls of the base station's housing. The controller extracts the corner pixel positions of these image markers. Since the physical coordinates of each corner point are pre-calibrated and known, the relative pose (including distance, direction, and orientation angle) between the cleaning equipment and the base station is calculated based on the correspondence between the corner pixel positions and their physical coordinates. The controller then plans the movement trajectory accordingly, controlling the cleaning equipment to autonomously move to a target area near the base station. This target area is a predetermined position that the cleaning equipment needs to reach before the robotic arm extends, ensuring the robotic arm can smoothly enter the base station's housing and that the second vision sensor at the end of the robotic arm can obtain an effective field of view.

[0090] S302, Control the robotic arm to extend, and during the process of the robotic arm extending into the cleaning tank, acquire a first image based on the second vision sensor on the end of the robotic arm, the first image including some or all image markers.

[0091] The robotic arm is mounted on a cleaning device and can extend or retract relative to the device. A second vision sensor and a cleaning component are located at the end of the robotic arm. The second vision sensor is used to acquire image markers on the wall of the base station housing at close range as the robotic arm extends into the housing.

[0092] After the cleaning equipment reaches the target area, the controller controls the robotic arm to extend from its retracted state. During the extension of the robotic arm into the cleaning tank, a second vision sensor (e.g., a monocular camera) acquires images, obtaining a first image. Since there are multiple image markers on the wall of the base station housing cavity and their physical spacing is known, the first image may contain all the image markers, or it may only contain some of them due to obstruction by the robotic arm itself, the base station housing cavity structure, or damage to the markers.

[0093] S303. Based on the first image, control the cleaning component to move to the target position within the cleaning tank.

[0094] The controller calculates the pose of the robotic arm end effector (i.e., the cleaning component) relative to the cleaning tank based on the corner pixel positions of the image markers identified in the first image, combined with the known physical coordinates of the corner points of each image marker, and the correspondence between the corner pixel coordinates and the physical coordinates. The controller then drives the robotic arm to move according to the calculated pose, gradually moving the cleaning component to the target position within the cleaning tank.

[0095] S304. Send a cleaning command to the base station to trigger the cleaning task for the cleaning components.

[0096] Once the cleaning component reaches the target position within the cleaning tank, the controller sends a cleaning command to the base station via wired or wireless communication. Upon receiving the cleaning command, the base station initiates the cleaning task, such as injecting cleaning fluid into the cleaning tank, driving the scrubbing components, or activating the spray device to clean the component.

[0097] Through the above steps, this embodiment sets vision sensors at the end of the cleaning device and the robotic arm, respectively, and uses image markers with known physical distances on the wall of the base station's receiving cavity as a reference system. First, the cleaning device guides itself to the target area near the base station using the first vision sensor, completing coarse positioning. Then, the second vision sensor at the end of the robotic arm acquires image marker information during insertion, guiding the cleaning component to the target position within the cleaning tank, completing fine positioning. An effective connection is formed between coarse and fine positioning, ensuring precise alignment between the cleaning component and the cleaning tank even when the image markers on the base station's receiving cavity wall are partially obscured or the cleaning device's docking is off-center, thus improving the cleaning success rate.

[0098] Based on the above, the following section explains the handling method when the number of identifiable image markers in the first image is insufficient.

[0099] In step S303 (controlling the cleaning component to move to the target position within the cleaning tank based on the first image), the following process is specifically included:

[0100] First, the controller acquires the corner pixel positions of identifiable image markers in the first image. The corner pixel positions refer to the two-dimensional coordinates of the four vertices of the image markers in the first image, which are extracted by a corner detection algorithm after the second vision sensor acquires the image.

[0101] When the number of identifiable image markers is less than a preset threshold (e.g., some image markers are obscured, damaged, or outside the field of view due to the shooting angle), the controller obtains the predicted corner pixel positions of the unidentified image markers based on the corner pixel positions of the identified image markers and the known physical coordinates of the corners of each image marker. The physical coordinates of the corners refer to the three-dimensional coordinates of the corners of the image markers in the base station's own coordinate system; these are known data obtained in advance through calibration and stored in the controller.

[0102] The calculation process for predicting corner pixel positions is as follows: First, based on the corner pixel positions and corresponding physical coordinates of the identified image markers, the coarse pose (including rotation matrix and translation vector) of the second vision sensor relative to the cleaning tank is solved using the perspective-n-point (PnP) algorithm. Then, the physical coordinates of each corner point of the unidentified image markers are substituted into this coarse pose relationship, and the theoretical pixel positions that these corner points should appear in the image are deduced through perspective projection transformation, which are the predicted corner pixel positions.

[0103] Optionally, for the predicted corner pixel positions, the controller can perform image feature detection within a small area around them: if a real corner is detected, the pixel position of the real corner is merged with the already identified corner pixel positions and used together for subsequent pose calculation; if no corner is detected, the predicted corner is discarded, and only the already identified corner pixel positions are used for pose calculation. If using only the already identified corner pixel positions is still insufficient to meet the minimum number of corners required for pose calculation, the current localization process is terminated and a prompt message is generated.

[0104] After obtaining the predicted corner pixel positions, the controller determines the pose of the robotic arm relative to the cleaning tank based on the identified corner pixel positions of the image markers, the predicted corner pixel positions, and the physical coordinates of each image marker's corner. Specifically, the controller merges the identified corner pixel positions with the predicted corner pixel positions to form a complete set of corner pixel coordinates. Then, it maps these pixel coordinates one-to-one with the physical coordinates corresponding to each corner and solves them uniformly using a visual positioning algorithm to obtain the pose of the robotic arm relative to the cleaning tank. That is, the pose calculation result represents the current position and orientation of the robotic arm's end effector.

[0105] Finally, based on the deviation between the calculated pose and the target pose, the controller controls the cleaning component to move to the target position within the cleaning tank.

[0106] Using the above method, when the number of identifiable image markers is insufficient, the system uses the corner pixel positions of the already identified image markers and their known physical coordinates to estimate the corner pixel positions of the occluded image markers. The predicted corner pixel positions are then used together with the already identified corner pixel positions for pose calculation. Since the physical coordinates of each image marker's corner are fixed, and the relative positional relationships between the markers are determined, the prediction results have high reliability. Thus, even when image markers are partially occluded, the system can still obtain a sufficient number of corner information for accurate positioning, avoiding positioning failures due to missing markers and ensuring that the cleaning component can be accurately delivered to its target position within the cleaning tank.

[0107] Based on the above, the specific implementation method for triggering the cleaning task will be further explained below.

[0108] In step S304 (sending a cleaning command to the base station to trigger the cleaning task of the cleaning component), after the cleaning component reaches the target position in the cleaning tank, the controller controls the robotic arm to rotate, causing the cleaning component to rotate in the cleaning tank.

[0109] Specifically, the controller sends a rotation command to the robotic arm drive unit, and the robotic arm drives the cleaning component to rotate within the cleaning tank through the rotational movement of its joints. During the rotation, the cleaning component generates relative motion with the cleaning medium (such as water or cleaning fluid) and the scrubbing structure on the inner wall of the cleaning tank, thereby peeling off and rinsing away the dirt attached to the surface of the cleaning component.

[0110] The controller can control the speed, direction, and duration of the robotic arm based on the type or degree of soiling of the part being cleaned. For example, it can use alternating forward and reverse rotation or uniform unidirectional rotation to ensure that all areas of the surface of the part being cleaned are cleaned evenly.

[0111] By controlling the rotation of the robotic arm, the cleaning components are driven to rotate in the cleaning tank. A stable relative motion is formed between the cleaning components and the cleaning medium in the cleaning tank, making the cleaning action more complete and uniform, and avoiding the problem of insufficient cleaning in some areas due to the cleaning components being stationary.

[0112] After the cleaning task is completed, the cleaning system also needs to return the cleaned parts to their designated locations. The cleaning system also includes a separate accessory compartment, independent of the base station, for storing the cleaned parts. The controller controls the robotic arm to remove the cleaned parts from the cleaning tank and then controls the robotic arm to move the cleaned parts back to the accessory compartment.

[0113] Specifically, after the cleaning task is completed, the controller first drives the robotic arm to remove the cleaned part from the cleaning tank and detach it from the base station's receiving cavity. Then, the controller controls the robotic arm to move the cleaned part to the opening of the accessory compartment and place it in the predetermined storage location within the accessory compartment.

[0114] The storage location of the parts compartment can be equipped with limiting grooves, guide surfaces, or elastic retaining elements to position and fix the cleaning parts, preventing them from shaking or falling out within the compartment. Optionally, image markers can also be set on the parts compartment for recognition by the first and second vision sensors, so as to perform positioning correction on the movement of the cleaning equipment and the placement of the robotic arm during the repositioning process, thereby improving the repositioning accuracy.

[0115] By setting up an accessory compartment independent of the base station, the cleaning parts can return to a dedicated storage location after cleaning. The separation of cleaning and storage makes the return location of the cleaning parts clear, the retrieval action stable, and facilitates the robotic arm to grab and use them again in the next cleaning task.

[0116] The following is a detailed explanation of the process of moving the cleaned parts back to the parts storage after the cleaning task is completed.

[0117] The front of the accessory compartment has at least two image markers with known physical spacing. The physical coordinates of the corner points of each image marker are pre-calibrated and stored in the memory of the cleaning device. The corner point physical coordinates can be understood with reference to the description in the previous embodiments, and will not be repeated here. Specifically, the process of moving the cleaning parts back to the accessory compartment includes the following two stages.

[0118] Phase 1: The cleaning equipment is moved to the target area near the parts warehouse.

[0119] The cleaning equipment acquires images containing the accessory compartment using a first vision sensor. The controller processes the images, identifies image markers on the front of the accessory compartment, and extracts the corner pixel positions of these image markers.

[0120] The controller establishes a correspondence between the extracted corner pixel positions and their corresponding physical coordinates. For example, if the physical coordinates of a corner point on the parts compartment are (X1, Y1, Z1), its corresponding pixel coordinates in the image are (u1, v1). With multiple such correspondences, the controller calculates the pose of the first visual sensor relative to the parts compartment using the PnP algorithm. Combining this with the fixed installation position of the first visual sensor on the cleaning equipment, the controller further calculates the relative pose between the cleaning equipment and the parts compartment, including the distance, direction, and orientation angle of the cleaning equipment relative to the parts compartment. Based on the calculated relative pose, the controller plans a movement path and controls the cleaning equipment to move to the target area near the parts compartment.

[0121] Optionally, an additional image marker can be placed below the accessory compartment, positioned below the image marker on the front of the accessory compartment as identified by the controller. When the cleaning device is at a distance, the front image marker may be out of view, but the image marker below it remains visible. Therefore, the cleaning device can prioritize identifying the image marker below it, achieving rapid coarse positioning.

[0122] Furthermore, the first visual sensor can be a binocular vision sensor. A binocular vision sensor acquires two images of the same scene using two cameras, left and right, and calculates the depth information of each pixel in the image using the principle of parallax, forming a binocular point cloud. Once the cleaning equipment approaches the parts compartment, the controller jointly calculates the binocular point cloud information with the corner pixel positions and physical coordinates marked in the lower image. This effectively eliminates the uncertainty in depth estimation inherent in monocular vision, further improving the accuracy of the relative pose calculation between the cleaning equipment and the parts compartment.

[0123] Phase 2: The robotic arm places the cleaned parts into the parts compartment.

[0124] After the cleaning equipment reaches the target area near the parts compartment, the second vision sensor at the end of the robotic arm activates. The controller acquires a second image through the second vision sensor, which contains image markers on the front of the parts compartment. The controller extracts the corner pixel positions of the image markers in the second image and, combined with the known physical coordinates of the corner points of each image marker, calculates the pose of the second vision sensor relative to the parts compartment using the same visual positioning algorithm (e.g., PnP algorithm). Since the second vision sensor is fixedly mounted on the end of the robotic arm, its relative positional relationship with the end of the robotic arm is known (often referred to as hand-eye calibration parameters). Based on this, the controller further calculates the pose of the end of the robotic arm relative to the parts compartment.

[0125] The controller compares the calculated pose of the robotic arm's end effector relative to the parts compartment (i.e., the current position and orientation of the end effector) with the target pose, and controls the robotic arm to move to the target pose based on the deviation between the two. This target pose is where the robotic arm is perpendicular to the face of the parts compartment, meaning the entry direction of the robotic arm's end effector is aligned with the normal direction of the plane containing the front of the parts compartment. This allows the robotic arm to smoothly place the cleaning parts into the parts compartment in an attitude perpendicular to the front of the compartment. This perpendicular entry attitude avoids collisions between the cleaning parts and the edge of the parts compartment or improper placement due to angled placement.

[0126] In this way, the cleaning equipment is responsible for bringing itself to the vicinity of the parts compartment, and the robotic arm is responsible for accurately placing the cleaned parts into the predetermined positions in the parts compartment. Even if there is a deviation in the parking position of the cleaning equipment, the robotic arm can independently calculate the pose and make compensation corrections through its own visual observation, so that the cleaned parts are accurately placed, avoiding placement failures or collisions with the parts compartment.

[0127] If the number of identifiable image markers in the second image is less than a preset threshold during the process of the robotic arm placing the cleaning parts into the accessory compartment, the following procedure will be followed.

[0128] When the number of identifiable image markers in the second image is less than a preset threshold (e.g., some image markers are obscured, damaged, or outside the field of view due to the shooting angle), the controller obtains the predicted corner pixel positions of the unidentified image markers based on the corner pixel positions of the identified image markers and the known physical coordinates of the corners of each image marker.

[0129] The calculation process for predicting the corner pixel position is similar to that described in the previous embodiment: First, based on the corner pixel position of the identified image marker and the corresponding physical coordinates of the corner, the PnP algorithm is used to solve the coarse pose of the second vision sensor relative to the parts compartment; then, the physical coordinates of each corner of the unidentified image marker are substituted into the coarse pose relationship, and the theoretical pixel position that these corners should appear in the image is deduced through perspective projection transformation, which is the predicted corner pixel position.

[0130] Optionally, for the predicted corner pixel position, the controller can perform image feature detection in a small area around it: if a real corner is detected, the pixel position of the real corner is used for subsequent pose calculation; if it is not detected, the predicted corner is discarded, and only the identified corner pixel position is used for pose calculation.

[0131] After obtaining the predicted corner pixel positions, the controller determines the pose of the robotic arm relative to the parts compartment based on the identified corner pixel positions of the image markers, the predicted corner pixel positions, and the physical coordinates of each corner. Specifically, the controller merges the identified corner pixel positions with the predicted corner pixel positions to form a complete set of corner pixel coordinates. Then, it maps these pixel coordinates one-to-one with the physical coordinates of each corner and solves them uniformly using the PnP algorithm to obtain the pose of the robotic arm relative to the parts compartment. That is, the pose calculation result represents the current position and orientation of the robotic arm's end effector.

[0132] The controller moves the robotic arm to the target pose based on the deviation between the calculated pose and the target pose, and places the cleaning parts in the accessory compartment.

[0133] Using the above method, even when some image markers on the front of the parts compartment are obscured, the system can still use the corner pixel positions of the identified image markers and the known physical coordinates of the corners of each image marker to calculate the predicted corner pixel positions of the obscured image markers. The predicted corner points are then used together with the identified corner points for pose calculation. This avoids the problem of incomplete image marker recognition leading to positioning failure or decreased positioning accuracy, allowing the robotic arm to accurately place the cleaning parts in the predetermined positions within the parts compartment even when image markers are partially obscured, thus improving the reliability and success rate of the placement process.

[0134] When the cleaning equipment moves to the target area near the base station, the first vision sensor can be a binocular vision sensor to improve positioning accuracy.

[0135] Specifically, a binocular vision sensor on the cleaning equipment acquires a third image, which includes image markers on the wall of the base station housing cavity. The binocular vision sensor contains two spaced-apart imaging units, and the depth value of a point can be calculated using the disparity information of the same feature point in the left and right images.

[0136] The controller extracts the corner pixel positions of the image markers in the third image and establishes a correspondence between these corner pixel positions and the corresponding corner physical coordinates. The PnP algorithm is used to calculate the first pose (including distance, direction and orientation angle) of the cleaning equipment relative to the base station.

[0137] The controller plans a movement path based on the calculated first pose and controls the cleaning equipment to move to the target area near the base station. During the movement, the binocular vision sensor can continuously acquire new third images, and the controller updates the first pose in real time, correcting the movement trajectory until the cleaning equipment reaches the target area.

[0138] By employing binocular vision sensors, the cleaning equipment can simultaneously acquire spatial information of base station image markers within a large field of view and continuously correct its relative pose during approach, making the relative relationship between the equipment's docking position and the base station more stable, thus providing a reliable initial positioning basis for the subsequent extension of the robotic arm into the cleaning tank.

[0139] Before the cleaning equipment moves to the target area near the base station, if the first pose does not meet the preset conditions, the positioning and moving steps need to be repeated until the preset conditions are met.

[0140] The preset conditions include: the distance between the cleaning equipment and the base station is greater than a preset distance threshold, and the deviation between the orientation of the cleaning equipment and the orientation of the base station is less than a preset angle threshold. The preset distance threshold is used to ensure that there is a sufficient distance between the cleaning equipment and the base station so that the robotic arm can extend smoothly; the preset angle threshold is used to ensure that the orientation of the cleaning equipment is basically consistent with the orientation of the base station. The distance threshold and angle threshold can be preset according to the docking accuracy requirements of the cleaning equipment and the working range of the robotic arm.

[0141] The specific steps for repeated execution are as follows:

[0142] First, a third image is reacquired using a binocular vision sensor. The third image contains image markings on the walls of the base station housing cavity.

[0143] Secondly, the updated first pose of the cleaning device relative to the base station is determined based on the image markers in the third image. The controller extracts the corner pixel positions of the image markers in the third image, and combines them with the known physical coordinates of the corner points of each image marker to calculate the distance and orientation angle between the current cleaning device and the base station using the PnP algorithm.

[0144] Then, the cleaning equipment is moved based on the updated first pose. The controller generates movement control commands based on the calculated distance and orientation deviation, driving the cleaning equipment to move towards the base station or adjust its direction.

[0145] Repeat the above steps until the first pose meets the preset conditions (i.e., the distance is greater than the preset distance threshold and the orientation deviation is less than the preset angle threshold). At this point, the cleaning equipment is in a suitable position and orientation for the robotic arm to extend into, and can proceed to the subsequent target area docking steps.

[0146] Optionally, if the first pose still does not meet the preset conditions after repeating the preset number of times, the current process is terminated and a prompt message is generated.

[0147] Through multiple pose iterations, the cleaning equipment can gradually correct the cumulative errors caused by a single positioning and movement as it approaches the base station, avoiding excessive deviation in a single positioning result that could affect the subsequent extension of the robotic arm. Entering the target area only after both distance and orientation meet threshold conditions allows for a more stable posture as the cleaning equipment approaches the base station, providing reliable initial conditions for the subsequent extension of the robotic arm into the cleaning tank.

[0148] Similar to the repeated positioning when the cleaning device approaches the base station in the previous embodiment, the following steps can also be repeated a preset number of times before the cleaning device is moved to the target position in the cleaning tank, so as to gradually approach the target position.

[0149] The specific steps for repeated execution are as follows:

[0150] First, a first image is acquired (or the first image is reacquired) using a second visual sensor. The first image contains image markings on the wall of the base station housing cavity.

[0151] Secondly, based on the corner pixel positions of the image markers in the first image, the pose of the robotic arm relative to the cleaning tank is re-determined. The controller extracts the corner pixel positions of the image markers in the first image, combines them with the known physical coordinates of the corner points of each image marker, and uses the PnP algorithm to calculate the pose of the current robotic arm end effector (i.e., the cleaning part) relative to the cleaning tank (i.e., the current position and orientation of the robotic arm end effector).

[0152] Then, the cleaning component is moved according to the redefined pose. The controller generates fine-tuning control commands based on the deviation between the calculated pose and the target pose, driving the robotic arm to move the cleaning component a short distance towards the target position in the cleaning tank.

[0153] Repeat the above steps until the preset number of times is reached (e.g., 2 or 3 times). Each time the robotic arm is repeated, its position and orientation are updated and corrected based on the latest acquired images, so that the cleaning part gradually approaches the target position.

[0154] Through multiple iterative corrections, the robotic arm can overcome visual recognition errors or motion control errors that may exist in a single positioning, enabling the cleaning parts to reach the target position in the cleaning tank with higher accuracy, and avoiding the cleaning parts failing to be properly delivered into the cleaning tank or interfering with the edge of the cleaning tank due to inaccurate positioning in one go.

[0155] As mentioned above Figure 2 The image markers described can be set on multiple walls of the base station housing cavity, specifically including at least two of the front wall, side wall, and top wall. Figure 2An exemplary layout is shown, in which three image markers are respectively set on the left, front, and right walls of the base station housing cavity, but the actual layout is not limited to this.

[0156] For example, image markers can be set on the front wall and side wall simultaneously, or on the front wall and top surface simultaneously, or on the side wall and top surface simultaneously, or on all three walls simultaneously. The physical coordinates of each image marker are pre-calibrated, and the spatial relationships between the markers are known.

[0157] By distributing image markers across multiple walls, when the robotic arm extends from the base station's receiving cavity opening, even if the robotic arm itself, the cleaning component, or the cavity edge obscures an image marker on one wall, the image markers on the other walls can still be observed by the second vision sensor. The spatial relationships between the image markers on different walls are pre-calibrated and known. The controller can use the image markers observed on any wall to establish the spatial relationship between the cleaning component and the cleaning tank, completing pose calculation, thereby improving the continuity and stability of pose calculation and increasing the positioning accuracy of the cleaning component entering the cleaning tank.

[0158] Based on the aforementioned embodiments, this embodiment describes the installation method of the second visual sensor and the handling method when image marker recognition is incomplete. The second visual sensor is side-mounted at the end of the robotic arm, that is, its imaging optical axis forms a certain angle with the axis of the robotic arm, so as to observe the image markers on the cavity wall from the side after the robotic arm extends into the base station receiving cavity. Side-mounting means that the second visual sensor is installed on the side of the end of the robotic arm, rather than directly in front, so that even if the front space of the end of the robotic arm is limited after the robotic arm extends into the base station receiving cavity, the image markers on the wall can still be observed from the side view of the base station receiving cavity.

[0159] When acquiring the first image through the second vision sensor, if the number of identifiable image markers in the first image is less than a preset threshold (e.g., some image markers are occluded or outside the field of view), the controller adjusts the position of the robotic arm according to the corner pixel position of the identified image markers, and then re-acquires the image for identifying the image markers through the second vision sensor.

[0160] Specifically, the controller reads the corner pixel positions of the identified image markers and, combined with the distribution direction of these image markers in the image, calculates the amount of displacement that the robotic arm's end effector needs to adjust (e.g., a small translation along the horizontal or vertical direction, or a small rotation about the end effector axis). Then, the controller converts the adjustment amount into motion commands for each joint of the robotic arm, driving the end effector to move to the new position.

[0161] After the robotic arm adjusts its position, the second vision sensor acquires images again. Because the position of the robotic arm's end effector has changed, the field of view of the second vision sensor also changes accordingly. Image markers that were originally obscured or outside the field of view may enter the field of view, thereby increasing the number of recognizable image markers.

[0162] In this way, when the number of identifiable image markers in the first image is insufficient, the system can actively adjust the position of the robotic arm's end effector, changing the field of view of the second vision sensor. This allows image markers that were originally obscured or outside the field of view to enter the visible area, thereby obtaining more identifiable image markers. Since the side-mounted method itself is prone to limited field of view, this embodiment compensates for this limitation by actively adjusting the position of the robotic arm, improving the recognition rate of image markers and the reliability of pose calculation. This avoids the situation where positioning is impossible due to poor initial field of view, allowing the robotic arm to continue to complete the alignment and placement operations of the cleaning parts.

[0163] The foregoing embodiments described the control method for the cleaning system provided in this application. To implement the functions of the above method, the cleaning system, as the executing entity, may include hardware structures and / or software modules, implementing the functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0164] Figure 4 This is a schematic diagram of the structure of a control device for a cleaning system provided in an embodiment of this application. Figure 4 As shown, the control device 500 of the cleaning system includes a control module 501. The control module 501 is used to execute the control method of the cleaning system described in any of the foregoing method embodiments.

[0165] Specifically, the control module 501 is used to: control the cleaning equipment to move to a target area near the base station based on a first vision sensor on the cleaning equipment; control the robotic arm to extend, and during the process of the robotic arm extending into the cleaning tank, acquire a first image based on a second vision sensor on the end of the robotic arm, the first image including some or all image markers; control the cleaning component to move to a target position in the cleaning tank based on the first image; and send a cleaning command to the base station to trigger a cleaning task for the cleaning component.

[0166] Optionally, the control module 501 is further configured to: obtain the corner pixel positions of identifiable image markers in the first image; if the number of identifiable image markers is less than a preset threshold, obtain the predicted corner pixel positions of unidentified image markers based on the corner pixel positions of the identified image markers and the physical coordinates of the corners of each image marker; determine the pose of the robotic arm relative to the cleaning tank based on the identified corner pixel positions, the predicted corner pixel positions, and the physical coordinates of the corners of each image marker; and control the cleaning component to move to the target position based on the pose.

[0167] Optionally, the control module 501 is also used to: control the rotation of the robotic arm to drive the cleaning parts to rotate in the cleaning tank.

[0168] Optionally, the control module 501 is also used to: after the cleaning task is completed, control the robotic arm to remove the cleaned parts out of the cleaning tank, and control the robotic arm to move the cleaned parts back to the parts compartment.

[0169] Optionally, the control module 501 is further configured to: control the cleaning equipment to move to a target area near the accessory compartment based on a first vision sensor on the cleaning equipment; acquire a second image through a second vision sensor on the end of the robotic arm, the second image including some or all of the image markers set on the accessory compartment; control the robotic arm to move to the target pose based on the second image, and control the robotic arm to place the cleaning parts in the accessory compartment according to the target pose.

[0170] Optionally, the control module 501 is further configured to: when the number of identifiable image markers in the second image is less than a preset threshold, obtain the predicted corner pixel positions of unidentified image markers based on the corner pixel positions of the identified image markers and the physical coordinates of the corners of each image marker; determine the pose of the robotic arm relative to the parts compartment based on the identified corner pixel positions, the predicted corner pixel positions, and the physical coordinates of the corners of each image marker; and control the robotic arm to move to the target pose based on the pose.

[0171] Optionally, the first vision sensor is a binocular vision sensor, and the control module 501 is further configured to: acquire a third image through the binocular vision sensor, the third image including image markers; determine the first pose of the cleaning device relative to the base station based on the corner pixel positions of the image markers in the third image and the physical coordinates of the corner points of each image marker; and control the cleaning device to move to the target area near the base station based on the first pose.

[0172] Optionally, the control module 501 is further configured to: before the cleaning device moves to the target area near the base station, if the first pose does not meet the preset conditions, repeatedly execute the steps of acquiring a third image through a binocular vision sensor, determining the updated first pose of the cleaning device relative to the base station based on the image markers in the third image, and controlling the movement of the cleaning device based on the updated first pose, until the first pose meets the preset conditions; wherein, the preset conditions include that the distance between the cleaning device and the base station is greater than a preset distance threshold, and the deviation between the orientation of the cleaning device and the orientation of the base station is less than a preset angle threshold.

[0173] Optionally, the control module 501 is further configured to: before controlling the cleaning component to move to the target position in the cleaning tank, repeatedly execute the steps of acquiring a first image through a second vision sensor, re-determining the pose of the robotic arm relative to the cleaning tank based on the corner pixel positions of the image markers in the first image, and controlling the movement of the cleaning component according to the re-determined pose, so as to gradually approach the target position.

[0174] Optionally, the control module 501 is further configured to: when acquiring the first image through the second vision sensor mounted on the end of the robotic arm, if the number of identifiable image markers in the first image is less than a preset threshold, adjust the position of the robotic arm according to the corner pixel position of the identified image markers, and reacquire the image for identifying the image markers through the second vision sensor.

[0175] It should be noted that the specific implementation principle and technical effect of the control device 500 of the above-mentioned cleaning system can be found in the relevant description in the foregoing method embodiments, and will not be repeated here.

[0176] This application also provides a cleaning system. For example... Figure 1 As shown, the cleaning system 100 includes a base station 200, a cleaning device 300, and a robotic arm 400. The cleaning device 300 is equipped with a first vision sensor 310, and the end of the robotic arm 400 is equipped with a second vision sensor 410. The base station 200 has a receiving cavity 210, and a cleaning tank 230 is provided on the bottom surface of the receiving cavity 210. At least two image markers with known physical spacing are provided on the wall surface of the receiving cavity 210 (220a and 220b schematically shown in the figure).

[0177] The cleaning system also includes a control module (not shown in the figure), which is used to execute the control method of the cleaning system described in any of the foregoing method embodiments. The specific control flow of the cleaning system can be referred to the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0178] This application also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the methods described in any of the foregoing embodiments of this application.

[0179] This application also provides a chip for executing instructions, which is used to perform the methods described in any of the foregoing embodiments of this application as executed by a cleaning device or cleaning system.

[0180] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the methods described in any of the foregoing embodiments of this application as performed by a cleaning device or cleaning system.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0182] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0183] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0184] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0185] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0186] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0187] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0188] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0189] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within a cleaning device or a main control device.

[0190] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0191] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0192] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0193] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A control method for a cleaning system, characterized in that, The cleaning system includes a cleaning device and a base station. The cleaning device is equipped with a robotic arm, and a cleaning component is disposed at the end of the robotic arm. The base station has a receiving cavity, the bottom surface of which is provided with a cleaning tank for receiving the cleaning component. At least two image markers with known physical spacing are disposed on the walls of the receiving cavity. The method includes: In response to a request to clean the cleaning device, the cleaning device is controlled to move to a target area near the base station based on a first visual sensor on the cleaning device; The robotic arm is controlled to extend, and during the process of the robotic arm extending into the cleaning tank, a first image is acquired based on a second vision sensor at the end of the robotic arm, the first image including some or all of the image markers; Based on the first image, control the cleaning component to move to the target position within the cleaning tank; A cleaning command is sent to the base station to trigger a cleaning task for the cleaning component.

2. The method according to claim 1, characterized in that, The step of controlling the cleaning component to move to the target position within the cleaning tank based on the first image includes: Obtain the corner pixel positions of identifiable image markers in the first image; If the number of identifiable image markers is less than a preset threshold, the predicted corner pixel positions of unidentified image markers are obtained based on the corner pixel positions of the identified image markers and the physical coordinates of the corners of each image marker. Based on the corner pixel positions of the identified image markers, the predicted corner pixel positions, and the physical coordinates of the corners of each image marker, the pose of the robotic arm relative to the cleaning tank is determined. The cleaning component is moved to the target position based on the stated pose.

3. The method according to claim 1, characterized in that, The triggering of the cleaning task for the cleaning component includes: The robotic arm is controlled to rotate, causing the cleaning component to rotate within the cleaning tank.

4. The method according to claim 3, characterized in that, The cleaning system also includes an accessory compartment independent of the base station, which is used to store the cleaning parts; after the cleaning task is completed, the robotic arm is controlled to remove the cleaning parts from the cleaning tank and then move the cleaning parts back to the accessory compartment.

5. The method according to claim 4, characterized in that, The control of the robotic arm to move the cleaning component back to the accessory compartment includes: Based on the first visual sensor on the cleaning equipment, the cleaning equipment is controlled to move to the target area near the accessory compartment; A second image is acquired by a second vision sensor at the end of the robotic arm, the second image including some or all of the image markers set on the accessory compartment; Based on the second image, the robotic arm is controlled to move to the target pose, and according to the target pose, the robotic arm is controlled to place the cleaning part into the accessory compartment.

6. The method according to claim 5, characterized in that, The step of controlling the robotic arm to move to the target pose based on the second image includes: If the number of identifiable image markers in the second image is less than a preset threshold, the predicted corner pixel positions of the unidentified image markers are obtained based on the corner pixel positions of the identified image markers and the physical coordinates of the corners of each image marker. Based on the corner pixel positions of the identified image markers, the predicted corner pixel positions, and the physical coordinates of the corners of each image marker, the pose of the robotic arm relative to the parts compartment is determined. The robotic arm is controlled to move to the target pose based on the pose.

7. The method according to claim 1, characterized in that, The first visual sensor is a binocular visual sensor; the control of the cleaning equipment to move to the target area near the base station based on the first visual sensor on the cleaning equipment includes: A third image is acquired using the binocular vision sensor, the third image including the image markers; Based on the corner pixel positions of the image markers in the third image and the physical coordinates of the corners of each image marker, the first pose of the cleaning device relative to the base station is determined. Based on the first pose, the cleaning equipment is controlled to move to a target area near the base station.

8. The method according to claim 7, characterized in that, Before the cleaning equipment moves to the target area near the base station, if the first pose does not meet the preset conditions, the following steps are repeated until the first pose meets the preset conditions: The third image is acquired using the binocular vision sensor; The updated first pose of the cleaning device relative to the base station is determined based on the image markers in the third image; The cleaning equipment is moved based on the updated first pose. The preset conditions include that the distance between the cleaning device and the base station is greater than a preset distance threshold, and the deviation between the orientation of the cleaning device and the orientation of the base station is less than a preset angle threshold.

9. The method according to claim 1, characterized in that, Before controlling the cleaning component to move to the target position within the cleaning tank, the following steps are repeated a preset number of times: The first image is acquired using the second visual sensor; Based on the corner pixel positions of the image markers in the first image, the pose of the robotic arm relative to the cleaning tank is re-determined; The cleaning component is moved according to the redefined pose to gradually approach the target position.

10. The method according to claim 1, characterized in that, The image markers are disposed on multiple walls of the receiving cavity, including at least two of the front wall, side wall, and top surface.

11. The method according to claim 1, characterized in that, The second vision sensor is mounted on the end of the robotic arm. When the first image is acquired through the second vision sensor, if the number of identifiable image markers in the first image is less than a preset threshold, the position of the robotic arm is adjusted according to the corner pixel position of the identified image markers, and the image for identifying the image markers is acquired again through the second vision sensor.

12. A control device for a cleaning system, characterized in that, The cleaning system includes a cleaning device and a base station. The cleaning device is equipped with a robotic arm, and a cleaning component is disposed at the end of the robotic arm. The base station has a receiving cavity, the bottom surface of which has a cleaning tank for receiving the cleaning component. At least two image markers with known physical spacing are disposed on the walls of the receiving cavity. The control device includes: A control module for performing the method as described in any one of claims 1 to 11.

13. A cleaning system, characterized in that, The cleaning system includes a cleaning device and a base station. The cleaning device is equipped with a robotic arm, and a cleaning component is provided at the end of the robotic arm. The base station has a receiving cavity, and a cleaning tank for receiving the cleaning component is provided on the bottom surface of the receiving cavity. At least two image markers with known physical spacing are provided on the wall surface of the receiving cavity. The cleaning system further includes a control module for performing the method as described in any one of claims 1 to 11.

Images