Algorithm for cleaning robot and mechanical arm to combine with visual identification and grab cleaning tool

Abstract

The invention discloses an algorithm combining a cleaning robot and a mechanical arm with visual recognition and grabbing a cleaning tool, relates to the field of cleaning tool grabbing, and provides the following scheme aiming at the problem that the existing cleaning tool replacement mode mostly depends on manual operation or a complex mechanical structure. A mechanical arm is connected to one side of the upper end of the movable machine body, an observation camera is connected to one end of the mechanical arm, image information collected by the observation camera is transmitted to a control system in the robot, a deep learning model in the system processes the image information and converts the image information into three-dimensional point cloud data, tool features are extracted and recognized, and the three-dimensional point cloud data are obtained. According to the robot, information data is transmitted to a control system of the mechanical arm by observing the image processing and deep learning technology in the camera, and the control system operates the rotating arm and the two-finger clamping jaw to accurately and rapidly grab the first finger tip fixing frame and the second finger tip fixing frame above.

Description

Technical Field

[0001] The present invention relates to the field of cleaning robots, and in particular to an algorithm for a cleaning robot and a mechanical arm to combine visual recognition and grasp cleaning tools. Background Art

[0002] In the field of automated cleaning, cleaning robots are becoming increasingly popular and playing an increasingly important role. Traditional methods of replacing cleaning tools mostly rely on manual operation or complex mechanical structures. For example, some cleaning robots use physical devices such as electromagnets to replace cleaning tools using magnetic suction. Although this method has achieved automation to a certain extent, it increases the complexity and cost of the system and may be affected by factors such as electromagnetic interference, resulting in insufficient replacement accuracy and stability. Summary of the invention

[0003] The cleaning robot and the mechanical arm proposed in the present invention combine visual recognition and an algorithm for grasping cleaning tools, thus solving the existing problems.

[0004] In order to achieve the above-mentioned objectives, the present invention adopts the following technical solutions: a cleaning robot, comprising a mobile body, a mechanical arm connected to one side of the upper end of the mobile body, and the mechanical arm is a multi-angle rotation structure, and an observation camera is connected to one end of the mechanical arm, the observation camera is used to collect image information in real time, the image information collected by the observation camera is transmitted to the internal control system of the robot, the deep learning model inside the system processes the image information and converts the image information into three-dimensional point cloud data, and extracts and identifies tool features, cleaning mechanisms are connected to both sides of the upper end of the mobile body, and the mechanical arm and the observation camera cooperate with each other to grasp and replace the cleaning mechanism.

[0005] Preferably, a moving roller is provided at the lower end of the moving body, and the moving roller drives the device to move, and the mechanical arm is connected to one side of the upper end of the moving body.

[0006] Preferably, a first placement groove is provided on one side of the upper end of the mobile body, and a placement rack is connected to one side of the upper end of the mobile body adjacent to the first placement groove.

[0007] Preferably, the robotic arm comprises a rotating arm, a plurality of the rotating arms are provided, and the plurality of rotating arms are rotatably connected to each other, one end of the rotating arm on one side is connected to a two-finger gripper, and an observation camera is connected to the outer side of one end of the rotating arm close to the two-finger gripper.

[0008] Preferably, the cleaning mechanism includes a first cleaning tool, a first clamping plate is connected to one side of the first cleaning tool, and the first clamping plate is matched and engaged with a first placement groove on one side of the upper end of the mobile body, and a first fingertip fixing frame is connected to the side of the first cleaning tool away from the first clamping plate, and rectangular slots are provided on both sides of the first fingertip fixing frame, and the rectangular slots are matched with the two-finger clamps in the robotic arm.

[0009] Preferably, the cleaning mechanism further comprises a placement rack, the placement rack is connected to one side of the upper end of the mobile body, three second placement slots are provided inside the placement rack, and sliding slots are provided on both sides of the three second placement slots.

[0010] Preferably, the insides of the three second placement grooves are all connected to a second cleaning tool, one side of the second cleaning tool is connected to a second clamping plate, and a second fingertip fixing frame is provided on a side of the second cleaning tool away from the second clamping plate.

[0011] Preferably, the second clamping plate and the slide groove are matched and engaged with each other, and rectangular grooves are provided on both sides of the upper end of the second fingertip fixing frame, and the rectangular grooves are matched and engaged with the two-finger clamps in the mechanical arm.

[0012] An algorithm for a robotic arm to combine visual recognition and grasping cleaning tools is applied to the above-mentioned cleaning robot. When the cleaning robot moves to the position that needs cleaning, it determines the cleaning task according to the cleaning scene, and selects the required tools for cleaning according to the cleaning task. Because the shapes of various cleaning tools are not uniform, in order to grasp the tools stably and accurately, it is necessary to estimate different tools from different perspectives and generate appropriate grasping postures. The designed algorithm for identifying and grasping tools is GTN, which is a deep learning model for object grasping. The deep learning model consists of multiple sub-modules with different functions.

[0013] Preferably, the deep learning model includes two stages: grasping perspective estimation and grasping posture generation;

[0014] The grasping perspective estimation stage includes two submodules: PointMamba and ApproachNet. PointMamba is the backbone network for point cloud data processing, which can extract the point cloud features of the tool from the point cloud data obtained from the camera. Approach Net is a module for grasping perspective estimation.

[0015] The grasping posture generation stage includes three sub-modules: Cloud Crop, Operation Net and Tolerance Net; Cloud Crop is used to crop and preprocess the input point cloud data; Operation Net is responsible for generating grasping postures based on the cropped point cloud data and predicting the optimal grasping points and postures; Tolerance Net is used to evaluate the tolerance and stability of the grasping posture, and can analyze the reliability of the grasping posture under different conditions to ensure that the generated grasping posture is feasible in actual operation.

[0016] The beneficial effects of the present invention are: acquiring image information through a camera, autonomously identifying and analyzing the shape, size and other information of the first cleaning tool and the second cleaning tool in the cleaning mechanism, and transmitting the information data to the control system of the robotic arm, the control system operates the rotating arm and the two-finger gripper to accurately and quickly grasp the first fingertip fixing frame and the second fingertip fixing frame above, thereby achieving rapid and accurate identification and grasping of the first cleaning tool and the second cleaning tool, and flexibly responding to a variety of cleaning scenarios and task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the structure of the cleaning robot of the present invention.

[0018] Figure 2 It is a schematic diagram of the mobile body of the present invention.

[0019] Figure 3 Schematic diagram of the robotic arm of the present invention.

[0020] Figure 4 It is a schematic diagram of the cleaning mechanism of the present invention.

[0021] Numbers in the figure: 1, mobile body; 101, first placement slot; 2, mechanical arm; 201, rotating arm; 202, two-finger gripper; 3, observation camera; 4, cleaning mechanism; 401, first cleaning tool; 402, first card plate; 403, first fingertip fixing frame; 404, placement rack; 405, second placement slot; 406, slide slot; 407, second cleaning tool; 408, second card plate; 409, second fingertip fixing frame. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0023] Reference Figure 1-Figure 4A cleaning robot comprises a mobile body 1, a mechanical arm 2 is connected to one side of the upper end of the mobile body 1, and the mechanical arm 2 is a multi-angle rotating structure, and an observation camera 3 is connected to one end of the mechanical arm 2, the observation camera 3 is used to collect image information in real time, and the image information collected by the observation camera 3 is transmitted to the internal control system of the robot, and the deep learning model inside the system processes the image information to convert the image information into three-dimensional point cloud data, and extracts and identifies tool features, cleaning mechanisms 4 are connected to both sides of the upper end of the mobile body 1, and the mechanical arm 2 and the observation camera 3 cooperate with each other to grab and replace the cleaning mechanism 4, and a moving roller is arranged at the lower end of the mobile body 1, and the moving roller drives the device to move The robot arm 2 is connected to one side of the upper end of the mobile body 1, a first placement slot 101 is provided on one side of the upper end of the mobile body 1, and a placement rack 404 is connected to one side of the upper end of the mobile body 1 adjacent to the first placement slot 101. The robot arm 2 includes a rotating arm 201, a plurality of rotating arms 201 are provided, and the plurality of rotating arms 201 are rotatably connected to each other, a two-finger gripper 202 is connected to one end of one rotating arm 201, an observation camera 3 is connected to the outer side of one end of the rotating arm 201 close to the two-finger gripper 202, a control system of the robot arm 2 calculates a motion trajectory according to image data collected by the observation camera 3, and the control system operates the plurality of rotating arms 201 and the two-finger gripper 202 at the end for mobile clamping and cleaning.

[0024] refer to Figure 3 , Figure 4 The cleaning mechanism 4 includes a first cleaning tool 401, a first card plate 402 is connected to one side of the first cleaning tool 401, and the first card plate 402 is matched with the first placement groove 101 on one side of the upper end of the mobile body 1, and the first cleaning tool 401 is connected to a first fingertip fixing frame 403 on the side away from the first card plate 402, and rectangular slots are provided on both sides of the first fingertip fixing frame 403, and the rectangular slots are matched with the two-finger clamps 202 in the robot arm 2, and the cleaning mechanism 4 also includes a placement rack 404, the placement rack 404 is connected to one side of the upper end of the mobile body 1, and three second placement grooves 405 are provided inside the placement rack 404, and slide grooves 406 are provided on both sides of the three second placement grooves 405. Each second placement slot 405 is connected to a second cleaning tool 407, one side of the second cleaning tool 407 is connected to a second clamping plate 408, a second fingertip fixing frame 409 is provided on the side of the second cleaning tool 407 away from the second clamping plate 408, the second clamping plate 408 and the slide slot 406 match and engage with each other, rectangular grooves are provided on both sides of the upper end of the second fingertip fixing frame 409, the rectangular grooves match and engage with the two-finger clamp 202 in the robot arm 2, the second fingertip fixing frame 409 and the first fingertip fixing frame 403 are grasped by the two-finger clamp 202, and the second cleaning tool 407 and the first cleaning tool 401 connected below are driven to move and be pulled out from the second placement slot 405 or the first placement slot 101.

[0025] In this embodiment, first, the robot arm 2 is composed of multiple rotating arms 201 that are rotatably connected to each other. One end of one of the rotating arms 201 is connected to a two-finger gripper 202, and an observation camera 3 is installed on the side close to the two-finger gripper 202, so that the observation camera 3 can capture the image of the second cleaning tool 407 or the first cleaning tool 401 without obstruction. The observation camera 3 is started to shoot and transmit the image of the cleaning tool in real time. The deep learning model processes the image data and outputs the grasping position and posture of the tool. The position and posture information is transmitted to the control system of the robot arm 2. The system calculates the motion trajectory of the robot arm 2 based on the obtained data information. The robot arm 2 moves according to the calculated trajectory and controls the two-finger gripper 202 at one end to grasp the cleaning tool. When replacing the cleaning tool, the above steps are repeated to ensure that the robot arm can complete the replacement action accurately and quickly.

[0026] Specifically, an algorithm for a robotic arm to combine visual recognition and grasp cleaning tools is applied to the above-mentioned cleaning robot, when the cleaning robot moves to the position that needs to be cleaned. The cleaning task is judged according to the cleaning scene, and the required tools are selected for cleaning according to the cleaning task. Because the shapes of various cleaning tools are not uniform, in order to grasp the tools stably and accurately, it is necessary to estimate different tools from different perspectives and generate appropriate grasping postures (positions and postures). The recognition and grasping tool algorithm designed according to the above requirements is GTN (Grasp Tool Net), which is a deep learning model for object recognition and grasping. This model is a multi-stage deep learning model, which mainly involves the processing of image and 3D point cloud data and the generation of object grasping postures (positions and postures). The pseudo code of this model is as follows:

[0027]

[0028]

[0029]

[0030]

[0031] The deep learning model consists of two stages: grasp view estimation and grasp pose generation.

[0032] 1. Grasp the perspective estimation stage:

[0033] This stage contains two sub-modules: Point Mamba and Approach Net. Point Mamba is the backbone network for point cloud data processing, which can extract the point cloud features of the tool from the point cloud data obtained from the camera. ApproachNet is a module used to capture view estimation.

[0034] (1) Point Mamba is a point cloud data feature processing module. Its input is point cloud data and its output is the features and positions of points. In the upsampling layer, the number of final output grasping points is determined by the npoint parameter of the SA2 layer. During the forward propagation process, the number of points sampled by sa2 will be passed and ultimately determine the number of fp2_xyz. Specifically, the npoint parameter of the SA2 layer determines the number of final output grasping points. In the entire GTN model, these points will be used for further grasping view estimation and grasping posture generation.

[0035] Return value:

[0036] Features: The upsampled features are used in the subsequent network for grasping view estimation and grasping posture generation. These features contain local and global geometric information in the point cloud, which helps to identify points suitable for grasping.

[0037] end_points['fp2_xyz']: The coordinates of the upsampled points are used to determine the location of the grasping points. These coordinates represent the points that the network considers to be the most suitable for grasping.

[0038] end_points: Contains all intermediate results, such as the output of the collection abstraction layer and the upsampling layer. These results are used for subsequent processing and debugging.

[0039] (2) Approach Net is a module used to estimate the grasping perspective. Its input is the features and position of the point, and its output is the grasping perspective. It is mainly used to guide Cloud Crop to extract features, and does not directly provide an accurate grasping angle. This perspective information provides a preliminary estimate of the grasping direction. Subsequent grasping configuration estimation modules (such as Operation Net and ToleranceNet) will further determine the specific grasping configuration based on these preliminary estimates. This step-by-step processing method ensures that the model has higher flexibility and accuracy in grasping tasks.

[0040] Return value:

[0041] end_points: A dictionary containing new results, mainly including the following keys:

[0042] objectness_score: object existence score, shape is (B, 2, num_seed).

[0043] view_score: view score, shape is (B, num_seed, num_view).

[0044] grasp_top_view_inds: index of the best grasping view, shape is (B, num_seed).

[0045] grasp_top_view_score: The score of the best grasping perspective, shape is (B, num_seed).

[0046] grasp_top_view_xyz: coordinates of the optimal grasping view, shape is (B, num_seed, 3).

[0047] grasp_top_view_rot: The rotation matrix of the optimal grasping perspective, with a shape of (B, num_seed, 3, 3).

[0048] 2. Grasping posture generation stage:

[0049] This stage contains three sub-modules: Cloud Crop, Operation Net and Tolerance Net; Cloud Crop is used to extract local area features from the point cloud; Operation Net is used to estimate the operational parameters of the grasping, including the grasping angle and depth; Tolerance Net is used to estimate the tolerance of the grasping.

[0050] (1) The Cloud Crop module aggregates and processes local features around the grasping point to generate features for grasping configuration estimation. These features simulate the grasping area of ​​the robot's fingers.

[0051] Return value:

[0052] vp_features: local features extracted from the point cloud, shape is (B, mlps[-1], num_seed, num_depth), where mlps[-1] is the output dimension of the last layer.

[0053] (2) Operation Net is used to estimate the grasping operation parameters, including grasping angle, depth, and score.

[0054] The in-plane rotation angle is discretized into 12 categories, each representing 30 degrees.

[0055] The grasping depth is discretized into 4 categories, each representing a specific depth value. Each depth category represents a different position of the grasper relative to the object surface and can be set according to specific task requirements.

[0056] Return Value

[0057] end_points: The updated dictionary with the following keys:

[0058] grasp_score_pred: grasp score prediction, shape is (B, num_angle, num_seed, num_depth).

[0059] grasp_angle_cls_pred: grasp angle classification prediction, shape is (B, num_angle, num_seed, num_depth).

[0060] grasp_width_pred: grasp width prediction, shape is (B, num_angle, num_seed, num_depth).

[0061] (3) Tolerance Net is used to estimate the tolerance of grasping, which indicates the robustness of the grasping operation. The module processes the input features through a multi-layer convolutional network and generates the grasping tolerance prediction results for each grasping point at different angles and depths. These results are used to evaluate the robustness of grasping and help the model select the optimal grasping configuration in actual operation.

[0062] (4) The pred_decode function is used to decode the prediction results and generate the final grasping posture. This function combines different prediction values ​​to generate the grasping posture, including grasping score, width, depth, rotation matrix, and grasping center.

[0063] Return the crawl prediction results:

[0064] Returns a list of processed grasp prediction results, including grasp score, grasp width, grasp height, grasp depth, rotation matrix, grasp center point and object ID.

[0065] This model consists of multiple modules. After acquiring image information through the camera, each module is responsible for feature extraction, view estimation, local feature extraction and grasping parameter estimation. Finally, the prediction results are combined into a complete grasping posture through the prediction decoding function. The complete grasping posture is input into the controller of the robot arm, and the robot arm performs grasping.

[0066] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cleaning robot, comprising a mobile body (1), characterized in that: A mechanical arm (2) is connected to one side of the upper end of the mobile body (1), and the mechanical arm (2) is a multi-angle rotating structure. An observation camera (3) is connected to one end of the mechanical arm (2), and the observation camera (3) is used to collect image information in real time. The image information collected by the observation camera (3) is transmitted to the internal control system of the robot. The deep learning model in the system processes the image information to convert it into three-dimensional point cloud data, and extracts and identifies tool features. Cleaning mechanisms (4) are connected to both sides of the upper end of the mobile body (1). The mechanical arm (2) and the observation camera (3) cooperate with each other to grasp and replace the cleaning mechanism (4).

2. A cleaning robot according to claim 1, characterized in that: A moving roller is arranged at the lower end of the moving body (1), and the moving roller drives the device to move.

3. According to the cleaning robot described in claim 2, a first placement groove (101) is opened on one side of the upper end of the mobile body (1), and a placement rack (404) is connected to one side of the upper end of the mobile body (1) adjacent to the first placement groove (101).

4. A cleaning robot according to claim 1, characterized in that: The mechanical arm (2) comprises a rotating arm (201), wherein a plurality of the rotating arms (201) are provided and the plurality of rotating arms (201) are rotatably connected to each other, one end of the rotating arm (201) on one side is connected to a two-finger gripper (202), and an observation camera (3) is connected to the outer side of one end of the rotating arm (201) close to the two-finger gripper (202).

5. The cleaning robot according to claim 1, characterized in that: The cleaning mechanism (4) comprises a first cleaning tool (401), one side of the first cleaning tool (401) is connected to a first clamping plate (402), and the first clamping plate (402) is matched and engaged with a first placement slot (101) on one side of the upper end of the mobile body (1), the first cleaning tool (401) is connected to a first fingertip fixing frame (403) on the side away from the first clamping plate (402), and rectangular slots are provided on both sides of the first fingertip fixing frame (403), and the rectangular slots are matched and engaged with the two-finger clamps (202) in the mechanical arm (2).

6. A cleaning robot according to claim 5, characterized in that: The cleaning mechanism (4) further comprises a placement rack (404), wherein the placement rack (404) is connected to one side of the upper end of the mobile body (1), and three second placement slots (405) are provided inside the placement rack (404), and slide slots (406) are provided on both sides of the three second placement slots (405).

7. A cleaning robot according to claim 6, characterized in that: The three second placement slots (405) are each connected to a second cleaning tool (407), one side of the second cleaning tool (407) is connected to a second clamping plate (408), and a second fingertip fixing frame (409) is provided on the side of the second cleaning tool (407) away from the second clamping plate (408).

8. A cleaning robot according to claim 7, characterized in that: The second clamping plate (408) and the slide groove (406) are matched and engaged with each other, and rectangular grooves are provided on both sides of the upper end of the second fingertip fixing frame (409), and the rectangular grooves are matched and engaged with the two-finger clamping claws (202) in the mechanical arm (2).

9. An algorithm for a robotic arm to combine visual recognition and grasp cleaning tools, characterized in that: Applied to the cleaning robot described in any one of claims 1 to 8 above, when the cleaning robot moves to the position that needs cleaning, the cleaning task is determined according to the cleaning scene, and the required tools are selected for cleaning according to the cleaning task. Because the shapes of various cleaning tools are not uniform, in order to stably and accurately grasp the tools, it is necessary to estimate different tools from different perspectives and generate appropriate grasping postures. The designed algorithm for identifying and grasping tools is GTN, which is a deep learning model for object grasping, and the deep learning model is composed of multiple sub-modules with different functions.

10. The algorithm for a robot arm to combine visual recognition and grasp cleaning tools according to claim 9, characterized in that: The deep learning model includes two stages: grasping perspective estimation and grasping posture generation; The grasping perspective estimation stage includes two submodules: Point Mamba and Approach Net. Point Mamba is the backbone network for point cloud data processing, which can extract the point cloud features of the tool from the point cloud data acquired by the camera. Approach Net is a module used for grasping perspective estimation. The grasping posture generation stage includes three sub-modules: Cloud Crop, Operation Net and ToleranceNet; Cloud Crop is used to crop and preprocess the input point cloud data; Operation Net is responsible for generating grasping postures based on the cropped point cloud data and predicting the optimal grasping points and postures; Tolerance Net is used to evaluate the tolerance and stability of the grasping posture, and can analyze the reliability of the grasping posture under different conditions to ensure that the generated grasping posture is feasible in actual operation.

Images