Control system, control method, robot and cleaning system for a robot

Abstract

The application provides a robot control system, a control method, a robot and a cleaning system. The control system comprises: a robot configured to perform availability identification on scene perception information to obtain an availability identification result; and a terminal configured to display an availability view based on a target operation type selected by a user, generate a target task based on the scene perception information and the availability identification result, and send the target task to the robot, so that the robot performs an operation action corresponding to a target operation object based on the target task, by using a target force on the target operation object according to an object type of the target operation object by a mechanical arm of the robot. The control system can display selectable operation objects on the terminal in a visual manner. The user can select a target operation object on the terminal to quickly and accurately control the robot to perform an operation action, greatly reduce the cognitive threshold, and improve the efficiency and accuracy of controlling the robot to perform a task.

Description

Technical Field

[0001] This application relates to the field of robotics, and more particularly to a control system, control method, robot, and cleaning system for a robot. Background Technology

[0002] Currently, during robot task execution, user assistance is typically required to allow the robot to contact objects. Related technologies usually involve remote operation via joysticks, visual remote operation using smart wearable devices, or point-and-click operation on two-dimensional images. However, these methods have a high cognitive threshold, preventing users from quickly and accurately controlling the robot to complete tasks. Summary of the Invention

[0003] This application provides a robot control system, control method, robot, and cleaning system to solve the problem that existing remote operation has a high cognitive threshold, which makes it impossible for users to quickly and accurately control the robot to complete the task.

[0004] This application provides a control system for a robot, the control system of which includes: A robot is used to acquire scene perception information corresponding to objects around the robot, and to perform availability identification on the scene perception information to obtain availability identification results. The availability identification results include at least one operation object and at least one operation action that can be performed on each operation object. The terminal is configured to, based on the target operation type selected by the user, and according to the scene perception information and the availability recognition result, display an availability view including candidate operation objects capable of performing the target operation type. Based on the target operation object selected by the user in the availability view, the terminal generates a target task corresponding to the target operation object and sends the target task to the robot. The robot, based on the target operation object in the target task, uses its robotic arm to execute the operation corresponding to the target operation object with different target forces according to the object type of the target operation object. The target force is dynamically adjusted based on pressure signals collected by a pressure sensor in the robotic arm.

[0005] In one embodiment, the robot includes a vision sensor disposed at the front of the robot's body; When acquiring scene perception information corresponding to objects around the robot, the robot performs the following: The visual sensor acquires color and depth images of objects around the robot. Semantic recognition is performed on the color image and the depth image to obtain the target object type and point cloud data; The scene perception information is determined based on the color image, the target object type, and the point cloud data.

[0006] In one embodiment, the scene perception information includes target object type and point cloud data; when performing availability identification on the scene perception information to obtain availability identification results, the robot performs the following: Availability identification is performed based on the target object type and the point cloud data to determine the availability identification result.

[0007] In one embodiment, when determining the availability identification result based on the target object type and the point cloud data, the robot performs the following: Based on the pre-defined correspondence between object types and availability recognition models, the target availability recognition model corresponding to the target object type is determined; The point cloud data is used to perform availability identification based on the target availability identification model to obtain the availability identification result.

[0008] In one embodiment, the availability identification model includes an operation action identification model and a risk assessment model, wherein the operation action identification model includes a grasping identification model, a placement identification model, and a wiping identification model, and the risk assessment model includes a collision cost identification model.

[0009] In one embodiment, the scene perception information includes the target object type and a color image; when performing availability identification on the scene perception information, the robot performs the following: The color image is segmented to obtain at least one segmented region; The at least one image segmentation region is classified according to the target object type to obtain the availability recognition result.

[0010] In one embodiment, the scene perception information includes point cloud data; When displaying an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene-aware information and the availability recognition result, the terminal is used to perform: Based on the availability identification results, the operable objects whose operation actions match the target operation type and the operable areas of the operable objects are identified as the candidate operation objects. Obtain the identifier data corresponding to the candidate operation object; The target point cloud data is obtained by superimposing the identification data corresponding to the candidate operation object onto the point cloud data through spatial interpolation. The target point cloud data is rendered to obtain the availability view.

[0011] In one embodiment, the scene perception information includes a color image; When displaying an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene-aware information and the availability recognition result, the terminal is used to perform: Based on the availability identification results, the operable objects whose operation actions match the target operation type and the operable areas of the operable objects are identified as the candidate operation objects. The availability view is obtained by overlaying the color image with the identifier of the candidate operation object.

[0012] In one embodiment, before overlaying the color image with the identifier of the candidate operation object to obtain the availability view, the terminal is further configured to perform: When the viewing angle of the color image is not a preset viewing angle, a weighted back projection is performed on the color image to obtain a color image with adjusted viewing angle. When overlaying the color image with the identifier of the candidate operation object to obtain the availability view, the terminal is also used to perform: The color image after the perspective adjustment is overlaid with the identifier of the candidate operation object to obtain the availability view.

[0013] In one embodiment, the availability identification result further includes a collision cost map, which includes the collision cost value corresponding to each of the operation objects; When displaying an availability view including candidate operation objects capable of performing the target operation type based on the scene-aware information and the availability recognition result, the terminal is further configured to perform: In the availability view, operation objects whose collision cost is greater than a preset cost threshold are set to an unselectable state.

[0014] In one embodiment, before performing the operation action corresponding to the target operation object using different target forces based on the object type of the target operation object, the robot is further configured to perform: Determine a collision-free path between the robotic arm and the target object; Based on the collision-free path, determine the degree of freedom information of the joints in the robotic arm; When performing the operation action corresponding to the target operation object using different target forces according to the object type of the target operation object, the robot is also used to perform: The robotic arm is controlled to move toward the target object according to the degree of freedom information, and different target forces are used to perform the corresponding operation actions of the target object according to the object type of the target object.

[0015] In one embodiment, the scene perception information includes object positions; when determining a collision-free path between the robotic arm and the target object, the robot is further configured to perform: Based on a preset obstacle map, a path is planned according to the object position of the target object and the position coordinates of the robotic arm to obtain the collision-free path.

[0016] In one embodiment, the pressure sensing device includes a pressure sensor; when performing an operation action corresponding to the target operation object using different target forces according to the object type of the target operation object, the robot is also used to perform: Obtain the pressure value collected by the pressure sensor; The target force is adjusted according to the pressure value.

[0017] In one embodiment, the pressure sensing device includes a motor disposed in the robotic arm. When performing an operation action corresponding to the target object using different target forces according to the object type of the target operation object, the robot is also used to perform: Obtain the operating current of the motor; Based on a preset correspondence between current and pressure, the working current is converted into the pressure value of the robotic arm, and the target force is adjusted according to the pressure value.

[0018] In one embodiment, the control system further includes a cloud server; the robot is also used to perform: Multi-dimensional data is recorded at preset intervals and sent to the cloud server. The multi-dimensional data includes at least two of the following: color image, depth image, availability view, end pose of the robotic arm, pressure value of the robotic arm, torque of the robotic arm, task tag, and timestamp.

[0019] In one embodiment, the terminal is further configured to perform: The multi-dimensional data is obtained from the cloud server, and scripted test instructions are generated based on the multi-dimensional data. The scripted test instructions are sent to the robot so that the robot can execute the scripted test instructions and return the movement trajectory of the robotic arm; The scripted test results are obtained by comparing the motion trajectory with the preset expected trajectory.

[0020] Secondly, this application also provides a robot control method, applied to a robot, the control method comprising: The robot acquires scene perception information corresponding to objects around it, and performs availability identification on the scene perception information to obtain availability identification results. The availability identification results include at least one operation object and at least one operation action to be performed for each operation object. The scene-aware information and the availability recognition result are sent to the terminal so that the terminal can display an availability view including candidate operation objects that can perform the target operation type based on the target operation type selected by the user and according to the scene-aware information and the availability recognition result. Obtain the target task, which is generated by the terminal based on the target operation object selected by the user in the availability view; Based on the target object in the target task, the robot's robotic arm performs the corresponding operation action of the target object using different target forces according to the object type of the target object, and the force is dynamically adjusted based on the pressure signal collected by the pressure sensing device in the robotic arm.

[0021] Thirdly, this application also provides a robot, which includes a robot body, a chassis, a vision sensor, a robotic arm, and a controller. The chassis is equipped with a wiping component and a brushing component, and the robotic arm includes a pressure sensing device. The controller is used to execute the above-described robot control method.

[0022] Fourthly, this application also provides a cleaning system, characterized in that it comprises: Cleaning robots; and A base station, which is capable of maintaining the cleaning robot, which is the robot described above.

[0023] The robot control system, control method, robot, and cleaning system provided in this application embodiment achieve the following: the robot performs availability identification on scene perception information of surrounding objects; the terminal displays an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene perception information and availability identification results; in response to the target operation object selected by the user in the availability view, a target task corresponding to the target operation object is generated and sent to the robot; the robot, through its robotic arm, performs the operation action corresponding to the target operation object using different target forces according to the object type of the target operation object. This allows for the visual display of selectable operation objects on the terminal, enabling users to quickly and accurately control the robot to perform the operation action corresponding to the target operation object simply by selecting the target operation object on the terminal, significantly reducing the cognitive threshold for users and improving the efficiency and accuracy of controlling the robot to perform tasks.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the embodiments of this application. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic block diagram of a robot provided in an embodiment of this application; Figure 2 This is a schematic diagram of a robot control system provided in an embodiment of this application; Figure 3 This is a schematic diagram of another robot control system provided in an embodiment of this application; Figure 4 This is a schematic flowchart of a robot control method provided in an embodiment of this application; Figure 5 This is a schematic diagram of a cleaning system provided in an embodiment of this application.

[0027] Explanation of reference numerals in the attached figures: 10. Robot; 100. Robot body; 101. Chassis; 102. Robotic arm; 103. Drive motor; 104. Sensor unit; 105. Controller; 106. Battery; 107. Walking unit; 108. Communication unit; 109. Interaction unit; 110. Mopping / wiping component; 111. Brushing / sweeping component; 20. Terminal; 30. Cloud server; 40. Base station; 1000. Cleaning system. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0030] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0031] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0032] Currently, during robot task execution, user assistance is typically required to enable the robot to contact objects. Related technologies usually involve user-controlled operation via joysticks, purely visual remote control using smart wearable devices, or point-based remote control on 2D images. Joystick operation works by outputting speed or pose increments through a 2D / 3D joystick, with the robotic arm following as a servo. The operation chain involves human eye → brain → hand → joystick → inverse kinematics → arm, resulting in a long operation chain, a high cognitive threshold, requiring extensive training for novice users, and a lack of intuitive aids, leading to a high rate of false collisions. Purely visual remote control using smart wearable devices utilizes color images for hand pose redirection and 6-DoF (six degrees of freedom) control of the robotic arm, but still relies on the user to determine the "graspable area," resulting in a high cognitive load and a lack of "affordance" semantics, making it difficult to support non-grasping actions such as "wiping a table" or "flushing a toilet." The working principle of point-selection teleoperation on a two-dimensional image is that the user clicks on the 2D image → back-projects into 3D → the robotic arm moves. However, these point-selection teleoperations have the problem of large ambiguity in the 2D to 3D mapping, and there is no object / part-level semantics, so it is impossible to distinguish the specific operation object.

[0033] To address the issue of high cognitive barriers in teleoperation, where users are unsure of the appropriate posture for the robot to contact objects, hindering their ability to quickly and accurately control the robot to complete tasks, this application addresses this problem. The robot performs availability identification based on scene perception information of surrounding objects. The terminal, based on the user-selected target operation type and the scene perception information and availability identification results, displays an availability view including candidate operation objects capable of performing the target operation type. In response to the user's selection of a target operation object in the availability view, a target task corresponding to the target operation object is generated and sent to the robot. The robot, through its robotic arm, executes the corresponding operation action using different target forces depending on the object type. This allows for a visual display of selectable operation objects on the terminal. Users only need to select the target operation object on the terminal to quickly and accurately control the robot to perform the corresponding operation action, significantly reducing the cognitive barrier for users and improving the efficiency and accuracy of controlling the robot to perform tasks.

[0034] For example, robots may include, but are not limited to, lawnmowers, snowplows, cleaning robots, food delivery robots, and welcoming robots. This application uses a cleaning robot as an example for illustration.

[0035] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0036] Please see Figure 1 , Figure 1 This is a schematic block diagram of a robot 10 provided in an embodiment of this application. Figure 1 As shown, the robot 10 includes a robot body 100, a chassis 101, a robotic arm 102, a drive motor 103, a sensor unit 104, a controller 105, a battery 106, a walking unit 107, a communication unit 108, an interaction unit 109, a wiping component 110, and a brushing component 111. The chassis 101 is equipped with the wiping component 110 and the brushing component 111. The robotic arm 102 includes a pressure sensing device. The controller 105 is used to execute the robot control method of any one of the embodiments of this application.

[0037] The robotic arm 102 can be a single robotic arm or a dual robotic arm.

[0038] The sensor unit 104 mounted on the robot body 100 includes various types of sensors, such as vision sensors (e.g., cameras), ultrasonic sensors, lidar, collision sensors, distance sensors, drop sensors, counters, and gyroscopes. The camera is mounted on the side of the robot body 100, for example, to acquire environmental images in the robot's forward direction. The ultrasonic sensor is mounted on the bottom of the robot body 100 to detect the presence of obstacles in the area directly opposite the ultrasonic sensor. The lidar is mounted on the top of the robot body 100 and, during operation, obtains information about the surrounding environment, such as the distance and angle of obstacles relative to the lidar. Alternatively, a camera can be used instead of lidar; by analyzing the images captured by the camera, the distance and angle of obstacles relative to the camera can also be obtained. The collision sensor includes a collision housing and a trigger sensor. When the robot 10 collides with an obstacle through the collision housing, the collision housing moves inwards from the robot 10 and compresses an elastic buffer. After the collision housing has moved a certain distance inwards from the robot 10, it contacts the trigger sensor, which is then triggered to generate a signal. This signal can be sent to a controller within the robot body 100 for processing. After colliding with an obstacle, robot 10 moves away from the obstacle, and the collision shell returns to its original position under the action of the elastic buffer. It is evident that the collision sensor can detect obstacles and, upon collision, act as a buffer. The distance sensor can be an infrared sensor, used to detect the distance from the obstacle to the sensor. The distance sensor is located on the side of the robot body 100, allowing the measurement of the distance from obstacles near the side of the robot 10 to the sensor. The distance sensor can also be an ultrasonic rangefinder, a laser rangefinder, or a depth sensor, etc. A drop sensor is located at the bottom edge of the robot body 100. When the robot 10 moves to the edge of the ground, the drop sensor detects the risk of the robot 10 falling from a height, thus executing corresponding anti-fall reactions, such as stopping the robot 10 or moving it away from the fall location. A counter and a gyroscope are also installed inside the robot body 100. The counter is used to detect the distance traveled by the robot 10. The gyroscope is used to detect the angle of rotation of the robot 10, thereby determining the robot 10's orientation.

[0039] The controller 105 is located inside the robot body and is used to control the robot 10 to perform specific operations. This controller 105 can be, for example, a central processing unit (CPU) or a microprocessor. Figure 1As shown, the controller 105 is electrically connected to components such as the robotic arm 102, drive motor 103, sensor unit 104, battery 106, walking unit 107, communication unit 108, interaction unit 109, mopping component 110, and brushing component 111 to control these components.

[0040] The battery 106 is disposed inside the robot body 100 and is used to provide power to the robot 10. The robot body 100 is also provided with a charging component, which is used to obtain power from an external device to charge the battery 106 of the robot 10.

[0041] The walking unit 107 is a component related to the movement of the robot 10, used to drive the robot 10 to move so that the mopping component 110 and / or the brushing component 111 mop the ground.

[0042] The communication unit 108 may include, but is not limited to, a Bluetooth module, an RF module, a Wi-Fi module, a 4G module, a 5G module, an NB-IoT module, or a LoRa module. For example, the robot 10 can communicate with the terminal 20 via a 5G module.

[0043] An interaction unit 109 is mounted on the robot body 100, allowing users to interact with the robot 10. The interaction unit 109 includes components such as a power button, speaker, microphone, and touch switch / screen. Users can control the robot 10 to start or stop working by pressing the power button or touching the screen, and can also display the robot 10's working status information on the touch screen. The robot 10 can play prompts to the user through the speaker, receive control commands from the user through the microphone, or locate the user's position by receiving the user's voice.

[0044] The mopping component 110 is used to mop the surface to be cleaned, and there can be one or more mopping components 110. The mopping component 110 is, for example, a mop cloth. The mopping component 110 is located at the bottom of the robot body 100, specifically at the rear of the bottom of the robot body 100. A drive motor 103 is located inside the robot body 100, and two rotating shafts extend from the bottom of the robot body 100. The mopping component 110 is fitted onto these rotating shafts. The drive motor 103 drives the rotating shafts to rotate, thereby causing the rotating shafts to rotate the mopping component 110.

[0045] In some embodiments, robot 10 is a sweeping and mopping robot, where the sweeping component 111 and the mopping component 110 can work together. The sweeping component 111 is used to clean the surface to be cleaned. For example, the sweeping component 111 and the mopping component 110 can work simultaneously or alternately. Of course, the sweeping component 111 and the mopping component 110 can also work separately, that is, the sweeping component 111 can perform sweeping work alone, or the mopping component 110 can perform mopping work alone.

[0046] It should be understood that the robot 10 described in the embodiments of this application is merely a specific example and does not constitute a specific limitation on the robot 10 of the embodiments of this application. The robot 10 of the embodiments of this application can also be implemented in other specific ways. For example, in other implementations, the robot 10 can be more... Figure 1 The robot 10 shown may have more or fewer parts.

[0047] Please see Figure 2 , Figure 2 This is a schematic diagram of a robot control system provided in an embodiment of this application, as shown below. Figure 2 As shown, the control system of the robot may include robot 10 and terminal 20, wherein robot 10 and terminal 20 can communicate via wireless communication or wired communication.

[0048] For example, terminal 20 may include smartphones, tablets, smart wearable devices, etc. Among them, smart wearable devices may be AR glasses, AR helmets, smartwatches, smart bracelets, etc.

[0049] In some embodiments, the robot acquires scene perception information corresponding to objects around the robot and performs availability identification on the scene perception information to obtain availability identification results. The availability identification results include at least one operable object and at least one operable action that can be performed on each operable object. The terminal displays an availability view including candidate operable objects capable of performing the target operable type, based on the target operable type selected by the user, according to the scene perception information and the availability identification results. Based on the target operable object selected by the user in the availability view, the terminal generates a target task corresponding to the target operable object and sends the target task to the robot. The robot then performs the operable action corresponding to the target operable object using different target forces according to the object type of the target operable object, based on the target operable object in the target task. The target force is dynamically adjusted based on the pressure signal collected by the pressure sensing device in the robotic arm.

[0050] For example, the object of manipulation may include the object to be manipulated and the corresponding operable area. It is understood that the object to be manipulated is the object that the robot's robotic arm can manipulate, and a certain area or part of that object. For example, the object to be manipulated may be a water cup, and the operable area may be the handle of the water cup. Another example is a toilet, and the operable area may be the toilet seat. In this embodiment, the robot can identify multiple operable areas at once, such as a toilet, bottle, or shoes in a bathroom, thereby reducing the overhead of repeated detection and improving efficiency.

[0051] For example, the operation may include, but is not limited to, grasping, placing, wiping, moving, etc. For example, for a water cup, the operation may be grasping. For a toilet, the operation may be wiping.

[0052] For example, after the robot performs availability identification on the scene perception information and obtains the availability identification results, it can send the availability identification results to the terminal. For example, the availability identification results may include: toilet, wipe; banana peel, pick up and wipe the floor; water cup, grasp, move, place; and so on.

[0053] For example, after receiving the availability recognition result, the terminal can display an availability view of candidate objects for which the robot can perform the target operation type, based on the user-selected target operation type and according to scene perception information and availability recognition results. For instance, the user can first select the desired operation type on the terminal, and then the terminal will generate and display an availability view of candidate objects for which the robot can perform the target operation type, based on the user-selected target operation type (e.g., grasping, placing, wiping, moving), and according to scene perception information and availability recognition results. For example, when the user selects a grasping operation, an availability view (also known as a heatmap) of candidate objects for which the grasping operation can be displayed, such as a banana peel and a water cup. As another example, when the user selects a wiping operation, an availability view of candidate objects for which the grasping operation can be displayed, such as a banana peel and a toilet.

[0054] For example, after displaying the availability view, the terminal can generate a target task corresponding to the target operation object selected by the user in the availability view. For instance, when the user selects a toilet as the target operation object in the availability view, a target task for cleaning the toilet seat can be generated. As another example, when the user selects a water cup as the target operation object in the availability view, a target task for picking up the water cup can be generated. In this embodiment, the user can autonomously select the execution order from multiple candidate operation objects, thereby enhancing the flexibility of human-computer interaction.

[0055] In some embodiments, the user's selection instruction for a target operation or target object may include, but is not limited to, touch instructions, gesture instructions, voice instructions, text instructions, and eye-tracking instructions. For example, in application scenarios where the terminal is an AR device (e.g., AR glasses, AR helmet), the selection instruction may be a gesture instruction, a voice instruction, or an eye-tracking instruction. In application scenarios where the terminal is a tablet or smartphone, the selection instruction may be a touch instruction, a voice instruction, or a text instruction.

[0056] In some embodiments, for application scenarios where a spilled milk tea cup is on the ground, the availability view that the terminal can render includes: the milk tea cup, corresponding to a green grab arrow; the liquid edge, corresponding to an orange wipeable surface + the number "estimate n round trips"; the trash can, corresponding to a blue cube that flashes; and the sweeping and mopping mode, corresponding to the window robot cleaning module that flashes. The user can click whether to perform chassis sweeping and mopping to clean the milk tea liquid. When the user clicks "milk tea cup" once to confirm "pick up", the terminal can automatically queue the subsequent "wiping → sweeping and mopping" operations and generate target tasks without requiring the user to click multiple times.

[0057] For example, after generating the target task corresponding to the target operation object, the terminal can send the target task to the robot. Upon receiving the target task, the robot can control its robotic arm to perform the corresponding operation action based on the target operation object in the task, using different target forces according to the object type. For instance, when the target task is cleaning a toilet seat, the robot can control its robotic arm to perform a wiping action on the seat using appropriate target forces. Similarly, when the target task is picking up a water cup, the robot can control its robotic arm to pick up the handle of the cup using appropriate target forces.

[0058] For example, when performing a target task, a robot can apply different levels of force depending on the type of the object being manipulated. This force can be dynamically adjusted based on pressure or current values ​​collected by pressure sensors in the robotic arm. For instance, when cleaning or grasping an object, different levels of force are applied depending on the object's type. Furthermore, the force is dynamically adjusted based on the pressure or current values ​​fed back from the pressure sensors. For example, when grasping a vase, light pressure is needed to avoid tipping it over; when wiping a toilet seat, appropriate pressure is required to ensure effective cleaning. By applying different levels of force to perform the corresponding actions based on the object's type, the probability of successfully completing the task is increased, and damage to the object is minimized.

[0059] In the above embodiments, by having the robot perform availability identification based on scene perception information of surrounding objects, the terminal displays an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene perception information and availability identification results. In response to the target operation object selected by the user in the availability view, a target task corresponding to the target operation object is generated and sent to the robot. The robot then uses its robotic arm to perform the operation action corresponding to the target operation object with different target forces according to the object type of the target operation object. This allows for the visual display of selectable operation objects on the terminal. Users only need to select the target operation object on the terminal to quickly and accurately control the robot to perform the operation action corresponding to the target operation object, significantly reducing the cognitive threshold for users and improving the efficiency and accuracy of controlling the robot to perform tasks.

[0060] In some embodiments, the robot includes a vision sensor configured in front of the robot's body; when acquiring scene perception information corresponding to objects around the robot, the robot performs the following actions: acquiring color images and depth images corresponding to objects around the robot collected by the vision sensor; performing semantic recognition on the color images and depth images to obtain target object type and point cloud data; and determining scene perception information based on the color images, target object type, and point cloud data.

[0061] For example, the vision sensor may include an RGB-D camera and / or a ToF (Time of Flight) camera. For instance, an RGB-D camera can be used to acquire color images of objects around the robot, and a ToF camera can be used to acquire depth images of those objects. As another example, an RGB-D camera can be used to acquire both color and depth images of objects around the robot.

[0062] For example, after acquiring color and depth images of objects around the robot, a pre-defined semantic recognition model can be used to perform semantic recognition on the color and depth images to obtain the target object type and point cloud data. The semantic recognition model can include, but is not limited to, the ERNIE model, BERT model, Seq2Seq model, graph neural network model, hidden Markov model, etc. For instance, a Seq2Seq model can be used to perform semantic recognition on the color and depth images to obtain the target object type and point cloud data. It should be noted that point cloud data is a collection of a large number of discrete points in three-dimensional space, and each point can include attribute information such as the object's three-dimensional coordinates, reflection intensity, color information, timestamp, and classification label.

[0063] For example, color images, target object types, and point cloud data can be identified as scene-aware information. Subsequently, availability identification can be performed on the target object types and point cloud data within the scene-aware information, or on the color images and target object types within the scene-aware information.

[0064] In some embodiments, scene perception information includes target object type and point cloud data; when performing availability identification on scene perception information to obtain availability identification results, the robot performs the following: performing availability identification based on target object type and point cloud data to determine availability identification results.

[0065] In the embodiments of this application, when performing availability identification on scene perception information, different types of availability identification models can be used to perform availability identification on target object types and point cloud data, and availability identification data for different operation objects can be obtained, such as operation actions corresponding to the operation objects.

[0066] In some embodiments, the availability identification model includes an operation action identification model and a risk assessment model. The operation action identification model may include a grasping identification model, a placement identification model, and a wiping identification model, and the risk assessment model may include a collision cost identification model. Of course, other types of models can be added to the operation action identification model according to the actual situation, such as a handling identification model, an assembly identification model, etc. The embodiments of this application do not limit the type of operation action identification model.

[0067] The grasping recognition model can be a GraspNet model, used to output all graspable methods for objects. The placement recognition model can be a PlaceNet model, used to output all placeable methods for objects, for example, outputting "recyclable areas" (e.g., kitchen trash cans) and "non-recyclable areas" (e.g., carpet edges) for the robotic arm to place objects. The wiping recognition model is used to output wiping methods for "wiping surfaces". The collision cost recognition model is used to output collision cost values ​​of 0 to 1 (e.g., 0 = safe, 1 = fragile). For example, the "roller brush area" and "side brush area" can be set to a collision cost value of 1 to prevent accidental user clicks that could cause the robotic arm to collide with the mop. In this embodiment, different availability recognition models can be used to perform availability recognition on point cloud data according to different object types. Of course, multiple availability recognition models can also be used simultaneously to perform availability recognition on point cloud data. By using multiple availability recognition models, multiple semantic (e.g., grasping, wiping, rinsing) operation actions can be output on the same color image, thereby reducing forward overhead.

[0068] In some embodiments, when determining the availability recognition result based on the target object type and point cloud data, the robot performs the following: determining the target availability recognition model corresponding to the target object type based on a preset correspondence between object types and availability recognition models; and performing availability recognition on the point cloud data based on the target availability recognition model to obtain the availability recognition result.

[0069] For example, when the target object is a toilet, the corresponding target availability recognition model can be a wiping recognition model and a collision cost recognition model. In this case, the availability recognition results obtained by performing availability recognition on the point cloud data can include the wiping action performed on the toilet seat and the collision cost map. When the target object is a banana peel, the corresponding target availability recognition model can be a placement recognition model, a wiping recognition model, and a collision cost recognition model. In this case, the availability recognition results obtained by performing availability recognition on the point cloud data can include the picking up and placing actions of the banana peel, the wiping action of the ground, and the collision cost map.

[0070] The above embodiments, by performing availability identification on point cloud data based on target availability identification models corresponding to different object types, can improve the accuracy and reliability of availability identification for different objects.

[0071] It should be noted that, in the embodiments of this application, in addition to using the availability recognition model to perform availability recognition on point cloud data, color images can also be segmented and classified to obtain availability recognition results.

[0072] In some embodiments, the scene perception information includes the target object type and a color image; when performing availability identification on the scene perception information, the robot performs the following: image segmentation on the color image to obtain at least one image segmentation region; and classification of the at least one image segmentation region according to the target object type to obtain availability identification results.

[0073] For example, a Segment Anything model can be used to segment a color image, obtaining at least one segmented region. A classification algorithm is then used to further classify the segmented region based on the type of the target object, yielding an availability assessment result. This classification algorithm can include, but is not limited to, Naive Bayes, logistic regression, decision trees, support vector machines, etc. For instance, a decision tree algorithm can be used to further classify the segmented region based on the type of the target object, resulting in different objects and at least one action that can be performed on each object.

[0074] In this embodiment, after the robot performs availability identification on the scene perception information and obtains the availability identification result, it can send the scene perception information and the availability identification result to the terminal. The terminal then generates and displays an availability view based on the scene perception information and the availability identification result. Specifically, the user can first select the type of target operation to be performed. The terminal then presents the objects that can perform the target operation type as candidate operation objects on the availability view. Next, the terminal generates a target task based on the target operation object selected by the user in the availability view. Finally, the terminal sends the target task to the robot for execution. The following will explain in detail how the terminal generates and displays the availability view.

[0075] In some embodiments, the scene-aware information includes point cloud data; when displaying an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene-aware information and availability recognition results, the terminal performs the following: based on the availability recognition results, determining the operable objects whose operation actions match the target operation type and the operable areas of the operable objects as candidate operation objects; obtaining the identification data corresponding to the candidate operation objects; superimposing the identification data corresponding to the candidate operation objects onto the point cloud data through spatial interpolation to obtain target point cloud data; and rendering the target point cloud data to obtain an availability view.

[0076] For example, in an application scenario where the terminal is an AR device (e.g., AR glasses, AR helmet), if the user selects a grasping operation as the target operation type, the operable objects matching the grasping operation and their operable areas can be identified as candidate operation objects based on the availability recognition results. Examples include banana peels and water cups. Then, the corresponding identifier data for the candidate operation objects is obtained. For example, the identifier could be arrows of different colors and shapes, or highlighted frames of different colors and shapes. Next, the identifier data corresponding to the candidate operation objects is superimposed onto the point cloud data using spatial interpolation to obtain the target point cloud data. Finally, the target point cloud data is rendered on the AR device to obtain an availability view.

[0077] The above embodiments, by acquiring the identification data corresponding to the candidate operation object, superimposing the identification data corresponding to the candidate operation object onto the point cloud data through spatial interpolation, and rendering the obtained target point cloud data, can realize the availability view of 3D rendering effect on AR terminal devices, which facilitates human-computer interaction through AR terminal devices, so as to conveniently control the robot to perform target tasks, and can effectively improve the efficiency and accuracy of controlling the robot to perform tasks.

[0078] It should be noted that, in this embodiment, in addition to rendering an availability view based on point cloud data, an availability view can also be generated based on a color image. The following will explain in detail how to generate an availability view based on a color image.

[0079] In some embodiments, the scene-aware information includes a color image; when displaying an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene-aware information and availability recognition results, the terminal is configured to: determine the operable objects whose operation actions match the target operation type and the operable areas of the operable objects as candidate operation objects according to the availability recognition results; and overlay the color image with the identifiers of the candidate operation objects to obtain the availability view.

[0080] For example, in application scenarios where the terminal is a tablet or smartphone, if the user selects a wiping operation as the target operation type, the operable objects that match the wiping operation and the operable areas of those objects can be identified as candidate operation objects based on the availability recognition results. For example, a toilet seat. Then, a color image is overlaid with the identifiers of the candidate operation objects to obtain an availability view. The identifiers of the candidate operation objects can be arrows of different colors and shapes, or highlighted frames of different colors and shapes, etc.

[0081] The above embodiments, by overlaying a color image with the identifier of the candidate operation object, obtain an availability view, which can realize a 2D rendering effect availability view on a tablet or smartphone, making it convenient for users to interact with the robot through a tablet or smartphone to control the robot to perform the target task. This can effectively improve the efficiency and accuracy of controlling the robot to perform the task.

[0082] In some embodiments, before overlaying the color image with the identifier of the candidate operation object to obtain an availability view, the terminal is further configured to perform: when the viewing angle of the color image is not a preset viewing angle, perform weighted back projection on the color image to obtain a color image with adjusted viewing angle; when overlaying the color image with the identifier of the candidate operation object to obtain an availability view, the terminal is further configured to perform: overlaying the color image with the identifier of the candidate operation object to obtain an availability view.

[0083] For example, the preset viewing angle can be set according to the actual situation, such as a top-down view. For instance, when the robot captures a color image of an object from a side view, a weighted back projection can be performed on the color image to make the adjusted color image a frontal view. Then, the adjusted color image is overlaid with the identifier of the candidate operation object to obtain an availability view. The weighted back projection involves back-projecting the projection data into the image space and weighting the projection data according to the projection angle and distance to improve the quality of the reconstructed image. The specific process of weighted back projection can be found in related technologies, and will not be elaborated here.

[0084] It should be noted that, in this embodiment of the application, to address the problem of blurred recognition in 2D color images due to occlusion or viewing angle, a weighted back-projection technique can be used to map 3D availability information back to the 2D view, thereby improving the clarity of the user's understanding. For example, a wiping area that is difficult to express in a side view can be rendered using a virtual top-down view and then back-projected onto the 2D interface.

[0085] The above embodiments, by performing weighted back projection on the color image when the viewing angle of the color image is not a preset viewing angle, and superimposing the color image with the identifier of the candidate operation object to obtain the availability view, can solve the problem of large mapping ambiguity in the 2D to 3D mapping in related technologies, ensuring that the user can accurately select the target operation object and control the robot to perform the target task corresponding to the target operation object, thereby effectively improving the accuracy of controlling the robot to perform tasks.

[0086] In some embodiments, the availability identification result further includes a collision cost map, which includes the collision cost value corresponding to each operation object; when displaying an availability view including candidate operation objects capable of performing the target operation type based on scene perception information and availability identification result, the terminal is also used to perform: in the availability view, setting operation objects with a collision cost value greater than a preset cost threshold to an unselectable state.

[0087] For example, the preset cost threshold can be set according to the actual situation, and the specific value is not limited here. For example, the preset cost threshold can be 0.9. In the availability view, operation objects with a collision cost value greater than 0.9 can be set to an unselectable state.

[0088] In the above embodiments, by setting the operation objects whose collision cost is greater than a preset cost threshold to an unselectable state in the availability view, it is possible to avoid users accidentally selecting fragile or dangerous areas, thereby reducing the possibility of the robotic arm colliding with fragile or dangerous areas, thus improving the safety of the robot, without increasing hardware costs.

[0089] In some embodiments, before performing the operation action corresponding to the target object using different target forces according to the object type of the target object, the robot is also used to perform: determining a collision-free path between the robotic arm and the target object; and determining the degree of freedom information of the joints in the robotic arm based on the collision-free path.

[0090] In some embodiments, scene perception information may include object positions; when determining a collision-free path between the robotic arm and the target object, the robot is also used to perform: based on a preset obstacle map, path planning is performed according to the object position of the target object and the position coordinates of the robotic arm to obtain a collision-free path.

[0091] It should be noted that an obstacle map is a map that marks obstacles, used to help robots quickly identify and confirm which objects are obstacles.

[0092] For example, based on a preset obstacle map, a path planning algorithm can be used to plan the path between the object position of the target object and the position coordinates of the robotic arm, generating a collision-free path that avoids collisions between the robotic arm and obstacles. The path planning algorithm can include, but is not limited to, Dijkstra's algorithm, A* algorithm, D* algorithm, etc.

[0093] For example, after generating a collision-free path that avoids collisions between the robotic arm and obstacles, the degrees of freedom information of the joints in the robotic arm can be determined based on the collision-free path. The degrees of freedom information can include the six degrees of freedom of the joints of the robotic arm, such as three-dimensional coordinates (X, Y, Z) and attitude angles (roll, pitch, yaw).

[0094] In the above embodiments, by determining a collision-free path between the robotic arm and the target object, and by determining the degree of freedom information of the joints in the robotic arm based on the collision-free path, it can be ensured that the robotic arm will not collide with obstacles when moving according to the degree of freedom information, thereby ensuring the safety of the robot.

[0095] In some embodiments, when performing the operation action corresponding to the target operation object using different target forces according to the object type of the target operation object, the robot is also used to: control the robotic arm to move towards the target operation object according to the degree of freedom information, and perform the operation action corresponding to the target operation object using different target forces according to the object type of the target operation object.

[0096] For example, the robotic arm can be controlled to move toward the target object according to the degree of freedom information (X, Y, Z, Roll, Pitch, Yaw) until the robotic arm reaches the location of the target object.

[0097] In the above embodiments, by controlling the robotic arm to move toward the target object according to the degree of freedom information, since the degree of freedom information is generated based on a collision-free path, it can be ensured that the robotic arm will not collide with obstacles when moving according to the degree of freedom information, thereby ensuring the safety of the robot.

[0098] In some embodiments, the pressure sensing device includes a pressure sensor; when performing an operation action corresponding to the target object using different target forces according to the object type of the target object, the robot is also used to: acquire the pressure value collected by the pressure sensor; and adjust the target force according to the pressure value.

[0099] For example, the pressure sensor can be a six-dimensional pressure sensor, which can be installed at the joints, end effector, or fingers of a robotic arm. In this embodiment, because the pressure sensor has high accuracy, adjusting the target force using the pressure value collected by the pressure sensor can effectively improve the accuracy of force adjustment.

[0100] In some embodiments, the pressure sensing device includes a motor disposed in the robotic arm. When performing an operation action corresponding to the target object with different target forces according to the object type of the target operation object, the robot is also used to: acquire the working current of the motor; convert the working current into the pressure value of the robotic arm based on a preset correspondence between current and pressure, and adjust the target force according to the pressure value.

[0101] For example, the motor can be located at the joints, end caps, or fingers of the robotic arm.

[0102] In this embodiment, a motor can be used instead of a pressure sensor, and the pressure value of the robotic arm can be estimated by monitoring the change in motor current through a current loop.

[0103] For example, the correspondence between the motor's operating current and the robotic arm's pressure can be tested and saved in advance. When the robot controls the robotic arm to perform a task, it can acquire the motor's operating current in real time. Based on the preset correspondence between current and pressure, the operating current is converted into the pressure value of the robotic arm. Then, the target force is adjusted according to the pressure value.

[0104] In the above embodiments, by converting the motor's operating current into the pressure value of the robotic arm and adjusting the target force according to the pressure value, the pressure value of the robotic arm can be estimated using a current loop, eliminating the need for a pressure sensor to detect the force, thereby reducing hardware costs.

[0105] Please see Figure 3 , Figure 3 This is a schematic diagram of another robot control system provided in an embodiment of this application, as shown below. Figure 3As shown, the robot's control system may also include a cloud server 30, wherein the cloud server 30 and the robot 10 can communicate wirelessly. The cloud server 30 can also communicate wirelessly with the terminal 20.

[0106] In some embodiments, such as Figure 3 As shown, robot 10 records multi-dimensional data at preset intervals and sends the multi-dimensional data to cloud server 30. The multi-dimensional data includes at least two of the following: color image, depth image, availability view, end pose of robotic arm, pressure value of robotic arm, torque of robotic arm, task label, and timestamp.

[0107] The preset duration can be set according to actual conditions, and the specific value is not limited here. For example, the preset duration can be 100ms.

[0108] It should be noted that the existing remote control methods cannot simultaneously output multi-dimensional data such as color images, depth images, availability views, end-effector pose, pressure values, torque, task labels, and timestamps, which means that additional manual data annotation is required in end-to-end imitation learning.

[0109] The above embodiments improve data collection efficiency by recording multi-dimensional data at preset intervals and sending the multi-dimensional data to the cloud server, thus solving the problem of low efficiency caused by the need for additional manual data annotation in end-to-end imitation learning in existing technologies.

[0110] In some embodiments, such as Figure 3 As shown, the terminal 20 is also used to perform: obtaining multi-dimensional data from the cloud server 30 and generating scripted test instructions based on the multi-dimensional data; sending the scripted test instructions to the robot 10 so that the robot 10 can execute the scripted test instructions and return the motion trajectory of the robotic arm; and comparing the motion trajectory with the preset expected trajectory to obtain the scripted test result.

[0111] For example, color images, depth images, availability views, the end effector pose of the robotic arm, the pressure value of the robotic arm, the torque of the robotic arm, task labels, and timestamps can be combined to generate multiple scripted test instructions covering the entire scene. Then, the robot executes the scripted test instructions, and the returned motion trajectory of the robotic arm is compared with the preset expected trajectory to obtain the scripted test results. Based on the scripted test results, the scripted test instructions are further optimized to improve the accuracy and robustness of the robot in performing tasks.

[0112] The above embodiments generate scripted test instructions based on multi-dimensional data and send these instructions to the robot, enabling the robot to execute them. This allows for coverage of various test combination scenarios without the need for manual joystick operation to traverse all test combinations. This not only improves the efficiency of robot testing but also reduces testing costs.

[0113] Please see Figure 4 , Figure 4 This is a schematic flowchart illustrating a robot control method provided in an embodiment of this application. Figure 4 As shown, the control method of the robot includes steps S201 to S204.

[0114] Step S201: Obtain scene perception information corresponding to objects around the robot, and perform availability identification on the scene perception information to obtain availability identification results. Availability identification results include at least one operation object and at least one operation action to be performed for each operation object.

[0115] For example, when acquiring scene perception information corresponding to objects around the robot, semantic recognition can be performed on the color images and depth images collected by the vision sensor to obtain the target object type and point cloud data, and the color images, target object type and point cloud data can be identified as scene perception information.

[0116] In some implementations, when performing availability identification on scene perception information, the robot can perform availability identification based on the target object type and point cloud data to determine the availability identification result.

[0117] For example, based on a pre-defined correspondence between object types and availability recognition models, a target availability recognition model corresponding to the target object type can be determined. Then, availability recognition is performed on the point cloud data according to the target availability recognition model to obtain the availability recognition result. The availability recognition model may include a grasping recognition model, a placement recognition model, a wiping recognition model, and a collision cost recognition model.

[0118] By using target availability identification models based on different object types to identify the availability of point cloud data, the accuracy and reliability of identifying the availability of different objects can be improved.

[0119] In other implementations, when performing availability identification on scene perception information, the robot can perform image segmentation on the color image to obtain at least one image segmentation region, classify the at least one image segmentation region according to the type of the target object, and obtain availability identification results.

[0120] Step S202: Send scene-aware information and availability recognition results to the terminal so that the terminal can display an availability view including candidate operation objects that can perform the target operation type based on the target operation type selected by the user and the scene-aware information and availability recognition results.

[0121] For example, after the robot performs availability identification on the scene perception information and obtains the availability identification result, it can send the scene perception information and the availability identification result to the terminal. The terminal then generates and displays an availability view based on the scene perception information and the availability identification result. The user can first select the type of target operation to be performed. The terminal then presents the objects that can perform the target operation type as candidate operation objects on the availability view. Next, the terminal generates a target task based on the target operation object selected by the user. Finally, the terminal sends the target task to the robot for execution.

[0122] In some implementations, the terminal can identify candidate operation objects based on the availability recognition results, such as operable objects whose operation actions match the target operation type and the operable areas of the operable objects; obtain the identification data corresponding to the candidate operation objects; superimpose the identification data corresponding to the candidate operation objects onto the point cloud data through spatial interpolation to obtain the target point cloud data; and render the target point cloud data to obtain the availability view.

[0123] In other implementations, the terminal can identify candidate operation objects based on the availability recognition results, such as operable objects whose operation actions match the target operation type and the operable areas of the operable objects; and overlay the color image with the identifiers of the candidate operation objects to obtain an availability view.

[0124] Step S203: Obtain the target task. The target task is generated by the terminal based on the target operation object selected by the user in the availability view.

[0125] For example, after displaying the availability view, the terminal can generate a target task corresponding to the target operation object selected by the user in the availability view, and then send the target task to the robot. For instance, when the user selects a toilet as the target operation object in the availability view, a target task for cleaning the toilet seat can be generated. As another example, when the user selects a water cup as the target operation object in the availability view, a target task for picking up the water cup can be generated.

[0126] Step S204: Based on the target operation object in the target task, the robot's robotic arm performs the corresponding operation action of the target operation object using different target forces according to the object type of the target operation object. The target force is dynamically adjusted based on the pressure signal collected by the pressure sensing device in the robotic arm.

[0127] For example, when a robot performs an operation corresponding to a target operation, it can first determine a collision-free path between the robotic arm and the target object. Based on the collision-free path, it determines the degree of freedom information of the joints in the robotic arm. Then, it controls the robotic arm to move towards the target object according to the degree of freedom information, and performs the operation corresponding to the target object using different target forces based on the object type. The specific process of determining the collision-free path can be found in the detailed description of the above embodiments, and will not be repeated here.

[0128] By controlling the robotic arm to move toward the target object according to the degree of freedom information, since the degree of freedom information is generated based on a collision-free path, it can be ensured that the robotic arm will not collide with obstacles when moving according to the degree of freedom information, thereby ensuring the safety of the robot.

[0129] In some implementations, when performing operations corresponding to the target object using different target forces based on the object type, the robot can also acquire pressure values ​​collected by pressure sensors; and adjust the target force according to the pressure values. By using pressure values ​​collected by pressure sensors to adjust the target force, the accuracy of force adjustment can be effectively improved due to the high precision of the pressure sensors.

[0130] In some implementations, when performing actions corresponding to the target object using different target forces based on the object type, the robot can also acquire the motor's operating current. Based on a preset correspondence between current and pressure, the operating current is converted into a pressure value for the robotic arm, and the target force is adjusted according to this pressure value. By converting the motor's operating current into a pressure value for the robotic arm and adjusting the target force accordingly, the pressure value experienced by the robotic arm can be estimated using a current loop, eliminating the need for a pressure sensor for force detection and thus reducing hardware costs.

[0131] This application provides a robot control method that involves the robot performing availability identification on scene perception information of surrounding objects. Based on the user-selected target operation type, the terminal displays an availability view including candidate operation objects capable of performing the target operation type, according to the scene perception information and availability identification results. In response to the user's selection of a target operation object in the availability view, a target task corresponding to the target operation object is generated and sent to the robot. The robot, through its robotic arm, performs the corresponding operation action of the target operation object using different target forces according to the object type. This method allows for the visual display of selectable operation objects on the terminal. Users only need to select the target operation object on the terminal to quickly and accurately control the robot to perform the corresponding operation action, significantly reducing the cognitive threshold for users and improving the efficiency and accuracy of controlling the robot to perform tasks.

[0132] Please see Figure 5 , Figure 5 This is a schematic diagram of a cleaning system 1000 provided in an embodiment of this application. Figure 5 As shown, the cleaning system 1000 includes one or more cleaning robots 10 and one or more base stations 40. The base station 40 is used for maintenance of the cleaning robots 10, including but not limited to charging, dust collection, adding clean water, emptying wastewater, and cleaning the cleaning components. For example, the base station 40 can charge the cleaning robots 10, and can provide a docking position for the cleaning robots 10. The base station 40 can also clean the mopping components of the cleaning robots 10, which are used to mop surfaces to be cleaned, such as the ground. The base station 40 can also provide a clean cloth module for the robotic arm of the cleaning robots 10, and clean the cloths for the cleaning robots 10, etc.

[0133] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," "mechanical coupling," and "coupling" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection. They can refer to a mechanical connection or an electrical connection. They can refer to a direct connection or an indirect connection through an intermediate medium, and can refer to the internal communication of two components or the interaction between two components. Mechanical coupling or coupling of two components includes direct coupling and indirect coupling, such as a direct fixed connection or a connection through a transmission mechanism. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0134] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0135] The foregoing disclosure provides many different embodiments or examples for implementing different structures of this application. To simplify the disclosure, specific examples of components and arrangements are described above. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, examples of various specific processes and materials are provided in this application, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0136] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific method step, feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific method steps, features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0137] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control system for a robot, characterized in that, The control system includes: A robot is used to acquire scene perception information corresponding to objects around the robot, and to perform availability identification on the scene perception information to obtain availability identification results. The availability identification results include at least one operation object, at least one operation action that can be performed on each operation object, and a collision cost map. The collision cost map includes the collision cost value corresponding to each operation object. The terminal is used to display an availability view, including candidate operation objects capable of performing the target operation type, based on the user-selected target operation type, the scene perception information, and the availability recognition result. In the availability view, operation objects with a collision cost greater than a preset cost threshold are set to an unselectable state. Based on the target operation object selected by the user in the availability view, a target task corresponding to the target operation object is generated and sent to the robot. The robot then uses its robotic arm to perform the operation action corresponding to the target operation object based on the target operation object in the target task, using different target forces according to the object type of the target operation object. The target force is dynamically adjusted based on the pressure signal collected by the pressure sensing device in the robotic arm.

2. The control system for the robot according to claim 1, characterized in that, The robot includes a vision sensor, which is positioned at the front of the robot's body. When acquiring scene perception information corresponding to objects around the robot, the robot performs the following: The visual sensor acquires color and depth images of objects around the robot. Semantic recognition is performed on the color image and the depth image to obtain the target object type and point cloud data; The scene perception information is determined based on the color image, the target object type, and the point cloud data.

3. The control system for the robot according to claim 1, characterized in that, The scene perception information includes target object type and point cloud data; when performing availability identification on the scene perception information to obtain availability identification results, the robot performs the following: Availability identification is performed based on the target object type and the point cloud data to determine the availability identification result.

4. The control system of the robot according to claim 3, characterized in that, When performing availability identification based on the target object type and the point cloud data to determine the availability identification result, the robot performs the following: Based on the pre-defined correspondence between object types and availability recognition models, the target availability recognition model corresponding to the target object type is determined; The point cloud data is used to perform availability identification based on the target availability identification model to obtain the availability identification result.

5. The control system of the robot according to claim 4, characterized in that, The availability identification model includes an operation action identification model and a risk assessment model. The operation action identification model includes a grasping identification model, a placement identification model, and a wiping identification model. The risk assessment model includes a collision cost identification model.

6. The control system for the robot according to claim 1, characterized in that, The scene perception information includes the target object type and a color image; when performing availability identification on the scene perception information, the robot performs the following: The color image is segmented to obtain at least one segmented region; The at least one image segmentation region is classified according to the target object type to obtain the availability recognition result.

7. The control system of the robot according to claim 1, characterized in that, The scene perception information includes point cloud data; When displaying an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene-aware information and the availability recognition result, the terminal is used to perform: Based on the availability identification results, the operable objects whose operation actions match the target operation type and the operable areas of the operable objects are identified as the candidate operation objects. Obtain the identifier data corresponding to the candidate operation object; The target point cloud data is obtained by superimposing the identification data corresponding to the candidate operation object onto the point cloud data through spatial interpolation. The target point cloud data is rendered to obtain the availability view.

8. The control system of the robot according to claim 1, characterized in that, The scene perception information includes color images; When displaying an availability view including candidate operation objects capable of performing the target operation type based on the user-selected target operation type, according to the scene-aware information and the availability recognition result, the terminal is used to perform: Based on the availability identification results, the operable objects whose operation actions match the target operation type and the operable areas of the operable objects are identified as the candidate operation objects. The availability view is obtained by overlaying the color image with the identifier of the candidate operation object.

9. The control system for the robot according to claim 8, characterized in that, Before overlaying the color image with the identifiers of the candidate operation objects to obtain the availability view, the terminal is also used to perform: When the viewing angle of the color image is not a preset viewing angle, a weighted back projection is performed on the color image to obtain a color image with adjusted viewing angle. When overlaying the color image with the identifier of the candidate operation object to obtain the availability view, the terminal is also used to perform: The color image after the perspective adjustment is overlaid with the identifier of the candidate operation object to obtain the availability view.

10. The control system for the robot according to claim 1, characterized in that, Before performing the operation action corresponding to the target operation object using different target forces based on the object type of the target operation object, the robot is also used to perform: Determine a collision-free path between the robotic arm and the target object; Based on the collision-free path, determine the degree of freedom information of the joints in the robotic arm; When performing the operation action corresponding to the target operation object using different target forces according to the object type of the target operation object, the robot is also used to perform: The robotic arm is controlled to move toward the target object according to the degree of freedom information, and different target forces are used to perform the corresponding operation actions of the target object according to the object type of the target object.

11. The control system for the robot according to claim 10, characterized in that, The scene perception information includes object positions; when determining a collision-free path between the robotic arm and the target object, the robot is also used to perform: Based on a preset obstacle map, a path is planned according to the object position of the target object and the position coordinates of the robotic arm to obtain the collision-free path.

12. The control system for the robot according to claim 1, characterized in that, The pressure sensing device includes a pressure sensor; when performing the operation action corresponding to the target operation object using different target forces according to the object type of the target operation object, the robot is also used to perform: Obtain the pressure value collected by the pressure sensor; The target force is adjusted according to the pressure value.

13. The control system for the robot according to claim 1, characterized in that, The pressure sensing device includes a motor installed in the robotic arm. When performing the operation action corresponding to the target object using different target forces according to the object type of the target operation object, the robot is also used to perform: Obtain the operating current of the motor; Based on a preset correspondence between current and pressure, the working current is converted into the pressure value of the robotic arm, and the target force is adjusted according to the pressure value.

14. The control system of the robot according to claim 1, characterized in that, The control system also includes a cloud server; the robot is also used to perform: Multi-dimensional data is recorded at preset intervals and sent to the cloud server. The multi-dimensional data includes at least two of the following: color image, depth image, availability view, end pose of the robotic arm, pressure value of the robotic arm, torque of the robotic arm, task tag, and timestamp.

15. The control system for the robot according to claim 14, characterized in that, The terminal is also used to perform: The multi-dimensional data is obtained from the cloud server, and scripted test instructions are generated based on the multi-dimensional data. The scripted test instructions are sent to the robot so that the robot can execute the scripted test instructions and return the movement trajectory of the robotic arm; The scripted test results are obtained by comparing the motion trajectory with the preset expected trajectory.

16. A method for controlling a robot, characterized in that, The control method, applied to a robot in a control system as described in any one of claims 1-15, comprises: The robot acquires scene perception information corresponding to objects around it, and performs availability identification on the scene perception information to obtain availability identification results. The availability identification results include at least one operation object and at least one operation action to be performed for each operation object. The scene-aware information and the availability recognition result are sent to the terminal so that the terminal can display an availability view including candidate operation objects that can perform the target operation type based on the target operation type selected by the user and according to the scene-aware information and the availability recognition result. Obtain the target task, which is generated by the terminal based on the target operation object selected by the user in the availability view; Based on the target object in the target task, the robot's robotic arm performs the corresponding operation action of the target object using different target forces according to the object type of the target object. The target force is dynamically adjusted based on the pressure signal collected by the pressure sensing device in the robotic arm.

17. A robot, characterized in that, The robot includes a robot body, a chassis, a vision sensor, a robotic arm, and a controller. The chassis is equipped with a dragging and wiping component and a brushing component. The robotic arm includes a pressure sensing device. The controller is used to execute the control method of the robot according to claim 16.

18. A cleaning system, characterized in that, include: Cleaning robots; as well as A base station, which is capable of maintaining the cleaning robot, wherein the cleaning robot is the robot as described in claim 17.

Images