Task execution method and device and electronic equipment

Abstract

The embodiment of the invention provides a task execution method and device and electronic equipment, and relates to the technical field of robot control, the task execution method and device are applied to a robot, and the robot is provided with a depth image acquisition device and a mechanical arm. Acquiring a first depth image of the surrounding environment through depth image acquisition equipment; when it is detected that the to-be-picked object exists in the first depth image, the robot travels from the first position where the robot is currently located to the picking position, and after the robot travels to the picking position, the to-be-picked object is picked through a mechanical arm; driving from the pickup position to a second position, and continuously driving according to the current task path after driving to the second position; the first position and the second position are the same position on the current task path, or the second position is a position behind the first position on the current task path. In this way, the autonomy of the robot for executing the picking task is improved.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a task execution method, apparatus and electronic device. Background Technology

[0002] With the rapid development of robotics technology, robots are widely used in various scenarios to replace human labor in performing various tasks to meet user needs. Among them, mobile robots equipped with robotic arms are widely used in scenarios where objects need to be picked up and moved.

[0003] However, when dealing with objects in a scene whose location is not fixed, staff in the scene need to control the robot to move to the location of the object and control the robot to pick up the object using a robotic arm, resulting in poor autonomy for the robot in performing the picking task. Summary of the Invention

[0004] The purpose of this application is to provide a task execution method, apparatus, and electronic device to improve the autonomy of a robot in performing a picking task. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a task execution method applied to a robot, the robot being equipped with a depth image acquisition device and a robotic arm, the method comprising:

[0006] While traveling along the current task path, the depth image acquisition device acquires a first depth image of the surrounding environment.

[0007] If an object to be picked up is detected in the first depth image, the robot moves from its current first position to the picking position, and after reaching the picking position, it uses the robotic arm to pick up the object to be picked up. The picking position is the position where the robot can pick up the object to be picked up.

[0008] The vehicle travels from the pick-up location to the second location, and continues along the current task path after reaching the second location.

[0009] Wherein, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that is after the first position.

[0010] In some embodiments, before moving from the robot's current first position to the pickup position, the method further includes: obtaining the position of the object to be picked up based on the robot's current first position, the installation position of the depth image acquisition device on the robot, and the pixel coordinates and depth information of the object to be picked up in the first depth image; and determining the pickup position based on the position of the object to be picked up and the pickup range supported by the robotic arm.

[0011] In some embodiments, the step of traveling from the robot's current first position to the pickup position, and then using the robotic arm to pick up the object to be picked up after reaching the pickup position, includes: when there are multiple objects to be picked up, determining the target object to be picked up from the objects to be picked up; traveling from the first position to the target pickup position, and then using the robotic arm to pick up the target object to be picked up after reaching the target pickup position, wherein the target pickup position is a position where the robot can pick up the target object to be picked up; and based on the target pickup position, determining the target object to be picked up from the unpicked objects to be picked up and picking it up, until all objects to be picked up have been traversed.

[0012] In some embodiments, for each of the objects to be picked up, the following steps are performed: acquiring a second depth image using the depth image acquisition device, the second depth image containing the object to be picked up; generating point cloud data based on the second depth image; performing a picking pose estimation based on the point cloud data to obtain the picking pose required by the end effector of the robotic arm to pick up the object to be picked up; performing inverse kinematics on the picking pose to obtain the joint angles required by each joint of the robotic arm; controlling each joint of the robotic arm to reach the joint angles, and picking up the object to be picked up.

[0013] In some embodiments, the bottom of the robot is provided with an adjustment component for adjusting the robot's pose; before acquiring a second depth image through the depth image acquisition device, the method further includes: changing the pose of the robot's depth image acquisition device by adjusting the state of the adjustment component, wherein the object to be picked up is located within the field of view of the current depth image acquisition device.

[0014] In some embodiments, the adjustment component includes a plurality of support structures, each support structure including at least one rotatable joint; and / or, in the case where a plurality of depth image acquisition devices are provided in the robot, the pose of the robot is changed so that the object to be picked up is located within the field of view of a designated depth image acquisition device among the plurality of depth image acquisition devices.

[0015] In some embodiments, the robot is further provided with a storage container for placing the picked-up object; the step of picking up the object using the robotic arm includes: picking up the object using the robotic arm when the space of the storage container is not saturated; and / or, the method further includes: when the space of the storage container is saturated, moving from the robot's current position to a target cleaning point and performing a cleaning task on the storage container; in response to picking up the object, moving to a second position to continue moving along the current task path; in response to not picking up the object, moving to the picking position to perform the step of picking up the object using the robotic arm.

[0016] In some embodiments, in response to the robot detecting that its own power meets the charging conditions, the robot travels to the charging point and charges. The charging conditions include: the robot's remaining power is lower than a preset threshold and / or not greater than a projected power, the projected power being determined based on the power required for the robot to travel along the shortest path between its current position and the charging point.

[0017] In some embodiments, the robot is further equipped with a radar sensor, which is used to detect whether there are obstacles in the part of the current task path to be traversed; the method further includes: when an obstacle in a moving state is detected, stopping movement and obtaining a target time when the obstacle leaves the current task path based on the moving direction and moving speed of the obstacle, and continuing to detect the obstacle through the radar sensor at the target time; when an obstacle in a non-moving state is detected, generating an obstacle avoidance path for the detected non-moving obstacle based on a preset obstacle avoidance path generation algorithm; replacing a part of the current task path with the obstacle avoidance path to obtain a new task path, wherein the starting position and ending position of the replaced part of the path are the same as the starting position and ending position of the obstacle avoidance path, respectively.

[0018] Secondly, embodiments of this application provide a robot control method applied to a control terminal, the method comprising:

[0019] Send the task path to the robot;

[0020] The task path is the path that the robot travels according to the task execution method described in the first aspect.

[0021] In some embodiments, prior to sending the task path to the robot, the method includes at least one of the following:

[0022] Based on the estimated power required to perform the picking task according to the task path, the robot is determined from the idle robots in the current scene whose actual power is greater than the power threshold, and the estimated power is not greater than the power threshold.

[0023] In response to the robot's inability to continue performing the pickup task according to the current task path, an idle robot in the current scene other than the robot is selected as the robot to continue performing the pickup task, and the untraveled part of the task path is selected as the current task path and sent to that robot.

[0024] Thirdly, embodiments of this application provide a robot control system, which includes a control terminal and a robot, wherein:

[0025] The robot is used to perform any of the task execution methods described in the first aspect above;

[0026] The control terminal is used to execute any of the task execution methods described in the second aspect above.

[0027] Fourthly, embodiments of this application provide a task execution device applied to a robot, wherein the robot is equipped with a depth image acquisition device and a robotic arm, and the device includes:

[0028] The first acquisition module is used to acquire a first depth image of the surrounding environment through the depth image acquisition device while traveling according to the current task path.

[0029] The picking control module is used to move from the robot's current first position to the picking position when the presence of an object to be picked in the first depth image is detected, and to pick up the object to be picked up using the robotic arm after moving to the picking position. The picking position is the position where the robot can pick up the object to be picked up.

[0030] The execution control module is used to travel from the pickup position to the second position, and continue to travel along the current task path after arriving at the second position;

[0031] Wherein, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that is after the first position.

[0032] Fifthly, embodiments of this application provide a task execution device applied to a control terminal, the device comprising:

[0033] The first path sending module is used to send the task path to the robot;

[0034] The task path is the path that the robot travels according to the task execution method described in the first aspect.

[0035] Sixthly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0036] Memory, used to store computer programs;

[0037] A processor, when executing a program stored in memory, implements the steps of the method described in the first or second aspect.

[0038] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0039] Eighthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the method described in the first or second aspect.

[0040] Beneficial effects of the embodiments in this application:

[0041] Based on the task execution method provided in this application embodiment, while the robot is traveling along the current task path, it acquires a first depth image of the surrounding environment using a depth image acquisition device. The acquired first depth image is then detected to determine whether an object to be picked up (i.e., an object to be picked up) exists in the robot's current surrounding environment. If it exists, the robot can travel to the picking location of the object to be picked up, and the robot can pick up the object using its robotic arm at that location. Then, it returns to a second position on the current task path and continues traveling along the task path. Here, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that follows the first position. Therefore, while traveling along the portion of the task path after the second position, the robot can continue to detect whether the object to be picked up exists in the surrounding environment, thus ensuring that the robot can continue to perform the picking task represented by the current task path. In other words, the robot can automatically detect and locate the object to be picked up by using a depth image acquisition device, and then pick up the object by a robotic arm. The robot can then execute the picking task represented by the current task path according to the above logic, thereby improving the robot's autonomy in performing picking tasks.

[0042] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0044] Figure 1 This is a first flowchart of a task execution method provided in an embodiment of this application;

[0045] Figure 2 A second flowchart of the task execution method provided in the embodiments of this application;

[0046] Figure 3 A third flowchart of the task execution method provided in the embodiments of this application;

[0047] Figure 4 A flowchart illustrating how a robotic arm picks up an object, as provided in this application embodiment;

[0048] Figure 5 This is a schematic diagram of the structure of a quadruped robot provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the structure of the robot control system provided in an embodiment of this application;

[0050] Figure 7 A flowchart illustrating the process of a robot performing a task, provided as an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the structure of a task execution device provided in an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0054] To improve the autonomy of robots in performing picking tasks, embodiments of this application provide a task execution method, apparatus, and electronic device. The following first describes a task execution method provided by an embodiment of this application.

[0055] The task execution method provided in this application can be applied to robots. The executing entity of this task execution method can be a controller or processor inside the robot.

[0056] In this application, the robot is a mobile robot, which is equipped with a depth image acquisition device and a robotic arm.

[0057] For example, the robot can be a quadruped robot, a tracked robot, or a wheeled robot, etc.

[0058] The depth image acquisition device can be an RGB-D (Red, Green, and Blue-Depth) camera, or a stereo camera, etc. The depth image acquisition device is used to acquire depth images of the surrounding environment. Correspondingly, the robot can acquire the depth images captured by the depth image acquisition device.

[0059] A robotic arm comprises components such as a lever, joints, and an end effector. Accordingly, the robot can adjust the pose of the end effector and control it to pick up objects by adjusting the joint angles of the robotic arm's joints. For example, the end effector of a robotic arm can be a gripper (such as a two-finger gripper or a three-finger gripper).

[0060] The application scenarios of the solution provided in this application are: various scenarios in which robots need to inspect and pick up objects within a preset area, and the types of objects can be set according to the actual needs of the application scenario.

[0061] The following will use several specific scenarios as examples to provide a more intuitive introduction.

[0062] Scene 1: Park inspection scene.

[0063] In this scenario, park paths, lakeside areas, and lawns may be littered with trash (such as scraps of paper and water bottles) discarded by visitors. Accordingly, the robot can conduct periodic inspections in various areas of the park. That is, while moving through different areas of the park, it detects the presence of such trash in the surrounding environment and promptly picks it up when detected. Subsequently, the collected trash can be transported to a waste station (i.e., the cleaning point in this application) to maintain the cleanliness of the park environment.

[0064] Scenario 2: Shopping mall inspection scenario.

[0065] In this scenario, customers may leave behind items (such as clothing, children's toys, etc.) on the floor, display stands, and other areas within the shopping mall. Accordingly, the robot can patrol various areas of the mall. That is, while moving through different areas of the mall, it detects whether the aforementioned items are present in the surrounding environment and picks them up promptly when detected. Subsequently, the picked-up items can be transported to the mall's work station (i.e., the cleaning point in this application) to ensure the safety of customers' lost items.

[0066] This solution is applicable to, but not limited to, the above scenarios. For example, the robot can also be used to inspect and pick up objects in streets, campuses, and other similar locations, which will not be elaborated here.

[0067] See Figure 1 , Figure 1 A first flowchart of a task execution method provided in this application embodiment, the method may include the following steps:

[0068] S101: While traveling along the current task path, acquire the first depth image of the surrounding environment using a depth image acquisition device.

[0069] S102: If an object to be picked up is detected in the first depth image, the robot moves from its current first position to the picking position, and after reaching the picking position, the robot uses its robotic arm to pick up the object to be picked up. The picking position is the position where the robot can pick up the object to be picked up.

[0070] S103: Move from the pickup location to the second location, and continue along the current mission path after reaching the second location.

[0071] Wherein, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that is after the first position.

[0072] Based on the task execution method provided in this application embodiment, while the robot is traveling along the current task path, it acquires a first depth image of the surrounding environment using a depth image acquisition device. The acquired first depth image is then detected to determine whether an object to be picked up (i.e., an object to be picked up) exists in the robot's current surrounding environment. If it exists, the robot can travel to the picking location of the object to be picked up, and the robot can pick up the object using its robotic arm at that location. Then, it returns to a second position on the current task path and continues traveling along the task path. Here, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that follows the first position. Therefore, while traveling along the portion of the task path after the second position, the robot can continue to detect whether the object to be picked up exists in the surrounding environment, thus ensuring that the robot can continue to perform the picking task represented by the current task path. In other words, the robot can automatically detect and locate the object to be picked up by using a depth image acquisition device, and then pick up the object by a robotic arm. The robot can then execute the picking task represented by the current task path according to the above logic, thereby improving the robot's autonomy in performing picking tasks.

[0073] For step S101, the current task path represents the path that the robot needs to travel to perform the task.

[0074] In one implementation, a robot can receive a task path sent to it by a control terminal included in the robot control system, and use it as the current task path. Further description of the robot control system and the control terminal can be found in subsequent embodiments.

[0075] Alternatively, the robot can provide an interactive interface for receiving user commands. After the robot is activated, the user (such as the staff in the above application scenario) can trigger user commands containing the path the robot needs to travel through the interface. For example, for a robot, the user can select one of the alternative paths displayed on the robot's interactive interface as the desired path. Accordingly, the robot can obtain the task path indicated by the user as the current task path.

[0076] In another implementation, the robot can update the current task path when it detects an obstacle in the part of the path to be traversed while it is moving along the current task path. The specific process of updating the current task path will be described in subsequent embodiments.

[0077] In some embodiments, the robot may record map data containing a preset inspection area.

[0078] The preset inspection area represents the maximum range that the robot can travel. Map data containing the preset inspection area can be represented as the area corresponding to the preset inspection area in a preset map coordinate system. The origin of the preset map coordinate system represents the charging point within the preset inspection area.

[0079] In this embodiment, the robot can receive map data containing a preset inspection area from the aforementioned control terminal. Correspondingly, the robot can record the map data containing the preset inspection area.

[0080] The preset inspection area can be pre-set by users who need the robot to perform picking tasks according to their actual needs.

[0081] In one implementation, the preset inspection area can represent a circular area centered on the charging point and with a first preset distance as its radius. The first preset distance can represent half of the maximum distance the robot can travel after a single charge, or it can represent the distance between the furthest point in the area to be inspected and the charging point.

[0082] Alternatively, the preset inspection area can be a closed polygonal area set according to the actual inspection scenario required by the robot.

[0083] For example, in the park inspection scenario described above, the actual area the robot needs to inspect is the park's interior. Accordingly, multiple real locations on the park's boundary can be identified, and the closed polygonal region formed by connecting these multiple real locations can be used as the preset inspection area.

[0084] In one implementation, the map data containing the preset inspection area may also record the coordinates of the preset obstacles in the preset map coordinate system.

[0085] Among them, the preset obstacles can represent: real obstacles such as walls and buildings existing in the preset inspection area.

[0086] Alternatively, a preset obstacle can also refer to a virtual obstacle pre-set in a preset map coordinate system. For example, there may be narrow passages within a preset inspection area that the robot can pass through, but to ensure the safety of the robot during operation, virtual obstacles can be set at the location of these narrow passages in the preset map coordinate system.

[0087] For example, in the park inspection scenario described above, there may be a body of water within the park, and no real obstacles may exist around it. To prevent the robot from navigating this body of water, virtual obstacles can be placed at locations representing the area around the body of water within a pre-defined map coordinate system.

[0088] Correspondingly, the preset coordinates of obstacles indicate the positions that the robot is not allowed to reach in the preset map coordinate system during its movement.

[0089] Accordingly, while traveling along the current task path, the robot can periodically perform a depth image detection process. That is, when a preset detection time is reached, the robot can acquire the latest depth image of the surrounding environment captured by the installed depth image acquisition device (i.e., the first depth image corresponding to the detection time), and detect whether there is an object to be picked up in the first depth image corresponding to the detection time.

[0090] The time interval between any two adjacent detection moments can be a preset fixed duration. This preset fixed duration can be in the millisecond range (e.g., 50 milliseconds) or in the second range (e.g., 1 second).

[0091] In one implementation, when a depth image acquisition device is installed in a robot, the device can be mounted on the front side of the robot and acquire depth images along the robot's forward direction. The side of the robot that always faces the direction of movement during its forward motion is considered the front side of the robot.

[0092] Accordingly, the depth image acquisition device can acquire depth images of the surrounding environment in the direction the robot is moving along the current task path.

[0093] In another implementation, to ensure comprehensive acquisition of the surrounding environment, a robot can be equipped with multiple depth image acquisition devices. Accordingly, when multiple depth image acquisition devices are installed in a robot, the lens orientation of each device can be set so that each device can acquire depth images of the surrounding environment from different positions relative to the robot.

[0094] For example, one or more depth image acquisition devices can be installed on the following parts of the robot: front, left, right, rear, or chassis.

[0095] It is understandable that when a robot is equipped with multiple depth image acquisition devices, during a single depth image detection process, the robot can acquire the latest depth images (i.e., the first depth images) acquired by each of the multiple depth image acquisition devices for detection.

[0096] Accordingly, for each first depth image, the first depth image can be input into a pre-trained object detection model. The object detection model can detect whether there is a preset type of object (i.e., the object to be picked) in the first depth image, and obtain an object detection result indicating whether the object to be picked exists in the first depth image.

[0097] In a first depth image, there may be one or more targets to be picked up. When the target detection result of a first depth image indicates that there is an object to be picked up in the first depth image, the target detection result of the first depth image also includes: the pixel coordinates of each detected object to be picked up in the first depth image.

[0098] The preset types are pre-set by the user according to actual needs. For example, in the above park inspection scenario, the preset types may include at least one of the following: paper scraps, water bottles, and garbage bags, etc.

[0099] The object detection model used in this embodiment can be obtained by training an initial object detection model using sample images and the corresponding labels for each sample image. The sample images are images containing objects of a preset type, and the label corresponding to each sample image indicates the type and ground truth bounding box of each object in that sample image.

[0100] For example, the object detection model can be a YOLO (You Only Look Once) model, such as the YOLOE-detection+tracking model, or the YOLOX model, etc. The object detection models used in the embodiments of this application include, but are not limited to, the YOLO model described above.

[0101] In one implementation, during the detection of a depth image, for each object to be picked up detected in the first depth image corresponding to the detection time, the robot can also determine the base class to which the type of the object to be picked up belongs based on the type of the object to be picked up.

[0102] Alternatively, during the detection of a depth image, for each object to be picked up detected in the first depth image corresponding to that detection time, the object detection model can output the base class to which the type of the object to be picked up belongs.

[0103] For example, in the park inspection scenario described above, the types of objects detected for pickup include: scraps of paper, water bottles, or garbage bags. Correspondingly, scraps of paper and water bottles belong to the base class of recyclable, while garbage bags belong to the base class of non-recyclable.

[0104] Understandably, the robot periodically performs depth image detection while traveling along its current task path. During a depth image detection process, if the object to be picked up is not found in the first depth image at that specific moment, it indicates that the required object is not present in the robot's current environment. At this point, the robot remains on its current task path and continues traveling along that path.

[0105] In this way, the robot can detect in real time whether there is a target to be picked up in the surrounding environment while traveling along the current task path. Subsequently, the robot can pick up the target to be picked up in the surrounding environment based on the detection results.

[0106] Regarding step S102, if an object to be picked up is detected in the first depth image, the robot can move from its current first position to the picking position.

[0107] The robot's current first position indicates its location on the current task path when an object to be picked up is detected in the first depth image. Accordingly, the robot can stop traveling along the current task path and leave it to pick up the object in the surrounding environment.

[0108] In one implementation, during the depth image detection process described above, if multiple objects to be picked up are detected in the surrounding environment, the robot can record the real-time state of each of these objects. The real-time state of an object to be picked up can be: picked up, or not picked up.

[0109] For example, the robot can assign a unique number to each object to be picked up, and the number and real-time status of each object can be linked together in the form of a linked list.

[0110] In some embodiments, see Figure 2 , Figure 2 This is a second flowchart of a task execution method provided in an embodiment of this application. Figure 1 Based on this, before step S102, the method further includes:

[0111] S104: Based on the robot's current first position, the installation position of the depth image acquisition device on the robot, and the pixel coordinates and depth information of the object to be picked up in the first depth image, the position of the object to be picked up is obtained.

[0112] S105: Determine the pickup position based on the location of the object to be picked up and the pickup range supported by the robotic arm.

[0113] In this embodiment of the application, the position of an object to be picked up represents the coordinate position of the object in a preset map coordinate system.

[0114] For a given object to be picked up, the robot can calculate the object's position using the following steps:

[0115] Step 1: Based on the camera intrinsic parameter matrix of the depth image acquisition device that acquires the first depth image to which the object to be picked belongs, transform the pixel coordinates and depth information of the object to be picked in the first depth image to obtain the coordinate position of the object to be picked in the camera coordinate system.

[0116] Step 2: By determining the installation position of the depth image acquisition device on the robot, which acquires the first depth image, determine the offset vector and rotation matrix of the camera coordinate system of the depth image acquisition device relative to the robot coordinate system. Based on the offset vector and rotation matrix, calculate the coordinate position of the object to be picked up in the robot coordinate system.

[0117] Step 3: Combine the robot's current position (i.e., the robot's current coordinate position in the preset map coordinate system) to convert the coordinate position of the object to be picked up in the robot coordinate system to the coordinate position of the object to be picked up in the preset map coordinate system, that is, obtain the position of the object to be picked up.

[0118] In one implementation, after obtaining the positions of all objects to be picked up in the surrounding environment, the robot can calculate the density of objects to be picked up per unit area.

[0119] If the calculated density is greater than a preset threshold (e.g., 3 objects / square meter), it indicates that there are many objects to be picked up in the surrounding environment. For each object to be picked up, the robot determines the distance between the object and the robot's current position. Then, objects to be picked up that are more than a preset radius (e.g., 2 meters) are deleted.

[0120] Based on the above processing, during subsequent object pickup, the robot only picks up objects within a preset radius of its current position. In this way, the robot can be controlled to pick up objects near its current task path based on the density of the objects, reducing the time required for a single inspection and improving inspection efficiency.

[0121] Accordingly, for each object to be picked up, after calculating the position of the object, the picking position that the robot needs to reach when picking up the object can be calculated based on the picking range supported by the robotic arm. The picking range supported by the robotic arm does not exceed the maximum picking range that the robotic arm can reach, which is determined based on the longest length that the robotic arm's hardware can reach during the picking process.

[0122] Accordingly, for an object to be picked up, a region within the picking range centered on the object's location can be defined as the optional picking area for that object. Then, a location can be selected within this optional picking area as the picking position for that object (i.e., the picking position the robot needs to reach to pick up the object).

[0123] For example, if the robotic arm supports a pickup range of 0.5m, in a preset map coordinate system, this pickup range corresponds to one unit length. Accordingly, for a given object to be picked up, an area within one unit length centered on the object's location can be determined within the preset map coordinate system as the selectable pickup area. Furthermore, within this selectable pickup area, a location closest to the robot's current position can be selected as the pickup location (i.e., the pickup location the robot needs to reach to pick up the object).

[0124] In one implementation, when the robot has recorded map data containing a preset inspection area, for each object to be picked up, the robot can calculate the picking position of the object and then determine whether the picking position of the object is within the preset inspection area.

[0125] If the pickup location of the object to be picked up is not within the preset inspection area, it means that the robot cannot travel to the pickup location of the object to be picked up, and the robot can delete the recorded object to be picked up.

[0126] In this way, the robot can be restricted to moving within the preset inspection area during the picking task, thus ensuring its safety.

[0127] Based on the above processing, after detecting the object to be picked up in the surrounding environment, the robot can determine the location of the object and then calculate the pickup location that the robot needs to reach to pick up the object. Subsequently, the robot can travel to the vicinity of the object according to the calculated pickup location and pick up the object using its robotic arm.

[0128] In some embodiments, see Figure 3 , Figure 3 This is a third flowchart illustrating the task execution method provided in an embodiment of this application. Figure 1 Based on this, step S102 includes:

[0129] S1021: When there are multiple objects to be picked up, determine the target object to be picked up from the objects to be picked up.

[0130] S1022: Move from the first position to the target pickup position, and after arriving at the target pickup position, use the robotic arm to pick up the target object to be picked up. The target pickup position is the position where the robot can pick up the target object to be picked up.

[0131] S1023: Based on the target picking position, determine the target object to be picked from the unpicked objects to be picked and pick it, until all objects to be picked are traversed.

[0132] In this embodiment of the application, when multiple objects to be picked up are detected in the first depth image, the robot can pick up each object one by one according to the preset picking order of the multiple objects to be picked up.

[0133] In one implementation, for each object to be picked up, if the robot has recorded the base class to which the type of the object to be picked up belongs, the preset picking order of each object to be picked up can be determined according to the base class to which the type of each object to be picked up belongs.

[0134] For example, in the park inspection scenario described above, the base class of the objects to be picked up can be categorized as either recyclable or non-recyclable. Accordingly, the robot can prioritize picking up objects whose base class is recyclable. After picking up all objects whose base class is recyclable, it will then pick up objects whose base class is non-recyclable.

[0135] In this way, the robot can prioritize picking up specified types of objects in the surrounding environment based on the user's personalized needs, thereby improving the flexibility of the robot in performing tasks.

[0136] In one implementation, for each object to be picked up, the preset picking order of each object can be determined according to the order of distance from the robot's current position to the farthest.

[0137] Accordingly, the robot can determine the target object to be picked up from among the multiple objects to be picked up according to the preset picking order of the multiple objects to be picked up.

[0138] Then, the robot can travel to the pickup location of the target object (i.e., the target pickup location where the target object can be picked up), and after arriving at the target pickup location, use the robotic arm to pick up the target object.

[0139] After picking up the target object, the robot can determine the next target object to be picked up according to the preset picking order, and move from the current position to the picking position of the new target object. The robot will then perform the picking process of the new target object according to steps S1022 to S1023 until there are no more unpicked objects.

[0140] Based on the above processing, during the depth image detection process, if multiple objects to be picked up are detected in the surrounding environment, the robot can pick them up one by one according to a preset picking order. This ensures the completeness of the robot's task execution by picking up all the objects.

[0141] Regarding step S103, after picking up the object to be picked up, the robot can move from the position where the object was last picked up to the second position on the current task path.

[0142] In one implementation, the second position and the first position are the same position on the current task path.

[0143] The second position can be the position where the robot last left the current task path (i.e., the first position). This can be understood as the robot's position on the current task path when the object to be picked up was found in the most recently detected depth image.

[0144] In another implementation, the second position is the position on the current task path that follows the first position.

[0145] For example, the second position can be a position on the current task path that is located at a second preset distance after the first position.

[0146] For example, during each pickup task, the robot can record the location of each object to be picked up. Based on this historical data, the robot can determine the most frequently occurring locations on the current task path. Therefore, the second location can be the closest available location on the current task path to the first location, following it.

[0147] After reaching the second position, the robot can continue along the current task path. It is understood that during subsequent travel, the robot can use the object detection scheme provided in the above embodiments to detect in real time whether there is an object to be picked up in the robot's surrounding environment, until it reaches the end of the current task path.

[0148] In some embodiments, see Figure 4 , Figure 4This application provides a flowchart of a process for picking up objects using a robotic arm, as described in an embodiment. Accordingly, for each object to be picked up, the following steps are performed:

[0149] S401: Acquire a second depth image using a depth image acquisition device. The second depth image contains the object to be picked up.

[0150] S402: Generate point cloud data based on the second depth image.

[0151] S403: Based on point cloud data, perform pickup pose estimation to obtain the pickup pose required by the end effector of the robotic arm to pick up the object to be picked up.

[0152] S404: Perform inverse kinematics on the picked pose to obtain the joint angles that each joint of the robotic arm needs to achieve.

[0153] S405: Control each joint of the robotic arm to reach the joint angle and pick up the object to be picked up.

[0154] In this embodiment of the application, after the robot travels to the vicinity of the object to be picked up, it can acquire a second depth image containing the object using a depth image acquisition device. For example, after the robot travels to the pickup location of the object to be picked up, it can acquire a second depth image containing the object using a depth image acquisition device.

[0155] The robot can convert the pixel coordinates and depth values ​​of each pixel in the acquired second depth image based on the camera intrinsic parameter matrix of the depth image acquisition device that acquired the second depth image, and obtain three-dimensional point cloud data containing the object to be picked up.

[0156] In one implementation, the robot can input point cloud data into a pre-trained pose estimation model to obtain the estimated pose output by the model, which is then used as the picking pose of the object to be picked up (i.e., the picking pose required to pick up the object using the end effector of the robotic arm).

[0157] The pre-trained pose estimation model can be obtained by training the pose estimation model of the initial structure using sample point cloud data and the corresponding labels for each sample point cloud data. The sample point cloud data represents point cloud data of various preset object types, and the label corresponding to one sample point cloud data indicates the pose required for the robotic arm's end effector to pick up that object.

[0158] It is understood that the labels used during the training process target the end effector in the robotic arm, which is the same type as the end effector in the robotic arm installed on the robot in this embodiment.

[0159] For example, the pose estimation model can be a GSNet (Geometric and Scene-aware Network) model. The pose estimation models used in the embodiments of this application include, but are not limited to, the GSNet model described above.

[0160] After obtaining the picking pose of the object to be picked up (i.e., the picking pose required for the robotic arm's end effector to pick up the object), the robot can use inverse kinematics algorithms (e.g., analytical and numerical methods) to solve for the picking pose of the object, obtaining the joint angles required for each joint of the robotic arm. Accordingly, the robot can control each joint of the robotic arm to reach the joint angles obtained from the inverse kinematics solution to pick up the object.

[0161] Based on the above processing, after the robot travels to the pickup location of the object to be picked up, it is now relatively close to the object. Accordingly, a depth image acquisition device can be used to re-acquire a depth image (second depth image) containing the object to be picked up. Since the second depth image was acquired at a closer location, the point cloud data generated based on this second depth image can accurately represent the shape of the object to be picked up. Consequently, using the generated point cloud data for pose estimation can improve the accuracy of estimating the pickup pose required by the end effector in the robotic arm, thereby improving the stability of the pickup process.

[0162] In some embodiments, the bottom of the robot is provided with an adjustment component for adjusting the robot's posture.

[0163] Before step S401, the method further includes:

[0164] By adjusting the state of the adjustment components, the pose of the robot's depth image acquisition device is changed, and the object to be picked up is located within the field of view of the current depth image acquisition device.

[0165] In this embodiment of the application, for a target object to be picked up, as the robot moves toward the picking position of the target object, the target object may move away from the field of view of the depth image acquisition device along with the robot.

[0166] In order for the robot to acquire a depth image containing the object to be picked up after it has traveled to the picking location, the robot can adjust its posture using an adjustment component on its bottom to ensure that the object to be picked up is within the field of view of the depth image acquisition device on the robot.

[0167] For example, the adjustment component could be a height-adjustable support frame located on the rear side of the robot's bottom. As the robot approaches the object to be picked up, the object may be below the field of view of the depth image acquisition device. Accordingly, the robot can raise the height of the support frame, causing the front side of the robot body, on which the depth image acquisition device is mounted, to tilt downwards, so that the object to be picked up is within the field of view of the depth image acquisition device mounted on the robot.

[0168] Subsequently, the robot can acquire a depth image (i.e., a second depth image) containing the object to be picked up using a depth image acquisition device.

[0169] Based on the above processing, the robot can adjust its pose using the adjustment components located at its bottom. Furthermore, by changing the robot's pose, the field of view of the depth image acquisition device mounted on the robot can be adjusted. This allows the solution provided in this embodiment to be applicable to picking up objects at different heights, improving the stability of the robot's task execution.

[0170] In one implementation, the adjusting component includes multiple support structures, each support structure including at least one rotatable joint.

[0171] In this embodiment, the robot's bottom may be provided with multiple support structures, each support structure including at least one rotatable joint. Accordingly, the robot can adjust the height of a support structure by controlling the rotatable joint included in a support structure, thereby changing the robot's pose.

[0172] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a quadruped robot provided in an embodiment of this application. Figure 5 The quadruped robot comprises a body 501 and four supporting legs 502. Depth image acquisition devices 503 are located on the front and left sides of the body, respectively. A robotic arm 504 and a storage container 505 are located on the top of the body. Additionally, a radar sensor 506 is located on the front of the body. The supporting legs constitute the aforementioned support structure.

[0173] Based on the above processing, the robot can flexibly adjust its pose by controlling multiple support structures at its bottom. Furthermore, by changing the robot's pose, the field of view of the depth image acquisition device mounted on the robot can be adjusted. Thus, the solution provided in this application embodiment can be applied to picking up objects at different heights, improving the stability of the robot's task execution.

[0174] In one implementation, when multiple depth image acquisition devices are installed in the robot, the robot's pose is changed so that the object to be picked is located within the field of view of a specified depth image acquisition device among the multiple depth image acquisition devices.

[0175] In this embodiment, after the robot travels to the pickup location of the object to be picked up, the object is typically located in front of the robot. Correspondingly, if the robot is equipped with multiple depth image acquisition devices, after traveling to the pickup location of the object, the robot can adjust the adjustment components to ensure that the object is within the field of view of a designated depth image acquisition device among the multiple depth image acquisition devices.

[0176] The specified depth image acquisition device can be a depth image acquisition device installed on the front of the robot. Alternatively, the specified depth image acquisition device can be any one of multiple depth image acquisition devices.

[0177] For example, for Figure 5 As the quadruped robot approaches the object to be picked up, the object may be located below the field of view of the depth image acquisition device mounted on the front of the robot. Accordingly, the robot can lower the support height of its two supporting legs by controlling the angle of the rotatable joints on the front of its body, thus achieving a robot-attached posture in front of the object. This lowers the field of view of the depth image acquisition device, ensuring that the object is within its field of view.

[0178] Based on the above processing, after the robot travels to the pickup location of the object to be picked up, the position of the robot is adjusted by controlling the support structure so that the designated depth image acquisition device can acquire a depth image containing the object to be picked up. This avoids the robot frequently moving and adjusting its position to acquire the depth image of the object to be picked up, ensuring the smoothness of the robot's pickup process and ensuring that the robotic arm can accurately and safely pick up the object.

[0179] In some embodiments, the robot is also provided with a storage container for placing the picked-up objects.

[0180] The steps described above for using a robotic arm to pick up the object include:

[0181] When the storage container is not full, a robotic arm is used to pick up the object to be picked up.

[0182] The method also includes:

[0183] Step 1: If the storage container is full, the robot will move from its current location to the target cleaning point and perform the cleaning task of the storage container.

[0184] Step 2: In response to having picked up the object to be picked up, move to the second position to continue following the current task path; in response to not having picked up the object to be picked up, move to the picking position to perform the step of picking up the object using the robotic arm.

[0185] In this embodiment of the application, the robot is also provided with a container for placing the picked-up objects.

[0186] For example, the storage container can be installed on top of the robot body, or it can be an open frame hung on the side of the robot body. Figure 5 As shown, the storage container 505 can be mounted on the upper part of the quadruped robot.

[0187] During the depth image detection process described above, if multiple objects to be picked up are detected in the surrounding environment, the robot can pick up each object one by one according to the preset picking order of the multiple objects to be picked up.

[0188] Accordingly, before picking up each object, the robot can determine whether the current container is saturated.

[0189] For example, a robot can be equipped with an image acquisition device to photograph the storage container. Accordingly, the robot can use the image acquired by the image acquisition device to detect whether the storage container is saturated.

[0190] If the space in the container is not saturated, the process of picking up the object to be picked up can be performed, that is, the robotic arm can be used to pick up the object to be picked up.

[0191] In one implementation, since the installation position of the container on the robot is fixed, the robot can determine the joint angles (which can be called the placement joint angles of each joint) required by the robotic arm when placing the object to be picked up into the container using the robotic arm, based on the installation position of the container on the robot.

[0192] Correspondingly, when the robotic arm needs to place the object to be picked up into the storage container, the robot can control the robotic arm to place the object into the storage container according to the pre-determined placement joint angles of each joint.

[0193] In one implementation, the map data pre-acquired by the robot, which includes a preset inspection area, records all cleaning points within the preset inspection area.

[0194] When the storage container is full, the robot can determine the target cleaning point.

[0195] In one implementation, the target cleaning point is determined based on the distance between the robot's current position and each cleaning point. For example, the target cleaning point can be the cleaning point closest to the robot's current position.

[0196] Alternatively, the target cleaning point can be determined based on the distance between the robot's current position and each cleaning point, as well as the status of each cleaning point. For example, the target cleaning point can be the cleaning point closest to the robot's current position among the cleaning points that support cleaning the already picked-up objects stored in the storage container.

[0197] For example, in the park inspection scenario described above, the cleaning point could be a garbage station. The robot can then travel to the nearest cleaning point and empty the garbage from the storage container.

[0198] Then, the robot can travel to the identified target cleaning point and, upon reaching the target cleaning point, perform the task of cleaning the container.

[0199] After completing the cleaning task, in response to the absence of any objects to be picked up (i.e., there are unpicked objects), indicating that not all of the previously detected objects were picked up, the robot can move to the picking location to perform the step of picking up the objects using its robotic arm. In other words, the robot can move to the target picking location of the target object to pick up using its robotic arm.

[0200] In response to the fact that an object to be picked up has been picked up (i.e., there are no unpicked objects to be picked up), it indicates that all the multiple objects to be picked up previously detected have been picked up. At this point, the robot can move to the second position to continue following the current task path.

[0201] Based on the above processing, the robot can store the picked-up objects in its installed container while traveling along the current task path. This eliminates the need for the robot to travel to a cleanup point after each object is picked up, thus improving task efficiency.

[0202] In some embodiments, in response to the robot detecting that its own power meets the charging conditions, the robot travels to the charging point and charges. The charging conditions include: the robot's remaining power is lower than a preset threshold and / or not greater than the expected power, which is determined based on the power required for the robot to travel along the shortest path between its current position and the charging point.

[0203] In one implementation, if the robot detects that its current battery level has dropped to a preset threshold while it is moving, it indicates that the robot needs to return to the charging point for recharging. That is, the robot can travel from its current position to the charging point to recharge. For example, the preset threshold could be 20%.

[0204] In another implementation, the robot can dynamically determine whether the remaining battery power is sufficient based on its current location and the distance to the charging point. Correspondingly, other methods include:

[0205] While traveling along the current task path, when the preset power detection time is reached, the robot determines the shortest path from its current position to the charging point, as well as the estimated power required to travel that shortest path, when traveling along the preset drivable path.

[0206] If the robot's current battery level is not greater than the expected battery level, it will travel to the charging point to recharge.

[0207] In this embodiment, while traveling along the current task path, the robot can periodically perform a battery level check. That is, the time interval between two adjacent preset battery level checks is fixed. For example, the robot can perform a battery level check every minute.

[0208] During the process of performing a power level check, the robot can determine the shortest path between its current position and the charging point when traveling along a preset drivable path.

[0209] In one implementation, the map data pre-acquired by the robot, which includes a preset inspection area, records a preset drivable path within the preset inspection area.

[0210] The preset drivable path refers to all paths within the preset inspection area that the robot can traverse.

[0211] Accordingly, after determining the shortest path between the robot's current location and the charging point, the estimated power required to travel that shortest path can be determined based on its length. Understandably, the longer the shortest path, the higher the estimated power required to travel it.

[0212] For example, the estimated power required to travel the shortest path can be: the minimum power required for the robot to travel the shortest path. Alternatively, the estimated power required to travel the shortest path can be: the minimum power required for the robot to travel the shortest path, plus a certain amount of additional power.

[0213] After determining the estimated power required to travel the shortest path, the robot can judge whether the current power level is greater than the determined estimated power level.

[0214] If the robot's current battery level is higher than the expected battery level, it means that the robot has sufficient power. If the robot continues to perform the picking task, the current battery level can support the robot to return to the charging point. Therefore, the robot can perform subsequent tasks and periodically perform the above battery level judgment process during the execution of subsequent tasks.

[0215] If the robot's current battery level is not greater than the expected battery level, it indicates that the robot's current battery level is low. If the robot continues to perform the picking task, it may run out of power before returning to the charging point. In order to ensure that the robot can travel back to the charging point, it should travel from its current position to the charging point to recharge.

[0216] Based on the above processing, the robot can determine whether it needs to return to the charging point for recharging by considering the shortest path back. This enables dynamic battery level detection based on the length of the return path. While ensuring the robot can travel from its current position to the charging point, this approach maximizes the completeness of the robot's task execution.

[0217] In some embodiments, the robot is further equipped with a radar sensor, which is used to detect whether there are obstacles in the portion of the current task path to be traversed. Correspondingly, the method also includes:

[0218] Step a: If a moving obstacle is detected, stop moving and, based on the obstacle's direction and speed of movement, determine the target time when the obstacle leaves the current mission path. At the target time, continue detection using the radar sensor.

[0219] Step b: When a non-moving obstacle is detected, generate an obstacle avoidance path for the detected non-moving obstacle based on a preset obstacle avoidance path generation algorithm.

[0220] Step c: Replace part of the current task path with the obstacle avoidance path to obtain a new task path, wherein the starting and ending positions of the replaced part of the path are the same as the starting and ending positions of the obstacle avoidance path, respectively.

[0221] In this embodiment of the application, while traveling along the current task path, the robot can use its radar sensors to detect whether there are obstacles in the part of the task path to be traversed. For example, the robot can determine that there are obstacles in the current task path if it detects an obstacle within 1.5 meters ahead, or if there are objects within 0.5 meters to either side of the robot's center.

[0222] In addition, the robot can also use radar sensors to detect the movement of obstacles in the section of the current task path that needs to be traversed.

[0223] For example, a robot can stop when it detects an obstacle in the part of its current task path it needs to traverse. Based on data collected by radar sensors in two adjacent frames after stopping, it can calculate whether the obstacle has moved relative to the robot. If the obstacle has moved relative to the robot, it can be determined that the obstacle is in a moving state, and the direction and speed of movement can be determined based on the data collected by radar sensors in two adjacent frames. If the obstacle has not moved, it can be determined that the obstacle is in a stationary state.

[0224] Accordingly, upon detecting a moving obstacle, the robot can stop moving and, based on the obstacle's direction and speed of movement, determine the target time when the obstacle leaves the current task path. Upon reaching the target time, the robot can resume detection using its radar sensors.

[0225] Understandably, after calculating the obstacle's direction and speed of movement, the robot can stop detecting the obstacle using its radar sensors before the target time. This avoids wasting the robot's battery power and reduces ineffective energy consumption during obstacle avoidance.

[0226] When a stationary obstacle is detected, the robot can trigger local obstacle avoidance processing. That is, it generates an obstacle avoidance path for the detected obstacle.

[0227] In one implementation, the robot can generate an obstacle avoidance path for detected obstacles based on the current task path, according to a preset obstacle avoidance path generation algorithm. The starting and ending points of the obstacle avoidance path are both located on the current task path. For example, the preset obstacle avoidance path generation algorithm could be the Dijkstra algorithm, or the A* algorithm, etc.

[0228] Furthermore, the robot can replace a portion of the current task path with an obstacle-avoidance path; that is, it can replace the portion of the current task path between the start and end points of the obstacle-avoidance path with the generated obstacle-avoidance path to obtain a new task path. In this way, the robot can update the current task path while traveling along it. Subsequently, it travels according to this new task path (i.e., the current task path). That is, for the new task path, the robot can perform the picking task represented by the new task path according to the task execution method provided in this application.

[0229] If no obstacles are detected, the robot can proceed along its current task path to the section where there are no obstacles. Conversely, during travel, the robot can use its radar sensors to detect whether there are obstacles in the section of its current task path. This enables dynamic obstacle avoidance during the robot's movement.

[0230] Based on the above processing, when the robot is traveling along the current task path, obstacle avoidance can be triggered if there are obstacles in the untraveled task path. Furthermore, when generating the obstacle avoidance path, it is ensured that both the start and end points of the path are located on the current task path. This allows the robot to return to the original task path after avoiding the obstacle and continue processing subsequent tasks.

[0231] In some embodiments, while traveling along the current task path, the robot can obtain its current position based on preset sensor data and a multi-sensor SLAM (Simultaneous Localization and Mapping) algorithm, and simultaneously send it to the control terminal. The preset sensor data includes data from the chassis odometer, inertial measurement unit, radar sensor, and image acquisition device.

[0232] Correspondingly, the control terminal can obtain the real-time position of each robot in the robot control system. For example, the robot can send its current position to the control terminal at preset intervals in the form of heartbeat messages.

[0233] In one implementation, to ensure the robot's safety during operation, if the robot detects that its current position is outside a preset inspection area, it can trigger an automatic return-to-home process. That is, the robot can return to the preset inspection area by following the path it took to reach its current position.

[0234] Based on the above processing, the robot can perform localization by combining data from multiple sensors. Compared to localization based solely on the number of radar sensors, the solution provided in this application avoids the problem of location loss in relatively open environments, improves the accuracy of robot localization, and ensures the accuracy of the real-time position determined by the robot.

[0235] In some embodiments, the robot is a quadruped robot.

[0236] During the robot's movement, the quadruped robot can achieve movement through the joints contained in its four supporting structures.

[0237] Quadruped robots can be continuously trained during movement using reinforcement learning, enabling them to control the joints of their supporting structures to perform actions such as moving forward, backward, left and right, and rotating at specified speeds.

[0238] Based on the same inventive concept, this application also provides a robot control method, applied to a control terminal, the method comprising:

[0239] Send the task path to the robot;

[0240] The task path is the path the robot follows according to the task execution method in the first aspect.

[0241] Based on the above processing, the control unit can assign a task path to the robot. The robot can use the received task path as its current task path. While traveling along the current task path, the robot acquires a first depth image of the surrounding environment using a depth image acquisition device. It then detects the acquired first depth image to determine if there is an object to be picked up (i.e., the object to be picked up) in the robot's current environment. If so, the robot can travel to the picking location of the object and use its robotic arm to pick it up. Then, it returns to the second position on the current task path and continues traveling along the task path. Here, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path after the first position. Therefore, while traveling along the portion of the task path after the second position, the robot can continue to detect whether there is an object to be picked up in the surrounding environment, thus ensuring that the robot can continue to perform the picking task represented by the current task path. In other words, the robot can automatically detect and locate the object to be picked up by using a depth image acquisition device, and then pick up the object by a robotic arm. The robot can then execute the picking task represented by the current task path according to the above logic, thereby improving the robot's autonomy in performing picking tasks.

[0242] In this embodiment of the application, there may be multiple task paths in the preset inspection area.

[0243] In one implementation, the control terminal can respond to a user-triggered task issuance command by sending a specified task path to an idle robot, so that the robot, upon receiving the task path, will travel along the task path and complete the picking task represented by the task path.

[0244] A robot control system can include multiple robots.

[0245] In some embodiments, the method further includes, before sending the task path to the robot:

[0246] Based on the estimated power required to perform the picking task according to the task path, a robot is selected from the idle robots in the current scene whose actual power is greater than the power threshold, and whose estimated power is not greater than the power threshold.

[0247] In this embodiment, before issuing the task path, the control terminal can obtain the actual battery level of each robot within the robot control system. Then, based on the estimated battery level required to perform the pickup task according to the task path, the control terminal can determine a robot from the idle robots in the current scene whose actual battery level is greater than a battery threshold. Furthermore, the estimated battery level is not greater than the battery threshold. For example, the battery threshold can be calculated by adding a certain amount of buffered battery level to the estimated battery level.

[0248] In this way, before issuing the task path, the control terminal can prioritize selecting the robot with sufficient power to perform the picking task represented by the task path from among the multiple robots in the scene, thus ensuring the integrity of the task execution.

[0249] In some embodiments, in response to the robot not being able to continue performing the pickup task according to the current task path, an idle robot other than the robot in the current scene is selected as the robot to continue performing the pickup task, and the untraveled part of the task path is selected as the current task path and sent to that robot.

[0250] In this embodiment, if the robot is unable to continue performing the picking task according to the current task path, the control terminal can select a robot from the idle robots in the current scene other than the robot itself, as the robot to continue performing the picking task.

[0251] For example, while a robot is traveling along its current task path, it can periodically execute the battery level determination process described in the above embodiment. If it detects that its current battery level meets the charging requirements, the robot can send a return message to the control terminal, carrying information about the untraveled parts of its current task path. Correspondingly, when the control terminal receives a return message from a robot, it can determine that the robot cannot continue performing the pickup task along its current task path. The control terminal can then retrieve the path carried in the return message.

[0252] Furthermore, the control unit can send the untraveled portion of the task path to the selected robot that continues to perform the picking task, so that the robot takes the received task path as the current task path and travels according to the current task path to complete the picking task represented by the untraveled portion.

[0253] Based on the above processing, if the robot is unable to continue executing the pickup task according to the current task path, the control unit can send the incomplete task path to other robots, so that the other robots can continue to travel according to the incomplete task path. In this way, the integrity of task execution can be guaranteed.

[0254] In one implementation, the control terminal and the robot can communicate using a preset encryption / decryption algorithm.

[0255] For example, the preset encryption and decryption algorithms can be: DES (Data Encryption Standard) algorithm, or AES (Advanced Encryption Standard) algorithm, etc.

[0256] In this embodiment, the control unit can encrypt the raw data (such as the task path and map data mentioned above) to be sent to the robot based on a preset encryption algorithm. Correspondingly, the robot can decrypt the received encrypted data according to a preset decryption algorithm to obtain the original data.

[0257] In some embodiments, for each robot, a user can control the robot via a control handle. Accordingly, the robot can communicate with the control handle.

[0258] For example, a user can trigger a return-to-charging command by pressing a preset button on the control handle, instructing the robot to return to the charging point. Upon receiving the return-to-charging command, the robot can then travel from its current location to the charging point.

[0259] In one implementation, the control handle can also provide an interactive interface for displaying the robot's real-time status.

[0260] Correspondingly, the control handle can display the real-time status of multiple robots in the robot control system on the interactive interface.

[0261] The real-time status of a robot includes: the robot's current location, the robot's current battery level, and whether the robot's storage container is full.

[0262] Users can also manage the robots included in the robot control system through the interactive interface. For example, they can perform operations such as deleting or adding robots.

[0263] Based on the same inventive concept, this application also provides a robot control system, which includes a control terminal and a robot.

[0264] A robot for performing any of the task execution methods described in the first aspect above;

[0265] The control terminal is used to execute any of the task execution methods described in the second aspect above.

[0266] See Figure 6 , Figure 6 This is an interactive schematic diagram of the robot control system provided in an embodiment of this application. Figure 6 In this system, the robot control system includes a control terminal, robot 1, and robot 2.

[0267] S601: The control terminal sends the task path to robot 1.

[0268] S602: Robot 1 takes the received task path as the current task path and moves along the current task path to perform the picking task.

[0269] S603: Robot 1 sends a message to the control terminal indicating that it will stop performing the task.

[0270] That is, if robot 1 detects that its current battery level meets the charging requirements, it can send a return message to the control terminal containing the untraveled paths in the current task path, indicating that robot 1 will stop executing the task.

[0271] S604: The control terminal sends the untraveled portion of the current task path of robot 1 to robot 2.

[0272] That is, if robot 2 is currently idle and its real-time battery power is greater than the estimated battery power required to perform the picking task according to the untraveled part, the path of the untraveled part can be sent to robot 2.

[0273] S605: Robot 2 takes the received task path as the current task path and moves along the current task path to perform the picking task.

[0274] See Figure 7 , Figure 7 This is a flowchart illustrating the process of a robot performing a task, as provided in an embodiment of this application.

[0275] First, during the initialization phase, the user can manually turn on the robot. Then, the user can press the power button on the remote control (control handle) and press the positioning initialization button to initialize the robot's current position. The robot can then determine if positioning was successful. If so (i.e., positioning successful), it enters the inspection phase (i.e., begins inspection). If not (i.e., positioning unsuccessful), the robot can display a positioning failure message via the control handle, prompting the user to re-initialize the positioning using the handle.

[0276] During the inspection phase, the robot starts from the charging station (i.e., the charging point) and picks up objects of a specified type from the surrounding environment according to a preset path. The preset path can be the task path sent to the robot from the control terminal. While traveling along the current task path, the robot can determine in real time whether the basket is full (i.e., whether the container is full). If not (i.e., the basket is not full), the robot can continue picking up objects according to the current task path until the inspection is completed and it reaches the charging station.

[0277] If the basket is full, the robot can pause picking up the detected items and indicate that the basket is full via the control handle. The robot can then proceed to the nearest cleaning point to wait. After cleaning the storage basket (i.e., the container), the robot can continue its inspection, returning to the pause point (the location it left last in the task path) and continuing to pick up items along the current task path until the inspection is complete and it reaches the charging station.

[0278] During the inspection phase, the robot can periodically execute the battery level determination process described in the above embodiment. When a low battery abnormality is detected, it will pause picking up items and automatically return to the charging station to recharge, indicating the low battery level via the control handle.

[0279] During the inspection phase, if the robot detects any abnormalities, such as tipping over, hardware damage, power outage, positioning failure, or navigation failure, it can alert the user via the control handle. The user can then press the remote control button on the control handle to remotely control the robot to return to the charging point (i.e., remotely return to the charging dock).

[0280] Based on the same inventive concept, this application provides a task execution device for use in a robot, wherein the robot is equipped with a depth image acquisition device and a robotic arm.

[0281] See Figure 8 , Figure 8This is a structural diagram of a task execution device provided in an embodiment of this application. The device includes:

[0282] The first acquisition module 801 is used to acquire a first depth image of the surrounding environment through the depth image acquisition device while traveling according to the current task path.

[0283] The picking control module 802 is used to move from the robot's current first position to the picking position when the presence of an object to be picked in the first depth image is detected, and to pick up the object to be picked up using the robotic arm after moving to the picking position. The picking position is the position where the robot can pick up the object to be picked up.

[0284] The execution control module 803 is used to travel from the pickup position to the second position, and continue to travel along the current task path after arriving at the second position;

[0285] Wherein, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that is after the first position.

[0286] In some embodiments, the apparatus further includes:

[0287] The first position determination module is used to determine the position of the object to be picked up based on the first position of the robot, the installation position of the depth image acquisition device on the robot, and the pixel coordinates and depth information of the object to be picked up in the first depth image before the robot moves from its current first position to the picking position.

[0288] The second position determination module is used to determine the pickup position based on the position of the object to be picked up and the pickup range supported by the robotic arm.

[0289] In some embodiments, the pickup control module 802 is specifically used for:

[0290] When there are multiple objects to be picked up, the target object to be picked up is determined from the objects to be picked up.

[0291] The robot moves from the first position to the target pickup position, and after reaching the target pickup position, it uses the robotic arm to pick up the target object to be picked up. The target pickup position is the position where the robot can pick up the target object to be picked up.

[0292] Based on the target pickup position, the target object to be picked is determined from the unpicked objects to be picked and picked, until the objects to be picked are traversed.

[0293] In some embodiments, for each object in the objects to be picked, the following steps are performed:

[0294] A second depth image is acquired using the depth image acquisition device, and the second depth image contains the object to be picked up;

[0295] Point cloud data is generated based on the second depth image;

[0296] Based on the point cloud data, the picking pose is estimated to obtain the picking pose required by the end effector of the robotic arm to pick up the object to be picked up.

[0297] The inverse kinematics of the pickup pose is used to obtain the joint angles that each joint of the robotic arm needs to achieve.

[0298] Control each joint of the robotic arm to reach the joint angle, and pick up the object to be picked up.

[0299] In some embodiments, the bottom of the robot is provided with an adjustment component for adjusting the robot's posture;

[0300] The device further includes:

[0301] The position adjustment module is used to change the pose of the robot's depth image acquisition device by adjusting the state of the adjustment component before the second depth image is acquired by the depth image acquisition device, so that the object to be picked up is located within the field of view of the current depth image acquisition device.

[0302] In some embodiments, the adjusting component includes a plurality of support structures, each support structure including at least one rotatable joint;

[0303] And / or,

[0304] When the robot is equipped with multiple depth image acquisition devices, the robot's pose is changed so that the object to be picked up is located within the field of view of a designated depth image acquisition device among the multiple depth image acquisition devices.

[0305] In some embodiments, the robot is further provided with a storage container for placing the picked-up objects;

[0306] The pickup control module 802 is specifically used for:

[0307] When the space in the storage container is not saturated, the robotic arm is used to pick up the object to be picked up.

[0308] And / or,

[0309] The device further includes:

[0310] The cleaning task execution module is used to move from the robot's current location to the target cleaning point and perform the cleaning task of the storage container when the space of the storage container is saturated.

[0311] The first response execution module is configured to, in response to having picked up the object to be picked up, travel to the second position to continue traveling along the current task path; and in response to not picking up the object to be picked up, travel to the picking position to perform the step of picking up the object to be picked up using the robotic arm.

[0312] And / or,

[0313] The second response execution module is used to respond to the robot's detection that its own power meets the charging conditions, and to drive to the charging point and charge. The charging conditions include: the robot's remaining power is lower than a preset threshold and / or not greater than the expected power. The expected power is determined based on the power required for the robot to travel along the shortest path between its current position and the charging point.

[0314] In some embodiments, the robot is also equipped with a radar sensor, which is used to detect whether there are obstacles in the part of the current task path to be traversed;

[0315] The device further includes:

[0316] The first detection module is used to stop moving when a moving obstacle is detected, and to obtain the target time when the obstacle leaves the current task path based on the moving direction and moving speed of the obstacle, and to continue detection by the radar sensor at the target time;

[0317] The obstacle avoidance path generation module is used to generate an obstacle avoidance path for the detected non-moving obstacle based on a preset obstacle avoidance path generation algorithm when a non-moving obstacle is detected.

[0318] The path update module is used to replace a portion of the current task path with the obstacle avoidance path to obtain a new task path, wherein the starting and ending positions of the replaced portion of the path are the same as the starting and ending positions of the obstacle avoidance path, respectively.

[0319] Based on the same inventive concept, embodiments of this application provide a task execution device applied to a control terminal, the device comprising:

[0320] The first path sending module is used to send the task path to the robot;

[0321] The task path is the path that the robot travels according to the task execution method described in the first aspect.

[0322] In some embodiments, the apparatus further includes at least one of the following:

[0323] The robot selection module is used to determine the robot from idle robots in the current scene whose actual power is greater than a power threshold based on the estimated power required to perform the picking task according to the task path before sending the task path to the robot, wherein the estimated power is not greater than the power threshold.

[0324] The forwarding module is used to respond to situations where the robot is unable to continue executing the pickup task according to the current task path, by selecting an idle robot in the current scene other than the robot to continue executing the pickup task, and sending the untraveled portion of the task path as the current task path to that robot.

[0325] This application also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0326] Memory 903 is used to store computer programs;

[0327] When the processor 901 executes the program stored in the memory 903, it implements the steps of any of the above-described task execution methods.

[0328] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0329] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0330] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0331] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0332] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the above-described task execution methods.

[0333] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the task execution methods described in the above embodiments.

[0334] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0335] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0336] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system, apparatus, electronic device, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0337] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A task execution method, characterized in that, Applied to a robot, the robot being equipped with a depth image acquisition device and a robotic arm, the method includes: While traveling along the current task path, the depth image acquisition device acquires a first depth image of the surrounding environment. If an object to be picked up is detected in the first depth image, the robot moves from its current first position to the picking position, and after reaching the picking position, it uses the robotic arm to pick up the object to be picked up. The picking position is the position where the robot can pick up the object to be picked up. The vehicle travels from the pick-up location to the second location, and continues along the current task path after reaching the second location. Wherein, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that is after the first position.

2. The method according to claim 1, characterized in that, Before moving from the robot's current first position to the pickup position, the method further includes: The position of the object to be picked up is obtained based on the robot's current first position, the installation position of the depth image acquisition device on the robot, and the pixel coordinates and depth information of the object to be picked up in the first depth image. The pickup position is determined based on the position of the object to be picked up and the pickup range supported by the robotic arm.

3. The method according to claim 1 or 2, characterized in that, The process of moving from the robot's current first position to the pickup position, and then using the robotic arm to pick up the object to be picked up after reaching the pickup position, includes: When there are multiple objects to be picked up, the target object to be picked up is determined from the objects to be picked up. The robot moves from the first position to the target pickup position, and after reaching the target pickup position, it uses the robotic arm to pick up the target object to be picked up. The target pickup position is the position where the robot can pick up the target object to be picked up. Based on the target pickup position, the target object to be picked is determined from the unpicked objects to be picked and picked, until the objects to be picked are traversed.

4. The method according to any one of claims 1 to 3, characterized in that, For each object in the list of objects to be picked, the following steps are performed: A second depth image is acquired using the depth image acquisition device, and the second depth image contains the object to be picked up; Point cloud data is generated based on the second depth image; Based on the point cloud data, the picking pose is estimated to obtain the picking pose required by the end effector of the robotic arm to pick up the object to be picked up. The inverse kinematics of the pickup pose is used to obtain the joint angles that each joint of the robotic arm needs to achieve. Control each joint of the robotic arm to reach the joint angle, and pick up the object to be picked up.

5. The method according to claim 4, characterized in that, The bottom of the robot is provided with an adjustment component for adjusting the robot's posture; Before acquiring the second depth image using the depth image acquisition device, the method further includes: By adjusting the state of the adjustment component, the pose of the depth image acquisition device of the robot is changed, and the object to be picked up is located within the field of view of the current depth image acquisition device.

6. The method according to claim 5, characterized in that, The adjustment component includes multiple support structures, and each support structure includes at least one rotatable joint. And / or, When the robot is equipped with multiple depth image acquisition devices, the robot's pose is changed so that the object to be picked up is located within the field of view of a designated depth image acquisition device among the multiple depth image acquisition devices.

7. The method according to any one of claims 1 to 6, characterized in that, The robot is also equipped with a storage container for placing the picked-up objects. The process of using the robotic arm to pick up the object to be picked up includes: When the space in the storage container is not saturated, the robotic arm is used to pick up the object to be picked up. And / or, The method further includes: If the space in the storage container is full, the robot will move from its current location to the target cleaning point and perform the cleaning task of the storage container. In response to having picked up the object to be picked up, the robot moves to the second position to continue following the current task path; in response to not picking up the object to be picked up, the robot moves to the picking position to perform the step of picking up the object to be picked up using the robotic arm. And / or, In response to the robot detecting that its own power meets the charging conditions, it travels to the charging point and begins charging. The charging conditions include: the robot's remaining power is below a preset threshold and / or not greater than the expected power, which is determined based on the power required for the robot to travel along the shortest path between its current position and the charging point.

8. The method according to any one of claims 1 to 7, characterized in that, The robot is also equipped with a radar sensor, which is used to detect whether there are obstacles in the part of the current task path to be traversed; The method further includes: If a moving obstacle is detected, the movement stops and the target time when the obstacle leaves the current task path is obtained based on the obstacle's direction of movement and speed. At the target time, the detection continues through the radar sensor. When a non-moving obstacle is detected, a bypass path is generated for the detected non-moving obstacle based on a preset obstacle bypass path generation algorithm. A new task path is obtained by replacing a portion of the current task path with the obstacle avoidance path, wherein the starting and ending positions of the replaced portion of the path are the same as the starting and ending positions of the obstacle avoidance path, respectively.

9. A task execution device, characterized in that, Applied to robots, the robots are equipped with depth image acquisition devices and robotic arms, and the device includes: The first acquisition module is used to acquire a first depth image of the surrounding environment through the depth image acquisition device while traveling according to the current task path. The picking control module is used to move from the robot's current first position to the picking position when the presence of an object to be picked in the first depth image is detected, and to pick up the object to be picked up using the robotic arm after moving to the picking position. The picking position is the position where the robot can pick up the object to be picked up. The execution control module is used to travel from the pickup position to the second position, and continue to travel along the current task path after arriving at the second position; Wherein, the first position and the second position are the same position on the current task path, or the second position is a position on the current task path that is after the first position.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method according to any one of claims 1 to 8.

Images