A robot motion planning method, device, robot and storage medium

Abstract

The present application relates to the field of robot technology, and in particular to a robot motion planning method, device, robot and storage medium, which are used to solve the problem of low success rate of operation execution in related technologies. The method is: based on the position of a first object and the current position of the robot, the robot performs motion planning for the operation to be performed on the first object, and moves toward the first object based on the obtained first planning result; during the movement, a second visual sensor deployed on the end effector of the robot continuously acquires a second image, and based on the position and the currently acquired second image, when it is determined that the first object is perceived by the second visual sensor, local motion planning is performed for the operation of the execution device for the first object based on the currently acquired second image and the subsequent second image acquired by the second visual sensor, and the execution device is controlled to act based on the obtained second planning result, thereby improving the success rate of the operation execution.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot motion planning method, device, robot and storage medium. Background Art

[0002] With the development of modern industry and the advancement of science and technology, robotics has penetrated into various fields. Usually, robots obtain information about the surrounding environment through deployed sensors, and convert the information into environmental data that can be understood by computers. Based on the environmental data, they build a terrain map, and then identify the position of objects in the environment through computer vision or other recognition technologies. Based on the position of the object, they perform path and / or motion planning, thereby realizing robot operations such as grasping and assembling objects.

[0003] Since the robot perceives the environment through deployed sensors, the accuracy of the robot's perception of the environment is strongly related to the accuracy of the sensors, and the accuracy of the sensors is related to the distance from the object, whether the object is blocked, etc. For example, in some scenes, the position of the object cannot be accurately estimated because the object is far away; in other scenes, the position of the object cannot be obtained because the object is blocked. Regardless of which of the above reasons leads to the inability to accurately estimate the position of the object, it will lead to the inability to accurately estimate the robot's operating posture, which directly affects the success rate of the operation execution. Summary of the invention

[0004] The embodiments of the present application provide a robot motion planning method, device, equipment and medium for realizing detailed motion planning of a robot arm and an end effector, thereby improving the accuracy of the robot's operating posture and thus improving the success rate of the robot's operating operation execution.

[0005] The specific technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a motion planning method for a robot, comprising:

[0007] Based on the position of the first object and the current position of the robot, motion planning is performed for the robot to perform an operation on the first object to obtain a first planning result, and moving toward the first object based on the first planning result, wherein the position is determined based on a first image acquired by a first visual sensor deployed on the head of the robot;

[0008] During the movement, a second visual sensor deployed on the end effector of the robot continuously acquires a second image, and the following operations are performed for each second image: based on the position and the currently acquired second image, determining whether the first object is sensed by the second visual sensor;

[0009] If so, based on the currently acquired second image and the subsequent second image acquired by the second visual sensor, local motion planning is performed on the robot's execution device for the operation of the first object to obtain a second planning result, and based on the second planning result, the execution device is controlled to perform action, and the execution device includes the end effector and an associated robotic arm.

[0010] A robot motion planning method provided in an embodiment of the present application can determine the position of a first object through a first image obtained by a first visual sensor deployed on the head of the robot, and then, based on the position and the current position of the robot, perform motion planning for the robot to perform an operation on the first object, thereby realizing the overall motion planning of the robot, thereby moving toward the first object based on the obtained first planning result, and during the movement, continuously obtain a second image through a second visual sensor deployed on the end effector of the robot, and determine whether the first object is perceived through the second visual sensor based on the position and the currently obtained second image. If so, it means that the first object has been perceived through the second visual sensor. Based on the currently obtained second image and the subsequent second image obtained by the second visual sensor, local motion planning is performed for the operation of the execution device on the first object, that is, switching to the image obtained by the second visual sensor for detailed motion planning of the robotic arm and the end effector, thereby improving the accuracy of the robot's operating posture, and also improving the efficiency and accuracy of the operation, thereby improving the success rate of the operation execution.

[0011] In a possible implementation manner, the position of the first object is determined by:

[0012] Based on the received task instruction, acquiring the first image by the first visual sensor;

[0013] Performing target recognition on each object in the first image to determine whether the first object exists in the first image;

[0014] If so, determining the position based on an image position of the first object in the first image;

[0015] If not, and a second object including the first object is identified to exist in the first image, the position is estimated based on the image position of the second object in the first image and the three-dimensional structure data of the second object.

[0016] The above method, through prior knowledge, that is, the three-dimensional structure data of the second object containing the first object, realizes the accurate identification of the position of the first object that the robot cannot directly perceive. Compared with the related technology, such scenarios cannot be operated by robots. More importantly, this solution provides a direction for the intelligent transformation of similar production lines in smart manufacturing scenarios. Therefore, this solution has important guiding significance.

[0017] In a possible implementation, based on the position of the first object and the current position of the robot, performing motion planning for the robot to perform an operation on the first object to obtain a first planning result includes:

[0018] Determine a standing position of the robot based on the position and the length of the robot's mechanical arm, and environmental information of the environment in which the robot is located;

[0019] Perform path planning based on the standing point and the current position to obtain a moving path;

[0020] Based on the moving path and the position, motion planning is performed on the execution device to obtain the first planning result.

[0021] In a possible implementation, the second planning result includes a plurality of sub-results, each sub-result corresponds to a target image, and the target image is the currently acquired second image or any subsequent second image. Any sub-result is obtained in the following manner:

[0022] Extracting abstract features of the first object from the currently acquired target image, wherein the abstract features include color features, texture features, shape features, and spatial relationship features, and,

[0023] Extracting the first pose feature of the end effector from the current pose data of the end effector corresponding to the acquisition time of the target image, and,

[0024] Extracting a second posture feature of the robotic arm from the current joint parameters of the robotic arm corresponding to the moment of acquiring the target image;

[0025] According to the abstract feature, the first posture feature and the second posture feature, the next posture data of the end effector and the next joint parameter of the robotic arm are predicted respectively to obtain any one of the sub-results.

[0026] The above method plans the operating posture of the robot based on the second image obtained by the second visual sensor deployed on the end effector, that is, based on the currently obtained target image, and the posture data of the end effector and the joint parameters of the robotic arm corresponding to the moment of obtaining the target image, end-to-end imitation learning is used to predict the next posture data and the next joint parameters, thereby improving the adaptability and operation accuracy of the robot's robotic arm and end effector in performing operating operations.

[0027] In a possible implementation manner, the determining, based on the position and the currently acquired second image, whether the first object is perceived by the second visual sensor includes:

[0028] determining a distance between the end effector and the first object based on the position and a real-time position of the end effector;

[0029] Based on the distance and the currently acquired second image, it is determined whether the first object is perceived by the second visual sensor.

[0030] The above method determines the distance between the position and the real-time position of the end effector, thereby ensuring the robot's operating space and avoiding the possibility of operating the wrong target due to the presence of objects similar to the first object in different operating spaces. Then, through the distance between the end effector and the first object and the second image currently acquired, jointly evaluate whether the first object is perceived by the second visual sensor, that is, evaluate whether the distance between the first object and the end effector meets the operating space requirements, and whether the second image acquired by the second visual sensor contains the first object, thereby improving the accuracy and precision of the robot's operating posture determined by the subsequent local motion planning based on the image acquired by the second visual sensor, thereby improving the success rate of the operation operation.

[0031] In a possible implementation, the real-time position of the end effector is obtained by:

[0032] Acquire a third image in real time by using the first visual sensor, and perform the following operations for each third image:

[0033] Performing target recognition for the end effector on any acquired third image;

[0034] If the end effector is identified, determining the real-time position of the end effector based on the identified position of the end effector in any one of the third images;

[0035] If the end effector is not identified, the real-time position of the end effector is determined based on the structural data of the robot and the joint parameters corresponding to the acquisition time of any one of the third images.

[0036] The above method, based on the different situations that may exist in the image obtained by the first visual sensor, realizes real-time calculation of the real-time position of the end sensor by combining forward kinematics (i.e., based on the robot's structural data) and visual object perception, so as to prepare for subsequent local motion planning.

[0037] In a possible implementation manner, determining whether the first object is perceived by the second visual sensor based on the distance and the currently acquired second image includes:

[0038] generating a heat map of the first object based on the currently acquired second image, the heat map including a probability that each pixel in the currently acquired second image belongs to the first object;

[0039] Based on the probability of each pixel in the heat map, determining a target region matching the first object from the heat map, and determining the confidence of the target region according to the probability of each pixel in the target region;

[0040] When it is determined that the confidence is greater than the confidence threshold, determining whether a weighted sum of the distance, the first weight value, the confidence, and the second weight value is greater than a replacement score threshold;

[0041] If yes, determining that the first object is perceived by the second visual sensor;

[0042] If not, it is determined that the first object is not sensed by the second visual sensor.

[0043] In the above method, by generating a heat map of the first object, the perception reliability of the first object is measured based on the heat map to ensure that the first object is indeed contained in the second image obtained by the second visual sensor, so as to make the subsequent local motion planning accurate. By setting the confidence threshold, it is further ensured that the second visual sensor can accurately perceive the first object. When it is determined that the confidence is greater than the confidence threshold, the replacement score is further determined by the replacement score formula, that is, the aforementioned weighted sum. Based on the relationship between the weighted sum and the replacement score threshold, it is determined whether the first object is perceived by the second visual sensor. After determining that the first object is perceived by the second visual sensor, the motion planning of the execution device for the operation of the first object is performed, which improves the accuracy of the robot's operating posture and the success rate of the operation execution. At the same time, it also simplifies the amount of calculation and improves the efficiency of the robot's data processing.

[0044] In a second aspect, an embodiment of the present application provides a motion planning device for a robot, comprising:

[0045] a first planning module, configured to perform motion planning for the robot to perform an operation on the first object based on a position of the first object and a current position of the robot, obtain a first planning result, and move toward the first object based on the first planning result, wherein the position is determined based on a first image acquired by a first visual sensor deployed on a head of the robot;

[0046] A recognition module, configured to continuously acquire a second image by a second visual sensor deployed on the end effector of the robot during movement, and perform the following operations for each second image: determine whether the first object is perceived by the second visual sensor based on the position and the second image currently acquired;

[0047] A second planning module is used to perform local motion planning on the robot's execution device for operating the first object based on the currently acquired second image and the subsequent second image acquired by the second visual sensor when the recognition module determines that the first object is perceived by the second visual sensor, obtain a second planning result, and control the execution device to perform actions based on the second planning result, wherein the execution device includes the end effector and an associated robotic arm.

[0048] In a third aspect, an embodiment of the present application provides a robot, comprising:

[0049] A first visual sensor, deployed on the head of the robot, for acquiring an image of the environment in which the robot is located;

[0050] a second visual sensor, deployed on the end effector of the robot, for acquiring an image around the end effector;

[0051] Memory, used to store computer programs or instructions;

[0052] A processor is used to execute the computer program or instructions in the memory so that any method in the first aspect described above is executed.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods described in the first aspect above.

[0054] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, enables the computer to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a flow chart of a motion planning method for a robot in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a process for determining a position of a first object in an embodiment of the present application;

[0057] Figure 3 This is a schematic diagram of an optional application scenario in an embodiment of the present application;

[0058] Figure 4 This is a schematic diagram of another optional application scenario in the embodiment of the present application;

[0059] Figure 5 A schematic diagram of a process for performing whole machine motion planning in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of a process for determining a standing position of a robot in an embodiment of the present application;

[0061] Figure 7 This is a schematic diagram of an optional position for deploying a second visual sensor in an embodiment of the present application;

[0062] Figure 8 A schematic diagram of a process of determining whether a first object is sensed by a second visual sensor in an embodiment of the present application;

[0063] Fig. 9 A schematic diagram of a process for determining a real-time position of an end effector in an embodiment of the present application;

[0064] Fig.10 This is a schematic diagram of a specific process of determining whether a first object is sensed by a second visual sensor in an embodiment of the present application;

[0065] Fig.11 It is a schematic diagram of the comparison effect of images acquired by the first visual sensor and the second visual sensor during the movement process of the robot in the embodiment of the present application;

[0066] Fig. 12A A schematic diagram of a process of performing local motion planning in an embodiment of the present application;

[0067] Fig. 12BThis is a schematic diagram of an image acquired by a second visual sensor in an embodiment of the present application;

[0068] Fig.13 A schematic diagram of the logical architecture of a motion planning device for a robot in an embodiment of the present application;

[0069] Fig.14 Schematic diagram of the physical structure of the robot in the embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0072] The following is a brief introduction to the design concept of the embodiments of the present application.

[0073] The present application relates to the field of robotics technology, and primarily to a robot motion planning method, device, robot, and storage medium.

[0074] Usually, robots obtain information about the surrounding environment through deployed sensors, and convert the information into environmental data that can be understood by computers. They build terrain maps based on the environmental data, and then use computer vision or other recognition technologies to identify objects in the environment and determine the object's posture, that is, position and posture. Based on the object's posture, they perform path and / or motion planning to enable the robot to grasp, assemble and other operations on objects.

[0075] Since the robot perceives the environment through deployed sensors, the accuracy of the robot's perception of the environment is strongly related to the accuracy of the sensors, and the accuracy of the sensors is related to the distance from the object, whether the object is blocked, etc. For example, in some scenes, the position of the object cannot be accurately estimated because the object is far away; in other scenes, the position of the object cannot be obtained because the object is blocked. In addition, the robot's hand-eye calibration error, the mechanical error of the robot arm control module, etc., will lead to the inability to accurately estimate the robot's operating posture, which directly affects the success rate of the operation execution.

[0076] In view of this, in order to solve the problem of low success rate of operation execution in the related art, an embodiment of the present application provides a robot motion planning method. In the embodiment of the present application, based on the position of the first object and the current position of the robot, the robot performs motion planning for the operation operation on the first object, obtains a first planning result, and moves toward the first object based on the first planning result, wherein the position is determined based on a first image acquired by a first visual sensor deployed on the head of the robot; during the movement, a second visual sensor deployed on the end effector of the robot continuously acquires a second image, and the following operations are performed for each second image: based on the position and the currently acquired second image, determine whether the first object is perceived by the second visual sensor; if so, based on the currently acquired second image and the subsequent second image acquired by the second visual sensor, perform local motion planning for the operation operation of the robot's execution device on the first object, obtain a second planning result, and control the execution device to act based on the second planning result to complete the operation operation on the first object, wherein the execution device includes an end effector and an associated robotic arm.

[0077] In this way, the position of the first object is determined by the first image obtained by the first visual sensor deployed on the head of the robot, and then, based on the position and the current position of the robot, motion planning is performed for the robot to perform an operation on the first object, thereby realizing the overall motion planning of the robot, thereby moving toward the first object based on the first planning result obtained. During the movement, the second image is continuously obtained by the second visual sensor deployed on the end effector of the robot, and based on the position and the currently obtained second image, it is determined whether the first object is perceived by the second visual sensor. If so, it means that the first object has indeed been perceived by the second visual sensor. Based on the currently obtained second image and the subsequent second image obtained by the second visual sensor, local motion planning is performed for the operation of the execution device on the first object, thereby realizing detailed motion planning of the robotic arm and the end effector, improving the accuracy of the operation posture, and also improving the efficiency and accuracy of the operation, thereby improving the success rate of the operation execution.

[0078] A robot motion planning method provided in an embodiment of the present application is applicable to robots, such as movable intelligent devices, multi-legged robots, mobile robots, humanoid / humanoid robots, bipedal robots, bionic humans, etc.

[0079] The preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0080] See also Figure 1 As shown, the embodiment of the present application provides a motion planning method for a robot, and the specific process of the method may include but is not limited to the following:

[0081] Step 100: Based on the position of the first object and the current position of the robot, motion planning is performed for the robot to perform an operation on the first object, a first planning result is obtained, and the robot moves toward the first object based on the first planning result, wherein the position is determined based on a first image obtained by a first visual sensor deployed on the head of the robot.

[0082] In some embodiments of this application, refer to Figure 2 As shown, the position of the first object can be determined specifically in the following manner:

[0083] Step 200: Based on the received task instruction, the first visual sensor acquires a first image.

[0084] The task instruction includes key information of the first object, such as keywords, features, etc.

[0085] In some embodiments of the present application, a user can send a task instruction to the robot to obtain a certain object, so that after receiving the task instruction, the robot calls the first visual sensor deployed on the head of the robot to capture images within the field of view based on the task instruction, thereby obtaining the aforementioned first image.

[0086] For example, see Figure 3 As shown, a home scene includes a robot 10, and a user interacts with the robot 10. The user issues a task instruction to the robot 10 to take book A located on the first layer below the bookshelf. Based on the task instruction, the robot 10 rotates its body and obtains a first image through a first visual sensor deployed on the head of the robot.

[0087] In other embodiments of the present application, the robot can receive a task instruction sent by a control end (such as a background server, other control devices, etc.). Based on the task instruction, the robot calls a first visual sensor deployed on the head of the robot to capture images within the field of view, thereby obtaining the aforementioned first image.

[0088] For example, see Figure 4 As shown, in the intelligent manufacturing scenario, the robot 10 is used to install screws in the drum of a washing machine. After receiving the task instruction of installing screws sent by the control terminal, the robot 10 acquires a first image through a first visual sensor deployed on the head of the robot based on the task instruction.

[0089] Step 210: Perform target recognition on each object in the first image to determine whether the first object exists in the first image; if so, execute step 220; if not, and a second object including the first object is recognized to exist in the first image, execute step 230.

[0090] In an embodiment of the present application, after acquiring the first image, step 210 is executed. When executing step 210, computer vision or other recognition algorithms can be called to perform target recognition on each object in the first image, so as to determine whether the first object exists in the first image.

[0091] In some embodiments of the present application, after executing step 210 , if it is determined that the first object exists in the first image, step 220 is executed.

[0092] In some other embodiments of the present application, after executing step 210, if it is determined that the first object does not exist in the first image, then it is further determined whether the second object exists in the first image. For example, the robot 10 is used to install screws in the drum of a washing machine, the second object is the washing machine, and the first object is the screw hole in the washing machine, that is, the first object is located inside the second object. If it is determined that the second object exists in the first image, step 230 is executed.

[0093] Step 220: Determine the position of the first object based on the image position of the first object in the first image.

[0094] In some embodiments of the present application, when executing step 210, target recognition is performed on each object in the first image. When the first object is recognized in the first image, the image position of the first object in the first image can be determined. If a traditional target recognition / detection algorithm is used, the bounding box information of the object in the image can be output after the object is recognized / detected, that is, the image position of the first object in the first image is obtained, and then step 220 is executed.

[0095] As a specific embodiment, when executing step 220, the position of the first object in the camera coordinate system can be obtained based on the image position of the first object in the first image, and the mapping relationship between the position in the pixel coordinate system and the position in the camera coordinate system, and the position can be mapped from the camera coordinate system to the world coordinate system, such as based on the mapping relationship between the position in the camera coordinate system and the position in the world coordinate system, to obtain the position of the first object in the world coordinate system, that is, the position of the first object.

[0096] Still Figure 3 As shown, assuming that the application scenario is still a home scenario, the robot 10 included in the home scene performs target recognition on each object in the first image. Assuming that book A is recognized to exist in the first image, then step 220 is executed to map the image position of book A in the first image to the position in the camera coordinate system, and then map the position in the camera coordinate system to the world coordinate system, thereby obtaining the position of book A in the world coordinate system, that is, the position of book A.

[0097] Step 230: Estimate the position of the first object based on the image position of the second object in the first image and the three-dimensional structure data of the second object.

[0098] In other embodiments of the present application, after executing step 210, if it is determined that the first object does not exist in the first image, it is further determined whether there is a second object in the first image. If so, the position of the first object can be estimated through prior knowledge, that is, step 230 is executed, wherein the prior knowledge may include three-dimensional structural data of the second object, etc.

[0099] In some embodiments of the present application, when executing step 230, the three-dimensional structural data of the second object can be superimposed and fused with the object contour of the second object in the first image based on the image position of the second object in the first image to obtain fused data, and then the position of the first object can be estimated based on the fused data.

[0100] Still reading Figure 4 As shown, in the intelligent manufacturing scenario, the robot 10 is used to install screws in the drum of a washing machine. The washing machine is the second object. Since the installation position of the screws (i.e., the screw holes) are located inside the drum of the washing machine, Figure 4 The screw hole (i.e., the first object) located near the observation window glass inside the washing machine is shown. Usually, the robot cannot directly identify the screw hole from the acquired first image. In the related art, the corresponding scene is usually not completed by the robot. However, in some embodiments of the present application, the problem of how to determine the position of the first object during the implementation process can be solved by pre-given three-dimensional structural data of the washing machine, that is, the position of any screw hole inside the washing machine can be estimated based on the three-dimensional structural data.

[0101] In a specific implementation, when it is determined that there is a washing machine in the first image, the image position of the washing machine in the first image is determined, such as the image positions of the four corners of the washing machine in the first image. Then, the three-dimensional structure diagram of the washing machine is superimposed and fused with the washing machine in the first image. In some embodiments of the present application, a corner point registration method can be used, that is, the four corner points on the front of the three-dimensional structure diagram of the washing machine are aligned with the four corner points of the washing machine in the first image, such as Figure 4 Dotted line shown.

[0102] Then, by obtaining the coordinates (image position) of the upper left corner of the front of the washing machine in the first image in the world coordinate system, for example, mapping the image position of the upper left corner of the washing machine in the first image to the world coordinate system, the position of the upper left corner of the washing machine in the world coordinate system is obtained; and then by superimposing the fused washing machine structure diagram, the coordinates (positions) of the screw holes in the washing machine in the world coordinate system are further calculated, so as to obtain the position of any screw hole in the washing machine, that is, the following can be obtained. Figure 4 The position of the screw hole (ie, the first object) located near the observation window glass inside the washing machine is shown.

[0103] The above method provides a direction for the intelligent transformation of similar production lines in intelligent manufacturing scenarios, which has important guiding significance. Similar application scenarios include cleaning of sanitary dead corners in home scenarios. To simply summarize the commonality of the above application scenarios: the robot cannot directly perceive the first object through the first visual sensor deployed on the head, that is, it cannot see the first object, but can estimate the approximate position of the first object through prior knowledge.

[0104] After obtaining the position of the first object, refer to Figure 5 As shown, when executing step 100, some or all of the following steps may be specifically performed to obtain the aforementioned first planning result:

[0105] Step 1001: Determine the standing position of the robot based on the position and length of the robot's mechanical arm and environmental information of the robot's environment.

[0106] In some embodiments of the present application, when executing step 1001, the standing position of the robot is determined based on the position and the length of the robot's mechanical arm in combination with the topographic map of the robot's environment, that is, the environmental information. The standing position needs to reserve activity space for the robot's execution device, that is, the operating space, so that the robot can contact the first object through the end effector, and can also ensure that the robot will not fall down or other problems during the operation.

[0107] For example, see Figure 6As shown, assuming that the position of the first object is represented by point O in the world coordinate system, then a sphere is created with point O of the first object in the world coordinate system as the center and the arm length R of the robot as the radius. The cross-section of the sphere and the plane α where the ground is located (assuming a circle with O1 as the center and r as the radius) is the area range of the robot's standing point. Combined with the environmental information of the robot's environment perceived by the first visual sensor and / or other sensors, such as the terrain conditions of the area, whether there are obstacles, whether it is a plane, etc., a foothold closest to O1 can be selected, and the selected foothold can be used as the standing point of the robot. The standing point can ensure that the end effector of the robot can contact the first object and perform any operation on the first object.

[0108] Step 1002: Perform path planning based on the standing point and the current position to obtain a moving path.

[0109] In some embodiments of the present application, when executing step 1002, path planning is performed for the robot based on the standing point and the current position of the robot, so that the robot can move to the vicinity of the first object.

[0110] In a specific embodiment, when executing step 1002, a path planning algorithm may be called according to the standing point and the current position of the robot to generate an optimal path, which is the aforementioned moving path. The path planning algorithm may include classic algorithms such as A* algorithm, Dijkstra algorithm, and Rapidly exploring Random Tree (RRT).

[0111] Optionally, in some embodiments of the present application, the path planning algorithm may select the RRT algorithm, so as to improve the performance and efficiency of the present application by virtue of the algorithm having good performance and efficiency in high-dimensional space. Specifically, the RRT algorithm randomly generates nodes in the search space and gradually expands the tree structure, and finally finds a path from the starting point (i.e., the current position of the robot) to the standing point, i.e., the moving path.

[0112] Step 1003: Based on the moving path and position, motion planning is performed on the execution device to obtain the aforementioned first planning result.

[0113] In some embodiments of the present application, after obtaining the moving path, step 1003 is executed, and a motion planning algorithm is used to perform motion planning for the robot to move to the vicinity of the first object (i.e., the standing point), and for the robot's execution device to perform operation planning on the first object. Specifically, motion planning is performed on the posture of the robot during movement based on the moving path, and the posture of the end effector when the robot moves to the standing point is obtained. Then, based on the posture and the position of the first object, a motion planning algorithm is used to plan the action (i.e., the operation posture) of the execution device for the operation of the first object, thereby obtaining the motion planning result of the robot reaching the standing point from the current position, and performing the operation on the first object at the standing point, i.e., the aforementioned first planning result. Among them, the motion planning algorithm may include Inverse Kinematics (IK), Probabilistic Roadmap Method (PRM), and Rapidly Expanding Random Tree (RRT), etc.

[0114] In some embodiments of the present application, the execution device may include a robotic arm and an end effector, and motion planning for the execution device refers to generating a motion trajectory from a current state to a target state.

[0115] Optionally, in some embodiments of the present application, during implementation, motion planning for the execution device can be performed based on a motion planning algorithm that combines inverse kinematics and an obstacle avoidance algorithm. On the one hand, inverse kinematics generates a sequence of joint angles by solving the position and posture of the end effector of the robotic arm, thereby providing a basic motion trajectory for the robotic arm; on the other hand, the obstacle avoidance algorithm adjusts and optimizes this trajectory based on real-time environmental information.

[0116] In some embodiments of the present application, after executing step 1003 to obtain the first planning result, the robot moves toward the first object based on the first planning result. Specifically, the robot can call the control module to execute the first planning result, that is, call the control module to convert the aforementioned moving path into an actual control command, drive the robot and the mechanical arm to perform corresponding actions, so as to realize the robot moving toward the first object. The control module may generally include but is not limited to submodules such as position control, speed control, and force control.

[0117] Optionally, a control strategy based on model predictive control (MPC) may be used in this application. MPC has good robustness and real-time performance and can be applied to complex robot control tasks.

[0118] Still Figure 3As shown, assuming that the application scenario is still a home scenario, the robot 10 included in the home scenario moves based on the first planning result. The book A to be picked up is located on the first layer below the bookshelf. When the robot moves to a certain distance from the bookshelf, due to the installation position of the first visual sensor on the head of the robot, the robot may not be able to obtain the image of book A through the first visual sensor. At this time, the relevant technology can only estimate the operation posture of the robot based on the structural data and joint parameters of the robot, as well as the pre-estimated position of the first object. The accumulated deviation based on the robot's execution of the control instruction, the estimated error of the position, etc., directly affect the success rate of the operation execution.

[0119] In some embodiments of the present application, a second visual sensor is deployed on the end effector of the robot. In the above scenario, the second visual sensor deployed on the end effector of the robot can obtain an image around the end effector, thereby determining the exact position of book A based on the image obtained by the second visual sensor, and performing local motion planning for the robot's execution device to pick up book A based on the image obtained by the second visual sensor, thereby achieving accurate picking up of book A. The specific process in the specific implementation is detailed in the description of subsequent steps 110 to 120.

[0120] Similar application scenarios include robots taking workpieces from lower shelves in smart manufacturing scenarios. To simply summarize the commonality of the above application scenarios, the robot can perceive the first object through the first visual sensor deployed on its head, but there may be visual obstruction after moving near the first object.

[0121] Step 110: During the movement, the second image is continuously acquired by the second visual sensor deployed on the end effector of the robot, and the following operations are performed for each second image: based on the position and the currently acquired second image, determine whether the first object is perceived by the second visual sensor; if so, execute step 120; if not, continue to execute step 110 to determine whether the first object is perceived by the second visual sensor based on the position and the next acquired second image.

[0122] In some embodiments of the present application, the second visual sensor may be deployed on the hand or end effector of the robot arm so as to obtain more detailed object perception information, such as Figure 7 As shown, Figure 7 An optional location for deploying a second visual sensor is shown in the figure, but it should be noted that this application does not specifically limit the specific installation location.

[0123] In some embodiments of this application, refer to Figure 8 As shown, when executing step 110, some or all of the following steps are specifically performed:

[0124] Step 1101 : Determine the distance between the end effector and the first object based on the position and the real-time position of the end effector.

[0125] In some embodiments of the present application, before executing step 1101, refer to Fig. 9 As shown, the real-time position of the end effector can be obtained by executing some or all of the following steps:

[0126] Step 900: Acquire a third image in real time through the first visual sensor, and perform the following operations for each third image:

[0127] Step 9001: performing target recognition for the end effector on any acquired third image;

[0128] Step 9002: if the end effector is identified, determining the real-time position of the end effector based on the identified position of the end effector in the third image;

[0129] Step 9003: If the end effector is not identified, the real-time position of the end effector is determined based on the structural data of the robot and the joint parameters corresponding to the time when the third image is acquired.

[0130] In some embodiments of the present application, the position of the first object in the world coordinate system has been determined by the aforementioned steps. Assuming that the coordinates are used, the position of the first object can be recorded as (x1, y1, z1). Then, the real-time position of the end effector of the robot arm can be determined by performing target recognition on the third image obtained by the first visual sensor, that is, executing step 9001. Specifically, based on whether the end effector is recognized from the third image, it includes but is not limited to the following two situations:

[0131] Case 1, corresponding to step 9002: If the end effector is within the field of view of the first visual sensor and is not blocked, then the end effector can be identified from the third image by performing target recognition on the end effector in the third image, thereby obtaining the real-time position of the end effector, that is, the real-time position mapped to the world coordinate system, which is recorded as (x2, y2, z2).

[0132] Case 2, corresponding to step 9003: If the end effector is not within the field of view of the first visual sensor, then forward kinematics calculation can be performed using the known robot structural data and the joint parameters at the current moment to obtain the real-time position of the end effector, recorded as (x2, y2, z2).

[0133] Through the above method, the real-time position of the end sensor can be calculated by combining forward kinematics (such as step 9003) and visual object perception (such as step 9002).

[0134] After the real-time position of the end sensor is obtained, the distance between the end sensor of the robot arm and the first object can be obtained based on the position of the first object and the real-time position of the end sensor, which is recorded as d.

[0135] In some embodiments of the present application, the distance d can be expressed by the following formula:

[0136]

[0137] Step 1102: Based on the distance and the currently acquired second image, determine whether the first object is perceived by the second visual sensor.

[0138] In some embodiments of this application, refer to Fig.10 As shown, when executing step 1102, some or all of the following steps may be specifically executed:

[0139] Step 1000: Generate a heat map of the first object based on the currently acquired second image, wherein the heat map includes the probability that each pixel in the currently acquired second image belongs to the first object.

[0140] In some embodiments of the present application, when executing step 1000, the currently acquired second image can be input into a convolutional neural network, and the second image can be processed by the convolutional neural network to generate a heat map of the first object, wherein the heat map can classify each pixel and output the probability that the pixel belongs to a certain object by classifying the image at the pixel level, that is, the heat map can represent the probability distribution of objects in the image, and the color of each pixel or area in the heat map represents the confidence that the position belongs to the first object, with higher confidence corresponding to brighter colors and lower confidence corresponding to darker colors; in this way, the position of the object in the image can be intuitively displayed through the heat map.

[0141] Step 1010: Based on the probability of each pixel in the heat map, determine a target area matching the first object from the heat map, and determine the confidence of the target area according to the probability of each pixel in the target area.

[0142] In some embodiments of the present application, when executing step 1010, based on the probability of each pixel in the heat map, the similarity between two pixels can be calculated using feature comparison methods such as Euclidean distance to determine whether the two pixels belong to the same object, thereby classifying the pixels in the second image, and then determining a target area that matches the first object from the various areas obtained after classification, and based on statistical values ​​of the probabilities of each pixel in the target area, such as the mean, determining the confidence of the target area, denoted as C, which represents the degree of certainty that the target area in the heat map is the first object, and is used to measure the reliability of the perception results, wherein C can be a value in the range of 0-1.

[0143] In some embodiments of the present application, a confidence threshold is pre-set to ensure that the second visual sensor can accurately perceive the first object. Then, after executing step 1010 to obtain the confidence of the target area, it is further determined whether the confidence is greater than the confidence threshold. If so, step 1020 is executed. If not, the processing flow for the second image is terminated, and the next second image is waited for and processed. The confidence threshold can be determined based on the specific sensor type, accuracy, etc., combined with specific experiments or experience.

[0144] Step 1020: When it is determined that the confidence is greater than the confidence threshold, determine whether the weighted sum of the distance, the first weight value, the confidence and the second weight value is greater than the replacement score threshold; if so, execute step 1030; if not, execute step 1040.

[0145] In some embodiments of the present application, after executing step 1010 to obtain the confidence of the target area, the following formula can be used to evaluate whether the first object is perceived by the second visual sensor, which is used to evaluate whether the current situation is suitable for local motion planning, that is, guiding the execution device (that is, the robot arm and the end effector) to perform actions based on the image obtained by the second visual sensor:

[0146]

[0147] Among them, s is the replacement score, which is used to measure the replacement feasibility of local motion planning; α and β are weight coefficients, respectively, denoted as the first weight value and the second weight value. In some embodiments of the present application, the value of the weight coefficient can be adjusted according to actual needs; C is the confidence level, and optionally, the confidence threshold can be 0.8; d is the distance between the aforementioned end effector and the first object.

[0148] Based on the above formula, in some embodiments of the present application, when it is determined that the confidence is greater than the confidence threshold, step 1020 is executed. When executing step 1020, the weighted sum (i.e., s) of the distance, the first weight value, the confidence, and the second weight value may be determined, and it may be determined whether the weighted sum (s) is greater than the replacement score threshold. If so, step 1030 is executed, and if not, step 1040 is executed. The replacement score threshold may be an empirical value or may be specifically determined through experimental data.

[0149] Step 1030: Determine whether the first object is perceived by the second visual sensor.

[0150] In some embodiments of the present application, if executing step 1020 determines that the weighted value is greater than the replacement score threshold, it means that the second visual sensor has indeed sensed the first object, that is, it is determined that the first object is sensed through the second visual sensor, then local motion planning can be performed on the robot's execution device for the operation of the first object.

[0151] In a specific implementation, after determining that the weighted value is greater than the replacement score threshold, the execution device is stopped from being controlled to perform actions based on the first planning result, and local motion planning is used to perform local motion planning on the execution device composed of the robot arm and the end effector, that is, the motion trajectory of the robot arm and the end effector for the operation of the first object is calculated, and the execution device is controlled to perform actions based on the local motion planning result.

[0152] Step 1040: Determine that the first object is not sensed by the second visual sensor.

[0153] In some embodiments of the present application, if step 1020 is executed to determine that the weighted value is less than or equal to the replacement score threshold, it means that the reliability of the second visual sensor in sensing the first object is poor, and step 1040 is executed to determine that the first object is not sensed by the second visual sensor. In this case, local motion planning is not performed on the robot's execution device. At this time, the execution device continues to be controlled to perform actions based on the first planning result.

[0154] Still Figure 4 In the application scenario shown, the robot is used to install screws in the drum of a washing machine. As the robot moves, suppose the robot moves near the washing machine, such as Fig.11As shown, the third image acquired by the first visual sensor is image 1. At this time, it is assumed that the robot's actuator can contact the drum of the washing machine. Then, the second image acquired by the second visual sensor is image 1'. In image 1, due to the installation position of the first visual sensor, its field of view only includes the upper half of the washing machine, and image 1' includes the internal image of the washing machine drum. At this time, an evaluation can be performed based on the position of the first object in step 110 and image 1 to determine whether the first object can be perceived by the second visual sensor, that is, Figure 4 In the specific implementation, the specific implementation can be referred to the aforementioned steps 1000 to 1040 and the description, which will not be repeated here.

[0155] As the robot moves further, it gets closer to the washing machine, and the actuator can gradually extend into the drum of the washing machine. Assume that the third image acquired by the first visual sensor is image n. Compared with image 1, the robot is closer to the washing machine. Due to the change in the field of view, image n only contains a small part of the upper part of the washing machine. At this time, the actuator of the robot can extend into the drum of the washing machine, as shown in FIG. Fig.11 The image n' obtained by the second visual sensor is shown, and the first object, i.e., the screw hole, can be vaguely seen in the image n'. Based on the position of the first object and the image n' in step 110, an evaluation is performed to determine whether the first object can be perceived by the second visual sensor, i.e. Figure 4 In the specific implementation, the specific implementation can be referred to the aforementioned steps 1000 to 1040 and the description, which will not be repeated here.

[0156] Step 120: Based on the second image currently acquired and the subsequent second image acquired by the second visual sensor, local motion planning is performed on the robot's execution device for the operation of the first object to obtain a second planning result, and the execution device is controlled to perform action based on the second planning result, wherein the execution device includes the aforementioned end effector and an associated robotic arm.

[0157] In some embodiments of the present application, local motion planning can use an end-to-end imitation learning algorithm, which is mainly performed by learning and imitating the behavior of humans or other experts. In a specific implementation, when executing step 120, end-to-end imitation learning is performed based on the image acquired by the second visual sensor, which can improve the adaptability and operation accuracy of the robot in performing tasks such as upper limb operation in a complex environment, wherein the image acquired by the second visual sensor may include the aforementioned currently acquired second image and each subsequent second image in the present application.

[0158] Optionally, in some embodiments of the present application, the aforementioned imitation learning algorithm can be developed by performing secondary training using an action chunking algorithm (ActionChunking with Transformers, ACT), a diffusion policy, etc. as a basic model.

[0159] Specifically, the end-to-end imitation learning model used in the implementation process can be trained in the following way:

[0160] 1. Data collection and preparation

[0161] A series of demonstration motion trajectories of the robot arm and end effector, as well as the RGBD data of the second visual sensor, are generated by experts or through some automated means (such as teleoperation, preset trajectories, etc.), which are subsequently recorded as demonstration data. These trajectories can cover various task scenarios and actions to provide rich learning samples.

[0162] 2. Imitation learning model construction

[0163] Choose a suitable deep learning model, such as Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), or Transformer, to learn the mapping from state space to action space. These models can process sequential data and capture temporal dependencies.

[0164] 3. Model training and optimization

[0165] Extract meaningful features from the demonstration data, such as extracting features from images acquired through the second visual sensor through a pre-trained deep learning model. The extracted abstract features of the target include but are not limited to color features, texture features, shape features, spatial relationship features, edge features, etc., the joint angles and speeds of the robotic arm (such as the subsequent first pose features), the pose of the end effector (such as the subsequent second pose features), etc.

[0166] These features are used as the input of the model, and the next state or action in the demonstration motion trajectory is used as the label; the extracted features and generated labels are used to train the aforementioned selected deep learning model through supervised learning. During the training process, the model parameters can be adjusted through training methods such as backpropagation algorithms and optimizers (such as gradient descent) to minimize the prediction error. After the convergence conditions of the model meet the expected requirements, the performance of the model is evaluated through an independent validation set. For example, various evaluation indicators such as mean-square error (MSE), accuracy, etc. can be used to evaluate the performance of the model, so that the model can be iterated and optimized according to the evaluation results until a model with better performance is obtained, and the model with better performance can be used for local motion planning in subsequent implementations.

[0167] In some embodiments of the present application, the aforementioned second planning result includes multiple sub-results, each sub-result corresponds to a target image, and the target image is the aforementioned currently acquired second image or any subsequent second image. Fig. 12A As shown, when executing step 120, any sub-result of the multiple sub-results included in the aforementioned second planning result can be obtained in the following manner:

[0168] Step 1201: extracting abstract features of the first object from the currently acquired target image, and extracting the first pose features of the end effector from the pose data of the end effector corresponding to the moment of acquiring the target image, and extracting the second pose features of the robotic arm from the joint parameters of the robotic arm corresponding to the moment of acquiring the target image, wherein the abstract features may include but are not limited to color features, texture features, shape features, and spatial relationship features.

[0169] In some embodiments of the present application, when executing step 1201, a first feature extraction algorithm can be called to perform feature extraction on the first object in the currently acquired target image, so as to obtain the abstract features of the first object, and a second feature extraction algorithm can be called to perform feature extraction on the pose data of the end effector corresponding to the moment of acquisition of the target image (i.e., the pose data of the end effector at the moment of acquisition of the target image), so as to obtain the first pose data of the end effector, and a third feature extraction algorithm can be called to perform feature extraction on the joint parameters of the robotic arm corresponding to the moment of acquisition of the target image (i.e., the joint parameters of the robotic arm at the moment of acquisition of the target image), so as to obtain the second pose features of the robotic arm.

[0170] It should be noted that the first feature extraction algorithm, the second feature extraction algorithm, and the third feature extraction algorithm may be the same feature extraction algorithm or may be independent feature extraction algorithms, and this application does not make any specific limitation.

[0171] Step 1202: Based on the abstract feature, the first pose feature and the second pose feature, the next pose data of the end effector and the next joint parameter of the robot arm are predicted respectively to obtain the aforementioned sub-results.

[0172] In some embodiments of the present application, when executing step 1202, the abstract feature, the first pose feature, and the second pose feature can be input into the aforementioned end-to-end imitation learning model (i.e., the aforementioned trained model with better performance), and the next pose data of the end effector and the next joint parameter of the robotic arm are predicted through the end-to-end imitation learning model to obtain the sub-result corresponding to the target image.

[0173] In some other embodiments of the present application, when executing step 120, motion planning can be performed on the robotic arm based on the currently acquired second image and the subsequent second image acquired by the second visual sensor, and then, based on the motion planning result of the robotic arm, motion planning can be performed on the operation of the end effector on the first object to obtain a second planning result.

[0174] Still Figure 4 In the application scenario shown, the robot is used to install screws in the drum of a washing machine. Assume that, through the aforementioned step 110, it is determined that the first object is sensed by the second visual sensor, and the target image obtained by the second visual sensor is as follows: Fig. 12B As shown, the target image may be the aforementioned currently acquired second image or any subsequent second image. Compared with the screw hole (i.e., the first object) in the first image acquired by the first visual sensor, the screw hole in the target image acquired by the second visual sensor is located at the center of the image. Then, step 120 is subsequently performed to perform local motion planning for the robot's execution device (i.e., the robot arm and the end effector) for the operation of the screw hole (i.e., the first object) based on the images continuously acquired by the second visual sensor, such as the second image / subsequent second image, so as to improve the success rate of the operation execution.

[0175] Specifically, local motion planning can be implemented by executing steps 1201 to 1202 to obtain each sub-result, thereby controlling the execution device to perform actions based on each sub-result.

[0176] In some embodiments of the present application, when executing step 1201, a first feature extraction algorithm can be used to extract features of the first object in the target image to obtain abstract features of the first object, and a second feature extraction algorithm can be used to extract features of the current posture data of the end effector to obtain first posture data of the end effector, and a third feature extraction algorithm can be used to extract features of the current joint parameters of the robotic arm to obtain second posture features of the robotic arm. Then, when executing step 1202, the abstract features, first posture features, and second posture features can be input into the aforementioned end-to-end imitation learning model (i.e., the aforementioned model with better performance obtained through training), and the next posture data of the end effector and the next joint parameters of the robotic arm can be predicted through the end-to-end imitation learning model to obtain a sub-result corresponding to the target image, thereby controlling the execution device to perform actions according to the posture data and joint parameters in the sub-result.

[0177] In this way, the robot motion planning method in this application is adopted, and the local motion planning of the robot arm and the end effector of the "last mile" is realized through the second visual sensor deployed on the end effector. Specifically, the whole machine motion of the robot is planned through the first visual sensor on the head of the robot, so as to realize the movement of the robot and the rough planning of the robot arm; in the process of moving according to the rough planning, it is evaluated in real time whether the current situation meets the conditions for switching to the second image obtained by the second visual sensor for local motion planning (i.e., fine planning), and when it is determined that it meets the local motion planning, the second image (subsequent second image) continuously obtained by the second visual sensor deployed on the end effector can accurately identify the abstract features of the first object, and then based on the abstract features of the first object, the local motion planning of the robot arm and the end effector is carried out, so as to realize the detailed action planning of the robot arm and the end effector, and improve the accuracy of the robot's operating posture. In summary, in this application, by combining the whole machine motion planning and the local motion planning, the operation operation can be realized efficiently and accurately, and the success rate of the operation operation execution can be improved.

[0178] Based on the same inventive concept, refer to Fig.13 As shown, in an embodiment of the present application, a motion planning device for a robot is provided, comprising:

[0179] A first planning module 1310 is used to perform motion planning for the robot to perform an operation on the first object based on the position of the first object and the current position of the robot, obtain a first planning result, and move toward the first object based on the first planning result, wherein the position is determined based on a first image obtained by a first visual sensor deployed on the head of the robot;

[0180] The recognition module 1320 is used to continuously acquire a second image by a second visual sensor deployed on the end effector of the robot during the movement, and perform the following operations for each second image: determine whether the first object is perceived by the second visual sensor based on the position and the second image currently acquired;

[0181] The second planning module 1330 is used to perform local motion planning on the robot's execution device for operating the first object based on the currently acquired second image and the subsequent second image acquired by the second visual sensor when the recognition module determines that the first object is perceived by the second visual sensor, obtain a second planning result, and control the execution device to perform actions based on the second planning result, the execution device including the end effector and an associated robotic arm.

[0182] In a possible implementation, the first planning module 1310 is configured to determine the position of the first object in the following manner:

[0183] Based on the received task instruction, acquiring the first image by the first visual sensor;

[0184] Performing target recognition on each object in the first image to determine whether the first object exists in the first image;

[0185] If so, determining the position based on an image position of the first object in the first image;

[0186] If not, and a second object including the first object is identified to exist in the first image, the position is estimated based on the image position of the second object in the first image and the three-dimensional structure data of the second object.

[0187] In a possible implementation, the first planning module 1310 is specifically configured to:

[0188] Determine a standing position of the robot based on the position and the length of the robot's mechanical arm, and environmental information of the environment in which the robot is located;

[0189] Perform path planning based on the standing point and the current position to obtain a moving path;

[0190] Based on the moving path and the position, motion planning is performed on the execution device to obtain the first planning result.

[0191] In a possible implementation, the second planning result includes a plurality of sub-results, each sub-result corresponds to a target image, and the target image is the currently acquired second image or any subsequent second image. The second planning module 1330 is specifically configured to obtain any sub-result in the following manner:

[0192] Extracting abstract features of the first object from the currently acquired target image, wherein the abstract features include color features, texture features, shape features, and spatial relationship features, and,

[0193] Extracting the first pose feature of the end effector from the current pose data of the end effector corresponding to the acquisition time of the target image, and,

[0194] Extracting a second posture feature of the robotic arm from the current joint parameters of the robotic arm corresponding to the moment of acquisition of the target image;

[0195] According to the abstract feature, the first posture feature and the second posture feature, the next posture data of the end effector and the next joint parameter of the robotic arm are predicted respectively to obtain any one of the sub-results.

[0196] In a possible implementation, the identification module 1320 is specifically configured to:

[0197] determining a distance between the end effector and the first object based on the position and a real-time position of the end effector;

[0198] Based on the distance and the currently acquired second image, it is determined whether the first object is perceived by the second visual sensor.

[0199] In a possible implementation, the identification module 1320 is used to obtain the real-time position of the end effector in the following manner:

[0200] Acquire a third image in real time by using the first visual sensor, and perform the following operations for each third image:

[0201] Performing target recognition for the end effector on any acquired third image;

[0202] If the end effector is identified, determining the real-time position of the end effector based on the identified position of the end effector in any one of the third images;

[0203] If the end effector is not identified, the real-time position of the end effector is determined based on the structural data of the robot and the joint parameters corresponding to the acquisition time of any one of the third images.

[0204] In a possible implementation, the identification module 1320 is specifically configured to:

[0205] generating a heat map of the first object based on the currently acquired second image, the heat map including a probability that each pixel in the currently acquired second image belongs to the first object;

[0206] Based on the probability of each pixel in the heat map, determining a target region matching the first object from the heat map, and determining the confidence of the target region according to the probability of each pixel in the target region;

[0207] When it is determined that the confidence is greater than the confidence threshold, determining whether a weighted sum of the distance, the first weight value, the confidence, and the second weight value is greater than a replacement score threshold;

[0208] If yes, determining that the first object is perceived by the second visual sensor;

[0209] If not, it is determined that the first object is not sensed by the second visual sensor.

[0210] Based on the same inventive concept, refer to Fig.14 As shown, an embodiment of the present application provides a robot, comprising:

[0211] A first visual sensor 141 is deployed on the head of the robot and is used to obtain an image of the environment where the robot is located;

[0212] A second visual sensor 142 is deployed on the end effector of the robot and is used to obtain an image around the end effector;

[0213] Memory 143, for storing computer programs or instructions;

[0214] The processor 144 is used to execute the computer program or instructions in the memory 143 so that any one of the methods in the above embodiments is executed.

[0215] The processor 144 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 143, so that any method in the above embodiments is executed;

[0216] It should be noted that the specific connection medium between the memory 143 and the processor 144 is not limited in the embodiment of the present application. Fig.14In the embodiment, the memory 143 and the processor 144 are connected via a bus 145, and the connection between other components is only for illustrative purposes and is not intended to be limiting. The bus 145 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0217] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments. Since the principle of solving the problem by the above computer-readable storage medium is similar to that of a robot motion planning method, the implementation of the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0218] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0219] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A robot motion planning method, characterized in that: include: Based on the position of the first object and the current position of the robot, motion planning is performed for the robot to perform an operation on the first object to obtain a first planning result, and moving toward the first object based on the first planning result, wherein the position is determined based on a first image acquired by a first visual sensor deployed on the head of the robot; During the movement, a second image is continuously acquired by a second visual sensor deployed on the end effector of the robot, and the following operations are performed for each second image: a heat map of the first object is generated based on the currently acquired second image, a target area matching the first object is determined from the heat map based on the probability of each pixel in the heat map, and when it is determined that the confidence of the target area is greater than a confidence threshold based on the probability of each pixel in the target area, whether the first object is perceived by the second visual sensor is determined based on the distance between the position and the real-time position of the end effector, a first weight value, a weighted sum of the confidence and the second weight value, and a replacement score threshold, wherein the heat map includes the probability that each pixel in the currently acquired second image belongs to the first object; If so, based on the currently acquired second image and the subsequent second image acquired by the second visual sensor, local motion planning is performed on the robot's execution device for the operation of the first object to obtain a second planning result, and based on the second planning result, the execution device is controlled to perform action, and the execution device includes the end effector and an associated robotic arm.

2. The method according to claim 1, characterized in that The position of the first object is determined by: Based on the received task instruction, acquiring the first image by the first visual sensor; Performing target recognition on each object in the first image to determine whether the first object exists in the first image; If so, determining the position based on an image position of the first object in the first image; If not, and a second object including the first object is identified to exist in the first image, the position is estimated based on the image position of the second object in the first image and the three-dimensional structure data of the second object.

3. The method according to claim 1, characterized in that The step of performing motion planning for the robot to perform an operation on the first object based on the position of the first object and the current position of the robot to obtain a first planning result includes: Determine a standing position of the robot based on the position and the length of the robot's mechanical arm, and environmental information of the environment in which the robot is located; Perform path planning based on the standing point and the current position to obtain a moving path; Based on the moving path and the position, motion planning is performed on the execution device to obtain the first planning result.

4. The method according to claim 1, characterized in that The second planning result includes a plurality of sub-results, each sub-result corresponds to a target image, and the target image is the currently acquired second image or any subsequent second image. Any sub-result is obtained by: Extracting abstract features of the first object from the currently acquired target image, wherein the abstract features include color features, texture features, shape features, and spatial relationship features, and, Extracting the first pose feature of the end effector from the pose data of the end effector corresponding to the acquisition time of the target image, and, Extracting a second posture feature of the robotic arm from the joint parameters of the robotic arm corresponding to the acquisition moment of the target image; According to the abstract feature, the first posture feature and the second posture feature, the next posture data of the end effector and the next joint parameter of the robotic arm are predicted respectively to obtain any one of the sub-results.

5. The method according to claim 1, characterized in that The real-time position of the end effector is obtained by: Acquire a third image in real time by using the first visual sensor, and perform the following operations for each third image: Performing target recognition for the end effector on any acquired third image; If the end effector is identified, determining the real-time position of the end effector based on the identified position of the end effector in any one of the third images; If the end effector is not identified, the real-time position of the end effector is determined based on the structural data of the robot and the joint parameters corresponding to the acquisition time of any one of the third images.

6. The method according to claim 1, characterized in that The determining whether the first object is perceived by the second visual sensor based on the distance between the position and the real-time position of the end effector, the first weight value, the weighted sum of the confidence and the second weight value, and the replacement score threshold, comprises: If it is determined that the weighted sum is greater than the replacement score threshold, it is determined that the first object is perceived by the second visual sensor; If it is determined that the weighted sum is less than or equal to the replacement score threshold, it is determined that the first object is not perceived by the second visual sensor.

7. A robot motion planning device, characterized in that: include: a first planning module, configured to perform motion planning for the robot to perform an operation on the first object based on a position of the first object and a current position of the robot, obtain a first planning result, and move toward the first object based on the first planning result, wherein the position is determined based on a first image acquired by a first visual sensor deployed on a head of the robot; a recognition module, configured to continuously acquire a second image by a second visual sensor deployed on the end effector of the robot during movement, and perform the following operations for each second image: generating a heat map of the first object based on the currently acquired second image, determining a target area matching the first object from the heat map based on the probability of each pixel in the heat map, and determining, based on the probability of each pixel in the target area, when the confidence of the target area is greater than a confidence threshold, determining whether the first object is sensed by the second visual sensor based on the distance between the position and the real-time position of the end effector, a first weight value, a weighted sum of the confidence and a second weight value, and a replacement score threshold, wherein the heat map includes the probability that each pixel in the currently acquired second image belongs to the first object; A second planning module is used to perform local motion planning on the robot's execution device for operating the first object based on the currently acquired second image and the subsequent second image acquired by the second visual sensor when the recognition module determines that the first object is perceived by the second visual sensor, obtain a second planning result, and control the execution device to perform actions based on the second planning result, wherein the execution device includes the end effector and an associated robotic arm.

8. A robot, characterized in that: include: A first visual sensor, deployed on the head of the robot, for acquiring an image of the environment in which the robot is located; A second visual sensor is deployed on the end effector of the robot and is used to acquire an image around the end effector; Memory, used to store computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 6.

Images