Robot control method and device, electronic equipment, storage medium and program product

Abstract

The application provides a robot control method and device, electronic equipment, storage medium and program product, wherein the method comprises the following steps: determining point cloud data of a target object in multi-view depth images based on a user instruction and the multi-view depth images; fusing the point cloud data of the target object in the multi-view depth images, and determining a target pose of a robot according to the fused point cloud data; and controlling the robot to perform a target task according to the target pose of the robot and a navigation path; the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to changes in an environment where the robot is located. The method fuses the point cloud data of the target object in the multi-view depth images, and updates the target pose and the navigation path in real time according to changes in the environment where the robot is located during navigation, thereby not only ensuring that the robot quickly and accurately performs a task in a complex environment, but also enabling real-time adjustment during task performance, and ensuring continuity and accuracy of the task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robots, and in particular to a robot control method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] With the rapid development of artificial intelligence and automation technology, intelligent robots have been widely applied in industrial manufacturing, logistics sorting, home service and other fields. Modern robot systems not only need to complete basic movement and obstacle avoidance functions, but also need to have complex operation capabilities such as target object recognition, grasping, placing and multi-task collaborative execution, especially in dynamically changing environments, such as object displacement, occlusion, external interference and other factors, which puts higher requirements on the perception, decision-making and execution capabilities of robots.

[0003] Currently, robot grasping and navigation technology mainly relies on depth perception devices to obtain spatial position information of target objects, and path planning and grasping control are based on this information. However, such systems generally have some problems.

[0004] Firstly, in terms of dynamic environment adaptability, most existing technologies are designed for static scenes, lacking effective perception and response mechanisms for real-time changes in the position of target objects. When the target object moves or is occluded by other objects, the robot has difficulty updating its spatial coordinates in a timely manner, resulting in failed grasping or invalid navigation paths.

[0005] Secondly, in terms of voice interaction and task switching, although some robot systems have integrated voice recognition modules to receive user instructions, these systems generally lack effective interrupt mechanisms and task priority management strategies. In the case of multiple task concurrency or sudden instruction input, the robot cannot quickly terminate the current task and switch to a new target, affecting the response efficiency and interaction experience of the system.

[0006] In addition, in terms of continuous execution of complex tasks, existing technologies often rely on fixed process settings when dealing with tasks involving multiple steps (such as first grasping object A and then placing it in region B), lacking the ability to adaptively adjust to changes in the environment during the process. Especially in scenarios with high requirements for multi-target recognition and spatial positioning accuracy, robots are prone to task interruption, motion conflict and other problems due to environmental disturbances.

[0007] In summary, how to solve the problem that existing robot systems cannot quickly and accurately execute tasks in dynamically complex environments, and cannot adapt to changes in the environment, leading to task interruption or incorrect execution, is an important issue in the field of intelligent robots that needs to be addressed. SUMMARY

[0008] The application provides a robot control method, device, electronic equipment, storage medium and program product, to overcome the defects that the existing robot system cannot quickly and accurately execute tasks in a dynamic complex environment, and cannot make adaptive adjustments when the environment changes, resulting in task interruption or incorrect execution, ensure that the robot can accurately execute tasks in a complex environment, and at the same time, can make real-time adjustments during task execution, guarantee the continuity and accuracy of the task.

[0009] In one aspect, the application provides a robot control method, comprising: collecting multi-view depth images of a target object according to a received user instruction; determining point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images; fusing the point cloud data of the target object in the multi-view depth images, and determining a target pose of the robot according to the fused point cloud data; obtaining a target three-dimensional coordinate corresponding to the target object in the multi-view depth images, and mapping the target three-dimensional coordinate to a map to generate a navigation path of the robot; controlling the robot to execute a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to the environment change of the robot.

[0010] Further, the user instruction is a voice instruction; accordingly, the determination of the point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images comprises: converting the voice instruction into a text instruction, and obtaining global point cloud data corresponding to the multi-view depth images; obtaining two-dimensional coordinates of the target object in the multi-view depth images according to the text instruction and the multi-view depth images; determining a mask of the target object in the multi-view depth images according to the two-dimensional coordinates of the target object in the multi-view depth images; determining the point cloud data of the target object from the global point cloud data corresponding to the multi-view depth images according to the mask of the target object in the multi-view depth images, to obtain the point cloud data of the target object in the multi-view depth images.

[0011] Further, the fusion of the point cloud data of the target object in the multi-view depth images comprises: mapping the point cloud data of the target object in each view depth image of the multi-view depth images to a unified coordinate system to obtain target point cloud data of the target object in the multi-view depth images; determining a weight value of each view depth image of the multi-view depth images; fusing the target point cloud data of the target object in the multi-view depth images by weighted Kalman filtering according to the weight value of each view depth image to obtain initial fused point cloud data; reducing the density of the initial fused point cloud data by voxel grid filtering to obtain the fused point cloud data.

[0012] Further, the obtaining the target three-dimensional coordinates corresponding to the target object in the multi-view depth image comprises: determining the target three-dimensional coordinates corresponding to the target object in the multi-view depth image according to the two-dimensional coordinates of the target object in the multi-view depth image, and internal parameters and depth values of a collection device of the multi-view depth image.

[0013] Further, the controlling the robot to perform the target task according to the target pose and the navigation path of the robot comprises: collecting a new multi-view depth image in real time, and obtaining current two-dimensional coordinates of the target object in the new multi-view depth image in real time; obtaining an offset difference value between the current two-dimensional coordinates and the two-dimensional coordinates; in a case where the offset difference value is greater than or equal to a preset dynamic threshold, updating the target pose and the navigation path of the robot, and controlling the robot to perform the target task according to the updated target pose and the navigation path; in a case where the offset difference value is less than the preset dynamic threshold, controlling the robot to perform the target task according to the target pose and the navigation path of the robot.

[0014] Further, the user instruction is a voice instruction; when the voice instruction is received, the robot is controlled to stop performing a current task and switch to performing the target task.

[0015] In a second aspect, the present application further provides a robot control device, comprising: a multi-view depth image collection module configured to collect a multi-view depth image of a target object according to a received user instruction; a target object point cloud data determination module configured to determine point cloud data of the target object in the multi-view depth image based on the user instruction and the multi-view depth image; a target pose determination module configured to fuse the point cloud data of the target object in the multi-view depth image, and determine a target pose of a robot according to the fused point cloud data; a navigation path generation module configured to obtain target three-dimensional coordinates corresponding to the target object in the multi-view depth image, and map the target three-dimensional coordinates to a map to generate a navigation path of the robot; and a target task execution module configured to control the robot to perform a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to changes in an environment in which the robot is located.

[0016] In a third aspect, the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the robot control method according to any one of the above aspects when executing the computer program.

[0017] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements any of the robot control methods described above.

[0018] In a fifth aspect, the present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements any of the robot control methods described above.

[0019] The robot control method provided by the present application comprises the following steps: collecting multi-view depth images of a target object according to a received user instruction; determining point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images; fusing the point cloud data of the target object in the multi-view depth images; determining a target pose of a robot according to the fused point cloud data; obtaining a target three-dimensional coordinate corresponding to the target object in the multi-view depth images; mapping the target three-dimensional coordinate to a map to generate a navigation path of the robot; and controlling the robot to perform a target task according to the target pose of the robot and the navigation path. The target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to changes in the environment where the robot is located. The method fuses the point cloud data of the target object in the multi-view depth images, cooperates with the target pose determination step and the navigation path generation step, and updates the target pose and the navigation path in real time according to changes in the environment where the robot is located during navigation. The method not only ensures that the robot can quickly and accurately perform a task in a complex environment, but also can perform real-time adjustment during task execution, thereby ensuring the continuity and accuracy of the task. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0021] Figure 1 is a flowchart of the robot control method provided by the embodiment of the present application.

[0022] Figure 2 is a schematic diagram of the overall flow of the robot control method provided by the embodiment of the present application.

[0023] Figure 3 is a structural schematic diagram of the robot control device provided by the embodiment of the present application.

[0024] Figure 4 is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection scope of the present application.

[0026] It should be noted that, with the continuous development of robot technology, intelligent robots are increasingly applied in industries, logistics, services and other fields. In these applications, robots often need to perform complex tasks such as grasping, placing, navigation, etc., especially in dynamic and challenging environments, such as object movement, environmental changes or external instruction interference. However, existing robot grasping and navigation systems face many difficulties in handling these complex tasks.

[0027] On the one hand, the limitations of existing robot grasping and navigation technology. Currently, many robot grasping and navigation systems rely on depth perception technology for target object positioning and operation. However, most of these systems can only handle static environments and lack real-time adaptability to changes in object position or environmental interference. For example, when an object moves or is blocked, existing technology often fails to update the position of the target object in time, resulting in task execution failure.

[0028] On the other hand, the existing robot voice interference and multi-task processing problems. Some existing robot systems have voice recognition capabilities for executing voice instructions. However, these systems often lack the ability to deal with voice interference. For example, during task execution, if the user issues a new voice instruction, the existing system may not be able to interrupt the current task and flexibly adjust the behavior. Without effective task management and interruption mechanisms, robots often cannot efficiently switch tasks or handle external interference.

[0029] In addition, existing robots are insufficient in handling multi-step tasks. In handling complex tasks involving multiple steps, existing robot systems also face great challenges in spatial positioning, grasping and placing tasks. Robots must be able to adapt to changes in object position and changes in the environment, but most existing technologies lack this flexibility. Existing spatial positioning technologies usually rely on static object detection and cannot update environmental changes in time, resulting in task interruption or error execution.

[0030] In view of this, the present application proposes a new robot control method, specifically, Figure 1 The flowchart of the robot control method provided by the embodiment of the present application is shown.

[0031] like Figure 1 As shown, the method includes: S110, acquiring multi-view depth images of a target object according to a received user instruction; S120, determining point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images; S130, fusing the point cloud data of the target object in the multi-view depth images and determining the target pose of the robot based on the fused point cloud data; S140, obtaining the target three-dimensional coordinates corresponding to the target object in the multi-view depth images and mapping the target three-dimensional coordinates to a map to generate a navigation path for the robot; S150, controlling the robot to execute a target task according to the target pose and navigation path; wherein, the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to changes in the robot's environment.

[0032] The following will provide a detailed description of steps S110-S150 and related steps.

[0033] S110 acquires multi-view depth images of the target object according to the received user instructions.

[0034] It's easy to understand that the robot receives / monitors instructions from the user in real time (user instructions). These user instructions can be voice instructions, text instructions, or visual recognition instructions (such as gestures), without specific limitations here. When the user instruction is something other than a text instruction, it needs to be converted into a text instruction and formatted so that the robot can understand its content.

[0035] Upon receiving a user command, the system controls multiple pre-installed acquisition devices to simultaneously capture images of the target object from different positions and angles. These acquisition devices can be ordinary RGB cameras paired with depth sensors, or dedicated depth cameras. For example, when photographing a large object, multiple cameras are placed around the object, each capturing an image, thus obtaining multi-view image data, i.e., multi-view depth images. This method can capture information from all sides of the target object, providing a rich data source of multi-view depth images for subsequent analysis.

[0036] The number and installation location of the data acquisition devices can be set according to actual needs and are not specifically limited here. For example, in a specific embodiment, the data acquisition devices include two (Realsense D435 and L515), one mounted on the robot's head and the other mounted on the robot's chest or third-person view.

[0037] The target object can be obtained through user commands. It can be a dynamic person or animal, or a static object, depending on the specific situation.

[0038] Multi-view depth images are composed of multiple single-view depth images, which provide more comprehensive three-dimensional scene information compared to ordinary depth images. Multi-view depth images can describe the shape, size and positional relationship of target objects from multiple angles, and can more accurately reflect the true form of target objects. In particular, multi-view depth images can avoid the occlusion problem that may occur in single-view depth images.

[0039] On the basis of step S110 of acquiring the multi-view depth image of the target object according to the received user instruction, further, step S120 is executed.

[0040] S120, based on the user instruction and the multi-view depth image, determining the point cloud data of the target object in the multi-view depth image.

[0041] It is easy to understand that for multiple single-view depth images in the multi-view depth image, since each pixel value in the single-view depth image represents the distance (depth value) from the camera to the target object surface corresponding to the pixel, the pixel coordinates on the two-dimensional single-view depth image can be converted into three-dimensional space coordinates through the internal parameters of the camera (such as focal length, optical center position, etc.), and then the global point cloud data corresponding to the single-view depth image is generated.

[0042] At the same time, by inputting the modal converted and formatted user instruction and the single-view depth image into the pre-trained Roborefer model, the output two-dimensional coordinates of the target object can be obtained. Then, taking the two-dimensional coordinates of the target object as input, the SAM (Segment Anything Model) is called to generate the mask of the target object.

[0043] The Roborefer model combines single-step spatial understanding and multi-step spatial reasoning to achieve precise control, especially in complex 3D environments.

[0044] Regarding the architecture of the Roborefer model. RoboRefer combines visual language models (VLMs) with 3D spatial perception, mainly including two key components: RGB encoder and depth encoder. The RGB encoder is used to process visual data (RGB image) to achieve spatial understanding; the depth encoder is used to extract 3D depth information separately to avoid interference with the RGB encoder, thereby enhancing spatial understanding without compromising image performance. The outputs of the two encoders are aligned through the projector with the large language model (LLM) to perform tasks such as question answering or point prediction to execute robot operation tasks.

[0045] The training process of the Roborefer model. The training of RoboRefer is divided into a supervised fine-tuning stage and a reinforcement fine-tuning stage. Among them, the goal of the supervised fine-tuning stage is to improve the model's understanding of spatial concepts, which specifically includes: Step 1: Depth alignment - align the depth information with the text input to ensure the accuracy of depth perception; Step 2: Spatial understanding enhancement - fine-tune the entire model on a pre-set dataset containing data that supports single-step spatial understanding and multi-step reasoning. The goal of the reinforcement fine-tuning stage is to enhance the model's ability to handle multi-step spatial reasoning, which specifically includes: training through a process reward function (such as accuracy reward and process format reward) to improve intermediate reasoning steps and improve the accuracy of the final prediction. Among them, the pre-set dataset can be the RefSpatial dataset, which contains 2.5 million samples and 20 million question and answer pairs, and also includes 31 spatial relationships, supporting multi-step reasoning tasks.

[0046] SAM is a new deep learning-based general image segmentation model developed by Meta AI, which can perform high-quality instance segmentation or semantic segmentation on any object in any image without further fine-tuning. SAM has three core capabilities: (1) zero-shot segmentation: no training is required to segment any object you specify in an image; (2) interactive segmentation: users can tell the model what to segment through points, boxes, etc. ; (3) full-scene generalization: suitable for various image types (natural images, medical images, remote sensing images, etc.).

[0047] SAM includes an image encoder, a prompt encoder, and a mask decoder. Among them, the image encoder uses VisionTransformer (such as ViT-B / 16) to encode the input image and extract high-dimensional feature representation. This part has a large computational overhead, but only needs to be run once. The prompt encoder encodes user prompts (such as click points, bounding boxes, and text descriptions) into vector form and combines them with image features. The mask decoder generates high-quality binary or probability mask images based on image features and prompt information to identify the pixel regions of target objects.

[0048] Subsequently, for a single-view depth image, the mask of the target object is used to filter the point cloud data of the target object in the single-view depth image from the global point cloud data corresponding to the single-view depth image.

[0049] Finally, according to the point cloud data of the target object in all single-view depth images, the point cloud data of the target object in the multi-view depth image can be obtained.

[0050] Based on the determination of the point cloud data of the target object in the multi-view depth image in step S120 based on the user instruction and the multi-view depth image, further, step S130 is performed.

[0051] S130, fuse the point cloud data of the target object in the multi-view depth image, and determine the target posture of the robot based on the fused point cloud data.

[0052] It is easy to understand that, for the point cloud data of the target object in the obtained multi-view depth image, the point cloud data of the target object in multiple single-view depth images are fused to obtain fused point cloud data, and then the target pose of the robot is determined based on the fused point cloud data.

[0053] Specifically, the point cloud data of the target object in multiple single-view depth images can be converted into the same coordinate system (such as the world coordinate system). Then, the corresponding weight values ​​are determined according to the depth accuracy and visual reliability of the multiple single-view depth images. Thus, the point cloud data of the target object in the multi-view depth images can be fused according to the weight values ​​of the multiple single-view depth images to obtain the preliminary fused point cloud data.

[0054] Then, by using the fused point cloud data as input to AnyGrasp, the robot's target pose in the third-view camera coordinate system can be generated. The target pose can be the grasping pose, including the position of the grasping point. and grab direction It can also refer to the placement posture; no specific restrictions are given here. Indicates the roll angle. Indicates pitch angle, Indicates the yaw angle.

[0055] AnyGrasp is a general algorithm framework for 6-DoF Grasp Detection, proposed by the Institute for Intelligent Industry (AIR) team at Tsinghua University. It is primarily used to efficiently and accurately predict effective grasping poses that robots can execute from 3D point cloud data. The core goal of AnyGrasp is to generate high-quality, executable 6-DoF Grasp poses for objects of arbitrary shapes, based solely on single-view or full-view point cloud data, without prior knowledge, object models, or training samples.

[0056] Finally, the robot's target pose in the third-view camera coordinate system is transformed to the robot coordinate system so that the robot can perform operations in the target pose.

[0057] It is worth mentioning that, compared with the existing technology, the point cloud data fusion of target objects in multi-view depth images proposed in this embodiment is superior to the point cloud data processing of target objects in single-view depth images, and can further improve accuracy.

[0058] S140, obtaining the target three-dimensional coordinates corresponding to the target object in the multi-view depth image, and mapping the target three-dimensional coordinates to a map to generate a navigation path of the robot.

[0059] It is easy to understand that for multiple single-view depth images in the multi-view depth image, the two-dimensional coordinates of the target object can be obtained by inputting the modal converted and formatted user instruction and the single-view depth image into the pre-trained Roborefer model.

[0060] Then, according to the two-dimensional coordinates of the target object, the three-dimensional coordinates of the target object in the world coordinate system, i.e., the target three-dimensional coordinates, can be obtained in combination with the internal parameters (such as focal length, optical center position, etc.) of the camera (acquisition device of the single-view depth image).

[0061] It should be noted that the target three-dimensional coordinates corresponding to the target object in the multi-view depth image can refer to the target three-dimensional coordinates corresponding to the target object in the single-view depth image in the multi-view depth image, or the fusion result of the target three-dimensional coordinates corresponding to the target object in multiple single-view depth images. The fusion strategy can also be realized according to the weight values of the multiple single-view depth images, which will not be expanded here.

[0062] After determining the target three-dimensional coordinates corresponding to the target object, SLAM (Simultaneous Localization And Mapping) is realized on the robot, and the target three-dimensional coordinates are mapped to a map, so as to generate a navigation path of the robot. The navigation path refers to the path segment of the robot from its current position to the target object.

[0063] Specifically, the robot constructs an environment map in real time in an unknown environment and simultaneously locates the position of the robot itself, converts the target three-dimensional coordinates of the target object into global map coordinates, and generates an executable navigation path based on the global environment map and the target point (i.e., the global map coordinates at the target object / conversion) using a path planning algorithm.

[0064] It should be noted that there is no strict execution order between steps S140 and steps S120-S130. In one specific embodiment, steps S140 and steps S120-S130 are executed cooperatively.

[0065] After steps S110-S140 are executed, step S150 is executed.

[0066] S150, controlling the robot to perform a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path will be updated in real time according to the environment where the robot is located.

[0067] It is easy to understand that the robot moves to the vicinity of the target object according to the navigation path, and performs the target task on the target object according to the target pose. The target task here is determined by analyzing the user instruction, which can be a target object grasping task or a target object placing task, which is not limited here.

[0068] It is worth mentioning that in the process of moving the robot to the vicinity of the target object according to the navigation path, the robot will detect the position change of the target object in real time. If the position offset is greater than or equal to the set threshold, the navigation path and the target pose will be regenerated, and the target task will be performed according to the new navigation path and the new target pose; if the position offset is less than the set threshold, the target task will continue to be performed according to the target pose and the navigation path.

[0069] The robot in this embodiment can be UR5, G1, or other types of intelligent robots, which are not limited here.

[0070] In this embodiment, by receiving the user instruction, the multi-view depth image of the target object is collected, and the point cloud data of the target object in the multi-view depth image is determined based on the user instruction and the multi-view depth image. Then the point cloud data of the target object in the multi-view depth image is fused, and the target pose of the robot is determined according to the fused point cloud data; at the same time, the target three-dimensional coordinates corresponding to the target object in the multi-view depth image are obtained, and the target three-dimensional coordinates are mapped to the map to generate the navigation path of the robot; then the target task is determined by the user instruction, and the target pose and the navigation path will be updated in real time according to the change of the environment where the robot is located. The method fuses the point cloud data of the target object in the multi-view depth image, cooperates with the target pose determination step and the navigation path generation step, and updates the target pose and the navigation path in real time according to the change of the environment where the robot is located during navigation. Not only ensures that the robot can quickly and accurately execute the task in a complex environment, but also can adjust in real time during task execution, and ensures the continuity and accuracy of the task.

[0071] On the basis of the above embodiment, further taking the user instruction as a voice instruction and the target pose as a grasping pose as an example, the target pose determination step of the robot will be described in detail.

[0072] First, according to the received user instruction, the multi-view depth image of the target object is collected.

[0073] Specifically, after receiving the voice instruction, the RGB-D images of the target object can be synchronously collected using the Realsense D435 installed on the head of the robot and the Realsense L515 installed on the chest of the robot or the third perspective, that is, multi-perspective depth images containing color information (RGB) and depth information (Depth) at the same time. The resolution of the multi-perspective depth image is 1280x720, and the depth accuracy is ±2mm.

[0074] Then, based on the user instruction and the multi-perspective depth image, the point cloud data of the target object in the multi-perspective depth image is determined, including: converting the voice instruction into a text instruction, and obtaining the global point cloud data corresponding to the multi-perspective depth image; obtaining the two-dimensional coordinates of the target object in the multi-perspective depth image according to the text instruction and the multi-perspective depth image; determining the mask of the target object in the multi-perspective depth image according to the two-dimensional coordinates of the target object in the multi-perspective depth image; and determining the point cloud data of the target object from the global point cloud data corresponding to the multi-perspective depth image according to the mask of the target object in the multi-perspective depth image, to obtain the point cloud data of the target object in the multi-perspective depth image.

[0075] Specifically, for multiple single-perspective depth images in the multi-perspective depth image, since each pixel value in the single-perspective depth image represents the distance (depth value) from the camera to the surface of the target object corresponding to the pixel, the pixel coordinates on the two-dimensional single-perspective depth image can be converted into three-dimensional space coordinates through the internal parameters of the camera (such as focal length, optical center position, etc.), and then the global point cloud data corresponding to the single-perspective depth image is generated, thereby obtaining the global point cloud data corresponding to the multi-perspective depth image.

[0076] For the received voice instruction, the voice instruction is first converted into a text instruction through a lightweight Whisper medium model, and then the text instruction can be translated into an English instruction through ChatGPT and formatted into a standard instruction, so that the robot can read the instruction content.

[0077] The formatted standard instruction and the single-perspective depth image are input into the pre-trained Roborefer model, and the two-dimensional coordinates of the target object in the output single-perspective depth image are obtained. Subsequently, the mask of the target object in the single-perspective depth image is generated by calling SAM with the two-dimensional coordinates of the target object as input. Thus, the mask of the target object in the multi-perspective depth image can be obtained.

[0078] According to the mask of the target object in each single-perspective depth image, the point cloud data of the target object is filtered out from the global point cloud data corresponding to the single-perspective depth image, so that the point cloud data of the target object in all single-perspective depth images, that is, the point cloud data of the target object in the multi-perspective depth image, is obtained.

[0079] Next, the point cloud data of the target object in the multi-view depth image is fused, including: mapping the point cloud data of the target object in each view depth image in the multi-view depth image to a unified coordinate system to obtain target point cloud data of the target object in the multi-view depth image; determining a weight value of each view depth image in the multi-view depth image; according to the weight value of each view depth image, using weighted Kalman filtering to fuse the target point cloud data of the target object in the multi-view depth image to obtain initial fused point cloud data; reducing the density of the initial fused point cloud data through voxel grid filtering to obtain fused point cloud data.

[0080] For the point cloud data of the target object in the multi-view depth image, the point cloud data of the target object in different single view depth images is mapped to a unified world coordinate system to obtain target point cloud data of the target object in the multi-view depth image.

[0081] For the multi-view depth image, the weight value of each single view depth image can be determined according to its depth accuracy and view reliability, for example, the weight value of the single view depth image collected by L515 is 0.6, and the weight value of the single view depth image collected by D435 is 0.4. Then, the weighted Kalman filtering is used to fuse the target point cloud data of the target object in each single view depth image to obtain the initial fused point cloud data. Further, the Voxel Grid filtering (voxel size is 0.01m) can also be used to reduce the density of the initial fused point cloud data to reduce the amount of calculation.

[0082] Finally, the target pose of the robot is determined according to the fused point cloud data.

[0083] Specifically, the fused point cloud data is input into AnyGrasp, and the grasping pose in the third view camera coordinate system can be generated, including the grasping point position and the grasping direction .

[0084] Further, through hand-eye calibration or robot base coordinate conversion, the grasping pose in the third view camera coordinate system is converted to the robot coordinate system, so as to facilitate the robot to perform the grasping operation of the target object.

[0085] In the embodiment, a multi-view point cloud fusion algorithm is adopted, and the existing single camera point cloud processing algorithm is further improved to improve the robot grasping accuracy and efficiency. At the same time, by combining the weighted Kalman filter with the Voxel Grid filter, the calculation amount is reduced by about 30%, which is also better than the traditional point cloud processing algorithm. In a specific embodiment, in the experimental environment of 50 grasping tests of UR5 and G1 robots, the grasping success rate of the embodiment is improved from 80% of the prior art to 90%; in the experimental environment of Jetson OrinNX and image resolution of 1280x720, the point cloud processing time of the embodiment is shortened from 0.8 seconds to 0.5 seconds, which is improved by 37.5%.

[0086] On the basis of the above embodiment, further, the navigation path generation step of the robot and the step of updating the target pose and the navigation path in real time during navigation will be described in detail.

[0087] First, the target three-dimensional coordinates corresponding to the target object in the multi-view depth image are obtained, including: determining the target three-dimensional coordinates corresponding to the target object in the multi-view depth image according to the two-dimensional coordinates of the target object in the multi-view depth image, and the internal parameters and depth values of the collection device of the multi-view depth image.

[0088] Specifically, for multiple single-view depth images in the multi-view depth image, by inputting the modal conversion and formatted user instruction and the single-view depth image into the pre-trained Roborefer model, the output two-dimensional coordinates of the target object (such as a table) can be obtained.

[0089] According to the two-dimensional coordinates of the target object, the three-dimensional coordinates of the target object in the world coordinate system, i.e. the target three-dimensional coordinates, can be obtained by combining the internal parameters (such as focal length, optical center position, etc.) of the camera (collection device of single-view depth image).

[0090] It should be noted that the target three-dimensional coordinates corresponding to the target object in the multi-view depth image can refer only to the target three-dimensional coordinates corresponding to the target object in the single-view depth image in the multi-view depth image, or can refer to the fusion result of the target three-dimensional coordinates corresponding to the target object in multiple single-view depth images. The fusion strategy can also be realized according to the weight values of the multiple single-view depth images, which will not be described in detail here.

[0091] Then, the target three-dimensional coordinates are mapped to the map to generate the navigation path of the robot.

[0092] Specifically, after determining the target three-dimensional coordinates corresponding to the target object, the robot implements SLAM, maps the target three-dimensional coordinates to a map, and thus generates a navigation path of the robot. In detail, the robot constructs an environment map in real time in an unknown environment and simultaneously locates the position of the robot itself, converts the target three-dimensional coordinates of the target object into global map coordinates, and generates an executable navigation path based on the global environment map and the target point (i.e., the global map coordinates converted from the target object) using a path planning algorithm.

[0093] Further, according to the target pose and the navigation path of the robot, the robot is controlled to perform the target task, including: collecting a new multi-view depth image in real time, and obtaining a current two-dimensional coordinate of the target object in the new multi-view depth image in real time; obtaining an offset difference value between the current two-dimensional coordinate and the two-dimensional coordinate; in a case where the offset difference value is greater than or equal to a preset dynamic threshold, updating the target pose and the navigation path of the robot, and controlling the robot to perform the target task according to the updated target pose and the navigation path; and in a case where the offset difference value is less than the preset dynamic threshold, controlling the robot to perform the target task according to the target pose and the navigation path of the robot.

[0094] Specifically, in the process in which the robot travels to the vicinity of the target object according to the navigation path, the multi-view depth image of the target object is collected in real time, and the current two-dimensional coordinate of the target object in the newly collected multi-view depth image is obtained in real time by calling the Roborefer model, so as to detect the position change of the target object.

[0095] If the offset difference value between the current two-dimensional coordinate of the target object and the two-dimensional coordinate of the target object obtained last time is greater than or equal to the preset dynamic threshold, the target pose and the navigation path of the robot are re-generated, and the robot is controlled to perform the target task according to the re-generated target pose and the navigation path.

[0096] If the offset difference value between the current two-dimensional coordinate of the target object and the two-dimensional coordinate of the target object obtained last time is less than the preset dynamic threshold, the target pose and the navigation path are not re-generated, and the target task can be performed according to the target pose and the navigation path determined previously.

[0097] The preset dynamic threshold can be set according to actual requirements, and is not specifically limited herein. For example, in a specific embodiment, the preset dynamic threshold is 0.1 m.

[0098] In this embodiment, by introducing a preset dynamic threshold, the target pose and navigation path are updated in real time during navigation, not only solving the problem of poor cooperation in dynamic environments, but also improving robustness. In a specific embodiment, the navigation accuracy is improved by 20%, and the positioning error is reduced from 0.15m to 0.12m (experimental environment: G1 robot, 5x5m dynamic environment). The success rate of grasping and navigation cooperation reaches 92%, supporting dynamic obstacle scenes (such as moving objects).

[0099] On the basis of the above-mentioned embodiments, further, taking the user instruction as a voice instruction as an example, the real-time voice interruption and task switching process provided by the present application will be described in detail below.

[0100] To realize real-time voice control in robot motion, this embodiment uses a state machine (FSM) to monitor voice instruction changes. If a new voice instruction is detected, a lightweight Whisper medium model (with 20% fewer parameters through model pruning) is used to process user input voice instructions in real time on a Jetson Orin NX, with a sampling rate of 16kHz and a frame length of 30ms.

[0101] Specifically, when the robot receives a voice instruction, it first converts the voice instruction to a text instruction through the lightweight Whisper medium model, then translates the text instruction to an English instruction through ChatGPT, and formats it into a standard instruction so that the robot can understand the instruction content. For example, "grasp the red object" is converted to "Grasp red object at position X".

[0102] For the received new standard instruction, the robot stops executing the current task (such as UR5 robot motion) through the terminal process, and switches to execute the target task according to the new standard instruction. That is, this embodiment can design a task priority queue through ROS2 Topic, with the priority rule being "new user instruction takes precedence over current task".

[0103] In this embodiment, combined with real-time voice recognition technology (such as using Whisper ASR model) and task interruption mechanism, new voice instructions can be monitored during task execution. When a new voice instruction is detected, the system will interrupt the current task and immediately re-plan the task execution according to the new voice instruction. This mechanism ensures that the robot can flexibly respond to user voice input while executing tasks, and better adjusts and executes tasks in complex environments.

[0104] In some embodiments, considering that the grasp direction may be upward or parallel to the tabletop when importing fused point cloud data into AnyGrasp to calculate the grasp pose, some improvements have been made in this embodiment.

[0105] Specifically, assuming that the target object is placed in the plate, the masks with the top three confidence values are taken as the mask of the target object + plate when the mask of the target object is determined using the SAM, which is also the final mask of the target object.

[0106] Assuming that the target object is not placed in the plate, the mask of the target object is expanded by one circle after the mask of the target object is determined using the SAM, so as to expand the range of the reserved point cloud. Specifically, the maximum inscribed circle of the expanded mask is taken as the final mask of the target object, with the center of the target object mask as the center of the circle.

[0107] wherein the horizontal coordinate of the center of the circle is the average of the horizontal coordinates of all points in the mask of the target object, and the vertical coordinate of the center of the circle is the average of the vertical coordinates of all points in the mask of the target object.

[0108] The purpose of the above processing of the mask of the target object is to make the fusion point cloud data input into the AnyGrasp reserve the point cloud data on the plane of the object.

[0109] According to the final mask of the target object in each single-view depth image, the point cloud data of the target object is filtered from the global point cloud data corresponding to the corresponding view depth image and fused, to obtain the fusion point cloud data.

[0110] Further, the fusion point cloud data is input into the AnyGrasp, and the grasping poses generated by the AnyGrasp are filtered, and the grasping poses with the grasping direction upward or parallel to the table are deleted. Specifically, the grasping poses are converted to the coordinate system of the robot arm through a conversion matrix, the grasping poses with too large pitch and yaw angles are deleted, and the grasping pose with the highest confidence value in the remaining grasping poses is taken as the target pose.

[0111] In some embodiments, Figure 2 The overall flowchart of the robot control method provided by the embodiment of the application is shown, which shows the determination process of the grasping pose, and the target tracking and dynamic adjustment process (i.e. the process of updating the grasping pose and the navigation path). The specific details in the figure can be found in the above embodiments, and will not be described here.

[0112] In one embodiment, in an industrial scene, using a UR5 robot arm and a Realsense L515 camera, the input voice instruction is "grab the red object". Whisper transcribes the instruction, ChatGPT4 translates it into English, Roborefer outputs the two-dimensional coordinates of the target object (0.5, 0.3), SAM generates the mask of the target object, AnyGrasp generates the grasping pose, and UR5 performs the grasping. Experimental results: 90% success rate in 50 grasps, response time 0.5 seconds.

[0113] In another embodiment, in a service scenario, G1 robot collects multi-view depth images of the target table through a D435 camera and a L515 camera for navigation guidance. A navigation path is generated through SLAM navigation, the position of the target object is updated in real time, and the grasping pose is generated based on the multi-view depth images collected by the D435 camera. Experimental results: navigation error 0.12m, grasping success rate 92%.

[0114] Through the above technical means, the robot control method provided by the embodiment of the application achieves the following effects: the voice response delay is reduced to 0.5 seconds, the task switching success rate is 95%, the point cloud processing time is shortened by 37.5%, the grasping success rate is improved to 90%, the navigation accuracy is improved by 20%, and the collaborative success rate in a dynamic environment is 92%.

[0115] Corresponding to the robot control method described in the above embodiments, the application also provides a robot control device. Specifically, Figure 3 The structure of the robot control device provided by the embodiment of the application is shown.

[0116] As Figure 3 shown, the device comprises: a multi-view depth image acquisition module 310 for acquiring multi-view depth images of a target object according to a received user instruction; a target object point cloud data determination module 320 for determining point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images; a target pose determination module 330 for fusing the point cloud data of the target object in the multi-view depth images and determining a target pose of a robot according to the fused point cloud data; a navigation path generation module 340 for obtaining a target three-dimensional coordinate corresponding to the target object in the multi-view depth images and mapping the target three-dimensional coordinate to a map to generate a navigation path of the robot; and a target task execution module 350 for controlling the robot to perform a target task according to the target pose and the navigation path of the robot, wherein the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to the environment where the robot is located.

[0117] In the embodiment, the multi-view depth image acquisition module 310 acquires the multi-view depth image of the target object according to the received user instruction, the target object point cloud data determination module 320 determines the point cloud data of the target object in the multi-view depth image based on the user instruction and the multi-view depth image, and then the target pose determination module 330 fuses the point cloud data of the target object in the multi-view depth image and determines the target pose of the robot according to the fused point cloud data; at the same time, the navigation path generation module 340 obtains the target three-dimensional coordinates corresponding to the target object in the multi-view depth image, and maps the target three-dimensional coordinates to a map to generate the navigation path of the robot; then the target task execution module 350 controls the robot to execute the target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path will be updated in real time according to the change of the environment where the robot is located. The device fuses the point cloud data of the target object in the multi-view depth image, cooperates with the target pose determination step and the navigation path generation step, and updates the target pose and the navigation path in real time according to the change of the environment where the robot is located during navigation, which not only ensures that the robot can quickly and accurately execute the task in a complex environment, but also can adjust in real time during the task execution process, ensuring the continuity and accuracy of the task.

[0118] It should be noted that the robot control device provided in the embodiments of the present application can be correspondingly referred to the robot control method described in the above embodiments, which will not be described here.

[0119] Figure 4 An example of a schematic diagram of a physical structure of an electronic device is shown as Figure 4 As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke the logical instructions in the memory 430 to execute the robot control method, which includes: acquiring a multi-view depth image of a target object according to a received user instruction; determining point cloud data of the target object in the multi-view depth image based on the user instruction and the multi-view depth image; fusing the point cloud data of the target object in the multi-view depth image and determining a target pose of a robot according to the fused point cloud data; obtaining target three-dimensional coordinates corresponding to the target object in the multi-view depth image, and mapping the target three-dimensional coordinates to a map to generate a navigation path of the robot; controlling the robot to execute a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path will be updated in real time according to the change of the environment where the robot is located.

[0120] Moreover, the logic instructions in the memory 430 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0121] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the robot control method provided by the above-mentioned method. The method comprises: collecting multi-view depth images of a target object according to received user instructions; determining point cloud data of the target object in the multi-view depth images based on the user instructions and the multi-view depth images; fusing the point cloud data of the target object in the multi-view depth images, and determining a target pose of a robot according to the fused point cloud data; obtaining a target three-dimensional coordinate corresponding to the target object in the multi-view depth images, and mapping the target three-dimensional coordinate to a map to generate a navigation path of the robot; controlling the robot to perform a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instructions, and the target pose and the navigation path will be updated in real time according to the environment of the robot.

[0122] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the robot control method provided by the above method, which comprises: collecting multi-view depth images of a target object according to a received user instruction; determining point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images; fusing the point cloud data of the target object in the multi-view depth images and determining a target pose of a robot according to the fused point cloud data; obtaining a target three-dimensional coordinate corresponding to the target object in the multi-view depth images and mapping the target three-dimensional coordinate to a map to generate a navigation path of the robot; controlling the robot to perform a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to the environment of the robot.

[0123] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0124] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot control method characterized by, The method comprises the following steps: According to the received user instruction, a multi-view depth image of the target object is collected; Based on the user instruction and the multi-view depth image, the point cloud data of the target object in the multi-view depth image is determined; Fusing the point cloud data of the target object in the multi-view depth image, and determining the target pose of the robot according to the fused point cloud data; Obtaining the target three-dimensional coordinates corresponding to the target object in the multi-view depth image, and mapping the target three-dimensional coordinates to a map to generate a navigation path of the robot; According to the target pose and navigation path of the robot, the robot is controlled to perform a target task; wherein the target task is determined by the user instruction, and the target pose and the navigation path will be updated in real time according to the environment where the robot is located; The fusion of the point cloud data of the target object in the multi-view depth image comprises: Mapping the point cloud data of the target object in each view depth image in the multi-view depth image to a unified coordinate system to obtain the target point cloud data of the target object in the multi-view depth image; According to the depth accuracy and visual reliability of multiple single-view depth images, the weight value of each visual depth image is determined; According to the weight value of each view depth image, the target point cloud data of the target object in the multi-view depth image is fused by using weighted Kalman filtering to obtain initial fused point cloud data; By voxel grid filtering, the density of the initial fused point cloud data is reduced to obtain the fused point cloud data.

2. The robot control method according to claim 1, characterized by, The user instruction is a voice instruction; Correspondingly, the determination of the point cloud data of the target object in the multi-view depth image based on the user instruction and the multi-view depth image comprises: Converting the voice instruction into a text instruction, and obtaining global point cloud data corresponding to the multi-view depth image; According to the text instruction and the multi-view depth image, the two-dimensional coordinates of the target object in the multi-view depth image are obtained; According to the two-dimensional coordinates of the target object in the multi-view depth image, a mask of the target object in the multi-view depth image is determined; According to the mask of the target object in the multi-view depth image, the point cloud data of the target object is determined from the global point cloud data corresponding to the multi-view depth image to obtain the point cloud data of the target object in the multi-view depth image.

3. The robot control method according to claim 2, wherein, The acquisition of the target three-dimensional coordinates corresponding to the target object in the multi-view depth image comprises: According to the two-dimensional coordinates of the target object in the multi-view depth image, and the internal parameters and depth values of the acquisition device of the multi-view depth image, the target three-dimensional coordinates corresponding to the target object in the multi-view depth image are determined.

4. The robot control method according to claim 2, wherein, The control of the robot to perform a target task according to the target pose and navigation path of the robot comprises: Real-time collection of new multi-view depth images, and real-time acquisition of current two-dimensional coordinates of the target object in the new multi-view depth images; Obtaining the offset difference value between the current two-dimensional coordinates and the two-dimensional coordinates; In a case where the offset difference is greater than or equal to a preset dynamic threshold, the target pose and the navigation path of the robot are updated, and the robot is controlled to perform a target task according to the updated target pose and the navigation path. In a case where the offset difference is less than the preset dynamic threshold, the robot is controlled to perform a target task according to the target pose and the navigation path of the robot.

5. The robot control method according to claim 1, wherein, The user instruction is a voice instruction. When the voice instruction is received, the robot is controlled to stop performing a current task and switch to performing the target task.

6. A robot control device characterized by comprising: Comprise: A multi-view depth image acquisition module configured to acquire multi-view depth images of a target object according to a received user instruction; A target object point cloud data determination module configured to determine point cloud data of the target object in the multi-view depth images based on the user instruction and the multi-view depth images; A target pose determination module configured to fuse the point cloud data of the target object in the multi-view depth images and determine a target pose of the robot according to the fused point cloud data; A navigation path generation module configured to obtain target three-dimensional coordinates corresponding to the target object in the multi-view depth images, map the target three-dimensional coordinates to a map, and generate a navigation path of the robot; A target task execution module configured to control the robot to perform a target task according to the target pose and the navigation path of the robot; wherein the target task is determined by the user instruction, and the target pose and the navigation path are updated in real time according to changes in an environment in which the robot is located; The fusion of the point cloud data of the target object in the multi-view depth images comprises: Mapping the point cloud data of the target object in each view depth image in the multi-view depth images to a unified coordinate system to obtain target point cloud data of the target object in the multi-view depth images; Determining a weight value of each visual depth image according to depth precision and visual reliability of a plurality of single view depth images; Fusing the target point cloud data of the target object in the multi-view depth images by using weighted Kalman filtering according to the weight value of each view depth image to obtain initial fused point cloud data; Reducing the density of the initial fused point cloud data by voxel grid filtering to obtain the fused point cloud data.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the robot control method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the robot control method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the robot control method according to any one of claims 1 to 5.

Images