Robot safety operation method and system based on dual-system architecture

Abstract

This application relates to a robot safety operation method and system based on a dual-system architecture, belonging to the field of robot operation. The method includes: parsing natural language commands to decompose the task in the natural language commands into multiple sub-tasks; for each sub-task, determining the target object point cloud and the observation scene point cloud based on robot visual information; determining the optimal grasping position and optimal grasping posture based on the target object point cloud and the observation scene point cloud, using a grasping posture generation network; constructing a path map, a posture map, and a gripper state map based on the optimal grasping position and optimal grasping posture; performing path planning based on the path map and posture planning based on the posture map to obtain the final trajectory; and controlling the robot to complete the sub-task based on the final trajectory and the gripper state map. This application combines semantic understanding capabilities with the execution robustness of traditional planning and control methods to achieve zero-shot, training-free task-driven operation.

Description

Technical Field

[0001] This application relates to the field of robot operation, and more specifically, to a robot safety operation method and system based on a dual-system architecture. Background Technology

[0002] With the rapid development of technology, robotics has been widely applied in industrial automation and daily life services, covering a range of repetitive or structured tasks, from precision assembly in industrial production to cleaning assistance in everyday household scenarios. Robots performing these operations typically involve several key technological aspects, including environmental perception, task semantic understanding, target localization, trajectory planning, and motion control.

[0003] Currently, the field of robot operation mainly relies on two types of algorithms: one is the path planning and execution framework based on traditional control. This type of method can achieve high-precision and robust closed-loop control at the execution layer, but it is difficult to adapt to diverse semantic tasks in open scenarios, and its generalization ability is limited. In recent years, large-scale pre-trained models based on the Transformer architecture have made significant progress in language understanding and multimodal fusion, bringing new opportunities to robot operating systems. Compared with traditional methods, large models have powerful cross-modal semantic understanding capabilities, and can uniformly represent natural language instructions and visual perception information, thereby significantly improving the system's understanding and generalization ability for diverse tasks. However, existing methods based on large models usually rely on large-scale labeled data for supervised training or fine-tuning, which is not only costly, but also sensitive to environmental changes (such as lighting, background, object placement posture, etc.). Once the scene shifts, the model performance often degrades significantly. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, this application provides a robot safety operation method and system based on a dual-system architecture.

[0005] Firstly, a robot safety operation method based on a dual-system architecture is provided, including:

[0006] Perform task parsing on natural speech commands, decomposing the tasks in the natural speech commands into multiple sub-tasks;

[0007] For each subtask, the point cloud of the target object and the point cloud of the observed scene are determined based on the robot's visual information;

[0008] Based on the point cloud of the target object and the point cloud of the observed scene, a grasping posture generation network is used to determine the optimal grasping position and optimal grasping posture.

[0009] Based on the optimal gripping position and optimal gripping posture, construct a path map, posture map, and gripper state map;

[0010] Path planning is performed based on the path map, and attitude planning is performed based on the attitude map to obtain the final trajectory. Each path point in the final trajectory has attitude information.

[0011] The robot is controlled based on the final trajectory and gripper status map to complete the sub-task.

[0012] In one embodiment, based on the target object point cloud and the observed scene point cloud, a grasping pose generation network is used to determine the optimal grasping position and optimal grasping pose, including:

[0013] Preprocess the point cloud of the target object to obtain the preprocessed point cloud of the target object.

[0014] The preprocessed target object point cloud and the observation scene point cloud are input into the grasping pose generation network to generate a grasping pose candidate set. The grasping pose candidate set includes multiple candidate poses, and each candidate pose corresponds to a grasping position and a quality score.

[0015] Perform model-independent collision detection on the candidate pose set to obtain candidate poses that pass the collision detection;

[0016] The candidate poses that pass the collision detection are initially screened using the non-maximum suppression method to obtain the screened candidate poses;

[0017] The candidate poses are further filtered based on quality scores and grasping positions to obtain the optimal grasping pose and optimal grasping position.

[0018] In one embodiment, a path map is constructed based on the optimal grasping position and optimal grasping posture, including:

[0019] The robot's workspace is discretized into a uniform three-dimensional grid. The uniform three-dimensional grid is initialized as free space. The point clouds of all objects except the target object, the robotic arm, and the gripper are mapped to the corresponding grid and marked as obstacles. The grid corresponding to the obstacle has a value of 1, and the other grids have a value of 0.

[0020] A cost map is constructed based on the obstacle distribution, and the cost map includes the cost of each grid cell:

[0021]

[0022] in, For grid The corresponding cost value Represents a grid The value, As the preset maximum cost, For grid The Euclidean distance to the grid corresponding to the nearest obstacle. To avoid dividing by zero for extremely small positive numbers;

[0023] The cost map is the path map, which also includes the current gripper position and the target position, with the target position being the optimal gripping position.

[0024] In one embodiment, a pose map is constructed based on the optimal grasping position and optimal grasping posture, including:

[0025] The attitude map and the path map maintain the same discrete space and coordinate mapping;

[0026] Set the starting position's orientation to the current gripper orientation, and the target position's orientation to the optimal gripping orientation.

[0027] In one embodiment, the method further includes:

[0028] After each subtask is completed, determine whether the capture was successful. If it was, execute the next subtask; otherwise, re-execute the current subtask.

[0029] In one embodiment, determining whether the crawling was successful includes:

[0030] After the subtask is completed, acquire the point cloud of the target object again and record it as the current point cloud; and determine the location of the current point cloud. If the current height statistic along the axis is greater than the height statistic before executing the subtask, and the difference between the two is greater than the first preset threshold, then the capture is successful.

[0031] In one embodiment, re-executing the current subtask includes:

[0032] After obtaining the optimal grasping pose and optimal grasping position, the optimal grasping position is adjusted: Let the distance between the optimal grasping position and the centroid of the target object's point cloud be d, then adjust the optimal grasping position to a distance of a from the centroid of the target object's point cloud. d, where a is a set ratio.

[0033] In one embodiment, the method is based on a language model program (LMP), which includes a high-level LMP, a mid-level LMP, and a low-level LMP.

[0034] High-level LMP is used to perform task parsing on natural speech instructions, decomposing the tasks in natural speech instructions into multiple subtasks;

[0035] The middle-layer LMP is used to generate a network based on the grasping posture, according to the point cloud of the target object and the point cloud of the observed scene, to determine the optimal grasping position and optimal grasping posture.

[0036] The underlying LMP is used to build path maps, pose maps, and gripper state maps based on the optimal gripping position and optimal gripping posture.

[0037] In one embodiment, the method is based on a discrete visual language model, which is used to: for each subtask, determine the point cloud of the target object and the point cloud of the observed scene based on robot visual information.

[0038] Secondly, a robot safety operating system based on a dual-system architecture is provided, including: a top-level decision-making system and a bottom-level execution system;

[0039] Top-level decision-making systems are used for:

[0040] Perform task parsing on natural speech commands, decomposing the tasks in the natural speech commands into multiple sub-tasks;

[0041] For each subtask, the point cloud of the target object and the point cloud of the observed scene are determined based on the robot's visual information;

[0042] Based on the point cloud of the target object and the point cloud of the observed scene, a grasping posture generation network is used to determine the optimal grasping position and optimal grasping posture.

[0043] Based on the optimal gripping position and optimal gripping posture, construct a path map, posture map, and gripper state map;

[0044] The underlying execution system is used for:

[0045] Path planning is performed based on the path map, and attitude planning is performed based on the attitude map to obtain the final trajectory. Each path point in the final trajectory has attitude information.

[0046] The robot is controlled based on the final trajectory and gripper status map to complete the sub-task.

[0047] Compared to existing technologies, this application offers the following advantages: It parses natural language instructions and aligns them with visual perception information to generate structured path and posture maps as intermediate environment representations. The underlying execution system uses these representations to perform path and posture planning, generating smooth and reliable end effector motion paths, and achieving safe robot operation based on the planned trajectories. This application combines semantic understanding capabilities with the robustness of traditional planning and control methods to achieve zero-shot, training-free task-driven operation. Attached Figure Description

[0048] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings:

[0049] Figure 1 A flowchart of a robot safety operation method based on a dual-system architecture is shown.

[0050] Figure 2 The trajectory planning results in a multi-obstacle environment are shown in the figure.

[0051] Figure 3 The diagram shows the motion trajectory of the robotic arm's end effector in a real-world scenario.

[0052] Figure 4 A schematic diagram of a robot safety operating system based on a dual-system architecture is shown. Detailed Implementation

[0053] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0054] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution of this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0055] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0056] This application provides a robot safety operation method based on a dual-system architecture. Figure 1 A flowchart illustrating a robot safety operation method based on a dual-system architecture is shown. (See attached diagram.) Figure 1 The method mainly includes the following steps:

[0057] Step S1: Perform task parsing on the natural speech command, decomposing the task in the natural speech command into multiple sub-tasks.

[0058] Step S2: For each subtask, determine the point cloud of the target object and the point cloud of the observed scene based on the robot's visual information.

[0059] Step S3: Based on the point cloud of the target object and the point cloud of the observed scene, a grasping posture generation network is used to determine the optimal grasping position and the optimal grasping posture.

[0060] Step S4: Based on the optimal gripping position and optimal gripping posture, construct a path map, posture map, and gripper state map.

[0061] Step S5: Perform path planning based on the path map and attitude planning based on the attitude map to obtain the final trajectory. Each path point in the final trajectory has attitude information. Figure 2 The results of trajectory planning in a multi-obstacle environment are shown in the figure.

[0062] Step S6: Control the robot to complete the sub-task based on the final trajectory and gripper state map.

[0063] The final trajectory includes a sequence of path points in the base coordinate system and corresponding pose quaternions. These path point sequences, pose quaternions, and gripper state sequences from the gripper state map are then converted into executable control commands and sent to the robot arm controller. The controller uses real-time inverse kinematics solving and a closed-loop trajectory tracking strategy to map the desired end-effector pose into joint space control variables, driving the robot arm to move smoothly along the planned trajectory. This establishes a stable closed-loop execution mechanism from high-level semantic commands to low-level continuous joint movements. Figure 3 The diagram shows the motion trajectory of the robotic arm's end effector in a real-world scenario.

[0064] In this embodiment, at the top-level decision-making stage, natural language instructions are parsed and aligned with visual perception information to generate structured path and attitude maps as intermediate environment representations. Based on these representations, the bottom-level execution system performs path and attitude planning, generating smooth and reliable end-effector motion paths. This embodiment combines semantic understanding capabilities with the robustness of traditional planning and control methods to achieve zero-shot, training-free task-driven operations.

[0065] Specifically, the robot safety operation method based on a dual-system architecture is based on Language Model Programs (LMPs). LMPs are executable code generated by a large language model (LLM) according to task instructions, conforming to the syntax of a programming language. This program uses a programming language as an intermediary to transform semantic understanding results into structured control logic. By calling predefined perception and motion planning interfaces, it achieves the mapping from abstract language goals to concrete control primitives. This embodiment constructs a three-layer LMP decision structure: a high-level LMP, a middle-level LMP, and a low-level LMP.

[0066] High-level LMP is used to perform task parsing on natural speech instructions, decomposing the tasks in natural speech instructions into multiple subtasks;

[0067] For example, after receiving natural language instructions (such as "put the fruit in the plate," "clean the table," or "move the chess piece"), the system calls a pre-trained large language model interface (such as GPT-4o or Qwen-Latest) for high-level task parsing. Based on preset task decomposition prompts, the large language model decomposes complex task instructions layer by layer into a series of sequentially executed atomic subtasks. For example, for the instruction "clean the table," the model can parse it into three basic steps: "grab object A," "determine if the grab was successful," "move to target location B," and "release the object," and execute these steps cyclically according to the scene state until the task is completed.

[0068] The middle-layer LMP is used to generate a network based on the grasping posture, according to the point cloud of the target object and the point cloud of the observed scene, to determine the optimal grasping position and optimal grasping posture.

[0069] The underlying LMP is used to build path maps, pose maps, and gripper state maps based on the optimal gripping position and optimal gripping posture.

[0070] Specifically, the robot safety operation method based on a dual-system architecture is based on a discrete visual language model, which is used to determine the point cloud of the target object and the point cloud of the observed scene based on the robot's visual information for each sub-task.

[0071] Real-time acquisition of RGB-D data streams from dual cameras is performed, and target objects are located using open vocabulary detection models (such as Grounding DINO or YOLO-World). The bounding boxes output by the open vocabulary detection models are further input into an efficient segmentation model (such as SAM2) to obtain pixel-level masks of the target objects. The pixel-level masks of the target objects are combined with depth information to obtain the target object point cloud. The depth information is combined with camera intrinsics to obtain the observation scene point cloud.

[0072] In one embodiment, step S3, based on the target object point cloud and the observed scene point cloud, determines the optimal grasping position and optimal grasping posture using a grasping posture generation network, including:

[0073] First, the target object point cloud is preprocessed to obtain the preprocessed target object point cloud. Here, the density-based clustering algorithm (DBSCAN) is used to extract the largest connected point cloud cluster to remove outlier noise points and background interference. Then, the local K-nearest neighbor adaptive density filtering method is used to filter out outlier points and noise, thereby preserving the dense and reliable point cloud of the main structure of the object.

[0074] Then, the preprocessed target object point cloud and the observed scene point cloud are input into the grasping pose generation network GraspNet to generate a grasping pose candidate set. The grasping pose candidate set includes multiple candidate poses, each of which corresponds to a grasping position and a quality score, as well as a grasping width.

[0075] Then, model-independent collision detection is performed on the candidate pose set to obtain candidate poses that pass the collision detection.

[0076] Here, model-independent collision detection utilizes the observed scene point cloud to construct a truncated directed distance field (TSDF) to exclude poses that interfere with other objects in the environment.

[0077] Then, the candidate poses that pass the collision detection are initially screened using the non-maximum suppression method to remove highly redundant grasping in space, and the screened candidate poses are obtained.

[0078] Then, the candidate poses are further filtered based on the quality score and the grasping position to obtain the optimal grasping pose and the optimal grasping position.

[0079] Here, the candidate poses after screening can be sorted from high to low according to their quality scores, and the top 20 candidate poses can be selected. Among the top 20 candidate poses, the one with the smallest distance between the grasping position and the centroid of the target object's point cloud is selected as the optimal grasping pose. The optimal grasping pose is represented in quaternion form. The grasping position corresponding to the optimal grasping pose is the optimal grasping position.

[0080] In one embodiment, step S4, based on the optimal grasping position and optimal grasping posture, constructs a path map, including:

[0081] Discretize the robot workspace as A uniform 3D grid is initialized as free space. The point clouds of all objects except the target object, the robotic arm, and the gripper are mapped to the corresponding grid and marked as obstacles. The grid corresponding to the obstacle has a value of 1, and the other grids have a value of 0.

[0082] A cost map is constructed based on the obstacle distribution, and the cost map includes the cost of each grid cell:

[0083]

[0084] in, For grid The corresponding cost value Represents a grid The value, As the maximum value preset, For grid Euclidean distance to the grid corresponding to the nearest obstacle. To avoid dividing by zero for extremely small positive numbers;

[0085] The cost map is the path map, which also includes the current gripper position and the target position, with the target position being the optimal gripping position.

[0086] This design ensures that the path avoids obstacles while staying as far away from the obstacle boundary as possible, forming a safe cost field with gradient guidance.

[0087] In one embodiment, step S4, based on the optimal grasping position and optimal grasping posture, constructs a posture map, including:

[0088] The attitude map and the path map maintain the same discrete space and coordinate mapping;

[0089] Set the initial position's pose as the current gripper pose and the target position's pose as the optimal gripping pose to complete the initial pose map creation. The initial position is the current gripper position.

[0090] Specifically, the gripper state map describes the opening and closing constraint area of ​​the gripper in space, that is, the gripper remains open outside a defined area near the target position and closed within a defined area near the target position. Here, the target position is the optimal gripping position, and the defined area near the target position is the area less than 0.1 cm away from the target position.

[0091] By maintaining consistency in the target position, it can be ensured that the robotic arm's posture and gripper movements synchronously meet the gripping requirements as the spatial path converges to the gripping point, thereby avoiding gripping failures caused by inconsistent target definitions.

[0092] Furthermore, since the grasping position and grasping posture are dynamically generated by the grasping network based on real-time sensing data, the results may differ at different times. If the latest grasping posture information is used in the generation of the path map, posture map, and gripper state map respectively, it may lead to inconsistencies in the target grasping posture used in the three maps, introducing target drift problems within the system. Therefore, at the beginning of each grasping subtask, the system must first save the latest optimal grasping posture and lock it throughout the entire planning cycle of that subtask. The path target, posture target, and gripper closed region are all generated based on this grasping posture. Through this locking mechanism, the consistency of the position, posture, and gripper state map targets can be guaranteed, thereby improving the stability and consistency of the grasping process.

[0093] In one embodiment, step S5, route planning based on the route map, includes:

[0094] To reasonably consider the impact of end effector (gripper) size on obstacle avoidance safety distance during subsequent safety passage construction, appropriate preprocessing of environmental geometry information is necessary during the path planning phase. Therefore, before path search, environmental obstacles are first equivalently inflated based on the target object size information obtained from the top-level perception system, and safety distance constraints are introduced based on the end effector and grasped object sizes. Subsequently, using the starting grid node in the path map as the starting point and the target grid node as the ending point, A... The search algorithm searches for a path. Upon reaching the target grid node, it extracts an initial path composed of discrete grid nodes by tracing back the parent node relationships. ,in Indicates the first position on the path There are 1 path points, where N is the number of path points.

[0095] The size of the raster map created at the top level can be 100. 100 100, use A The search algorithm finds a very dense network of path points. To reduce the impact of redundant nodes on subsequent channel construction and improve the overall quality of the path, this embodiment performs path point filtering after obtaining the initial path, including:

[0096] For the current path point Search for the furthest directly connected path point within its forward local window. ,in .here This indicates the upper limit of the step size of the local window, used to limit the maximum range spanned in a single filtering operation. If the path point... and If the line segment connecting the points lies entirely within free space and does not collide with any obstacles, then the point is retained. And delete all redundant path points in between. If a collision occurs with the farthest point, the next path point is selected from the farthest to the nearest. Repeat the above process to prioritize the farthest path point that satisfies the visibility constraints within the local window.

[0097] Then, based on the filtered paths, existing path planning methods are used for further processing to obtain the final path planning result. Here, the existing path planning method can be the one published in the journal "VAP: Safe Corridor-Based Velocity Adjustment Planning for RoboticArm" (2025 IEEE International Conference on Robotics and Biomimetics (ROBIO)).

[0098] Specifically, attitude planning includes:

[0099] The attitude map contains the current gripper attitude. (Attitude at the starting position) and attitude at the target position In order to plan a smooth and continuous attitude change between the starting position and the target position, the attitude planning module uses the quaternion-based SLERP interpolation method to calculate the rotation information corresponding to the path points, and uses this information to update the attitude map.

[0100] It is important to note that the starting pose and target pose defined in the original pose map correspond to the key nodes in the path map. However, during actual trajectory generation, due to path optimization, the final path point sequence may no longer be strictly aligned with the original path points at the discrete level, especially with slight offsets at the path endpoints. To ensure the accuracy of the trajectory at the execution level, consistency constraints on the poses of the path endpoints are necessary. Therefore, this embodiment uses the current gripper pose... attitude relative to the target position Assign values ​​to the start and end poses of the planned path respectively, and record them again. and This ensures that the trajectory remains consistent with the task semantics during the start and end phases. Then, using a quaternion-based SLERP interpolation method, the pose corresponding to each waypoint is calculated, and the pose map is updated.

[0101] Based on the original method, a quaternion is stored at the position corresponding to each path point to represent the rotational attitude at that position, ensuring that the robotic arm can smoothly transition to the target attitude along the path during actual control.

[0102] Through the aforementioned path planning and attitude planning processes, a discrete desired pose sequence of the end effector in the task space can be obtained, including the path point positions and corresponding attitude parameters. Since the path map, attitude map, and gripper state map are all constructed in the same base coordinate system and expressed using a unified spatial discretization method, they maintain a one-to-one correspondence at the spatial index level. The position of a path point in space directly corresponds to the gripper state information in the gripper state map. Therefore, each path point... Each can be associated with its corresponding attitude parameters. With gripper state The final sequence information sent to the underlying controller is in the following format: , Here, N is the path point number, and N is the number of path points.

[0103] In one embodiment, in a multi-stage operation task, the execution result of the grasping subtask has a decisive impact on subsequent operations. If a grasping failure is not promptly identified, subsequent placement, movement, and other actions will inevitably accumulate errors, potentially leading to overall task failure. Therefore, this embodiment introduces a grasping result feedback mechanism based on 3D vision perception after each grasping subtask is executed during the task decomposition phase. This mechanism is used to determine online whether the grasping was successful and trigger an adaptive re-grabbing process upon failure. That is, after each subtask is completed, it is determined whether the grasping was successful; if so, the next subtask is executed; otherwise, the current subtask is re-executed.

[0104] Specifically, determining whether a crawl was successful includes:

[0105] After the subtask is completed, acquire the point cloud of the target object again and record it as the current point cloud; and determine the location of the current point cloud. Current height statistics along the axis (e.g.) Minimum value in the axial direction , Mean along the axis If the current height statistic is greater than the height statistic before the subtask was executed, and the difference between the two is greater than the first preset threshold, then the capture is successful.

[0106] Specifically, re-execute the current subtask, including:

[0107] The grasping pose output directly by the grasping network may deviate from the geometric center of the object, resulting in unstable gripping. This embodiment introduces a grasping pose correction strategy based on the point cloud center.

[0108] After obtaining the optimal grasping pose and optimal grasping position using step S3 of the aforementioned embodiment, the optimal grasping position is adjusted:

[0109] Let d be the distance between the optimal grasping position and the centroid of the target object's point cloud. Adjust the optimal grasping position along the principal axis of the point cloud so that the distance between the optimal grasping position and the centroid of the target object's point cloud is a. d, where a is a set ratio, which can be 60% to improve the stability and success rate of the capture.

[0110] Based on the optimal grasping pose and the adjusted optimal grasping position, the grasping is performed again to verify the grasping result; if successful, the subsequent subtasks are executed; otherwise, the above grasping retry process is repeated within a set number of times.

[0111] In this embodiment, visual closed-loop feedback is introduced to correct the grasping posture and placement position online, thereby significantly improving the success rate of operation and the system adaptability in complex scenarios.

[0112] This application also provides a robot safety operating system based on a dual-system architecture. Figure 4 A schematic diagram of a robot safety operating system based on a dual-system architecture is shown, which includes a top-level decision-making system and a bottom-level execution system.

[0113] Top-level decision-making systems are used for:

[0114] Perform task parsing on natural speech commands, decomposing the tasks in the natural speech commands into multiple sub-tasks;

[0115] For each subtask, the point cloud of the target object and the point cloud of the observed scene are determined based on the robot's visual information;

[0116] Based on the point cloud of the target object and the point cloud of the observed scene, a grasping posture generation network is used to determine the optimal grasping position and optimal grasping posture.

[0117] Based on the optimal gripping position and optimal gripping posture, construct a path map, posture map, and gripper state map;

[0118] The underlying execution system is used for:

[0119] Path planning is performed based on the path map, and attitude planning is performed based on the attitude map to obtain the final trajectory. Each path point in the final trajectory has attitude information.

[0120] The robot is controlled based on the final trajectory and gripper status map to complete the sub-task.

[0121] This embodiment centralizes task understanding and environmental modeling capabilities in the top-level decision-making system, while unifying trajectory generation and motion control under the bottom-level execution system. This dual-system architecture achieves clear division of labor and functional decoupling. On one hand, the top-level large model does not need to concern itself with specific robotic arm models, dynamic parameters, or control details, reducing the system's dependence on model inference stability. On the other hand, the bottom-level execution system can flexibly adjust planning and control strategies according to the actual execution scenario while maintaining interface consistency. This avoids introducing unnecessary computational complexity in simple scenarios and fully utilizes safety information and speed constraints from the planning stage in complex environments. This architecture effectively achieves synergistic unity between intelligent decision-making capabilities and bottom-level motion execution performance, balancing system execution efficiency, adaptability, and safety.

[0122] The robot safety operating system based on the dual-system architecture in this embodiment has the same inventive concept as the robot safety operation method based on the dual-system architecture described above. Therefore, the specific implementation of this system can be found in the embodiment section of the robot safety operation method based on the dual-system architecture described above, and its technical effects correspond to the technical effects of the above method, so it will not be repeated here.

[0123] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot safety operation method based on a dual-system architecture, characterized in that, include: Perform task parsing on natural speech commands, and decompose the tasks in the natural speech commands into multiple sub-tasks; For each subtask, the point cloud of the target object and the point cloud of the observed scene are determined based on the robot's visual information; Based on the point cloud of the target object and the point cloud of the observed scene, the optimal grasping position and the optimal grasping posture are determined using a grasping posture generation network. Based on the optimal gripping position and the optimal gripping posture, construct a path map, a posture map, and a gripper state map; Path planning is performed based on the path map, and attitude planning is performed based on the attitude map to obtain the final trajectory. Each path point in the final trajectory has attitude information. The robot is controlled based on the final trajectory and the gripper state map to complete the sub-task; Specifically, based on the target object point cloud and the observed scene point cloud, and using a grasping posture generation network, the optimal grasping position and optimal grasping posture are determined, including: The target object point cloud is preprocessed to obtain the preprocessed target object point cloud. The preprocessed target object point cloud and the observed scene point cloud are input into the grasping posture generation network to generate a grasping posture candidate set; the grasping posture candidate set includes multiple candidate postures, and each candidate posture corresponds to a grasping position and a quality score. Model-independent collision detection is performed on the candidate set of grabbing poses to obtain candidate poses that pass the collision detection; The candidate poses that pass the collision detection are initially screened using a non-maximum suppression method to obtain the screened candidate poses. The candidate postures that have been filtered are further filtered based on the quality score and the grasping position to obtain the optimal grasping posture and the optimal grasping position. The path map is constructed based on the optimal grasping position and the optimal grasping posture, including: The robot's workspace is discretized into a uniform three-dimensional grid. The uniform three-dimensional grid is initialized as free space. The point clouds of all objects except the target object, the robotic arm, and the gripper are mapped to the corresponding grid and marked as obstacles. The grid corresponding to the obstacle has a value of 1, and the other grids have a value of 0. A cost value map is constructed based on the obstacle distribution, wherein the cost value map includes the cost value of each grid cell: in, For grid The corresponding cost, Represents grid The value, As the maximum value preset, For grid Euclidean distance to the grid corresponding to the nearest obstacle. To avoid dividing by zero for extremely small positive numbers; The cost map is the path map, which also includes the current gripper position and the target position, where the target position is the optimal gripping position. The method further includes: After each subtask is completed, determine whether the fetching was successful. If it was, execute the next subtask; otherwise, re-execute the current subtask.

2. The method as described in claim 1, characterized in that, in, Based on the optimal grasping position and the optimal grasping posture, a posture map is constructed, including: The attitude map and the path map maintain the same discrete space and coordinate mapping; Set the starting position's orientation to the current gripper orientation, and set the target position's orientation to the optimal gripping orientation.

3. The method as described in claim 1, characterized in that, The determination of whether the crawling was successful includes: After the subtask is completed, acquire the point cloud of the target object again and record it as the current point cloud; and determine the location of the current point cloud. If the current height statistic along the axis is greater than the height statistic before executing the subtask, and the difference between the two is greater than the first preset threshold, then the capture is successful.

4. The method as described in claim 1, characterized in that, The re-execution of the current subtask includes: After obtaining the optimal grasping pose and optimal grasping position, the optimal grasping position is adjusted: Let the distance between the optimal grasping position and the centroid of the target object's point cloud be d, then adjust the optimal grasping position to a distance of a from the centroid of the target object's point cloud. d, where a is a set ratio.

5. The method as described in claim 1, characterized in that, The method is based on a Language Model Program (LMP), which includes a high-level LMP, a mid-level LMP, and a low-level LMP. The high-level LMP is used to perform task parsing on natural speech instructions, decomposing the tasks in the natural speech instructions into multiple sub-tasks. The middle-layer LMP is used to generate a network based on the grasping posture, according to the point cloud of the target object and the point cloud of the observed scene, to determine the optimal grasping position and the optimal grasping posture. The underlying LMP is used to construct a path map, an attitude map, and a gripper state map based on the optimal gripping position and the optimal gripping posture.

6. The method as described in claim 1, characterized in that, The method is based on a discrete visual language model, which is used to: for each subtask, determine the point cloud of the target object and the point cloud of the observed scene based on the robot's visual information.

7. A robot safety operating system based on a dual-system architecture, characterized in that, include: Top-level decision-making system and bottom-level execution system; The top-level decision-making system is used for: Perform task parsing on natural speech commands, and decompose the tasks in the natural speech commands into multiple sub-tasks; For each subtask, the point cloud of the target object and the point cloud of the observed scene are determined based on the robot's visual information; Based on the point cloud of the target object and the point cloud of the observed scene, the optimal grasping position and the optimal grasping posture are determined using a grasping posture generation network. Based on the optimal gripping position and the optimal gripping posture, construct a path map, a posture map, and a gripper state map; The underlying execution system is used for: Path planning is performed based on the path map, and attitude planning is performed based on the attitude map to obtain the final trajectory. Each path point in the final trajectory has attitude information. The robot is controlled based on the final trajectory and the gripper state map to complete the sub-task; Specifically, based on the target object point cloud and the observed scene point cloud, and using a grasping posture generation network, the optimal grasping position and optimal grasping posture are determined, including: The target object point cloud is preprocessed to obtain the preprocessed target object point cloud. The preprocessed target object point cloud and the observed scene point cloud are input into the grasping posture generation network to generate a grasping posture candidate set; the grasping posture candidate set includes multiple candidate postures, and each candidate posture corresponds to a grasping position and a quality score. Model-independent collision detection is performed on the candidate set of grabbing poses to obtain candidate poses that pass the collision detection; The candidate poses that pass the collision detection are initially screened using a non-maximum suppression method to obtain the screened candidate poses. The candidate postures that have been filtered are further filtered based on the quality score and the grasping position to obtain the optimal grasping posture and the optimal grasping position. The path map is constructed based on the optimal grasping position and the optimal grasping posture, including: The robot's workspace is discretized into a uniform three-dimensional grid. The uniform three-dimensional grid is initialized as free space. The point clouds of all objects except the target object, the robotic arm, and the gripper are mapped to the corresponding grid and marked as obstacles. The grid corresponding to the obstacle has a value of 1, and the other grids have a value of 0. A cost value map is constructed based on the obstacle distribution, wherein the cost value map includes the cost value of each grid cell: in, For grid The corresponding cost, Represents grid The value, As the maximum value preset, For grid Euclidean distance to the grid corresponding to the nearest obstacle. To avoid dividing by zero for extremely small positive numbers; The cost map is the path map, which also includes the current gripper position and the target position, where the target position is the optimal gripping position. The system is also used for: After each subtask is completed, determine whether the fetching was successful. If it was, execute the next subtask; otherwise, re-execute the current subtask.

Images