Embodied intelligent robot sampling system and method based on multi-modal perception

Abstract

The application discloses a somatic intelligent robot sample taking and preparing system and method based on multi-modal perception, and relates to the technical field of intelligent robots.The system comprises a multi-modal somatic environment modeling module, a sample pose determination module, a cooperative work planning module, a closed-loop execution control module and an adaptive strategy updating module.The application establishes a multi-modal somatic environment model by fusing visual information, depth information, force sensation information and robot body pose information, establishes a dynamic pose mapping relationship between a target sample and a robot operation space, realizes cooperative control of a robot moving path and an end execution trajectory, and performs somatic adaptive closed-loop adjustment in combination with target sample state, robot pose and contact force information, and continuously updates the environment model according to somatic experience in the robot execution process, thereby improving the environment perception precision and grasping positioning accuracy of the robot sample taking and preparing in a complex unstructured environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a sampling system and method for embodied intelligent robots based on multimodal perception. Background Technology

[0002] Compared to traditional manual sampling methods, embodied intelligent robots can reduce the intensity of manual labor, improve sampling efficiency and operational safety, and adapt to the need for continuous operation in complex environments. Therefore, they have become an important development direction for intelligent automated sampling.

[0003] Existing sampling robots are typically equipped with sensing devices such as vision cameras, depth sensors, and force sensors to achieve automatic sampling through target recognition, path planning, and robot motion control. However, current technologies still mainly employ independent control methods for vision recognition, path planning, and end-effector control, lacking a unified information fusion and collaborative decision-making mechanism between modules. This makes it difficult to fully leverage the autonomous interaction capabilities of embodied intelligent robots in complex environments.

[0004] Specifically, existing technologies have the following shortcomings: Current robots primarily rely on visual information or a combination of visual and depth information for environmental modeling. There is a lack of deep fusion between multimodal perception data, and the robot's own pose, contact state, and other embodied information are not involved in environmental modeling. This results in poor robustness of environmental perception in unstructured environments with complex lighting, occlusion, and dust. Current sample recognition and grasping pose determination are mainly based on the geometric or visual features of the target sample, without comprehensively considering the robot's current motion state, operational capabilities, and environmental constraints to establish an embodied operational space. This means that while the robot can recognize the target sample, it is difficult to guarantee that the generated grasping pose has good executability. Current mobile path planning and end-effector trajectory planning are usually planned separately and then combined for control. There is a lack of temporal coupling between the robot's body motion and the end-effector actions. The robot only begins adjusting its end-effector posture after reaching the target position, resulting in discontinuous sampling actions and affecting overall operational efficiency. Current robot control systems often employ fixed control parameters or rely solely on force feedback for closed-loop control. They fail to dynamically adjust control strategies based on changes in robot pose and target sample state. Consequently, problems such as sample damage, gripping failure, or repeated grasping can easily occur when the target sample slips, tilts, or deforms.

[0005] Therefore, how to construct a sampling method for embodied intelligent robots that can integrate multimodal perception information, robot embodied state, and environmental constraints to achieve continuous adaptive updates of perception, planning, control, and environmental models, and improve the accuracy, stability, and autonomous adaptability of sampling in complex environments, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention provides an embodied intelligent robot sampling system and method based on multimodal perception to solve the problems of inaccurate environmental perception, insufficient collaborative control capability and poor environmental adaptability of robot sampling in the prior art, and realize accurate perception, adaptive collaborative control and autonomous continuous operation of robot sampling in complex environments.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: Firstly, embodiments of the present invention provide a sampling system for an embodied intelligent robot based on multimodal perception, comprising: a multimodal embodied environment modeling module, used to acquire visual information, depth information, force information, and robot body pose information of the robot's operating area, perform spatiotemporal registration of the visual information, depth information, force information, and robot body pose information, and construct a multimodal embodied environment model including environmental constraint information and robot embodied state information; a sample pose determination module, used to extract the appearance features, spatial structure features, and contact features of the target sample based on the multimodal embodied environment model, and establish a pose mapping relationship between the target sample and the robot's embodied operating space in conjunction with the robot's current embodied state, and determine the grasping pose of the target sample; and a collaborative operation planning module, used for... Based on the grasping pose, robot embodied state, and environmental constraint information, a collaborative operation plan for the robot's movement path and end-effector trajectory is generated, and the robot's body movement and end-effector actions are synchronously controlled. A closed-loop execution control module is used to execute the sampling operation according to the collaborative operation plan, collecting end-effector contact force information, robot pose information, and target sample state information in real time, and performing closed-loop adjustments to the robot's motion posture and end-effector contact force based on the end-effector contact force information, robot pose information, and target sample state information. An adaptive strategy update module is used to evaluate the current sampling execution state based on the closed-loop adjustment results. When a change in the target sample state or environmental constraints is detected, the multimodal embodied environment model is updated, and the grasping pose determination, collaborative operation plan generation, and closed-loop adjustment process are re-executed until the target sample sampling is completed.

[0008] As a preferred embodiment of the multimodal perception-based embodied intelligent robot sampling system described in this invention, the following steps are taken: Spatiotemporal registration is performed on visual information, depth information, force information, and robot pose information to construct a multimodal embodied environment model containing environmental constraint information and robot embodied state information. This includes: acquiring the acquisition timestamps of visual information, depth information, force information, and robot pose information; synchronizing different modal information according to a unified time reference; and establishing spatial coordinate transformation relationships between different modal information based on the robot's coordinate system. A work area is generated based on the synchronized visual and depth information. The three-dimensional environmental point cloud is obtained, and dynamic pose compensation is performed on the three-dimensional environmental point cloud based on the robot's pose information to obtain an environmental space model consistent with the robot's motion state. Force information is mapped to the contact positions in the environmental space model to generate an environmental contact constraint distribution, and a robot reachable space model is established by combining the robot's kinematic constraints. Based on the environmental space model, environmental contact constraint distribution, and robot reachable space model, embodied consistency correlation analysis is performed to eliminate environmental information that does not match the robot's motion capabilities, and a multimodal embodied environment model containing environmental constraint information, contact constraint information, and robot embodied state information is generated.

[0009] As a preferred embodiment of the embodied intelligent robot sampling system based on multimodal perception described in this invention, the following steps are taken: Embodied consistency correlation analysis is performed based on an environmental space model, an environmental contact constraint distribution, and a robot reachable space model. This includes: establishing a correspondence between spatial units in the environmental space model and the robot reachable space, and determining candidate operation areas that the robot can reach; calculating the contact stability index of each spatial unit within the candidate operation area based on the environmental contact constraint distribution, and calculating the operation feasibility index of the spatial unit based on the robot joint motion range, end-effector posture constraints, and collision constraints; correlating and fusing the contact stability index and the operation feasibility index to generate an embodied operation evaluation value for the spatial unit, and eliminating spatial units below a preset threshold based on the embodied operation evaluation value; combining the remaining spatial units to form a robot embodied operation space, and extracting the environmental constraint information, contact constraint information, and robot embodied state information of the robot embodied operation space to generate a multimodal embodied environment model.

[0010] As a preferred embodiment of the embodied intelligent robot sampling system based on multimodal perception described in this invention, the following steps are taken: The system extracts the appearance features, spatial structure features, and contact features of the target sample based on a multimodal embodied environment model, and establishes a pose mapping relationship between the target sample and the robot's embodied operating space in conjunction with the robot's current embodied state to determine the grasping pose of the target sample. This includes: acquiring the target sample in the multimodal embodied environment model and extracting its appearance features, spatial structure features, and contact features to construct a multidimensional feature representation of the target sample; determining candidate grasping areas of the target sample based on the robot's embodied operating space, and establishing a robot operating space model based on the robot's current pose, joint motion state, and end effector posture; mapping the multidimensional feature representation of the target sample to the robot operating space model, calculating the spatial relationship between the target sample and the robot's end effector, and generating multiple candidate grasping poses; comprehensively evaluating the reachability, posture continuity, and contact stability corresponding to each candidate grasping pose, selecting target grasping poses that meet preset evaluation conditions, and outputting the target grasping pose.

[0011] As a preferred embodiment of the embodied intelligent robot sampling system based on multimodal perception described in this invention, the following steps are included: generating a collaborative operation scheme for the robot's movement path and end-effector trajectory, and synchronously controlling the robot's body movement and end-effector actions, comprising: generating a robot's body movement path based on the target grasping pose, the robot's current position, and environmental constraint information; generating the end-effector's grasping trajectory based on the target grasping pose; dividing the robot's body movement path into multiple continuous movement stages, and establishing an end-effector posture constraint sequence based on the robot's pose in each movement stage; segmenting and mapping the grasping trajectory based on the end-effector posture constraint sequence, with the end-effector synchronously adjusting its posture during the robot's body movement, establishing a temporal coupling relationship between the robot's body movement path and the end-effector trajectory; generating a collaborative operation scheme based on the temporal coupling relationship, and synchronously controlling the robot's body movement and end-effector actions based on the collaborative operation scheme.

[0012] As a preferred embodiment of the embodied intelligent robot sampling system based on multimodal perception described in this invention, the following steps are performed: Closed-loop adjustment of the robot's motion posture and end-effector contact force is conducted based on end-effector contact force information, robot pose information, and target sample state information. This includes: calculating the contact force deviation between the end effector and the target sample based on the end-effector contact force information, calculating the pose deviation between the robot end effector and the target grasping pose based on the robot pose information, and determining the target sample state stability coefficient based on the target sample state information; and calculating an embodied adaptive closed-loop adjustment factor based on the contact force deviation, pose deviation, and target sample state stability coefficient, wherein the embodied adaptive closed-loop adjustment factor satisfies: ,in, It is a self-adaptive closed-loop adjustment factor. For real-time contact force, For the target contact force, For correction factor, For the real-time pose of the robot's end effector, Capture the pose of the target. This is the maximum achievable working radius of the robot. The stability coefficient of the target sample state. , , To adjust the weights, the robot's closed-loop control output is calculated based on the specific adaptive closed-loop adjustment factor: ,in, This is the output for the robot's closed-loop control. For contact force adjustment, This is the robot's end-effector posture compensation amount. For force-controlled gain, This is the attitude control gain.

[0013] As a preferred embodiment of the embodied intelligent robot sampling system based on multimodal perception described in this invention, the sampling execution state is evaluated based on the closed-loop adjustment results. When a change in the target sample state or environmental constraints is detected, the multimodal embodied environment model is updated, including: obtaining the robot's motion trajectory, target sample state, and environmental state based on the robot's closed-loop control results, and calculating the robot's motion deviation, target sample state deviation, and environmental constraint deviation respectively; establishing an embodied experience increment model based on the robot's motion deviation, target sample state deviation, and environmental constraint deviation, and calculating the embodied experience increment. ,in, For embodied experience increment, Due to environmental constraint deviation, For robot motion deviation, For the target sample state deviation, Environmental constraint impact coefficient, To perform credibility coefficients for robots, This is the sensitivity coefficient for the sample state.

[0014] Secondly, embodiments of the present invention provide a sampling method for an embodied intelligent robot based on multimodal perception, comprising: acquiring visual information, depth information, force information, and robot pose information of the robot's working area; performing spatiotemporal registration on the visual information, depth information, force information, and robot pose information to construct a multimodal embodied environment model including environmental constraint information and robot embodied state information; extracting appearance features, spatial structure features, and contact features of the target sample based on the multimodal embodied environment model, and establishing a pose mapping relationship between the target sample and the robot's operating space in conjunction with the robot's current embodied state to determine the grasping pose of the target sample; and determining the grasping pose based on the grasping pose, robot embodied state, and environment. Constraint information is used to generate a collaborative operation plan for the robot's movement path and end effector trajectory, and to synchronously control the robot's body movement and end effector actions. Sampling operations are performed according to the collaborative operation plan, and end effector contact force information, robot pose information, and target sample state information are collected in real time. Based on the end effector contact force information, robot pose information, and target sample state information, the robot's motion posture and end effector contact force are adjusted in a closed loop. The sampling execution status is evaluated based on the closed loop adjustment results. When a change in the target sample state or environmental constraints is detected, the multimodal embodied environment model is updated, and the grasping pose determination, collaborative operation plan generation, and closed loop adjustment process are re-executed until the target sample sampling is completed.

[0015] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of the embodied intelligent robot sampling system based on multimodal perception as described in the first aspect of the present invention.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of the embodied intelligent robot sampling system based on multimodal perception as described in the first aspect of the present invention.

[0017] The beneficial effects of this invention are as follows: By constructing a multimodal embodied environment model that integrates visual information, depth information, force information, and robot body pose information, this invention establishes a dynamic pose mapping relationship between the target sample and the robot's operating space, realizes the coordinated control of the robot's movement path and end-effector trajectory, and performs embodied adaptive closed-loop adjustment by combining the target sample state, robot pose, and contact force information. At the same time, the environmental model is continuously updated based on the embodied experience during the robot's execution process, thereby improving the environmental perception accuracy and grasping and positioning accuracy of the robot in complex unstructured environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the system structure of the present invention. Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1: Refer to Figures 1-2 This is the first embodiment of the present invention. This embodiment provides a multimodal perception-based embodied intelligent robot sampling system. The system consists of a multimodal embodied environment modeling module, a sample pose determination module, a collaborative operation planning module, a closed-loop execution control module, and an adaptive strategy update module.

[0024] Specifically, the multimodal embodied environment modeling module is used to acquire visual information, depth information, force information, and robot pose information of the robot's working area, and to perform spatiotemporal registration of the visual information, depth information, force information, and robot pose information to construct a multimodal embodied environment model that includes environmental constraint information and robot embodied state information.

[0025] Furthermore, the acquisition timestamps of visual information, depth information, force information, and robot pose information are obtained. Different modal information is synchronized in time according to a unified time reference, and spatial coordinate transformation relationships between different modal information are established based on the robot's body coordinate system. A three-dimensional environmental point cloud of the work area is generated based on the synchronized visual and depth information. Dynamic pose compensation is performed on the three-dimensional environmental point cloud based on the robot's pose information to obtain an environmental space model with consistent robot motion states. Force information is mapped to contact positions in the environmental space model to generate an environmental contact constraint distribution, and a robot reachable space model is established by combining it with robot kinematic constraints. An embodied consistency correlation analysis is performed based on the environmental space model, environmental contact constraint distribution, and robot reachable space model to eliminate environmental information that does not match the robot's motion capabilities, generating a multimodal embodied environment model containing environmental constraint information, contact constraint information, and robot embodied state information.

[0026] Furthermore, a correspondence is established between spatial units in the environmental space model and the robot's reachable space to determine candidate operation areas that the robot can reach. The contact stability index of each spatial unit within the candidate operation area is calculated based on the environmental contact constraint distribution, and the operational feasibility index of the spatial unit is calculated based on the robot's joint range of motion, end-effector posture constraints, and collision constraints. The contact stability index and operational feasibility index are correlated and fused to generate an embodied operation evaluation value for the spatial unit, and spatial units below a preset threshold are eliminated based on the embodied operation evaluation value. The remaining spatial units are combined to form the robot's embodied operation space, and the environmental constraint information, contact constraint information, and robot embodied state information of the robot's embodied operation space are extracted to generate a multimodal embodied environment model.

[0027] It should be noted that this embodiment uses an industrial laboratory sampling scenario as an example. The robot's working area is 4m×3m×2m in size. An RGB camera, a ToF depth camera, a six-dimensional force sensor, and a robot controller are used to collect visual information, depth information, force information, and robot pose information, respectively. Specifically: the RGB image resolution is 1920×1080; the depth map resolution is 640×480; the force sampling frequency is 1000Hz; and the robot pose refresh rate is 500Hz.

[0028] First, using the system clock output by the robot controller as a unified time reference, visual information, depth information, force information, and robot pose information are synchronized. When the time error between different modal data exceeds 2ms, a nearest neighbor time compensation strategy is used to complete the synchronization. Then, using the robot's base coordinate system {B} as a unified reference coordinate system, coordinate transformation is performed based on the extrinsic parameter matrix obtained from calibration. After coordinate unification, a 3D environment point cloud is generated based on the RGB image and depth map. In this embodiment, approximately 2.6 × 10⁻⁶ points are obtained. 5 Point cloud points; motion compensation is performed on the point cloud to convert point clouds collected at different times to the current pose of the robot, forming a unified environmental space model.

[0029] Subsequently, the environmental space was divided into spatial units with dimensions of 20mm × 20mm × 20mm. In this embodiment, a total of approximately 1.2 × 10 5 Each spatial unit; for each spatial unit, calculate: environmental accessibility: ,in, As for environmental accessibility, This represents the number of obstacle points within the spatial unit. This represents the total number of points in the spatial unit.

[0030] For example, if there are 8 obstacle points and a total of 20 points, then A i =1-8 / 20=0.60; This indicates that there is occlusion in this area; the robot's reachability is defined as: ,in, For the degree of reach of robots, The distance from the current position of the robot's end effector to the center of the spatial unit. This is the robot's maximum working radius.

[0031] For example, if the robot's working radius L = 1.8m and the spatial distance d = 0.45m, then R i =e -0.45 / 1.8 =0.779; indicating that the robot can easily reach it; subsequently, the contact stability is calculated based on the results collected by the six-dimensional force sensor: Among them, contact stability, For real-time contact force, For target contact force.

[0032] For example, if the target contact force is 6N and the actual contact force is 5.4N, then T i =1-0.6 / 6=0.90; indicating that the contact state is stable.

[0033] Specifically, the sample pose determination module is used to extract the appearance features, spatial structure features and contact features of the target sample based on the multimodal embodied environment model, and establish the pose mapping relationship between the target sample and the robot's embodied operation space in combination with the robot's current embodied state, so as to determine the grasping pose of the target sample.

[0034] Furthermore, target samples are acquired from the multimodal embodied environment model, and their appearance, spatial structure, and contact features are extracted to construct a multidimensional feature representation of the target samples. Candidate grasping regions for the target samples are determined based on the robot's embodied operational space, and a robot operational space model is established based on the robot's current pose, joint motion state, and end effector posture. The multidimensional feature representation of the target samples is mapped to the robot operational space model, and the spatial relationship between the target samples and the robot's end effector is calculated to generate multiple candidate grasping poses. A comprehensive evaluation is performed based on the reachability, posture continuity, and contact stability of each candidate grasping pose to select target grasping poses that meet preset evaluation conditions, and the target grasping poses are output.

[0035] It should be noted that in this embodiment, the region where the target sample is located is determined based on the multimodal embodied environment model, and the point cloud data corresponding to the target sample is extracted using a three-dimensional point cloud segmentation algorithm. At the same time, the color texture and edge contour information of the target sample are obtained by combining visual images to form the appearance features of the target sample. Based on the size, surface normal and spatial geometry of the target sample, the spatial structure features of the target sample are formed. Combined with the contact constraint information in the multimodal embodied environment model, the contactable area and contact direction of the target sample are extracted to form the contact features of the target sample. The above features are then fused to construct a multidimensional feature expression of the target sample.

[0036] In this embodiment, taking an ore sample as an example, the target sample size is approximately 85mm × 62mm × 48mm, and approximately 4.8 × 10⁻⁶ mm is extracted. 4 Based on one valid point cloud dataset, the system generates 15 candidate grasping regions from the top, sidewalls, and edge areas of the sample.

[0037] Subsequently, a robot operating space model is established based on the robot's current physical state. The robot controller acquires the robot chassis pose, the state of each joint of the robotic arm, and the attitude of the end effector in real time, and determines the robot's current physical operating space by combining the robot's joint range of motion, the end effector's gripping direction, and the currently accessible space. In this embodiment, the robot's current operable space is approximately 0.86m. 3 .

[0038] Furthermore, the multidimensional feature representation of the target sample is mapped to the robot's embodied operational space, and the spatial correspondence between each candidate grasping region and the robot is calculated. Among them, reachability reflects whether the robot's end effector can successfully reach the corresponding grasping region; posture continuity reflects the smoothness of motion required for the robot to adjust from the current posture to the target grasping posture; and contact stability reflects the ability of the target sample to establish stable contact in the corresponding grasping region.

[0039] This invention proposes a comprehensive evaluation model for grasping poses, which calculates the comprehensive evaluation value of candidate grasping poses based on accessibility, posture continuity, and contact stability. In this embodiment, the comprehensive evaluation values ​​of 15 candidate grasping poses are calculated, among which the grasping pose with the highest comprehensive evaluation value is 0.86. The system presets the grasping evaluation threshold to 0.75, so the candidate grasping pose with a comprehensive evaluation value of 0.86 is selected as the target grasping pose.

[0040] Specifically, the collaborative operation planning module is used to generate a collaborative operation plan for the robot's movement path and end effector trajectory based on the grasping pose, robot body state and environmental constraint information, and to synchronously control the robot's body movement and end effector actions.

[0041] Furthermore, a robot body movement path is generated based on the target grasping pose, the robot's current position, and environmental constraint information, and a grasping trajectory of the end effector is generated based on the target grasping pose. The robot body movement path is divided into multiple continuous motion stages, and an end effector posture constraint sequence is established based on the robot pose of each motion stage. Based on the end effector posture constraint sequence, the grasping trajectory is segmented and mapped, and the end effector synchronously completes posture adjustment during the robot body movement, establishing a temporal coupling relationship between the robot body movement path and the end effector trajectory. A collaborative operation scheme is generated based on the temporal coupling relationship, and the robot body movement and end effector actions are synchronously controlled based on the collaborative operation scheme.

[0042] It should be noted that in this embodiment, the robot's movement path is generated based on the target grasping pose, the robot's current position, and the environmental constraint information in the multimodal embodied environment model. Simultaneously, the end effector grasping trajectory is generated based on the target grasping pose and the target sample space orientation. The robot's movement path guides the robot chassis and robotic arm to move as a whole to the target work area, while the end effector grasping trajectory controls the end effector to gradually adjust its posture.

[0043] In this embodiment, the robot's current position is 2.8m away from the target sample. Based on the environmental constraints, a robot body movement path with a length of 3.15m is planned, and the corresponding end effector trajectory is generated. Subsequently, the robot body movement path is divided into 12 continuous movement stages according to the movement distance. Each movement stage corresponds to the robot's current pose, robot movement speed, and end effector target pose.

[0044] Furthermore, a corresponding end effector posture constraint sequence is established for each motion stage, enabling the end effector to gradually complete posture adjustment during robot movement, rather than correcting posture after the robot stops moving. For example, when the robot moves to the fourth motion stage, the end effector has completed approximately 35% of the posture adjustment; when it moves to the eighth motion stage, it has completed approximately 75% of the posture adjustment; when the robot reaches the target grasping position, the end effector has completed all posture adjustments and directly enters the grasping state, thereby reducing robot waiting time and improving sampling efficiency.

[0045] Furthermore, this invention establishes a temporal coupling relationship between the robot's body movement path and its end effector trajectory, and uses a cooperative operation evaluation model to calculate the degree of cooperative execution in each motion phase of the robot: in, To collaboratively execute evaluation values, The number of motion stages obtained by dividing the robot's movement path. Let i be the degree of completion of the robot's body motion in the i-th motion stage. Let the end effector's attitude completion be the i-th motion phase. Let be the motion weights of the robot body in the i-th motion stage. Let be the attitude weights of the end effector in the i-th motion phase.

[0046] In this embodiment, the robot's movement path is divided into 12 motion stages. The calculated collaborative execution evaluation value K = 0.91, and the system's preset collaborative execution threshold is 0.85. Since K is greater than the preset collaborative execution threshold, it indicates that the robot's movement and the end effector's posture adjustment have a high degree of consistency. Therefore, a corresponding collaborative operation plan is generated, and the robot's movement and end effector actions are synchronously controlled according to the collaborative operation plan. When the collaborative execution evaluation value is lower than the preset collaborative execution threshold, the robot's movement path or end effector trajectory is readjusted until the collaborative execution requirements are met.

[0047] Specifically, the closed-loop execution control module is used to perform sampling operations according to the collaborative operation plan, collect end contact force information, robot pose information and target sample status information in real time, and make closed-loop adjustments to the robot's motion posture and end contact force based on the end contact force information, robot pose information and target sample status information.

[0048] Furthermore, the contact force deviation between the end effector and the target sample is calculated based on the end-effector contact force information, and the pose deviation between the robot end effector and the target grasping pose is calculated based on the robot pose information. Simultaneously, the target sample state stability coefficient is determined based on the target sample state information. A self-adaptive closed-loop adjustment factor is calculated based on the contact force deviation, pose deviation, and target sample state stability coefficient, wherein the self-adaptive closed-loop adjustment factor satisfies the following: in, It is a self-adaptive closed-loop adjustment factor. For real-time contact force, For the target contact force, For correction factor, For the real-time pose of the robot's end effector, Capture the pose of the target. This is the maximum achievable working radius of the robot. The stability coefficient of the target sample state. , , To adjust the weights.

[0049] The robot's closed-loop control output is calculated based on the aforementioned adaptive closed-loop adjustment factor: in, This is the output for the robot's closed-loop control. For contact force adjustment, This is the robot's end-effector posture compensation amount. For force-controlled gain, This is the attitude control gain.

[0050] It should be noted that in this embodiment, the robot performs the sampling operation according to the collaborative operation plan. When the end effector approaches the target sample, the closed-loop execution control module collects end-effector contact force information, robot pose information, and target sample state information in real time. Specifically, the end-effector contact force information is acquired by a six-dimensional force sensor installed on the end effector, with a sampling frequency of 1000Hz; the robot pose information is output in real time by the robot controller, including the end effector's spatial position and attitude angle, with an update frequency of 500Hz; and the target sample state information is jointly acquired by vision and depth sensors, including sample position, pose, and whether slippage, tilting, or deformation has occurred.

[0051] In this embodiment, when the robot starts to grasp the ore sample, the preset target contact force is 6N, and the force sensor detects a real-time contact force of 5.6N; the spatial deviation between the actual pose of the robot end and the target grasping pose is 12 mm, and the maximum working radius of the robot is 1.5 m; according to the visual and depth fusion analysis, the target sample state stability coefficient is 0.93, indicating that the target sample remains stable with only slight posture changes.

[0052] In this embodiment, the expressions for the embodied adaptive closed-loop adjustment factor are set to α=0.40, β=0.35, and γ=0.25. Substituting these values, we obtain the embodied adaptive closed-loop adjustment factor Φ=0.11. This adjustment factor reflects the current interaction state between the robot and the target sample. When the contact force deviation, pose deviation, or sample state change increases, the adjustment factor increases accordingly; conversely, the adjustment amplitude decreases, thereby preventing the robot from over-adjusting.

[0053] Furthermore, in this embodiment, based on the control output results, the robot's end-effector contact force and end-effector posture are simultaneously corrected, adjusting the real-time contact force from 5.6 N to 5.98 N and reducing the end-effector posture deviation from 12 mm to 1.5 mm. Throughout the adjustment process, the robot can complete posture correction and force control compensation without stopping its movement, ultimately achieving stable grasping of the target sample.

[0054] Specifically, the adaptive strategy update module is used to evaluate the current sampling execution status based on the closed-loop adjustment results. When a change in the target sample status or environmental constraints is detected, the multimodal embodied environment model is updated, and the grasping pose determination, collaborative operation scheme generation, and closed-loop adjustment process are re-executed until the target sample sampling is completed.

[0055] Furthermore, based on the robot's closed-loop control results, the robot's motion trajectory, target sample state, and environmental state are obtained, and the robot's motion deviation, target sample state deviation, and environmental constraint deviation are calculated respectively. Based on these deviations, an embodied experience increment model is established, and the embodied experience increment is calculated. ,in, For embodied experience increment, Due to environmental constraint deviation, For robot motion deviation, For the target sample state deviation, Environmental constraint impact coefficient, To perform credibility coefficients for robots, This is the sensitivity coefficient for the sample state.

[0056] It should be noted that in this embodiment, after the robot completes a sampling operation, the adaptive strategy update module evaluates the sampling execution status based on the control results output by the closed-loop execution control module. During the evaluation process, the robot's actual motion trajectory, the target sample state, and the current environmental state are acquired and compared with the planned trajectory, the initial state of the target sample, and the environmental constraint information in the collaborative operation scheme. The robot's motion deviation, the target sample state deviation, and the environmental constraint deviation are then calculated.

[0057] Among them, robot motion deviation is used to reflect the degree of deviation between the robot's actual motion trajectory and the planned trajectory; target sample state deviation is used to reflect the degree of position change, posture change or deformation of the target sample during the sampling process; environmental constraint deviation is used to reflect the changes in the position of obstacles, changes in the work space or changes in environmental constraints during the robot's execution.

[0058] In this embodiment, taking the automatic ore sampling as an example, after the robot completes the first grasp, the average deviation between the robot's actual trajectory and the planned trajectory is detected to be 8.5 mm, and the target sample has a positional shift of about 4.2 mm. During the robot's gripping process, the surrounding sample accumulation state changes, causing changes in the local environmental constraints.

[0059] In this embodiment, based on historical execution data, it is found that changes in environmental constraints have a relatively small impact on continuous sampling tasks; therefore, the environmental constraint impact coefficient is set to 0.25. Robot motion deviation directly affects subsequent trajectory planning; therefore, the robot execution reliability coefficient is set to 0.45. Changes in the target sample state directly affect the determination of the next grasping pose; therefore, the sample state sensitivity coefficient is set to 0.30. The calculated embodied experience increment E is... inc =0.38.

[0060] Subsequently, the adaptive strategy update module updates only the local regions in the multimodal embodied environment model that have changed, based on the embodied experience increment. Specifically, when the robot's motion deviation exceeds a preset threshold, the robot's embodied operating space and motion capability parameters are updated; when the target sample's state deviation exceeds a preset threshold, the target sample's pose and contact constraint information are updated; when the environmental constraint deviation exceeds a preset threshold, the environmental constraint information of the corresponding region is updated, while the unchanged environmental regions remain unchanged. This avoids rebuilding the entire environment model and improves the efficiency of environment model updates.

[0061] After the update, the system re-executes the target grasping pose determination, collaborative operation scheme generation, and closed-loop execution control process based on the new multimodal embodied environment model. This allows the robot to adaptively correct subsequent grasping strategies based on the embodied experience gained from the previous sampling round. For example, in this embodiment, during the second grasping process, after the robot regenerates the grasping pose, the end effector grasping position deviation decreases from 8.5mm in the first round to 2.1mm, and the success rate of grasping the target sample on the first attempt increases from 92.4% to 98.1%. This demonstrates that the incremental embodied experience can effectively guide subsequent sampling decisions and improve the robot's continuous autonomous operation capability.

[0062] This embodiment also provides a sampling method for embodied intelligent robots based on multimodal perception, including: acquiring visual information, depth information, force information, and robot pose information of the robot's working area; performing spatiotemporal registration on the visual information, depth information, force information, and robot pose information to construct a multimodal embodied environment model containing environmental constraint information and robot embodied state information; extracting appearance features, spatial structure features, and contact features of the target sample based on the multimodal embodied environment model, and establishing a pose mapping relationship between the target sample and the robot's operating space in conjunction with the robot's current embodied state to determine the grasping pose of the target sample; and determining the grasping pose based on the grasping pose, robot embodied state, and environmental constraint information. The system generates a collaborative operation plan for the robot's movement path and end-effector trajectory, and synchronously controls the robot's body movement and end-effector actions. It then performs a sampling operation according to the collaborative operation plan, collecting end-effector contact force information, robot pose information, and target sample state information in real time. Based on the end-effector contact force information, robot pose information, and target sample state information, it performs closed-loop adjustments to the robot's motion posture and end-effector contact force. The sampling execution status is evaluated based on the closed-loop adjustment results. When a change in the target sample state or environmental constraints is detected, the multimodal embodied environment model is updated, and the grasping pose determination, collaborative operation plan generation, and closed-loop adjustment process are re-executed until the target sample sampling is completed.

[0063] This embodiment also provides a computer device applicable to the case of a multimodal perception-based embodied intelligent robot sampling system, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the multimodal perception-based embodied intelligent robot sampling system as proposed in the above embodiment.

[0064] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0065] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the embodied intelligent robot sampling system based on multimodal perception as proposed in the above embodiments.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal perception-based embodied intelligent robot sampling system, characterized in that, include: The multimodal embodied environment modeling module acquires visual, depth, force, and robot pose information of the robot's work area, performs spatiotemporal registration of the visual, depth, force, and robot pose information, and constructs a multimodal embodied environment model that includes environmental constraint information and robot embodied state information. The sample pose determination module extracts the appearance, spatial structure, and contact features of the target sample based on the multimodal embodied environment model, and establishes a pose mapping relationship between the target sample and the robot's embodied operation space in conjunction with the robot's current embodied state to determine the grasping pose of the target sample. The collaborative operation planning module generates a collaborative operation plan for the robot's movement path and end-effector trajectory based on the grasping pose, robot embodied state, and environmental constraint information, and synchronously controls the robot's body movement and end-effector actions. The closed-loop execution control module executes the sampling operation according to the collaborative operation plan, collects end-effector contact force information, robot pose information, and target sample state information in real time, and performs closed-loop adjustment of the robot's motion posture and end-effector contact force based on the end-effector contact force information, robot pose information, and target sample state information. The adaptive strategy update module is used to evaluate the current sampling execution status based on the closed-loop adjustment results. When a change in the target sample status or environmental constraints is detected, the multimodal embodied environment model is updated, and the grasping pose determination, collaborative operation scheme generation, and closed-loop adjustment process are re-executed until the target sample is obtained.

2. The embodied intelligent robot sampling system based on multimodal perception as described in claim 1, characterized in that, The process of spatiotemporally registering visual, depth, force, and robot pose information to construct a multimodal embodied environment model containing environmental constraint information and robot embodied state information includes: acquiring the acquisition timestamps of visual, depth, force, and robot pose information; synchronizing different modal information according to a unified time base; establishing spatial coordinate transformation relationships between different modal information based on the robot's body coordinate system; generating a 3D environmental point cloud of the work area based on the synchronized visual and depth information; performing dynamic pose compensation on the 3D environmental point cloud based on the robot's pose information to obtain an environmental space model with consistent robot motion states; mapping force information to contact positions in the environmental space model to generate an environmental contact constraint distribution; and establishing a robot reachable space model based on robot kinematic constraints; performing embodied consistency correlation analysis based on the environmental space model, environmental contact constraint distribution, and robot reachable space model to eliminate environmental information that does not match the robot's motion capabilities, thereby generating a multimodal embodied environment model containing environmental constraint information, contact constraint information, and robot embodied state information.

3. The embodied intelligent robot sampling system based on multimodal perception as described in claim 2, characterized in that, The embodiment consistency correlation analysis based on the environmental space model, environmental contact constraint distribution, and robot reachable space model includes: establishing the correspondence between spatial units in the environmental space model and the robot reachable space, and determining candidate operation areas that the robot can reach; calculating the contact stability index of each spatial unit in the candidate operation area according to the environmental contact constraint distribution, and calculating the operation feasibility index of the spatial unit according to the robot joint motion range, end-effector posture constraints, and collision constraints; correlating and fusing the contact stability index and the operation feasibility index to generate the embodiment operation evaluation value of the spatial unit, and eliminating spatial units below a preset threshold according to the embodiment operation evaluation value; combining the remaining spatial units to form the robot's embodiment operation space, and extracting the environmental constraint information, contact constraint information, and robot embodiment state information of the robot's embodiment operation space to generate a multimodal embodiment environment model.

4. The embodied intelligent robot sampling system based on multimodal perception as described in claim 1, characterized in that, The process of extracting the appearance, spatial structure, and contact features of the target sample based on a multimodal embodied environment model, and establishing a pose mapping relationship between the target sample and the robot's embodied operation space in conjunction with the robot's current embodied state to determine the grasping pose of the target sample includes: acquiring the target sample in the multimodal embodied environment model and extracting its appearance, spatial structure, and contact features to construct a multidimensional feature representation of the target sample; determining candidate grasping regions of the target sample based on the robot's embodied operation space, and establishing a robot operation space model based on the robot's current pose, joint motion state, and end effector posture; mapping the multidimensional feature representation of the target sample to the robot operation space model, calculating the spatial relationship between the target sample and the robot's end effector, and generating multiple candidate grasping poses; comprehensively evaluating the reachability, posture continuity, and contact stability corresponding to each candidate grasping pose, selecting target grasping poses that meet preset evaluation conditions, and outputting the target grasping pose.

5. The embodied intelligent robot sampling system based on multimodal perception as described in claim 1, characterized in that, The collaborative operation scheme for generating the robot's movement path and end-effector trajectory, and synchronously controlling the robot's motion and end-effector actions, includes: generating the robot's movement path based on the target grasping pose, the robot's current position, and environmental constraints; generating the end-effector's grasping trajectory based on the target grasping pose; dividing the robot's movement path into multiple continuous motion stages, and establishing an end-effector attitude constraint sequence based on the robot's pose in each motion stage; segmenting and mapping the grasping trajectory based on the end-effector attitude constraint sequence, with the end-effector synchronously adjusting its attitude during the robot's movement, establishing a temporal coupling relationship between the robot's movement path and the end-effector trajectory; generating a collaborative operation scheme based on the temporal coupling relationship, and synchronously controlling the robot's motion and end-effector actions based on the collaborative operation scheme.

6. The embodied intelligent robot sampling system based on multimodal perception as described in claim 1, characterized in that, The closed-loop adjustment of the robot's motion posture and end-effector contact force based on end-effector contact force information, robot pose information, and target sample state information includes: calculating the contact force deviation between the end-effector and the target sample based on the end-effector contact force information, calculating the pose deviation between the robot end-effector and the target grasping pose based on the robot pose information, and determining the target sample state stability coefficient based on the target sample state information; calculating a body-adaptive closed-loop adjustment factor based on the contact force deviation, pose deviation, and target sample state stability coefficient, wherein the body-adaptive closed-loop adjustment factor satisfies: ,in, It is a self-adaptive closed-loop adjustment factor. For real-time contact force, For the target contact force, For correction factor, For the real-time pose of the robot's end effector, Capture the pose of the target. This is the maximum achievable working radius of the robot. The stability coefficient of the target sample state. , , To adjust the weights; calculate the robot's closed-loop control output based on the specific adaptive closed-loop adjustment factor. ,in, This is the output for the robot's closed-loop control. For contact force adjustment, This is the robot's end-effector posture compensation amount. For force-controlled gain, This is the attitude control gain.

7. The embodied intelligent robot sampling system based on multimodal perception as described in claim 1, characterized in that, The evaluation of the sampling execution status based on the closed-loop adjustment results, and the updating of the multimodal embodied environment model when a change in the target sample state or environmental constraints is detected, includes: obtaining the robot's motion trajectory, target sample state, and environmental state based on the robot's closed-loop control results, and calculating the robot motion deviation, target sample state deviation, and environmental constraint deviation respectively; establishing an embodied experience increment model based on the robot motion deviation, target sample state deviation, and environmental constraint deviation, and calculating the embodied experience increment. ,in, For embodied experience increment, Due to environmental constraint deviation, For robot motion deviation, For the target sample state deviation, Environmental constraint impact coefficient, To perform credibility coefficients for robots, This is the sensitivity coefficient for the sample state.

8. A sampling method for embodied intelligent robots based on multimodal perception, characterized in that, include: Visual, depth, force, and robot pose information of the robot's work area are acquired. Spatiotemporal registration of these information is performed to construct a multimodal embodied environment model that includes environmental constraints and robot embodied state information. Based on this model, appearance, spatial structure, and contact features of the target sample are extracted. A pose mapping relationship between the target sample and the robot's operating space is established using the robot's current embodied state to determine the target sample's grasping pose. Based on the grasping pose, robot embodied state, and environmental constraints, the robot's movement path and end effector are generated. The system employs a collaborative operation scheme for trajectory determination and synchronizes the robot's body motion with its end effector. It performs sampling operations according to the scheme, collecting end-effector contact force information, robot pose information, and target sample state information in real time. Based on these information, it performs closed-loop adjustments to the robot's motion posture and end-effector contact force. The system evaluates the sampling execution status based on the closed-loop adjustment results. When a change in the target sample state or environmental constraints is detected, it updates the multimodal embodied environment model and re-executes the grasping pose determination, collaborative operation scheme generation, and closed-loop adjustment process until the target sample is successfully sampled.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the embodied intelligent robot sampling system based on multimodal perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the embodied intelligent robot sampling system based on multimodal perception as described in any one of claims 1 to 7.

Images