Mechanical arm grabbing method and system based on multi-modal information fusion

Abstract

The invention provides a mechanical arm grabbing method and system based on multi-modal information fusion, and belongs to the technical field of robot intelligent control. Comprising the steps that the conversion relation between a camera coordinate system and a mechanical arm base coordinate system is established through camera calibration, a deep learning neural network is used for conducting grabbing pose estimation on an obtained RGB-D image, and multiple candidate grabbing poses are determined; analyzing a natural language instruction input by a user based on a multi-modal large model, and recognizing a target object region from the RGB-D image by combining a target detection and image segmentation technology; based on the obtained candidate grabbing poses and the target object area, an optimal grabbing pose is screened through a scoring mechanism and mapped to a mechanical arm base coordinate system; and then a dynamic grabbing path is generated by adopting an imitation learning algorithm, and the mechanical arm is controlled to execute grabbing operation. Through multi-modal semantic understanding, accurate grabbing of the mechanical arm in a complex environment can be achieved.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control technology, and in particular relates to a robotic arm grasping method and system based on multimodal information fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Robotic intelligent grasping technology has been widely used in industrial automation, logistics sorting, and service robots in recent years. Existing robotic arm grasping solutions mostly rely on vision-based object recognition and localization methods, achieving grasping tasks by pre-setting grasping points in structured environments or relying on template matching. However, these methods suffer from insufficient robustness and poor generalization ability when facing unstructured scenes, stacked objects, or dynamic environments.

[0004] Furthermore, existing robotic arm grasping methods typically rely on a single type of sensor input, lacking the ability to understand higher-level task intentions such as natural language commands, thus failing to achieve a "say and grab" human-computer interaction experience. Although some research has attempted to introduce deep learning for grasping detection, limitations remain in multimodal information fusion and semantic task integration, including low system integration and weak environmental adaptability. Therefore, the main problems with existing technologies are: 1) Grasping solutions that rely on a single visual modality have limited recognition accuracy in complex scenes and are difficult to effectively cope with changes in object material, lighting, and occlusion interference.

[0005] 2) The lack of a multimodal interaction mechanism that can integrate voice, vision and motion control results in insufficient intelligence of the robotic arm system, which cannot adapt to the control requirements of flexibility and human-machine collaboration. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention provides a robotic arm grasping method and system based on multimodal information fusion, which enables precise grasping of the robotic arm in complex environments through multimodal semantic understanding.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a robotic arm grasping method based on multimodal information fusion.

[0008] A robotic arm grasping method based on multimodal information fusion includes: The transformation relationship between the camera coordinate system and the robot arm base coordinate system is established through camera calibration; wherein, the camera calibration is based on multimodal sensor data fusion to achieve high-precision coordinate mapping; An RGB-D image of the target environment is acquired, and a deep learning neural network is used to estimate the grasping pose of the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information; Based on the multimodal large model, the user input natural language commands are parsed, and combined with object detection and image segmentation techniques, the target object region is identified from the RGB-D image, and object mask and semantic label are generated. Based on the obtained candidate grasping poses and target object regions, the optimal grasping pose is selected through a scoring mechanism and mapped to the robot arm's base coordinate system. According to the optimal grasping pose, a dynamic grasping path is generated using an imitation learning algorithm, and the robot arm is controlled to perform the grasping operation.

[0009] Furthermore, during the camera calibration process, the Halcon calibration tool and the ArUcoMark method are used to iteratively optimize the intrinsic and extrinsic parameter matrices, thereby achieving the transformation from the camera coordinate system to the world coordinate system and then to the robot arm base coordinate system.

[0010] Furthermore, the deep learning neural network adopts an encoder-decoder structure to extract multi-scale features of the RGB-D image and output six degrees of freedom parameters of the grasping pose and confidence score.

[0011] Furthermore, the multimodal large model supports voice, image, and text input, and fuses multimodal features through an attention mechanism to achieve semantic parsing of natural language instructions and association with visual data.

[0012] Furthermore, the scoring mechanism includes vertical constraints and stability assessment; wherein the vertical constraints require the grasping direction to be aligned with the normal vector of the target object's surface, and the stability assessment is based on grasping force analysis and historical grasping success rate.

[0013] Furthermore, during the generation of the dynamic grasping path, the imitation learning algorithm trains the policy network using demonstration data to output a smooth and obstacle-avoiding grasping trajectory; simultaneously, reinforcement learning is introduced for online optimization.

[0014] Furthermore, after controlling the robotic arm to perform the grasping operation, the sensor provides feedback on whether the grasping was successful, and a retry mechanism is triggered when the grasping fails, and the grasping path is replanned based on the multimodal large model.

[0015] A second aspect of the present invention provides a robotic arm grasping system based on multimodal information fusion.

[0016] A robotic arm grasping system based on multimodal information fusion includes: The coordinate transformation module is configured to: establish the transformation relationship between the camera coordinate system and the robot arm base coordinate system through camera calibration; wherein, the camera calibration is based on multimodal sensor data fusion to achieve high-precision coordinate mapping; The pose estimation module is configured to: acquire an RGB-D image of the target environment, and use a deep learning neural network to perform grasping pose estimation on the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information. The model parsing module is configured to: parse the natural language commands input by the user based on a multimodal large model, and combine object detection and image segmentation techniques to identify the target object region from the RGB-D image and generate object masks and semantic labels; The grasping path generation module is configured to: based on the obtained candidate grasping poses and the target object region, select the optimal grasping pose through a scoring mechanism and map it into the robot arm base coordinate system; according to the optimal grasping pose, generate a dynamic grasping path using an imitation learning algorithm and control the robot arm to perform the grasping operation. A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a robotic arm grasping method based on multimodal information fusion as described in the first aspect of the present invention.

[0017] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a robotic arm grasping method based on multimodal information fusion as described in the first aspect of the present invention.

[0018] The above one or more technical solutions have the following beneficial effects: 1) This invention establishes the transformation relationship between the camera coordinate system and the robotic arm's base coordinate system through camera calibration. Upon acquiring an RGB-D image of the target environment, a deep learning neural network is used to estimate the grasping pose. Simultaneously, a multimodal large model is used to parse the user's input natural language commands to achieve target object region recognition. Compared to existing technologies, this invention, by fusing depth information from RGB-D images with a deep learning neural network and combining it with a multimodal large model's understanding of semantic context, can achieve high-precision recognition and stable grasping pose estimation of target objects in unstructured environments, improving the robotic arm's adaptability and grasping success rate in complex scenarios.

[0019] 2) By introducing a multimodal large model that supports voice, image and text input, this invention can parse natural language instructions and associate them with visual perception information, realizing semantic-level human-computer interaction and flexible task configuration. This enables the robotic arm to have a better level of intelligence and human-computer collaboration capabilities, thereby adapting to the needs of various flexible production and service scenarios.

[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of a robotic arm grasping method based on multimodal information fusion in Embodiment 1 of the present invention.

[0023] Figure 2 This is a flowchart of camera calibration and coordinate system transformation in Embodiment 1 of the present invention.

[0024] Figure 3 This is a flowchart of the pose estimation process in Embodiment 1 of the present invention. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] Example 1 This embodiment discloses a robotic arm grasping method based on multimodal information fusion.

[0029] like Figure 1 As shown, a robotic arm grasping method based on multimodal information fusion includes: Step S1: Establish the transformation relationship between the camera coordinate system and the robot arm base coordinate system through camera calibration; wherein, the camera calibration is based on multi-modal sensor data fusion to achieve high-precision coordinate mapping; Step S2: Obtain an RGB-D image of the target environment, and use a deep learning neural network to estimate the grasping pose of the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information; Step S3: Based on the multimodal large model, parse the natural language instructions input by the user, and combine object detection and image segmentation technology to identify the target object region from the RGB-D image and generate object mask and semantic label; Step S4: Based on the obtained candidate grasping poses and target object regions, the optimal grasping pose is selected through a scoring mechanism and mapped to the robot arm's base coordinate system; according to the optimal grasping pose, a dynamic grasping path is generated using an imitation learning algorithm, and the robot arm is controlled to perform the grasping operation.

[0030] Based on the above process, this invention, through multimodal semantic understanding, enables robotic arms to achieve precise grasping in complex environments. To facilitate understanding of the technical solution of this invention, the specific implementation methods of this invention will be further explained and described below.

[0031] In step S1, the transformation relationship between the camera coordinate system and the robot arm base coordinate system is established through camera calibration.

[0032] like Figure 2As shown, during camera calibration, the Halcon calibration tool and the ArUcoMark method are used to iteratively optimize the intrinsic and extrinsic parameter matrices, achieving the transformation from the camera coordinate system to the world coordinate system and then to the robot arm's base coordinate system. Camera extrinsic parameter calibration is the core step in establishing the mapping relationship between the image coordinate system and the world coordinate system, and its accuracy directly affects the accuracy of the robot arm's grasping under vision guidance. After data acquisition, the robot arm's pose corresponding to the image is converted to Halcon data format using the robot arm's end-effector pose saving code. This process is a crucial step in building accurate coordinate associations. The Halcon data format, with its unified and standardized specifications, can accurately describe the robot arm's pose information, providing a reliable coordinate basis for subsequent extrinsic parameter calculations and ensuring accurate mapping between the image and the physical space position. During the ArUcoMark acquisition process, the camera continuously captures the calibration board image at a frame rate of 6 Hz and a resolution of 1280*720, dynamically extracting the corner information of the ArUcoMark in real time, and using the PnP (Perspective-n-Point) algorithm to calculate the 3D pose of the calibration board relative to the camera coordinate system. These data are simultaneously saved for subsequent extrinsic parameter matrix calculations. In the actual calibration process, the poses of the calibration board and the end effector of the robotic arm are recorded and saved by sequentially changing the robotic arm's pose. The position transformations cover different areas of the robotic arm's workspace, ensuring that the collected data has sufficient diversity and representativeness. Based on the known pose of the robotic arm's end effector in the base coordinate system, the hand-eye calibration matrix, the observed pose of the ArUcoMark calibration board in the camera coordinate system, and partial poses of the calibration board in the base coordinate system captured by different robotic arm poses, the accuracy of the hand-eye calibration matrix can be determined based on the stability of this pose. ; in, This indicates the pose of the robotic arm's end effector in the base coordinate system. Represents the hand-eye calibration matrix. This indicates the observation pose of the ArUcoMark calibration board in the camera coordinate system. This represents the partial pose of the calibration plate in the base coordinate system as captured by different robotic arm poses.

[0033] The capture pose in the camera coordinate system is a 4×4 rotation matrix consisting of a rotation matrix (3×3) and a translation vector (3×1), i.e.: ; in, This represents the capture pose in the camera coordinate system. This represents the rotation matrix in the camera coordinate system. This represents the translation vector in the camera coordinate system.

[0034] The robotic arm and camera are mounted in an "eye-on-hand" configuration, meaning the camera (equivalent to the eye) is fixed to the end effector of the robotic arm (equivalent to the hand), and the camera is stationary relative to the end effector, moving with the robotic arm. To solve for the grasping pose in the robotic arm's base coordinate system, the transformation matrix from the gripper coordinate system to the actual robotic arm gripper needs to be derived. From the rotation transformation of the matrix, we know that: ; The grasping pose in the camera coordinate system relative to the robotic arm gripper coordinate system (movement) needs to be right-multiplied by the transformation matrix from the inferred gripper to the actual gripper. This allows the six-DOF grasping pose in the camera coordinate system inferred by the Graspnet neural network to be converted into the pose relative to the Xarm6 robotic arm gripper coordinate system of this invention. ; To determine the pose relative to the robot arm's base coordinate system (fixed), the grasping pose in the camera coordinate system needs to be determined. Multiply by the transformation matrix from the camera to the robotic arm end effector and the transformation matrix from the robotic arm end effector to the base obtained from the hand-eye calibration. The calculation formula is as follows: ; ; ; in, This represents the transformation matrix from the robotic arm's end effector to the base. This represents the transformation matrix from the camera to the end effector of the robotic arm; This represents the rotation matrix from the end effector of the robotic arm to the base. This represents the translation vector from the end effector of the robotic arm to the base; This represents the rotation matrix from the camera to the end effector of the robotic arm. This represents the grasping pose of the robotic arm in the base coordinate system.

[0035] Based on this, the grasping pose of the robotic arm in the base coordinate system can be obtained as follows: ; Furthermore, camera calibration is based on multimodal sensor data fusion to achieve high-precision coordinate mapping. Multimodal sensor data fusion refers to simultaneously utilizing RGB image information from the depth camera, depth point cloud information, and spatial location information from the ArUcoMark label, combining visual texture features with spatial geometric constraints to jointly participate in solving the calibration equations, thereby improving the accuracy and robustness of coordinate system transformation.

[0036] In step S2, an RGB-D image of the target environment is acquired, and a deep learning neural network is used to estimate the grasping pose of the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information.

[0037] The process involves acquiring an RGB-D image (i.e., a depth image) of the target environment. Specifically, this includes simultaneously acquiring a color image, a depth image, and a working space mask, and then registering and aligning the color and depth images. This ensures that the depth and color images match in resolution and maintain spatial alignment. In this embodiment, a Realsense D435 binocular depth camera is used to acquire a 1280*720 resolution color and depth image. After registering the color and depth images using the Align library in OpenCV, these images are used as raw data input to the neural network for estimating and inferring the six-DOF grasping pose.

[0038] like Figure 3 As shown, a deep learning neural network is used to estimate the grasping pose of the obtained RGB-D image, resulting in multiple candidate grasping poses. Specifically, based on the camera intrinsics obtained from Halcon and the depth scaling factor obtained from the RealSense library, point cloud data is created from the depth image to construct the basic point cloud representation of the 3D model. Valid point clouds are obtained through workspace masking and depth value filtering, and then random sampling is performed on these points according to a set number to obtain the final sampled point cloud and its corresponding color information for subsequent processing. The sampled point cloud data is converted into a format suitable for the input network, and a GraspNet network with initialized and pre-trained weights is used to perform forward propagation calculations on the processed input data, obtaining the network output in a gradient-free calculation mode.

[0039] Furthermore, the deep learning neural network employs an encoder-decoder structure to extract multi-scale features from the RGB-D image and output the six degrees of freedom parameters and confidence score of the grasping pose. Specifically, by decoding the network output, the predicted grasping pose is obtained, including the score, grasping position, pose, and grasping width. The 6D grasping poses in the camera coordinate system output by the deep learning GraspNet algorithm are sorted by score and visualized using Open3D, showing the top 40 grasping poses and the optimal grasping pose. When the robotic arm actually performs the grasping task, the inferred optimal grasping pose is calculated in the robotic arm's base coordinate system.

[0040] In step S3, the user's input natural language instructions are parsed based on a multimodal large model, and object detection and image segmentation techniques are combined to identify the target object region from the RGB-D image and generate object masks and semantic labels.

[0041] This embodiment uses a multimodal large model to parse user-input natural language commands. The QwenVL multimodal large model is employed to analyze image information and return corresponding data and text information that can be used for localization and classification. The VLM multimodal large model is used to understand and analyze the user's natural language.

[0042] The multimodal large model supports speech, image, and text input, and fuses multimodal features through an attention mechanism to achieve semantic parsing of natural language instructions and association with visual data. Internally, text instructions are converted into text feature sequences through a word embedding layer, and RGB images are converted into image feature sequences through a visual encoder. Through a cross-modal attention mechanism, the model calculates the relevance weights between text and image features, achieving semantic alignment. Finally, the model outputs the feature representation of the image region most relevant to the instruction. This feature drives subsequent object detection and segmentation modules, accurately locating the pixel region containing the "red cup" and outputting its corresponding binary mask and the semantic label "cup."

[0043] In step S4, based on the obtained candidate grasping poses and target object regions, the optimal grasping pose is selected through a scoring mechanism and mapped to the robot arm's base coordinate system. According to the optimal grasping pose, a dynamic grasping path is generated using an imitation learning algorithm, and the robot arm is controlled to perform the grasping operation.

[0044] Based on the obtained candidate grasping poses and target object regions, a scoring mechanism is used to select the optimal grasping pose, which is then mapped to the robotic arm's base coordinate system. Specifically, the grasping generation network takes 3D point cloud data as input and performs feature extraction and hierarchical learning based on the advanced PointNet++ architecture. A series of convolutional and pooling layers are used to downsample the original point cloud, progressively extracting global and local features. Deconvolutional layers and feature propagation mechanisms are then used for upsampling to restore the point cloud resolution, predicting possible grasping positions, directions, and widths at each point, generating a large number of grasping candidate points. The grasping evaluation network's main function is to evaluate the quality of the numerous grasping candidate points output by the grasping generation network. This network also takes point cloud data and corresponding grasping candidate point features as input, performs feature fusion and classification prediction through a multilayer perceptron (MLP) structure, and outputs a successful grasping probability value for each grasping candidate point, thus selecting the optimal grasping scheme.

[0045] The scoring mechanism used to select the optimal grasping pose includes vertical constraints and stability assessment. The vertical constraints require the grasping direction to align with the normal vector of the target object's surface, while the stability assessment is based on grasping force analysis and historical grasping success rates. Specifically, grasping poses often involve the robotic arm tilting at a large angle and being close to the table. To fully consider the kinematic constraints of the robotic arm itself, environmental space limitations, the actual movement capabilities of the robotic arm, and the physical boundary conditions of the working environment, vertical constraints are required on the grasping pose. This constraint mechanism establishes a mathematical model of the robotic arm's joint range of motion and the safe height of the operating plane, embedding the vertical motion constraints into the grasping pose prediction algorithm, and performing vertical constraint filtering on the grasping pose inferred by the algorithm. The Open3D visualization results clearly show that after vertical constraint optimization, the posture of the robotic arm's end effector is more reasonable, avoiding dangerous poses with large tilt angles, thereby improving the safety and feasibility of the grasping operation. After mapping the selected optimal grasping pose to the robotic arm's base coordinate system, a policy network is trained using demonstration data based on an imitation learning algorithm to output a smooth and obstacle-avoiding grasping trajectory; at the same time, reinforcement learning is introduced for online optimization.

[0046] After the robotic arm performs a grasping operation, sensors provide feedback on whether the grasp was successful. If the grasp fails, a retry mechanism is triggered, and the grasping path is replanned based on a multimodal large model. Sensor feedback includes, but is not limited to: force and torque sensors mounted on the robotic arm's end effector, which detect whether the expected gripping force threshold has been reached to determine if the object has been held; and vision sensors, which capture a new image of the scene after grasping and use a target detection model to determine if the target object has been removed, indirectly determining the success of the grasp. If the grasp fails, the retry mechanism is triggered. When this retry mechanism is triggered, the multimodal large model, combined with the current new environmental observation data (new RGB-D image) and the original natural language instructions, re-executes steps S2 to S4, i.e., re-performs grasping pose estimation, target region identification, optimal pose selection, and path planning, thereby generating a new grasping path with a potentially higher success rate and attempting to grasp again.

[0047] Example 2 This embodiment discloses a robotic arm grasping system based on multimodal information fusion.

[0048] A robotic arm grasping system based on multimodal information fusion includes: The coordinate transformation module is configured to: establish the transformation relationship between the camera coordinate system and the robot arm base coordinate system through camera calibration; wherein, the camera calibration is based on multimodal sensor data fusion to achieve high-precision coordinate mapping; The pose estimation module is configured to: acquire an RGB-D image of the target environment, and use a deep learning neural network to perform grasping pose estimation on the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information. The model parsing module is configured to: parse the natural language commands input by the user based on a multimodal large model, and combine object detection and image segmentation techniques to identify the target object region from the RGB-D image and generate object masks and semantic labels; The grasping path generation module is configured to: based on the obtained candidate grasping poses and the target object region, select the optimal grasping pose through a scoring mechanism and map it into the robot arm base coordinate system; according to the optimal grasping pose, generate a dynamic grasping path using an imitation learning algorithm and control the robot arm to perform the grasping operation. Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0049] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a robotic arm grasping method based on multimodal information fusion as described in Embodiment 1 of this disclosure.

[0050] Example 4 The purpose of this embodiment is to provide an electronic device.

[0051] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a robotic arm grasping method based on multimodal information fusion as described in Embodiment 1 of this disclosure.

[0052] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0053] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0054] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robotic arm grasping method based on multimodal information fusion, characterized in that, include: The transformation relationship between the camera coordinate system and the robot arm base coordinate system is established through camera calibration; wherein, the camera calibration is based on multimodal sensor data fusion to achieve high-precision coordinate mapping; An RGB-D image of the target environment is acquired, and a deep learning neural network is used to estimate the grasping pose of the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information; Based on the multimodal large model, the user input natural language commands are parsed, and combined with object detection and image segmentation techniques, the target object region is identified from the RGB-D image, and object mask and semantic label are generated. Based on the obtained candidate grasping poses and target object regions, the optimal grasping pose is selected through a scoring mechanism and mapped to the robot arm's base coordinate system. According to the optimal grasping pose, a dynamic grasping path is generated using an imitation learning algorithm, and the robot arm is controlled to perform the grasping operation.

2. The robotic arm grasping method based on multimodal information fusion as described in claim 1, characterized in that, During camera calibration, the Halcon calibration tool and the ArUcoMark method are used to iteratively optimize the intrinsic and extrinsic parameter matrices, thereby achieving the transformation from the camera coordinate system to the world coordinate system and then to the robot arm base coordinate system.

3. The robotic arm grasping method based on multimodal information fusion as described in claim 1, characterized in that, The deep learning neural network adopts an encoder-decoder structure to extract multi-scale features of RGB-D images and output six degrees of freedom parameters of the grasping pose and confidence score.

4. The robotic arm grasping method based on multimodal information fusion as described in claim 1, characterized in that, The multimodal large model supports voice, image and text input, and integrates multimodal features through an attention mechanism to achieve semantic parsing of natural language instructions and association with visual data.

5. The robotic arm grasping method based on multimodal information fusion as described in claim 1, characterized in that, The scoring mechanism includes vertical constraints and stability assessment; wherein, the vertical constraints require the grasping direction to be aligned with the normal vector of the target object's surface, and the stability assessment is based on grasping force analysis and historical grasping success rate.

6. The robotic arm grasping method based on multimodal information fusion as described in claim 1, characterized in that, During the generation of the dynamic grasping path, the imitation learning algorithm trains the policy network using demonstration data and outputs a smooth grasping trajectory that avoids obstacles; at the same time, reinforcement learning is introduced for online optimization.

7. The robotic arm grasping method based on multimodal information fusion as described in claim 1, characterized in that, After controlling the robotic arm to perform a grasping operation, the sensor provides feedback on whether the grasping was successful. If the grasping fails, a retry mechanism is triggered, and the grasping path is replanned based on a multimodal large model.

8. A robotic arm grasping system based on multimodal information fusion, characterized in that, include: The coordinate transformation module is configured to: establish the transformation relationship between the camera coordinate system and the robot arm base coordinate system through camera calibration; wherein, the camera calibration is based on multimodal sensor data fusion to achieve high-precision coordinate mapping; The pose estimation module is configured to: acquire an RGB-D image of the target environment, and use a deep learning neural network to perform grasping pose estimation on the obtained RGB-D image to obtain multiple candidate grasping poses; wherein, the grasping pose estimation integrates spatial geometric features and semantic context information. The model parsing module is configured to: parse the natural language commands input by the user based on a multimodal large model, and combine object detection and image segmentation techniques to identify the target object region from the RGB-D image and generate object masks and semantic labels; The grasping path generation module is configured to: based on the obtained candidate grasping poses and the target object region, select the optimal grasping pose through a scoring mechanism and map it into the robot arm base coordinate system; according to the optimal grasping pose, generate a dynamic grasping path using an imitation learning algorithm and control the robot arm to perform the grasping operation.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the robotic arm grasping method based on multimodal information fusion as described in any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the robotic arm grasping method based on multimodal information fusion as described in any one of claims 1-7.

Images