Robotic arm grasping system and method based on multi-modal large model and neural radiance field

Abstract

This invention relates to the field of robotic arm technology, and more particularly to a robotic arm grasping system and method based on a multimodal large model and neural radiation field. The system includes a robotic arm, a PyBullet module, a NeRF module, a network module, a multimodal large model module, a data acquisition module, an input module, and a client. The data acquisition module is mounted on the robotic arm. The PyBullet module is connected to the robotic arm, the NeRF module is connected to the data acquisition module, the network module is connected to the NeRF module, the network module is connected to the PyBullet module, and the multimodal large model module is connected to the network module. The input module is embedded in the client and is connected to the multimodal large model module. This approach solves the technical problem that existing methods cannot obtain the suitable grasping position and posture of the object being grasped, and can only grasp using a preset robotic arm posture.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and in particular to a robotic arm grasping system and method based on a multimodal large model and neural radiation field. Background Technology

[0002] Traditional robotic arms operate by pre-programming the expected trajectory and workflow in a teach pendant for fixed-point grasping. This method only works when the robotic arm is mounted at a fixed point. With the development of machine learning and deep learning, robotic arm grasping has gradually shifted from fixed-point grasping to unordered grasping. With the participation of 2D cameras, 2D vision detection combined with nine-point calibration has solved the problem of grasping some non-fixed objects by performing fixed-height unordered grasping. With the addition of 3D cameras, after performing hand-eye calibration between the robotic arm and the camera, unordered grasping at non-fixed heights has also been improved to some extent.

[0003] In summary, existing methods cannot obtain the appropriate grasping position and posture of the object to be grasped; they can only grasp it using a preset robotic arm posture. Summary of the Invention

[0004] The purpose of this invention is to provide a robotic arm grasping system and method based on a multimodal large model and neural radiation field, which aims to solve the technical problem that existing methods cannot obtain the suitable grasping position and grasping posture of the object to be grasped, and can only grasp it with a preset robotic arm posture.

[0005] To achieve the above objectives, this invention employs a robotic arm grasping system based on a multimodal large model and neural radiation field, comprising a robotic arm, a PyBullet module, a NeRF module, a network module, a multimodal large model module, a data acquisition module, an input module, and a client. The data acquisition module is mounted on the robotic arm, the PyBullet module is connected to the robotic arm, the NeRF module is connected to the data acquisition module, the network module is connected to the NeRF module, the network module is connected to the PyBullet module, the multimodal large model module is connected to the network module, and the input module is embedded in the client and connected to the multimodal large model module.

[0006] The acquisition module is used to acquire images of the working area from multiple perspectives;

[0007] The NeRF module is used to perform 3D reconstruction of the work area image obtained by the acquisition module to obtain the point cloud information of the scene;

[0008] The robotic arm is used to grasp objects.

[0009] The robotic arm grasping system based on multimodal large model and neural radiation field also includes a cleaning module, which is connected to the acquisition module.

[0010] The robotic arm grasping system based on multimodal large model and neural radiation field also includes an adjustment module, which is connected to the client.

[0011] The robotic arm grasping system based on multimodal large model and neural radiation field also includes a login module, a verification module and an alarm module. The login module is embedded in the client, the verification module is connected to the login module, and the alarm module is connected to the verification module.

[0012] This invention also provides a robotic arm grasping method based on a multimodal large model and neural radiation field, employing the robotic arm grasping system based on a multimodal large model and neural radiation field as described above.

[0013] Includes the following steps:

[0014] Step 1: First, obtain simulation data of the object to be grasped through mathematical modeling. Input the object information and the robotic arm information into the simulation environment of the pybullet module. All information of the object can be obtained in the simulation environment.

[0015] Step 2: Acquire images of the working area from multiple perspectives using the acquisition module, and use the Nerf module to perform 3D reconstruction to obtain the point cloud information of the scene;

[0016] Step 3: Input the point cloud information from Step 2 into the network module to obtain the grasping posture of the target object to be grasped;

[0017] Step 4: Combining the grasping posture in Step 3 with the spatial xyz coordinates of the object relative to the robotic arm obtained by the pybullet module, we can obtain the six elements X, Y, Z, A, B, and C required for the robotic arm's movement. The inverse kinematics algorithm is then used to plan the robotic arm's motion trajectory to grasp the object.

[0018] The specific method by which the Nerf module obtains the point cloud information of the scene through 3D reconstruction is as follows: the scene is captured in a series of ways under the existing viewpoint, and then the image under the new viewpoint can be synthesized based only on the pose intrinsic parameters and the image, without the need for an intermediate 3D reconstruction process.

[0019] In the NeRF-based representation, three-dimensional space is represented as a set of learnable and continuous radiation fields, which are learned by inputting viewpoint and position, and finally the density and color are obtained.

[0020] The physical equations corresponding to the rendering of neural radiation fields in three-dimensional space are:

[0021] L0(x,d)=L e (x,d)+∫ Ω f r (x,d,w i )Li(x,w1)cosθdw i ;

[0022] In the formula, the first part is the amount of radiation in the d direction when x is the light source point, and the second part is the amount of radiation in the d direction after the light source at point x illuminates other surfaces.

[0023] The radiation field describes the color and mention density of each point in the scene and in each viewing direction, expressed by the formula:

[0024]

[0025] Where x represents the scene coordinates xyz, θ, σ represents the azimuth and polar angle of the camera, c represents RGB color, and σ represents volume density.

[0026] This invention relates to a robotic arm grasping system and method based on a multimodal large model and neural radiation field. The network module is GraspNet, the multimodal large model module is CLIP, the acquisition module is a 2D camera, and the input module is a Tokenizer. First, simulation data of the object to be grasped is obtained through mathematical modeling. The object information and robotic arm information are input into the simulation environment of the pybullet module, where all information about the object can be obtained. The acquisition module acquires images of the working area from multiple perspectives, and the Nerf module is used for 3D reconstruction. The point cloud information of the scene is obtained and input into the network module to obtain the grasping posture of the target object to be grasped. The grasping posture is combined with the spatial xyz coordinates of the object relative to the robotic arm obtained by the pybullet module to obtain the six elements X, Y, Z, A, B, and C required for the movement of the robotic arm. The inverse kinematics algorithm is used to plan the motion trajectory of the robotic arm to grasp the object. This method solves the technical problem that existing methods cannot obtain the suitable grasping position and grasping posture of the object to be grasped, and can only grasp it with a preset robotic arm posture.

[0027] The present invention has the following beneficial effects: 1. By generating a robotic arm grasping dataset through a simulation environment, the difficulty of dataset collection is reduced; 2. The acquisition module selects a two-dimensional camera, takes pictures from different angles, and then uses neural radiation fields for three-dimensional reconstruction, without using a depth camera, thus reducing costs; 3. Through the Clip multimodal model, the robotic arm has the ability to understand and analyze multimodal information such as images and text information; 4. When the robotic arm reaches a new application scenario, it only needs a small amount of fine-tuning with real-world datasets to quickly meet the application requirements. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the robotic arm grasping system based on a multimodal large model and neural radiation field of the present invention.

[0030] 101-Robotic arm, 102-PyBullet module, 103-NeRF module, 104-Network module, 105-Multimodal large model module, 106-Acquisition module, 107-Input module, 108-Client, 109-Cleaning module, 110-Adjustment module, 111-Login module, 112-Verification module, 113-Alarm module, 114-Sensing module, 115-Adjustment module, 116-Heat dissipation module. Detailed Implementation

[0031] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0032] Please see Figure 1 , Figure 1 This is a schematic diagram of the robotic arm grasping system based on a multimodal large model and neural radiation field of the present invention.

[0033] This invention provides a robotic arm grasping system based on a multimodal large model and neural radiation field, including a robotic arm 101, a PyBullet module 102, a NeRF module 103, a network module 104, a multimodal large model module 105, a data acquisition module 106, an input module 107, and a client 108. The data acquisition module 106 is mounted on the robotic arm. The PyBullet module 102 is connected to the robotic arm 101. The NeRF module 103 is connected to the data acquisition module 106. The network module 104 is connected to the NeRF module 103. The network module 104 is connected to the PyBullet module 102. The multimodal large model module 105 is connected to the network module 104. The input module 107 is embedded in the client 108 and is connected to the multimodal large model module 105.

[0034] The acquisition module 106 is used to acquire images of the working area from multiple perspectives;

[0035] The NeRF module 103 is used to perform three-dimensional reconstruction of the work area image obtained by the acquisition module 106 to obtain the point cloud information of the scene;

[0036] The robotic arm is used to grasp objects.

[0037] In this specific embodiment, the network module 104 is GraspNet, the multimodal large model module 105 is CLIP, the acquisition module 106 is a 2D camera, and the input module 107 is a Tokenizer. First, simulation data of the object to be grasped is obtained through mathematical modeling. The object information and robotic arm information are then input into the simulation environment of the PyBullet module 102. In the simulation environment, all information about the object can be obtained. The acquisition module 106 acquires multi-view images of the working area, and the NeRF module 103 performs 3D reconstruction to obtain the desired data. The point cloud information of the scene is input into the network module 104 to obtain the grasping posture of the target object to be grasped. The grasping posture is combined with the spatial xyz coordinates of the object relative to the robotic arm obtained by the PyBullet module 102 to obtain the six elements X, Y, Z, A, B, and C required for the movement of the robotic arm. The inverse kinematics algorithm is used to plan the motion trajectory of the robotic arm to grasp the object. This method solves the technical problem that existing methods cannot obtain the suitable grasping position and grasping posture of the object to be grasped, and can only grasp it with a preset robotic arm posture.

[0038] The robotic arm grasping system based on multimodal large model and neural radiation field also includes a cleaning module 109, which is connected to the acquisition module 106.

[0039] In this specific embodiment, the cleaning module 109 can clean the acquisition module 106 to ensure the acquisition effect.

[0040] Secondly, the robotic arm grasping system based on multimodal large model and neural radiation field also includes an adjustment module 110, which is connected to the client 108.

[0041] In this specific embodiment, the adjustment module 110 can adjust the height of the client 108.

[0042] Meanwhile, the robotic arm grasping system based on multimodal large model and neural radiation field also includes a login module 111, a verification module 112 and an alarm module 113. The login module 111 is embedded in the client 108, the verification module 112 is connected to the login module 111, and the alarm module 113 is connected to the verification module 112.

[0043] In this specific embodiment, the login module 111 is used to log in to the client 108, and the verification module 112 verifies the identity of the logged-in personnel. If the verification fails three times, an alarm will be triggered through the alarm module 113.

[0044] In addition, the robotic arm grasping system based on multimodal large model and neural radiation field also includes a sensing module 114 and an adjustment module 115. The sensing module 114 is connected to the client 108, and the adjustment module 115 is connected to the sensing module 114.

[0045] In this specific embodiment, the sensing module 114 will sense whether the client 108 is being used. If no one is using it, the brightness of the client 108 will be adjusted by the adjustment module 115.

[0046] Furthermore, the robotic arm grasping system based on multimodal large model and neural radiation field also includes a heat dissipation module 116, which is connected to the client 108.

[0047] In this specific embodiment, the heat dissipation module 116 is used to dissipate the heat generated when the client 108 is working.

[0048] This invention also provides a robotic arm grasping method based on a multimodal large model and neural radiation field, employing the robotic arm grasping system based on a multimodal large model and neural radiation field as described above.

[0049] Includes the following steps:

[0050] Step 1: First, obtain simulation data of the object to be grasped through mathematical modeling. Input the object information and the robotic arm information into the simulation environment of the PyBullet module 102. All information of the object can be obtained in the simulation environment.

[0051] Step 2: Acquire images of the working area from multiple perspectives using the acquisition module 106, and use the NeRF module 103 to perform 3D reconstruction to obtain the point cloud information of the scene;

[0052] Step 3: Input the point cloud information from Step 2 into the network module 104 to obtain the grasping posture of the target object to be grasped;

[0053] Step 4: The grasping posture in Step 3 is combined with the spatial xyz coordinates of the object relative to the robotic arm obtained by the PyBullet module 102 to obtain the six elements X, Y, Z, A, B, and C required for the movement of the robotic arm. The inverse kinematics algorithm is then used to plan the movement trajectory of the robotic arm to grasp the object.

[0054] The present invention has the following beneficial effects: 1. By generating a robotic arm grasping dataset through a simulation environment, the difficulty of dataset collection is reduced; 2. The acquisition module 106 selects a two-dimensional camera, takes pictures from different angles, and then uses a neural radiation field for three-dimensional reconstruction, without using a depth camera, thus reducing costs; 3. Through the Clip multimodal model, the robotic arm has the ability to understand and analyze multimodal information such as images and text information; 4. After the robotic arm reaches a new application scenario, it only needs a small amount of fine-tuning of the real dataset to quickly meet the application requirements.

[0055] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A robotic arm grasping system based on a multimodal large model and neural radiation field, characterized in that, The system includes a robotic arm, a PyBullet module, a NeRF module, a network module, a multimodal large model module, a data acquisition module, an input module, and a client. The data acquisition module is mounted on the robotic arm. The PyBullet module is connected to the robotic arm. The NeRF module is connected to the data acquisition module. The network module is connected to the NeRF module. The network module is connected to the PyBullet module. The multimodal large model module is connected to the network module. The input module is embedded in the client and is connected to the multimodal large model module. The acquisition module is used to acquire images of the working area from multiple perspectives; The NeRF module is used to perform 3D reconstruction of the work area image obtained by the acquisition module to obtain the point cloud information of the scene; The robotic arm is used to grasp objects; The grasping method of the robotic arm grasping system based on multimodal large model and neural radiation field includes the following steps: Step 1: First, obtain simulation data of the object to be grasped through mathematical modeling. Input the object information and the robotic arm information into the simulation environment of the pybullet module. All information of the object can be obtained in the simulation environment. Step 2: Acquire images of the working area from multiple perspectives using the acquisition module, and use the Nerf module to perform 3D reconstruction to obtain the point cloud information of the scene; Step 3: Input the point cloud information from Step 2 into the network module to obtain the grasping posture of the target object to be grasped; Step 4: The grasping posture in Step 3 is combined with the spatial xyz coordinates of the object relative to the robotic arm obtained by the pybullet module to obtain the six elements X, Y, Z, A, B, and C required for the movement of the robotic arm. The inverse kinematics algorithm is then used to plan the movement trajectory of the robotic arm to grasp the object. The Nerf module obtains the point cloud information of the scene through 3D reconstruction in the following way: it captures the scene in a series of ways from the existing viewpoint, and then synthesizes the image from the new viewpoint based only on the pose intrinsic parameters and the image, without the need for an intermediate 3D reconstruction process. In the NeRF-based representation, three-dimensional space is represented as a set of learnable and continuous radiation fields, which are learned by inputting viewpoint and position, and finally the density and color are obtained. The physical equations corresponding to the rendering of neural radiation fields in three-dimensional space are: ; In the formula, the first part is the amount of radiation in the d direction when x is the light source point, and the second part is the amount of radiation in the d direction after the light source at point x illuminates other surfaces. The radiation field describes the color and mention density of each point in the scene and each viewing direction, expressed by the formula: ； Where x represents the scene coordinates xyz. This represents the camera's azimuth and polar angles; 'c' represents RGB color. Represents volume density.

2. The robotic arm grasping system based on multimodal large model and neural radiation field as described in claim 1, characterized in that, The robotic arm grasping system based on multimodal large model and neural radiation field also includes a cleaning module, which is connected to the acquisition module.

3. The robotic arm grasping system based on a multimodal large model and neural radiation field as described in claim 2, characterized in that, The robotic arm grasping system based on multimodal large model and neural radiation field also includes an adjustment module, which is connected to the client.

4. The robotic arm grasping system based on a multimodal large model and neural radiation field as described in claim 3, characterized in that, The robotic arm grasping system based on multimodal large model and neural radiation field also includes a login module, a verification module and an alarm module. The login module is embedded in the client, the verification module is connected to the login module, and the alarm module is connected to the verification module.

Images