A robot arm push-grasp collaborative operation system based on language interaction and its control method

Abstract

The present invention discloses a robot arm push-grasp cooperative operation system based on language interaction, comprising: an image data acquisition module for acquiring a color image I of a working area; rgb and depth image I depth ; Visual language localization module, used to input scene color image I rgb and natural language description L, effectively mapping images and language to target masks M; driving the decision module for inputting scene color images I rgb and depth image I depth , using the DQN reinforcement learning algorithm to output the pushing action to be performed by the robot; the grasping action prediction module is used to input the target mask M, thereby outputting the predicted grasping action of the robot; the robot motion control module is used to plan the motion trajectory after receiving the target position output by the algorithm, and then output the control signal; the robot grasping unit is used to receive the control signal and move according to the planned motion trajectory to grasp or push the object. The system's linear fitting grasping method based on the target mask greatly improves the success rate of target-oriented grasping.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm grasping control, and in particular to a robot arm push-grabbing collaborative operation system based on language interaction and a control method thereof. Background Art

[0002] Intelligent grasping technology of robotic arms has attracted much attention in the fields of home assistants and service robots. However, most existing robotic grasping technologies are mainly suitable for grasping posture detection of single objects and lack human-computer interaction capabilities. In real environments, it is of great significance to guide robots to understand and perform various target-level grasping tasks through natural language, such as taking specific tools from a desktop, organizing items, or performing simple assembly work. However, current technology cannot adapt to complex environments composed of multiple objects stacked together, in which case the target in the language is invisible. The present invention proposes a robotic arm push-grasp collaborative operation system based on language interaction and its control method. The system introduces visual grounding to understand language instructions and locate the target object at the same time. By pushing to separate the target from the complex scene, the success rate of target-oriented grasping in a cluttered environment is greatly improved, which has important theoretical value and practical significance. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention proposes a robot arm push-grasp collaborative operation system based on language interaction and a control method thereof, and a straight line fitting grasping method based on a target mask greatly improves the success rate of target-oriented grasping.

[0004] In order to solve the above technical problems, the technical solution of the present invention is:

[0005] A robot arm push-grasp collaborative operating system based on language interaction, comprising:

[0006] Image data acquisition module, used to collect color images of the working area rgb and depth image I depth The data collected is collected by Realsense D435 camera, and the resolution of the collected data is 640×480. The depth map is normalized and its value range is scaled to between 0-255. The camera is installed at the end of the robotic arm and collects images from top to bottom;

[0007] The visual language localization module is used to input the scene color image I rgb and natural language description L, using the visual grounding model to effectively map the image and language to the target mask M;

[0008] The driving decision module is used to input the scene color image I rgb and depth image I depth, and scale the resolution to 224×224, using the DQN reinforcement learning algorithm to output the push action to be performed by the robot arm;

[0009] The grasping action prediction module is used to input the target mask M, continuously update the parameters based on the least squares method, obtain the straight line Y that fits the target mask M, fuse the information features of the straight line and the mask to represent the object distribution, and thus output the predicted robotic arm grasping action;

[0010] The robot arm motion control module is used to plan the motion trajectory after receiving the target position output by the algorithm, and then output the control signal;

[0011] The robot arm grasping unit is used to receive control signals, move along the planned motion trajectory, and grasp or push objects.

[0012] Preferably, the image data acquisition module is further used to acquire the color image I of the acquisition working interval. rgb and depth image data I depth Perform preprocessing.

[0013] Preferably, the robotic arm grasping unit includes a UR10 six-degree-of-freedom robotic arm, a Robotiq two-finger gripper and an end force sensor. The robotic arm plans a motion trajectory after receiving the target position output by the algorithm, and completes the grasping or pushing of the object together with the two-finger gripper. The end force sensor is used to measure the applied force when controlling the closing of the two-finger gripper to prevent the object from slipping or deforming.

[0014] The present invention also provides a control method for a robot arm push-grasp cooperative operation system based on language interaction, comprising the following steps:

[0015] Step 1: Initialize the robotic arm and move it to the starting position;

[0016] Step 2: The image data acquisition module acquires the color image I of the working area. rgb and depth image I depth And pre-process, and then save it locally;

[0017] Step 3: Input color image I rgb And natural language description L, use the Referring Transformer network to perform the visual grounding task, so as to obtain the target mask M corresponding to the natural language;

[0018] Step 4: Determine the feasibility of grasping according to the morphological integrity of the target mask M and the surrounding blank area;

[0019] Step 5: Perform grasp prediction based on the target mask M and the grasp feasibility judgment result;

[0020] Step 6: Use ROS to control the robot arm to plan the trajectory and execute it, completing the target-oriented grasping in a cluttered environment based on language interaction.

[0021] Preferably, the Referring Transformer network uses Resnet101 and BERT large language model as the backbone network.

[0022] Preferably, in step 3, the method for obtaining the target mask M corresponding to the natural language is:

[0023] Step 3.1: Extract image and text features through Resnet101 and BERT large language models respectively, and extract target noun phrase features using part-of-speech tagging library based on text features, and extract feature vectors of a certain number of words before and after the target noun to form phrase-specific context information;

[0024] Step 3.2, extract image and text features and fuse them through the visual language encoder to obtain visual language multimodal features;

[0025] Step 3.3: Use the vector obtained by projecting the phrase features and phrase-specific context information through the linear layer as a reference query;

[0026] Step 3.4: The reference query and visual language multimodal features are fused and input into the segmentation task head, and the features of different stages of Resnet101 are skipped to obtain global and local information.

[0027] Step 3.5: Finally, the upsampled feature map is converted into a single-channel heat map to obtain the target mask M corresponding to the natural language.

[0028] Preferably, in step 4, the feasibility of grasping is determined by the integrity of the target mask M and whether there is a blank area around it for the gripper to extend into.

[0029] Preferably, in step 5, based on the grab prediction of the mask, after pushing, the target mask is clearly visible and there is a blank area around the object, which meets the grab condition. The image processing technology and the straight line fitting method are used to predict the grab pixel position (x g ,y g ) and the gripping angle θ g . It includes mask data processing, linear fitting parameter update and capture information extraction. The specific method is:

[0030] First, use median filtering to eliminate Gaussian noise and error data in the original mask data, retain the edge information of the object and detect its contour C as a sample point set for straight line fitting, and randomly initialize the straight line Y=kx+b, where k and b are the slope and intercept of the straight line;

[0031] The contour is composed of a series of pixel points, which provide information such as the shape, size and direction of the outer edge of the object. Using contour C as a sample point set can effectively reflect the overall distribution trend of the object, and the data size is small and convenient for calculation. Using contour C as a sample point set, continuously traverse the data points on the sample set, calculate the mean square error to represent the error coefficient between the sample set and the target line, iterate and update the parameters, and finally minimize the distance between the sample point and the fitting line, so as to determine k and b.

[0032]

[0033] Where T represents the objective function under the residual sum of squares, (x i ,y i ) represents the i-th sample data on contour C;

[0034] Finally, the intersection point (x 1 ,y 1 )...(x j ,y j )...(x m ,y m ), the calculation formula is as follows:

[0035]

[0036] θ=arctan(k) / π×180

[0037] Find the average value as the final captured pixel position (x g ,y g ), the straight line fits the distribution trend of the object, and the arc tangent value of its slope (converted to angle system) is used as the grasping angle θ g .

[0038] Preferably, in step 5, if the grabbing feasibility is judged not to meet the grabbing conditions, the color image I rgb and depth image I depth As input, the action decision is reinforced by a deep reinforcement learning-based driving action neural network, wherein the deep reinforcement learning-based driving action neural network includes: a multimodal information fusion layer, a feature extraction layer and an upsampling layer, and the feature extraction layer is composed of a Batch Normalization layer, a Rectified Linear Unit layer and a Convolutional 2D layer.

[0039] Preferably, the reinforcement learning method is:

[0040] The color image I rgb and depth image I depthThe convolutional layer of the Densenet-121 network pre-trained on ImageNet is used for feature extraction to obtain the color feature map I rgb_feat And the deep feature map I depth_feat The fusion feature map I is obtained by splicing the channels fusion_feat ;

[0041] I fusion_feat After two convolutional units, the image is upsampled and restored to a resolution of 224×224, and the network prediction graph Q(s t ,a;θ * ), where θ * represents the network parameters, a represents the action space of the driving action; the color map I rgb and depth map I depth S t , as the state space, where t represents the current time.

[0042] The network prediction graph Q(s) with a resolution of 224×224 t ,a;θ * ) constitutes a huge action set, which increases the complexity of the network and the difficulty of training. Threshold processing is performed on the depth map to extract and segment the foreground information. In fact, the network only needs to focus on the surrounding area of ​​the object. In order to reduce the ineffective exploration of a large number of blank backgrounds, the target contour is used as an effective prior constraint action space. depth Edge detection and binarization processing are used to obtain a contour image I with unit values ​​only at the object contour. outline , the network prediction graph Q(s t ,a;θ * ) and contour map I outline Channel multiplication operation, I outline The value of the blank background is zero, and finally there is only a predicted value at the object outline;

[0043] Perform argmax operation on each channel of the predicted value and compare them to obtain the maximum action prediction value a t , that is, push the pixel position (x p ,y p ) and the driving angle θ p .

[0044] Preferably, the specific method of step 6 is:

[0045] Predicted grabbing / pushing pixel action information (x pix ,y pix ,θ), combined with the depth map I depth The three-dimensional coordinates (x cam ,y cam ,zcam ), and obtain the coordinates in the real world through hand-eye calibration (x world ,y world ,z world ), and with the help of ROS service communication, it is sent to MoveIt to plan the path of the robot arm's movement, and finally control the opening and closing of the gripper to achieve push-grasp coordination.

[0046] As an example, the three-dimensional coordinates (x cam ,y cam ,z cam ), the expression is as follows:

[0047]

[0048] Where (f x ,f y ) and (c x ,c y ) represents the focal length and center pixel coordinates in the camera intrinsic parameters.

[0049] As a preferred method, the coordinates (x world ,y world ,z world ), the expression is as follows:

[0050]

[0051] Represents the rotation and translation transformation matrix obtained after hand-eye calibration, 0 1×3 represents a zero vector with one row and three columns.

[0052] The present invention has the following characteristics and beneficial effects:

[0053] Using the above technical solution, the present invention uses visual grounding technology to parse natural language to locate the target object, and uses deep reinforcement learning methods to guide the intelligent agent to learn autonomous pushing skills. Through the contour-based action space prior constraints, the trial and error cost of invalid exploration is effectively reduced, and the training efficiency is significantly improved, so that the network can focus on performing pushing actions in tight areas in complex stacking environments. In addition, the straight line fitting grasping method based on the target mask greatly improves the success rate of target-oriented grasping. The system adopts a modular design with a clear structural hierarchy and has important theoretical value and practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 The figure is a flow chart of a control method according to an embodiment of the present invention.

[0056] Figure 2 For Figure 1 System hardware diagram of the control method. DETAILED DESCRIPTION

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0059] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0060] Example 1

[0061] This embodiment provides a robot arm push-grasp collaborative operating system based on language interaction, such as Figure 1 and Figure 2 As shown, the system includes: an image data acquisition module, a visual language positioning module, a push action decision module, a grasping action prediction module and a robotic arm motion control module.

[0062] Specifically, the image data acquisition module uses a Realsense D435 camera to collect color images of the working area. rgb and depth image I depth , with a resolution of 640×480. The depth map is normalized and its value range is scaled to between 0 and 255. The camera is installed at the end of the robotic arm to collect images from top to bottom.

[0063] Visual language localization module: Input scene color image I rgb and natural language description L, and utilize the visual grounding model to efficiently map the image and language to the target mask M.

[0064] Driving decision-making module: Input scene color image I rgb and depth image I depth , and scale the resolution to 224×224, using the DQN reinforcement learning algorithm to output the pushing action to be performed by the robotic arm.

[0065] Grasping action prediction module: Input the target mask M, continuously update the parameters based on the least squares method, and obtain the straight line Y that fits the target area M. The information features of the straight line and the mask are fused to represent the object distribution, thereby outputting the predicted robotic arm grasping action.

[0066] Robotic arm motion control module: includes UR10 six-degree-of-freedom robot arm, Robotiq two-finger gripper and end force sensor. The robot arm plans the motion trajectory after receiving the target position output by the algorithm, and works with the two-finger gripper to grasp or push the object. The end force sensor is used to measure the force applied when the two-finger gripper is closed to prevent the object from slipping or deforming.

[0067] Example 2

[0068] This embodiment provides a control method for a robot arm push-grasp collaborative operation system based on language interaction, such as Figure 1 As shown, it is used to perform the task of grabbing a specified target in natural language. The main technical solution adopted by the present invention includes the following steps:

[0069] Step 1: Initialize the robot arm, and the image data acquisition module collects the color image I of the working area rgb and depth image I depth And preprocess it, then save it locally. After entering the natural language, go to step 2 to guide the robot arm to grasp the target.

[0070] Step 2: Visual language localization, using the Referring Transformer network to perform visual grounding tasks, processing stored image data and language text descriptions. The reference mask of each phrase expression is predicted by matching specific areas in the image with corresponding elements in the text description. Referring Transformer network is a single-stage learning method. Resnet101 and BERT large language model are selected as the backbone to extract image and text features respectively. The two data are fused through the visual language encoder to obtain multimodal features. The vector obtained by projecting the phrase features and phrase-specific context information through the linear layer is used as the reference query, and the target mask is focused in the cross-modal attention mechanism. The reference query and visual language multimodal features are fused and input into the segmentation task head, and the features of different stages of Resnet101 are skipped to obtain global and local information. Finally, the upsampled feature map is converted into a single-channel heat map to obtain the target mask M corresponding to the natural language.

[0071] The feasibility of grasping is determined based on the morphological integrity of the target mask M and the surrounding blank areas. If the mask is complete and there is a blank area around it for the gripper to extend into, the grasping conditions are met, and the process can proceed to step 3 for mask-based grasping prediction.

[0072] Step 3: Grasp prediction based on the mask. After pushing, the target mask is clearly visible and there is a blank area around the object, which meets the grasping conditions. Image processing technology and straight line fitting method are used to predict the grasping pixel points (x g ,y g ) and the gripping angle θ g . It includes mask data processing, linear fitting parameter update and capture information extraction.

[0073] 3.1 Mask denoising and contour detection

[0074] Use median filtering to eliminate Gaussian noise and error data in the original mask data, retain the edge information of the object and detect its contour C as a sample point set for line fitting. Randomly initialize the line Y = kx + b, k and b are the slope and intercept of the line.

[0075] 3.2 Iteratively update the straight line parameters to fit the target area

[0076] The contour is composed of a series of pixels, which provide information such as the shape, size and direction of the outer edge of the object. Using the contour C as a sample point set can effectively reflect the overall distribution trend of the object, and the data size is small and convenient for calculation. The data points on the sample set are continuously traversed, and the mean square error is calculated to represent the error coefficient between the sample set and the target line. The parameters are updated iteratively, and finally the distance between the sample point and the fitting line is minimized, thereby determining k and b.

[0077]

[0078] Where T represents the objective function under the residual sum of squares, (x i ,y i ) represents the i-th sample data on contour C.

[0079] 3.3 Fusion of line and mask information to determine capture configuration

[0080] Calculate the intersection point (x 1 ,y 1 )...(x j ,y j )...(x m ,y m ), and find the average value as the final grasping position (x g ,y g). The straight line fits the distribution trend of the object, and the arc tangent value of its slope is converted into an angle to make the grab angle θ g .

[0081]

[0082] Step 4: Push-grasp unified modeling motion control, the robot arm motion control module receives the algorithm's predicted action information (x pix ,y pix ,θ), combined with the depth map I depth The three-dimensional coordinates (x cam ,y cam ,z cam ), and obtain the coordinates in the real world through hand-eye calibration (x world ,y world ,z world ), and with the help of ROS service communication, it is sent to MoveIt to plan the path of the robot arm's movement, and finally control the opening and closing of the gripper to achieve push-grasp coordination.

[0083] It should be noted that x pix and pix They refer to the two-dimensional pixel positions respectively.

[0084] The matrix form is expressed as follows, where (f x ,f y ) and (c x ,c y ) represents the focal length and center pixel coordinates in the camera intrinsic parameters. Represents the rotation and translation transformation matrix obtained after hand-eye calibration. 1×3 represents a zero vector with one row and three columns.

[0085]

[0086] Example 3

[0087] The difference between this embodiment and embodiment 2 is that in step 2, if only a local target mask is segmented, it indicates that there is occlusion in the surrounding environment, the target object is interfered by other objects, and the grasping condition is not met. It is necessary to train the network through reinforcement learning so that the robot can learn to push and separate the target object in complex scenes to improve the feasibility of grasping and avoid grasping failure caused by physical collision factors. After the algorithm outputs the action information of pushing or grasping, it goes to step 4 motion control, uses ROS to control the robot arm to plan the trajectory and execute it, and completes the target-oriented grasping in a cluttered environment based on language interaction.

[0088] Specifically, the action decision under reinforcement learning is to convert the color image I rgb and depth map I depth St , as the state space, where t represents the current time. Action space A t Contains the two-dimensional pixel point (x, y) representing the push position and the push angle θ.

[0089] (1) Design of network structure

[0090] In this embodiment, the reinforcement learning training network adopts a deep reinforcement learning-based boosting neural network including: a multimodal information fusion layer, a feature extraction layer, and an upsampling layer. rgb and depth map I depth The convolutional layer of the Densenet-121 network pre-trained on ImageNet is used for feature extraction to obtain the color feature map I rgb_feat And the deep feature map I depth_feat The fusion feature map I is obtained by splicing the channels fusion_feat The feature extraction convolution unit is composed of Batch Normalization (batch normalization layer), Rectified Linear Unit (linear activation layer) and Convolutional 2D (two-dimensional convolution layer). fusion_feat After two convolutional units, the image is upsampled and restored to a resolution of 224×224, and the network prediction graph Q(s t ,a;θ * ). where θ * represents the network parameters, and a represents the action space of the driving action. The robot continuously optimizes its behavior through the reward function. The structure of the network is formulated as follows:

[0091]

[0092] Q(s t ,a;θ * )=F upsample [I conv_2 (I conv_1 )]=F upsample [I conv_2 (γgσ(BN(I fusion_feat )))]

[0093] in Indicates splicing along the channel, F conv represents the convolution processing of the Densenet-121 network, BN(g) represents BatchNormalization, σ(g) represents the ReLu activation function, and γ is the parameter to be learned by the network. conv_1 and I conv_2 represents the convolution unit, F upsample represents the upsampling function.

[0094] (2) Design of prior constraints on action space

[0095] The network prediction graph Q(s) with a resolution of 224×224 t ,a;θ * ) constitutes a huge action set, which increases the complexity of the network and the difficulty of training. Threshold processing is performed on the depth map to extract and segment the foreground information. In fact, the network only needs to focus on the surrounding area of ​​the object. In order to reduce the ineffective exploration of a large number of blank backgrounds, the target contour is used as an effective prior constraint action space. depth Edge detection and binarization processing are used to obtain a contour image I with unit values ​​only at the object contour. outline , the network prediction graph Q(s t ,a;θ * ) and contour map I outline Channel multiplication operation, I outline The value of the blank background is zero, and finally there is a predicted value only at the object contour. Perform argmax operation along each channel on the new prediction map and compare them to obtain the maximum action prediction value a t , that is, the two-dimensional pixel point (x p ,y p ) and the driving angle θ p .

[0096]

[0097] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A control method for a robotic arm push-grasp collaborative operation system based on language interaction, It is characterized in that The steps include: Step 1: Initialize the robotic arm and move it to the starting position; Step 2: The image data acquisition module acquires the color image of the working area and depth image And pre-process, and then save it locally; Step 3: Input color image and natural language description , using the Referring Transformer network to perform the visual grounding task, thereby obtaining the target mask corresponding to the natural language ; The target mask corresponding to the natural language The method to obtain is: Step 3.1, extract image and text features through Resnet101 and BERT large language models respectively, and use the part-of-speech tagging library to extract the target noun phrase features based on the text features, and extract the feature vectors of a certain number of words before and after the target noun to form phrase-specific context information; Step 3.2, extract image and text features and fuse them through the visual language encoder to obtain visual language multimodal features; Step 3.3: Use the vector obtained by projecting the phrase features and phrase-specific context information through the linear layer as a reference query; Step 3.4: The reference query and visual language multimodal features are fused and input into the segmentation task head, and the features of different stages of Resnet101 are skipped to obtain global and local information. Step 3.5: Finally, convert the upsampled feature map into a single-channel heat map to obtain the target mask corresponding to the natural language. ; The Referring Transformer network uses Resnet101 and BERT large language model as the backbone network; Step 4: According to the target mask The feasibility of grasping is determined by the morphological integrity and the surrounding blank areas; The feasibility of grasping can be determined by the integrity of the gripper and whether there is a blank area around it for the gripper to extend into. Step 5: According to the target mask And the grasping feasibility judgment results are used to make grasping predictions; The crawl feasibility judgment meets the crawl conditions, and the crawl prediction method is: First, median filtering is used to eliminate Gaussian noise and error data in the original mask data, retain the edge information of the object and detect its contour Make a sample point set for straight line fitting and randomly initialize the straight line , and are the slope and intercept of the straight line; Then, the contour As a sample point set, the data points on the sample set are continuously traversed, the mean square error is calculated to represent the error coefficient between the sample set and the target line, the parameters are updated iteratively, and finally the distance between the sample point and the fitting line is minimized, thereby determining and , ； ； ； Where T represents the objective function under the residual sum of squares, Representing contours The i-th sample data on ; Finally, calculate the intersection of the line and the mask , the calculation formula is as follows: ； ； Find the average value as the final captured pixel position The straight line fits the distribution trend of the object, and the arc tangent value of its slope is converted into an angle system as the grasping angle ; Step 6: Use ROS to control the robot arm to plan the trajectory and execute it, completing the target-oriented grasping in a cluttered environment based on language interaction.

2. According to the control method of a robot arm push-grasp collaborative operation system based on language interaction according to claim 1, It is characterized in that In step 5, the grasping feasibility judgment does not meet the grasping conditions, and the color image and depth image As input, a neural network reinforcement learning action decision is made through a boost based on deep reinforcement learning.

3. According to the control method of the robot arm push-grasp collaborative operation system based on language interaction according to claim 2, It is characterized in that The reinforcement learning method is: Color image and depth image go through The convolutional layer of the pre-trained Densenet-121 network is used for feature extraction to obtain a color feature map. and deep feature maps The fusion feature map is obtained by splicing the back channel ; After two convolutional units, the image is upsampled back to a resolution of 224×224 to obtain the network prediction image. ,in represents the network parameters, The action space for pushing action; From the depth map Edge detection and binarization processing obtain a contour image with unit value only at the object contour , the network prediction graph With contour map Channel multiplication operation, The value of the blank background is zero, and finally there is only a predicted value at the object outline; The predicted value is added to each channel Operate and compare to get the maximum action prediction value , that is, push the pixel position and push angle .

4. According to the control method of the robot arm push-grasp collaborative operation system based on language interaction according to claim 3, It is characterized in that The specific method of step 6 is: Predicted grab / push pixel action information , combined with the depth map And the camera imaging principle to obtain the three-dimensional coordinates in the camera coordinate system , obtain the coordinates in the real world through hand-eye calibration , using ROS service communication to send to Plan the path of the robot arm's movement, and finally control the opening and closing of the gripper to achieve push-grasp coordination.

5. According to the control method of the robot arm push-grasp collaborative operation system based on language interaction according to claim 4, It is characterized in that The three-dimensional coordinates are obtained by calculation , the expression is as follows: ； in and Represents the focal length and center pixel coordinates in the camera intrinsic parameters.

6. According to the control method of the robot arm push-grasp collaborative operation system based on language interaction according to claim 4, It is characterized in that Get the real-world coordinates through hand-eye calibration , the expression is as follows: ； Represents the rotation and translation transformation matrix obtained after hand-eye calibration, represents a zero vector with one row and three columns.

7. A robot arm push-grasp collaborative operation system based on language interaction for implementing the method described in any one of claims 1 to 6, It is characterized in that include: Image data acquisition module, used to collect color images of the working area and depth image ; Visual language localization module, used to input scene color image data and natural language description , using visual grounding models to efficiently map images and language to object masks superior; Driving decision module, for input scene color image and depth image , and scale the resolution to 224×224, using the DQN reinforcement learning algorithm to output the push action to be performed by the robot arm; Grasp action prediction module, used to input target mask , based on the least squares method, continuously update the parameters to obtain the fitted target mask The straight line , the information features of the line and the mask are fused to represent the object distribution, thereby outputting the predicted robot arm grasping action; The robot arm motion control module is used to plan the motion trajectory after receiving the target position output by the algorithm, and then output the control signal; The robot arm grasping unit is used to receive control signals, move along the planned motion trajectory, and grasp or push objects.

8. The robot arm push-grasp collaborative operation system based on language interaction according to claim 7, It is characterized in that The image data acquisition module is also used to acquire the color image of the acquisition working interval and depth image data Perform preprocessing.

9. The robot arm push-grasp collaborative operation system based on language interaction according to claim 7, It is characterized in that The robot arm grasping unit comprises Six-degree-of-freedom robotic arm, Two-finger gripper and end force sensor.

Images