Mechanical arm instruction grabbing method based on vision-force sense

Abstract

The invention belongs to the technical field of mechanical arm grabbing, and discloses a visual sense-force sense-based mechanical arm instruction grabbing method, which comprises the following steps of: acquiring an RGB-D image of a scene, performing cutting processing on the RGB-D image through OpenCV, and extracting visual features of each object in the RGB-D image; a text command input by a user is coded through BERT, and language features are output; inputting the language features and the visual features into a CLIP model for fusion and alignment, and performing further alignment by adopting a Co-TRM structure to generate bimodal features; generating a point cloud image according to the depth map of the RGB-D image, and based on the point cloud image, combining visual features of an object, and generating a grabbing attitude through Graspnet; the problem that in the prior art, when grabbing is conducted based on vision and instructions, self-adaptive adjustment on grabbing force is not convenient is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot arm grasping, and in particular relates to a robot arm instruction grasping method based on vision-force perception. Background Art

[0002] With the continuous development of science and technology, robotic arm grasping is an important part of industrial automation production, which can effectively improve production efficiency and quality. In the past, the commonly used robotic arm grasping method was vision-based grasping, and the robot obtained complete information about the object to be grasped. However, in the actual object grasping, among the factors of robotic arm grasping, the operator is more inclined to use text instructions to operate, such as "help me grab an apple". At present, the commonly used visual and text grasping methods mainly use simple feature fusion methods, such as the VL-Grasp method, to splice and fuse vision and text commands. These methods cannot effectively fuse visual and text feature information. In addition, the grasping force of the robotic arm is crucial to achieve stable grasping operations. Appropriate grasping force can ensure that the object will not be damaged, slip off or grasped loosely. The control methods of grasping force are mainly divided into traditional control methods and sensor feedback control. The traditional method relies on the design of the mechanical structure and achieves control by adjusting the opening and closing degree and tightness of the robotic gripper, but this method has low control accuracy. Sensor feedback control achieves precise control of grasping force by installing pressure sensors or force sensors, although this will increase the complexity and cost of the system. When the robot is actually grasping, the target object may slip out due to insufficient grasping force, or fragile objects such as cardboard boxes may be damaged due to excessive grasping force. Therefore, the existing technology has the problem of inconvenience in adaptively adjusting the grasping force when grasping based on vision and instructions. Summary of the invention

[0003] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a robot arm command grasping method based on vision-force perception, which solves the problem in the prior art that it is inconvenient to adaptively adjust the grasping force when grasping based on vision and instructions.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A method for grasping instructions of a robot arm based on vision-force perception comprises the following steps:

[0006] Get the RGB-D image of the scene, cut the RGB-D image through OpenCV, and extract the visual features of each object in the RGB-D image;

[0007] Encode the text commands entered by the user through BERT and output language features;

[0008] The language features and visual features are input into the CLIP model for fusion and alignment, and then further aligned using the Co-TRM structure to generate bimodal features;

[0009] Generate a point cloud image based on the depth map of the RGB-D image. Based on the point cloud image and the visual features of the object, generate the grasping posture through Graspnet.

[0010] Acquire the position features of each object according to the depth image;

[0011] Construct a feature fusion model to fuse the grasping posture, position features and bimodal features and generate multimodal features;

[0012] Based on the SAC algorithm, the feature fusion model is trained through reinforcement learning to optimize and generate the final grasping strategy;

[0013] Adaptively adjust the gripping force of the robot arm and complete object grasping in a multi-object environment.

[0014] Use OpenCV to slice the RGB-D image and extract the visual features of each object in the RGB-D image. The specific steps include:

[0015] Identify objects in the image, cut the RGB-D image according to the object, split the RGB-D image into object bounding boxes centered on the object, and generate a mask for each object;

[0016] Use OpenCV function to analyze the mask and obtain the connected domain information of each object, which includes the index and area of ​​the bounding box;

[0017] Extract the sub-image corresponding to each object from the RGB-D image according to the bounding box, and extract the object features in each sub-image;

[0018] The pixel coordinates of the center point of each bounding box in the image are calculated, and the pixel coordinates of each object are converted into three-dimensional coordinates by combining the pixel size and working space constraints.

[0019] BERT is used to encode the text commands input by the user and output language features. The specific steps include:

[0020] Get the text command input by the user and perform word segmentation;

[0021] For each character / word, three learnable vectors need to be obtained: 1) the character / word vector itself; 2) the position vector; 3) the sentence vector;

[0022] Concatenate three learnable vectors;

[0023] Multiple Transformer encoders are used to process the connected vector sequence to obtain richer text sequence context and semantic information, and finally output language features.

[0024] Let the visual features be I1,I2,...,I n , the language features are T1,T2,...,T n ;

[0025] Input the visual features and language features into the CLIP model to obtain the visual feature I 1a , I 2a ...I na , language feature T 1a , T 2a , ....T na , obtain the preliminary bimodal features by initially integrating and aligning the visual features and language features;

[0026] Then, the preliminary bimodal features are input into the Co-TRM structure, and the key and value of any feature in the visual feature and the language feature are passed as input to the attention module of the other feature. The attention module generates an attention pooling feature based on the condition of the other feature for any feature, and obtains the visual feature I 1b , I 2b ...I nb , language feature T 1b , T 2b , ....T nb , complete the further fusion and alignment of the visual features and language features in the preliminary bimodal features to generate bimodal features.

[0027] The grasping posture, position features and bimodal features are integrated to generate multimodal features. The specific steps are as follows:

[0028] The visual features and language features of n objects are concatenated to generate the visual-language features V of n objects. vis-text ;

[0029] The visual features and position features of n objects are concatenated to generate the visual-position features of n objects;

[0030] K grasping features are generated through the multi-layer perceptron MLP;

[0031] Map the position features of each object into CLIP, use K captured features as queries, and the visual-linguistic features V of n objects vis-text As values, visual-position features are used as keys, and the query, key, and value are calculated through a cross-attention mechanism to generate k multimodal features.

[0032] The SAC algorithm includes a policy network, four value networks, an experience replay pool, and an evaluator network. The four value networks include two Q-value networks and two V-value networks.

[0033] The two Q value networks are Q θ1 (s t ), Q θ2 (s t );

[0034] The two V value networks are state value network Target State Value Network

[0035] Reinforce learning training is performed on the feature fusion model to optimize and generate the final grasping strategy, which includes the following steps:

[0036] The strategic network Target State Value Network And the experience replay pool is initialized;

[0037] The two Q-value network outputs are used to evaluate the value function Q of the grasping posture under given multimodal features;

[0038] By inputting K multimodal features, the strategy network outputs the logits of K grasping strategies, and the two V value networks output the Q values ​​of each function Q corresponding to the K grasping strategies. Based on the logits and Q values, the final grasping strategy is screened and generated.

[0039] Adaptively adjust the gripping force of the robot arm and complete object grasping in a multi-object environment. The specific steps are as follows:

[0040] Establish a robot arm grasping force model, including the robot arm and force detector;

[0041] Solve the optimization problem constraints through logarithmic obstacle function and perform impedance control on the grasping force based on position characteristics;

[0042] The admittance parameter self-tuning method is adopted to dynamically adjust the impedance parameters according to the real-time data of the force detector when grasping the object, and then the grasping force is adaptively adjusted, and the object grasping in a multi-object environment is completed through the robotic arm.

[0043] Beneficial effects of the present invention:

[0044] The present invention obtains the RGB-D image of the scene, and after processing the RGB-D image, the visual features of each object can be obtained. The input text command is encoded by BERT to output the language features, and then the visual features and language features are fused and aligned by the CLIP model. The Co-TRM structure is then used to further align the visual features and language features to generate bimodal features, so that the scene can be understood more effectively, and the understanding and reasoning ability of the relationship between objects and language can be improved, which helps to enhance learning ability and improve the accuracy and efficiency of scene understanding and object recognition.

[0045] A point cloud image is generated according to the depth map of the RGB-D image and combined with the visual features. The grasping posture is generated through Graspnet. Then, the grasping posture is fused with the bimodal features through the feature fusion model to generate multimodal features. The feature fusion model is trained by reinforcement learning based on the SAC algorithm to optimize and generate the final grasping strategy in order to generate the optimal grasping strategy. The constraints of the optimization problem are solved through the logarithmic obstacle function. The admittance parameter self-tuning method is adopted. According to the real-time data of the force detector when grasping the object, the impedance parameters are dynamically adjusted, and then the grasping force is adaptively adjusted. The robot arm is used to complete the object grasping in a multi-object environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flow chart of the robot arm instruction grabbing method of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] like Figure 1 As shown, a method for grasping instructions of a robot arm based on vision-force perception comprises the following steps:

[0050] Get the RGB-D image of the scene, cut the RGB-D image through OpenCV, and extract the visual features of each object in the RGB-D image;

[0051] Encode the text commands input by the user through BERT and output language features;

[0052] The language features and visual features are input into the CLIP model for fusion and alignment, and then further aligned using the Co-TRM structure to generate bimodal features;

[0053] Generate a point cloud image based on the depth map of the RGB-D image. Based on the point cloud image and the visual features of the object, generate the grasping posture through Graspnet.

[0054] Acquire the position features of each object according to the depth image;

[0055] Construct a feature fusion model to fuse the grasping posture, position features and bimodal features and generate multimodal features;

[0056] Based on the SAC algorithm, the feature fusion model is trained through reinforcement learning to optimize and generate the final grasping strategy;

[0057] Adaptively adjust the gripping force of the robot arm and complete object grasping in a multi-object environment;

[0058] Preferably, the present application uses a KinectV2 depth camera to capture and acquire an RGB-D image of the scene.

[0059] Use OpenCV to slice the RGB-D image and extract the visual features of each object in the RGB-D image. The specific steps include:

[0060] Identify objects in the image, cut the RGB-D image according to the object, split the RGB-D image into object bounding boxes centered on the object, and generate a mask for each object;

[0061] Use OpenCV function to analyze the mask and obtain the connected domain information of each object, which includes the index and area of ​​the bounding box;

[0062] Extract the sub-image corresponding to each object from the RGB-D image according to the bounding box, and extract the object features in each sub-image;

[0063] Calculate the pixel coordinates of the center point of each bounding box in the image, and convert the pixel coordinates of each object into three-dimensional coordinates by combining the pixel size and working space constraints;

[0064] Preferably, the present application uses the Vit-B / 16 model to extract the object features in each sub-image;

[0065] BERT is used to encode the text commands input by the user and output language features. The specific steps include:

[0066] Get the text command input by the user and perform word segmentation;

[0067] For each character / word, three learnable vectors need to be obtained: 1) the character / word vector itself; 2) the position vector; 3) the sentence vector;

[0068] Concatenate three learnable vectors;

[0069] Use multiple Transformer encoders to process the connected vector sequence to obtain richer text sequence context and semantic information, and finally output language features;

[0070] The character / word vector itself is the vector representation of the character / word itself, which is obtained by looking up a predefined vocabulary. Each character or word has a corresponding vector representation in the vocabulary; these three learnable vectors will be optimized during the pre-training process to better capture the semantic information of the vocabulary; the position vector encodes the position information of the character / word into a feature vector; the sentence vector is used to distinguish the vector representation of two sentences. When two sentences are input, the character / word of the first sentence is marked as 0, and the character / word of the second sentence is marked as 1; after connecting these three vectors, multiple Transformer encoders are used to obtain richer text sequence context and semantic information.

[0071] Let the visual features be I1,I2,...,I n , the language features are T1,T2,...,T n ;

[0072] Input the visual features and language features into the CLIP model to obtain the visual feature I 1a , I 2a ...I na , language feature T 1a , T 2a , ....T na , obtain the preliminary bimodal features by initially integrating and aligning the visual features and language features;

[0073] Then, the preliminary bimodal features are input into the Co-TRM structure, and the key and value of any feature in the visual feature and the language feature are passed as input to the attention module of the other feature. The attention module generates an attention pooling feature based on the condition of the other feature for any feature, and obtains the visual feature I 1b , I 2b ...I nb , language feature T 1b , T 2b , ....T nb, complete the further fusion and alignment of the visual features and language features in the preliminary bimodal features to generate bimodal features;

[0074] After the CLIP model performs preliminary fusion and alignment of visual features and language features, this application uses the Co-TRM structure to further align visual features and language features. Specifically, in the visual stream, image-conditioned language attention is implemented, while in the language stream, language-conditioned image attention is performed.

[0075] The ViT-B / 16 model is used for visual feature processing. The input size of visual processing is B×N×3×32×32, where B represents the batch size, N represents the number of objects segmented in the scene, 3 represents the RGB value, and 32×32 represents the size of the object frame. The output visual feature F v , whose dimensions are B×N×D i , D i Indicates the dimension of the feature, which is set to 512 in the present invention.

[0076] The processing of language features first concatenates the word vector, position vector, and sentence vector, and then sends them to a multi-head Transformer group. The Transformer group consists of 8 Transformer layers, and its input size is B×L×D t ,L represents the length of the text, D t The feature dimension of the text is also set to 512 in the present invention, and the output is the text feature F t , whose output feature dimension is B×D t .

[0077] After obtaining the visual features and language features, the present invention adopts the Co-TRM structure to further fuse the two and obtain the visual language feature F v ′ and language visual features F t ′, and its size can be expressed as B×N×512; then the two are concatenated to obtain the visual-language fusion feature V vis-text During the encoding process, the input data of the grasping posture (shape B×K grasp ×Grasp dim , where K grasp Indicates the number of grabs, Grasp dim It represents the output dimension of the capture, which is set to 7 in the present invention. The size is gradually converted through three linear layers, and the feature dimension is increased from the original size to 256, 512, and 512, and the final generated shape is B×K grasp ×512 grasping posture features V grasp .

[0078] The grasping posture, position features and bimodal features are integrated to generate multimodal features. The specific steps are as follows:

[0079] The visual features and language features of n objects are concatenated to generate the visual-language features V of n objects. vis-text ;

[0080] The visual features and position features of n objects are concatenated to generate the visual-position features of n objects;

[0081] K grasping features are generated through the multi-layer perceptron MLP;

[0082] Acquire the position features of each object according to the depth image;

[0083] Map the position features of each object to the visual-linguistic feature domain of CLIP, take the K captured features as queries, and the visual-linguistic features V of n objects vis-text As the value, the visual-position feature is used as the key, and the query, key and value are calculated through the cross-attention mechanism to generate k multimodal features;

[0084] The position features of the object are processed. First, the position features are B×N×3. First, they are processed by three linear layers to increase the feature dimensions to 256, 512, and 512, forming a position feature V of size B×N×512. pos Next, the position feature V pos With visual features F v Superimpose to form the visual-position joint feature V vis-pos , so that the model can use both the position and visual information of the object. grasp As the query, the visual-linguistic features V of N items vis-text As the value, the visual-position feature V vis-pos As the key, the cross-attention mechanism is used to obtain the final cross-modal fusion feature F fusion .

[0085] The SAC algorithm includes a policy network, four value networks, an experience replay pool, and an evaluator network. The four value networks include two Q-value networks and two V-value networks.

[0086] The two Q value networks are Q θ1 (s t ), Q θ2 (s t );

[0087] The two V value networks are state value network Target State Value Network

[0088] In the training process of the simulation environment, a two-stage training process is adopted. In the first stage, a loose environment is designed, in which n items are contained and the number of grasping times per round is limited to 8. In this stage, the reward function is set to

[0089]

[0090] In the second stage, a highly cluttered environment is constructed, in which m objects are randomly distributed in the workspace. In this stage, each round requires the robot to grasp and remove obstacles that hinder the approach to the target object, so as to gradually approach and finally successfully grasp the target. In order to prevent the robot from over-grasping, the number of grasps in each round is limited to 10 times. At the same time, this stage encourages the robot to prioritize grasping obstacles close to the target object to improve grasping efficiency. To achieve this goal, a smaller penalty will be imposed when the obstacle is closer to the target object. Based on the above design considerations, the corresponding reward function is constructed:

[0091]

[0092] Where n is 5, m is 8, and the regularization parameter α is initialized to 0.2, and an automatic entropy adjustment strategy is adopted, and the decay degree γ is set to 0.63. First, initialize the policy network π(s,) and two Q value networks: Q θ1 (s t ) and Q θ2 (s t );Two V value networks: state value network and the target state value network V0(s t ), the Adam optimizer was used in the training process, and the learning rate was set to 5×10^(-5), the first stage was set to 500 rounds, and the total number of rounds was 3000.

[0093] Reinforce learning training is performed on the feature fusion model to optimize and generate the final grasping strategy, which includes the following steps:

[0094] Multimodal features as state s t Indicates that the grasping posture is defined as action a t , the grasping strategy is defined as π(a t |s t );

[0095] The optimal grabbing strategy formula can be expressed as:

[0096]

[0097] Among them, π represents the strategy that has found the maximum total reward; r(s t ,a t ) represents the reward function; represents the entropy of the grasping strategy in state St, which allows the model to explore more states and makes the algorithm more robust; α represents the regularization coefficient, which is used to quantify the importance of entropy; t represents discrete actions;

[0098] The policy network, Q θ1 (s t ), target state value network And the experience replay pool is initialized;

[0099] The two Q-value network outputs are used to evaluate the value function Q of the grasping posture under given multimodal features;

[0100] By inputting K multimodal features, the strategy network outputs the logits of K grasping strategies, and the two V value networks output the Q values ​​of each function Q corresponding to the K grasping strategies. According to the logits and Q values, the final grasping strategy is screened and generated;

[0101] In a discrete environment, the function Q is defined as Q:S→R |A| ;

[0102] Its Q value can be defined as:

[0103]

[0104] In the formula, D represents the experience replay pool;

[0105] In order to obtain the optimal grasping strategy, the objective function of the policy network is:

[0106]

[0107] The Q value is updated using Bellman error:

[0108]

[0109] y(r t ,s t ,s t+1 ,a t ,a t+1 )=r t +γ(π θ (a t ∣s t ) T V θ (s t+1 ,a t+1 ))

[0110] In addition, the expression for optimizing the regularization coefficient α is:

[0111]

[0112] in, is a constant vector, equivalent to the hyperparameter of the target entropy.

[0113] Adaptively adjust the gripping force of the robot arm and complete object grasping in a multi-object environment. The specific steps are as follows:

[0114] Establish a robot arm grasping force model, including the robot arm and force detector;

[0115] Solve the optimization problem constraints through logarithmic obstacle function and perform impedance control on the grasping force based on position characteristics;

[0116] Adopting the self-tuning method of admittance parameters, the impedance parameters are dynamically adjusted according to the real-time data of the force detector when grasping the object, and then the grasping force is adaptively adjusted, and the object grasping in the multi-object environment is completed by the robot arm;

[0117] Obtain the grasping force and current position, then calculate the grasping force error, dynamically adjust the impedance parameters through the self-tuning adjustment method of the impedance parameters, so as to obtain the updated acceleration, the updated speed, and the updated latest position, so as to better complete the grasping task.

[0118] Preferably, the force detector is a 6-dimensional force sensor;

[0119] Assume that the object is grasped at N contact points, and the positions of the contact points are p (i) , at the contact point position p (i) The gripping force applied is:

[0120] It represents the tangential force decomposed on the coordinate axis, the x-axis and y-axis are tangent to the object at the contact point, and the z-axis points to the inside of the object at the contact point;

[0121] The friction cone constraint of an object can be expressed as:

[0122]

[0123] In the formula, μ i Represents the friction coefficient.

[0124] It can be seen from the formula that the friction cone is a second-order cone constraint. Then the constraint can be written as f (t) ∈K, assuming Q i represents the transformation matrix that transforms the local coordinate system into the global coordinate system, and the force balance constraint is:

[0125] Q1 (1) +Qf (2) +…+Q N f (N) +f ext =0

[0126] In the formula, f ext Represents the resultant external force.

[0127] The torque applied on the contact point can be expressed as From this, the moment balance constraint can be expressed as:

[0128] QUR (1) +S2Q (2) +…+S N Q N f (N) +τ ext =0

[0129] where τ ext represents the overall torque applied externally;

[0130]

[0131] To define a more compact notation for the force balance constraint, let f = (f (1) ,f (2) ,…,f (N) );

[0132] Define the moment balance constraint as A i , expressed as:

[0133] The overall contact matrix is ​​A = (A1, A2, ..., A N ), and then define the force screw as ω ext =(f ext ,τ ext ) to obtain the force balance condition Af+ω ext =0; the force optimization problem is to find a set of f that is as small as possible (i) The friction cone condition and the force balance condition are satisfied at the same time, so the problem can be expressed as follows:

[0134] minimizeF

[0135] subjectto(f (i) ,F)∈K

[0136] f (i) ∈K i

[0137] Af+ω et =0

[0138]

[0139] Then the optimization problem in the grasping process is optimized. In this process, the logarithmic barrier function is used to solve it. The barrier function subproblem can be expressed as:

[0140]

[0141] subjecttoAf+ω ext =0

[0142] in It is expressed as a logarithmic barrier function. When t→+∞, the Dikin search direction is recorded as:

[0143]

[0144] If (f, F) strictly satisfies the cone condition, then the original residual, that is, the error in the force and moment equilibrium condition, is expressed as r p =Af+ω ext , the dual residual can be expressed as r d :

[0145]

[0146] where ν∈R 6 represents the Lagrange multiplier, r f ∈R 3M is the dual residual component related to the grasping force f, r F ∈R is the double residual component associated with F, where r f It can be decomposed into the form of all contact points. The optimal condition of the sub-problem can be expressed as: p =0, r d =0.

[0147] In this paper, Newton's method is used to solve this subproblem. The initial value of Newton's method is (f, F, ν), where c needs to strictly satisfy the cone constraint, Af+ω ext = 0, the external force balance condition does not need to be satisfied. In each iteration, the Newton step length (df, dF, dν) at the current point (f, F, ν) needs to be calculated, and the appropriate step length β is determined in the interval (0, 1) by backtracking straight line search method to ensure that each iteration is within the feasible range, and the new point is updated as:

[0148] f=f+γdf F=F+γdFν=ν+γdν

[0149] Where γ = β k , k is satisfied The smallest non-negative integer, It is the original residual and dual residual calculated by the updated point, usually α=0.1,β=0.5. According to Karush–Kuhn–Tucker (KKT), (df,dF,dν) can be solved:

[0150]

[0151] in,

[0152]

[0153] Where x∈R 2 ,y∈R, and because The size of is 6×3, so it can be decomposed and calculated Get F doal :

[0154]

[0155] Set up an admittance controller, where the parameters of admittance control include F, F d ,X are all known quantities, and the unknown quantity is the desired position of the gripper, X s =(XX d ), which represents the current position of the gripper, X d represents the desired position of the gripper. The overall control consists of an outer loop of admittance control and an inner loop of position control. Let the following formula be obtained: in the formula X e =(XX d ), with X e =(X·X d )=[Δp,Δω] as an example. When calculating the position deviation, the current position and the target position can be directly subtracted. However, since the posture cannot be directly subtracted, first check whether the direction of the current posture quaternion is consistent with the direction of the expected posture quaternion, and adjust the direction of the current posture if necessary. Then, by calculating the product of the current posture quaternion and the inverse of the expected posture quaternion, a quaternion representing the posture difference is obtained. In order to ensure the accuracy of the calculation, the quaternion is normalized. Subsequently, the posture difference quaternion is converted into axis-angle form, and the rotation axis and rotation angle are extracted. Finally, the rotation axis is multiplied by the rotation angle to obtain a 3D vector to approximate the posture error, so that X can be obtained. e .

[0156] Further integrating the formula, the position correction is obtained by numerical iteration method, and the result is:

[0157]

[0158] In the formula, Δt represents the calculation cycle of the program, (.) t Represents the known quantity in the previous cycle;

[0159] After the admittance controller is built, an adaptive function of the admittance parameters is constructed, which includes inertia parameters, damping parameters and rigidity parameters. The corresponding adaptive change function can be designed to make the contact force match the grasping force more quickly and effectively. The adaptive change functions are set as follows: The formula represents the hyperbolic secant function. For the inertia parameter, it should ensure that the system can respond quickly to changes in external forces without causing oscillations in the contact force. To this end, an adaptive change function designed in this paper gradually decreases the inertia parameter as time t increases, and initially reaches a maximum value a m , and as time tends to positive infinity, it gradually converges to the minimum value a m -β m This design ensures that the system can maintain stable control during the entire operation process while avoiding oscillation of contact force.

[0160] In practical applications, in the initial stage, a larger inertia parameter enables the system to respond to the contact force more quickly, and then stabilizes at a smaller value, so that the overshoot of the contact force is as small as possible;

[0161] The selection of damping parameters needs to strike a balance between ensuring that the contact force oscillation is avoided while maintaining a small overshoot and a fast response speed. Therefore, in the variation function, as time t passes, the damping parameter should first reach the minimum value a at the initial moment. b , and then gradually increases until it reaches a maximum value α when time approaches positive infinity b +β b ;

[0162] In the initial stage, an initial damping parameter value that can avoid oscillation and maintain fast response needs to be selected, and then slightly increased to improve system performance; the rigidity parameter is used to ensure the fast response speed of the system contact force in the initial stage, and then gradually decreases to maintain the contact force at the final stable value. As time t increases, the adaptive change function gradually decreases the inertia parameter and obtains the maximum value α from the initial moment. k , when time approaches positive infinity, it reaches the minimum value α k -β k .

[0163] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0164] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for grasping robot commands based on vision and force perception, characterized in that: The following steps are involved: Get the RGB-D image of the scene, cut the RGB-D image through OpenCV, and extract the visual features of each object in the RGB-D image; Encode the text commands input by the user through BERT and output language features; The language features and visual features are input into the CLIP model for fusion and alignment, and then further aligned using the Co-TRM structure to generate bimodal features; Generate a point cloud image based on the depth map of the RGB-D image. Based on the point cloud image and the visual features of the object, generate the grasping posture through Graspnet. Acquire the position features of each object according to the depth image; Construct a feature fusion model to fuse the grasping posture, position features and bimodal features and generate multimodal features; Based on the SAC algorithm, the feature fusion model is trained through reinforcement learning to optimize and generate the final grasping strategy; Adaptively adjust the gripping force of the robot arm and complete object grasping in a multi-object environment.

2. The method for grasping robot commands based on vision-force perception according to claim 1, characterized in that: Use OpenCV to slice the RGB-D image and extract the visual features of each object in the RGB-D image. The specific steps include: Identify objects in the image, cut the RGB-D image according to the object, split the RGB-D image into object bounding boxes centered on the object, and generate a mask for each object; Use OpenCV function to analyze the mask and obtain the connected domain information of each object, which includes the index and area of ​​the bounding box; Extract the sub-image corresponding to each object from the RGB-D image according to the bounding box, and extract the object features in each sub-image; The pixel coordinates of the center point of each bounding box in the image are calculated, and the pixel coordinates of each object are converted into three-dimensional coordinates by combining the pixel size and working space constraints.

3. The method for grasping robot arm instructions based on vision-force perception according to claim 2, characterized in that: BERT is used to encode the text commands input by the user and output language features. The specific steps include: Get the text command input by the user and perform word segmentation; For each character / word, three learnable vectors need to be obtained: 1) the character / word vector itself; 2) the position vector; 3) the sentence vector; Concatenate three learnable vectors; Multiple Transformer encoders are used to process the connected vector sequence to obtain richer text sequence context and semantic information, and finally output language features.

4. The method for grasping robot commands based on vision-force perception according to claim 3, characterized in that: The specific steps to generate bimodal features are as follows: Assume that the visual features are I1,I2,…,I n , the language features are T1, T2, …, T n ; Input the visual features and language features into the CLIP model to obtain the visual feature I 1a , I 2a ...I na , language feature T 1a , T 2a , ....T na , obtain the preliminary bimodal features by initially integrating and aligning the visual features and language features; Then the preliminary bimodal features are input into the Co-TRM structure, and the key and value of any feature in the visual feature and the language feature are passed as input to the attention module of the other feature. The attention module generates an attention pooling feature based on the condition of the other feature for any feature, and obtains the visual feature I 1b , I 2b ...I nb , language feature T 1b , T 2b , ....T nb , complete the further fusion and alignment of the visual features and language features in the preliminary bimodal features to generate bimodal features.

5. The method for grasping robot arm instructions based on vision-force perception according to claim 4, characterized in that: The grasping posture, position features and bimodal features are integrated to generate multimodal features. The specific steps are as follows: The visual features and language features of n objects are concatenated to generate the visual-language features V of n objects. vis-text ; The visual features and position features of n objects are concatenated to generate the visual-position features of n objects; K grasping features are generated through the multi-layer perceptron MLP; Map the position features of each object into CLIP, use K captured features as queries, and the visual-linguistic features V of n objects vis-text As values, visual-position features are used as keys, and the query, key, and value are calculated through a cross-attention mechanism to generate k multimodal features.

6. The method for grasping robot commands based on vision-force perception according to claim 5, characterized in that: The SAC algorithm includes a policy network, four value networks, and an experience replay pool. The four value networks include two Q-value networks and two V-value networks. The two Q value networks are Q θ1 (s t ), Q θ2 (s t ); The two V value networks are state value network Target State Value Network 7. The method for grasping robot commands based on vision-force perception according to claim 6, characterized in that: Reinforce learning training is performed on the feature fusion model to optimize and generate the final grasping strategy, which includes the following steps: The policy network, Q θ1 (s t ), target state value network And the experience replay pool is initialized; The two Q-value network outputs are used to evaluate the value function Q of the grasping posture under given multimodal features; By inputting K multimodal features, the strategy network outputs the logits of K grasping strategies, and the two V value networks output the Q values ​​of each function Q corresponding to the K grasping strategies. Based on the logits and Q values, the final grasping strategy is screened and generated.

8. The method for grasping robot commands based on vision-force perception according to claim 7, characterized in that: Adaptively adjust the gripping force of the robot arm and complete object grasping in a multi-object environment. The specific steps are as follows: Establish a robot arm grasping force model, including the robot arm and force detector; Solve the optimization problem constraints through logarithmic obstacle function and perform impedance control on the grasping force based on position characteristics; The admittance parameter self-tuning method is adopted to dynamically adjust the impedance parameters according to the real-time data of the force detector when grasping the object, and then the grasping force is adaptively adjusted, and the object grasping in a multi-object environment is completed through the robotic arm.

Images