Industrial robot grabbing control system

Abstract

The invention discloses an industrial robot grabbing control system. The system comprises a grabbing mechanism, a driving mechanism, a visual acquisition unit, a flexible touch array and an operation processing unit. The visual acquisition unit is used for acquiring a multi-view depth map of a to-be-sorted object; the flexible tactile array is arranged on the grabbing mechanism, is in contact with an object to be sorted during grabbing and is used for acquiring contact force matrix data; and the operation processing unit is used for receiving the multi-view depth map and the contact force matrix data, extracting visual features and tactile features, carrying out cross-modal attention fusion on the extracted visual features and tactile features, obtaining a fusion feature matrix, and using the fusion feature matrix to generate a grabbing strategy for the to-be-sorted object based on a double-layer GRU model.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial robots, and particularly relates to an industrial robot grasping control system. Background Art

[0002] In the prior art, the industrial robot grasping control system usually relies solely on vision or fixed programs to control the robot for grasping. However, in application scenarios such as logistics sorting, due to the large differences in the shapes and hardnesses of the objects to be sorted, the above-mentioned grasping methods may lead to grasping failures or unstable grasping. Summary of the Invention

[0003] The present invention provides an industrial robot grasping control system, which includes a grasping mechanism, a driving mechanism, a visual acquisition unit, a flexible tactile array, and an arithmetic processing unit;

[0004] The visual acquisition unit is used to obtain multi-view depth maps of the objects to be sorted;

[0005] The flexible tactile array is arranged on the grasping mechanism and contacts the objects to be sorted during grasping, and is used to obtain contact force matrix data;

[0006] The grasping mechanism is used to grasp the objects to be sorted;

[0007] The driving mechanism is used to drive the grasping mechanism to grasp the objects to be sorted according to the grasping strategy;

[0008] The grasping mechanism, the driving mechanism, the visual acquisition unit, and the flexible tactile array are all connected to the arithmetic processing unit;

[0009] The arithmetic processing unit is used to receive the multi-view depth maps and the contact force matrix data, extract visual features and tactile features, perform cross-modal attention fusion on the extracted visual features and tactile features to obtain a fused feature matrix, and use the fused feature matrix to generate a grasping strategy for the objects to be sorted based on a double-layer GRU model.

[0010] The present invention also relates to an industrial robot grasping control method for the above-mentioned industrial robot grasping control system, characterized in that the method specifically includes the following steps:

[0011] S1. The visual acquisition unit obtains multi-view two-dimensional depth maps of the objects to be sorted and sends them to the arithmetic processing unit, and the flexible tactile array obtains contact force matrix data and sends it to the arithmetic processing unit;

[0012] S2. The arithmetic processing unit processes the multi-view depth maps and the contact force matrix data to extract visual features and tactile features;

[0013] S3. The operation processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fused feature matrix;

[0014] S4. The operation processing unit uses the fused feature matrix and, based on a double-layer GRU model, generates a grasping strategy for the object to be sorted;

[0015] S5. The driving mechanism applies the grasping force and grasping angle given by the grasping strategy to drive the grasping mechanism to grasp the object to be sorted.

[0016] The present invention also relates to a computer program product, which includes a computer program. The computer program is executed by a processor and is used to execute the control method described above.

[0017] The present invention also relates to a computer-readable storage medium, which is used to store a computer program. The computer program is executed by a processor and is used to execute the control method described above.

[0018] The technical solution of the present invention enables a robot to fuse visual and tactile sensor information for object recognition by using multi-modal AI technology, and to perceive the physical properties such as the shape and hardness of an object in real time when grasping the object. By deeply integrating multi-modal AI technology into the grasping and sorting process of the robot, it breaks through the limitation of traditional robots that rely solely on vision or fixed programs for grasping, and can grasp different types and shapes of objects more intelligently and stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of the structure of the industrial robot grasping control system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The present invention will be further described below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and cannot be used to limit the protection scope of the present invention. It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention.

[0021] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0022] Embodiment 1 of the present invention relates to an industrial robot grasping control system, which includes a grasping mechanism, a driving mechanism, a vision acquisition unit, a flexible tactile array, and an arithmetic processing unit. The vision acquisition unit is used to obtain multi-view depth maps of the object to be sorted, and the flexible tactile array is arranged on the grasping mechanism and contacts the object to be sorted during grasping. The grasping mechanism is used to grasp the object to be sorted, and the driving mechanism is used to drive the grasping mechanism to grasp the object to be sorted according to the grasping strategy. The grasping mechanism, the driving mechanism, the vision acquisition unit, and the flexible tactile array are all connected to the arithmetic processing unit.

[0023] The vision acquisition unit is used to obtain multi-view depth maps of the object to be sorted and send them to the arithmetic processing unit. The flexible tactile array acquires contact force matrix data and sends the contact force matrix data to the arithmetic processing unit. The contact force matrix data is matrix-form data acquired by the flexible tactile array.

[0024] The arithmetic processing unit receives the multi-view depth maps and the contact force matrix data, processes them respectively, and extracts visual features and tactile features.

[0025] Visual features The extraction process is as follows.

[0026]

[0027] Among them, is the feature map of the th view, is the number of views of the vision acquisition unit, is the weight of the th view.

[0028] Specifically, for the depth map of each view , features are extracted through a residual block. Let the input of the th residual block be , and the output be , then the residual block formula is:

[0029]

[0030]

[0031] Among them, is the residual function, is the trainable weight parameter of the first convolutional layer in the th residual block, is the trainable weight parameter of the second convolutional layer in the th residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function.

[0032] After residual blocks, the high-level feature map is obtained:

[0033]

[0034] Among them, represents a residual network with residual blocks.

[0035] Apply adaptive pooling to each feature map to adjust it to a fixed size , and obtain:

[0036]

[0037] Among them, is the feature map of the th perspective, represents adaptive pooling. Adaptive pooling ensures a consistent output size by dynamically adjusting the pooling window size and stride.

[0038] Contact force matrix data is a matrix of . The extraction process of the tactile feature is as follows,

[0039]

[0040]

[0041] Among them, represents the dimension of the pooled feature, represents the pooling window size, is the stride in the row direction, is the stride in the column direction.

[0042]

[0043] In the formula,

[0044]

[0045]

[0046]

[0047]

[0048] Among them, Represents the tactile feature For each output position 、 Is the independent variable

[0049] The operation processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fused feature matrix

[0050] Obtain the fused feature The specific process is as follows

[0051]

[0052]

[0053]

[0054] Among them Is the visual feature Is the tactile feature Is the projection matrix parameter of the visual feature Is the projection matrix parameter of the tactile feature Is the key vector after the visual feature is projected Is the key vector after the tactile feature is projected Is the key vector Of the transpose Is the scaling factor parameter Is the tactile feature weighting parameter matrix Is the fused feature matrix after fusion

[0055] The operation processing unit uses the fused feature matrix and generates a grasping strategy for the object to be sorted based on a two-layer GRU model

[0056] The specific process is as follows

[0057]

[0058] Among them Is the Grasping force magnitude in the Is the Grasping angle in the Represents a two-layer Recurrent neural network Represents the number of candidate grasping strategies

[0059] Set the optimization objective function for obtaining the grasping strategy To be

[0060]

[0061] Among them, is the fusion feature matrix, is the ideal clamping force, is the ideal grasping angle, , are the weight coefficients.

[0062] The driving mechanism gives the grasping force and the grasping angle according to the grasping strategy, and drives the grasping mechanism to grasp the object to be sorted.

[0063] Embodiment 2 of the present invention relates to an industrial robot grasping control method for the industrial robot grasping control system of Embodiment 1. The method specifically includes the following steps:

[0064] S1. The vision acquisition unit acquires the multi-view two-dimensional depth map of the object to be sorted and sends it to the operation processing unit, and the flexible tactile array acquires the contact force matrix data and sends it to the operation processing unit.

[0065] S2. The operation processing unit processes the multi-view depth map and the contact force matrix data to extract visual features and tactile features.

[0066] The extraction of visual features includes:

[0067] S211. For the depth map of each view , features are extracted through the residual block. Let the input of the th residual block be , and the output be . Then the residual block formula is:

[0068]

[0069]

[0070] Among them, is the residual function, is the trainable weight parameter of the first convolutional layer in the th residual block, is the trainable weight parameter of the second convolutional layer in the th residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function.

[0071] S212. After residual blocks, the high-level feature map is obtained:

[0072]

[0073] Among them, represents a residual network with residual blocks.

[0074] S213. Apply adaptive pooling to each feature map and adjust it to a fixed size to obtain:

[0075]

[0076] Among them, is the feature map of the th perspective, represents adaptive pooling. Adaptive pooling ensures a consistent output size by dynamically adjusting the pooling window size and stride.

[0077] S214. Extract visual features :

[0078]

[0079] Among them, is the feature map of the th perspective, is the number of perspectives of the visual acquisition unit, is the weight of the th perspective.

[0080] Extracting tactile features includes:

[0081] S221. Contact force matrix data is a matrix of , and calculate the pooled feature dimension:

[0082]

[0083]

[0084] Among them, represents the pooled feature dimension, represents the pooling window size, is the stride in the row direction, is the stride in the column direction.

[0085] S222. Calculate

[0086]

[0087] In the formula,

[0088]

[0089]

[0090]

[0091]

[0092] Among them, represents the output positions of the tactile features and are independent variables. , For the independent variable.

[0093] S3. The operation processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fused feature matrix.

[0094] Obtaining the fused features The specific process includes:

[0095]

[0096]

[0097]

[0098] Among them, is the visual feature, is the tactile feature, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after the visual feature is projected, is the key vector after the tactile feature is projected, is the key vector transpose, is the scaling factor parameter, is the weighted parameter matrix of the tactile feature, is the fused feature matrix after fusion.

[0099] S4. The operation processing unit uses the fused feature matrix and generates a grasping strategy for the object to be sorted based on the double-layer GRU model.

[0100] The specific process includes:

[0101]

[0102] Among them, is the grasping force magnitude in the th candidate grasping strategy, is the grasping angle in the th candidate grasping strategy, represents the double-layer recurrent neural network, Indicates the number of candidate grasping strategies.

[0103] Set to obtain the grasping strategy The optimized objective function is:

[0104]

[0105] Where is the fusion feature matrix, is the ideal clamping force, is the ideal grasping angle, , are the weight coefficients.

[0106] S5. The driving mechanism drives the grasping mechanism to grasp the object to be sorted according to the grasping force and the grasping angle given by the grasping strategy.

[0107] Embodiment 3 of the present invention relates to a computer program product, the computer program product includes a computer program, and the computer program is executed by a processor to execute the industrial robot grasping control method of Embodiment 2.

[0108] Embodiment 4 of the present invention relates to a computer-readable storage medium, the computer-readable storage medium is used to store a computer program, and the computer program is executed by a processor to execute the industrial robot grasping control method of Embodiment 2.

[0109] As described above, it is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. An industrial robot grasping control system, characterized in that, The system includes a grasping mechanism, a driving mechanism, a visual acquisition unit, a flexible tactile array, and an arithmetic processing unit; The visual acquisition unit is used to obtain multi-view depth maps of the object to be sorted; The flexible tactile array is arranged on the grasping mechanism and contacts the object to be sorted during grasping, and is used to obtain contact force matrix data; The grasping mechanism is used to grasp the object to be sorted; The driving mechanism is used to drive the grasping mechanism to grasp the object to be sorted according to the grasping strategy; The grasping mechanism, the driving mechanism, the visual acquisition unit, and the flexible tactile array are all connected to the arithmetic processing unit; The arithmetic processing unit is used to receive the multi-view depth maps and the contact force matrix data, extract visual features and tactile features, perform cross-modal attention fusion on the extracted visual features and tactile features to obtain a fused feature matrix, and use the fused feature matrix to generate a grasping strategy for the object to be sorted based on a double-layer GRU model.

2. The industrial robot grasping control system according to claim 1, characterized in that, The specific process of extracting visual features includes: The operation processing unit processes the depth map of each perspective , extracts features through residual blocks, and after passing through residual blocks, a high-level feature map is obtained. The operation processing unit applies adaptive pooling to each feature map to adjust it to a fixed size, obtaining a feature map , and then extracts visual features through the following method : ； Among them, is the feature map of the th perspective, is the number of perspectives of the visual acquisition unit, is the weight of the th perspective.

3. The industrial robot grasping control system according to claim 1, wherein The specific process of extracting tactile features includes: Received contact force matrix data is a matrix. Obtain the pooled feature dimension ； ； Among them, represents the feature dimension after pooling, represents the pooling window size, is the stride in the row direction, is the stride in the column direction; Extract visual features , wherein ； In the formula, ； ； ； ； Among them, represents each output position of the tactile feature , , being the independent variables.

4. The industrial robot grasping control system according to claim 1, characterized in that, Obtain the fused features Specifically include: ； ； ； Among them, is a visual feature, is a tactile feature, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after the visual feature is projected, is the key vector after the tactile feature is projected, is the key vector is the transpose of, is the scaling factor parameter, is the tactile feature weighting parameter matrix, is the fused feature matrix after fusion.

5. The grasping control system of an industrial robot according to claim 1, characterized in that, The specific process of generating a grasping strategy for the object to be sorted includes: ； Among them, is the grasping force magnitude in the th candidate grasping strategy, is the grasping angle in the th candidate grasping strategy, represents a double recurrent neural network, represents the number of candidate grasping strategies; Set the acquisition of the scraping strategy The optimized objective function is as follows: ； Among them, is the fusion feature matrix, is the ideal clamping force, is the ideal grasping angle, and are the weight coefficients.

6. The industrial robot grasping control system according to claim 5, wherein, The driving mechanism drives the grasping mechanism to grasp the object to be sorted according to the grasping strategy, specifically including: the driving mechanism gives the grasping force according to the grasping strategy , the grasping angle , and drives the grasping mechanism to grasp the object to be sorted.

7. An industrial robot grasping control method for the industrial robot grasping control system according to any one of claims 1-6, characterized in that, The method specifically includes the following steps: S1. The visual acquisition unit obtains multi-view two-dimensional depth maps of the object to be sorted and sends them to the arithmetic processing unit, and the flexible tactile array obtains contact force matrix data and sends it to the arithmetic processing unit; S2. The arithmetic processing unit processes the multi-view depth maps and the contact force matrix data to extract visual features and tactile features; S3. The arithmetic processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fused feature matrix; S4. The arithmetic processing unit uses the fused feature matrix to generate a grasping strategy for the object to be sorted based on a double-layer GRU model; S5. The driving mechanism applies the grasping force given by the grasping strategy and the grasping angle to drive the grasping mechanism to grasp the object to be sorted.

8. The industrial robot grasping control method according to claim 7, wherein, In step S2, the extraction of visual features includes: S211. For the depth map of each perspective , extract features through residual blocks. Let the input of the -th residual block be , and the output be . Then the residual block formula is: ； ； Among them, is the residual function, is the trainable weight parameter of the first convolutional layer in the -th residual block, is the trainable weight parameter of the second convolutional layer in the -th residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function; S212. After passing through residual blocks, a high-level feature map is obtained: ； Among them, represents a residual network with residual blocks; S213. For each feature map apply adaptive pooling to adjust it to a fixed size , and obtain: ； Among them, is the feature map of the th perspective, represents adaptive pooling. Adaptive pooling ensures consistent output sizes by dynamically adjusting the pooling window size and stride; S214. Extract visual features : ； Among them, is the feature map of the th perspective, is the number of perspectives of the visual acquisition unit, is the weight of the th perspective; The extraction of tactile features includes: S221. Contact force matrix data is a matrix, calculate the pooled feature dimension: ； ； Among them, represents the feature dimension after pooling, represents the pooling window size, is the stride in the row direction, is the stride in the column direction; S222. Calculate ； In the formula, ； ； ； ； Among them, represents the respective output positions of the tactile feature , and are independent variables; Obtain the fused features The specific process includes: ； ； ； Among them, is a visual feature, is a tactile feature, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after the visual feature is projected, is the key vector after the tactile feature is projected, is the key vector is the transpose of, is the scaling factor parameter, is the tactile feature weighting parameter matrix, is the fused feature matrix after fusion; The specific process of generating a grasping strategy for the object to be sorted includes: ； Among them, is the grasping force magnitude in the th candidate grasping strategy, is the grasping angle in the th candidate grasping strategy, represents a double-layer recurrent neural network, represents the number of candidate grasping strategies; Set to obtain the crawling strategy The optimized objective function is as follows: ； Among them, is the fusion feature matrix, is the ideal clamping force, is the ideal grasping angle, , are the weight coefficients.

9. A computer program product, characterized in that, The computer program product includes a computer program, and the computer program is executed by a processor and is used to execute the control method as claimed in claim 7 or 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, and the computer program is executed by a processor and is used to execute the control method as claimed in claim 7 or 8.

Images