An industrial robot grasping control system

Abstract

The present invention discloses an industrial robot grasping control system, which includes a grasping mechanism, a driving mechanism, a vision acquisition unit, a flexible tactile array, and an operation processing unit; the vision acquisition unit is used to obtain a multi-perspective depth map of an object to be sorted; the flexible tactile array is arranged on the grasping mechanism and contacts the object to be sorted during grasping, and is used to obtain contact force matrix data; the operation processing unit is used to receive the multi-perspective depth map and the contact force matrix data, extract visual features and tactile features, perform cross-modal attention fusion on the extracted visual features and tactile features, obtain a fused feature matrix, and use the fused feature matrix to generate a grasping strategy for the object to be sorted based on a double-layer GRU model.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial robots, and in particular to an industrial robot grasping control system. Background Art

[0002] In the existing technology, industrial robot grasping control systems usually rely solely on vision or fixed programs to control the robot to grasp. However, in application scenarios such as logistics sorting, due to the large differences in the shape and hardness of the objects to be sorted, the above grasping method may cause grasping failure or unstable grasping. Summary of the Invention

[0003] The present invention provides an industrial robot grasping control system, the system comprising a grasping mechanism, a driving mechanism, a visual acquisition unit, a flexible tactile array, and an operation processing unit;

[0004] The visual acquisition unit is used to obtain multi-view depth maps of objects to be sorted;

[0005] The flexible tactile array is arranged on the grasping mechanism and contacts the object to be sorted during grasping, so as to obtain contact force matrix data;

[0006] The gripping mechanism is used to grip objects to be sorted;

[0007] The driving mechanism is used to drive the grasping mechanism to grasp the objects to be sorted according to the grasping strategy;

[0008] The grasping mechanism, driving mechanism, visual acquisition unit, and flexible tactile array are all connected to the computing processing unit;

[0009] The operation processing unit is used to receive multi-view depth maps and contact force matrix data, extract visual features and tactile features, perform cross-modal attention fusion on the extracted visual features and tactile features, obtain a fusion feature matrix, and use the fusion feature matrix to generate a grasping strategy for the objects to be sorted based on a two-layer GRU model.

[0010] The present invention also relates to an industrial robot grasping control method for the above-mentioned industrial robot grasping control system, characterized in that the method specifically comprises the following steps:

[0011] S1. The visual acquisition unit obtains a multi-view two-dimensional depth map of the object to be sorted and sends it to the processing unit. The flexible tactile array obtains the contact force matrix data and sends it to the processing unit.

[0012] S2. The processing unit processes the multi-view depth map and contact force matrix data to extract visual features and tactile features;

[0013] S3. The processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fusion feature matrix;

[0014] S4. The processing unit uses the fused feature matrix and a two-layer GRU model to generate a grasping strategy for the object to be sorted.

[0015] S5. The gripping force given by the driving mechanism according to the gripping strategy , grab angle , driving the grasping mechanism to grasp the objects to be sorted.

[0016] The present invention further relates to a computer program product, comprising a computer program, which is executed by a processor for performing the control method as described above.

[0017] The present invention further relates to a computer-readable storage medium for storing a computer program, wherein the computer program is executed by a processor for performing the control method as described above.

[0018] The technical solution of the present invention uses multimodal AI technology to enable the robot to integrate visual and tactile sensor information for object recognition, and perceive the shape, hardness and physical properties of the object in real time when grasping the object. The multimodal AI technology is deeply integrated into the robot's grasping and sorting process, breaking through the limitations of traditional robots that rely solely on vision or fixed programs for grasping, and can grasp objects of different types and shapes more intelligently and stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a structural block diagram of the industrial robot grasping control system of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below with reference to the accompanying drawings. The following examples are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention.

[0021] Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. It should be noted that the terms used herein are intended only to describe specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0022] Embodiment 1 of the present invention relates to an industrial robot grasping control system, comprising a grasping mechanism, a drive mechanism, a visual acquisition unit, a flexible tactile array, and an arithmetic processing unit. The visual acquisition unit is used to acquire a multi-perspective depth map of an object to be sorted, and the flexible tactile array is disposed on the grasping mechanism and contacts the object to be sorted during grasping. The grasping mechanism is used to grasp the object to be sorted, and the drive mechanism is used to drive the grasping mechanism to grasp the object to be sorted according to a grasping strategy. The grasping mechanism, drive mechanism, visual acquisition unit, and flexible tactile array are all connected to the arithmetic processing unit.

[0023] The visual acquisition unit is used to obtain multi-view depth maps of the objects to be sorted and send them to the processing unit. The flexible tactile array obtains contact force matrix data and sends it to the processing unit. The contact force matrix data is a matrix-like data obtained by the flexible tactile array.

[0024] The processing unit receives the multi-view depth map and contact force matrix data, processes them respectively, and extracts visual features and tactile features.

[0025] Visual features The extraction process is as follows.

[0026]

[0027] in, For the The feature map of each viewpoint, is the number of viewing angles of the visual acquisition unit, For the The weight of each perspective.

[0028] Specifically, the depth map for each perspective , extract features through residual blocks, set The input of the residual block is , the output is , then the residual block formula is:

[0029]

[0030]

[0031] in, is the residual function, For the The trainable weight parameters of the first convolutional layer in the residual block, For the The trainable weight parameters of the second convolutional layer in the residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function.

[0032] go through After the residual blocks, the high-level feature map is obtained:

[0033]

[0034] in, Indicates that Residual network with residual blocks.

[0035] For each feature map Apply adaptive pooling, adjusted to a fixed size ,get:

[0036]

[0037] in, For the The feature map of each viewpoint, Represents adaptive pooling. Adaptive pooling ensures consistent output size by dynamically adjusting the pooling window size and stride.

[0038] Contact force matrix data for Matrix, tactile features The extraction process is as follows:

[0039]

[0040]

[0041] in, represents the feature dimension after pooling, represents the pooling window size, is the step length in the row direction, is the step size in the column direction.

[0042]

[0043] In the formula,

[0044]

[0045]

[0046]

[0047]

[0048] in, Representing tactile features Each output position of 、 is the independent variable.

[0049] The processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fusion feature matrix.

[0050] Get fusion features The specific process is as follows.

[0051]

[0052]

[0053]

[0054] in, For visual features, For tactile characteristics, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after visual feature projection, is the key vector after tactile feature projection, is the key vector The transpose of is the scaling factor parameter, is the tactile feature weighted parameter matrix, is the fused feature matrix after fusion.

[0055] The processing unit uses the fused feature matrix and a two-layer GRU model to generate a grasping strategy for the objects to be sorted.

[0056] The specific process is as follows:

[0057]

[0058] in, For the The grasping force of the candidate grasping strategies, For the The grasping angles in the candidate grasping strategies, Indicates double layer Recurrent Neural Networks, Indicates the number of candidate crawling strategies.

[0059] Setting the acquisition and crawling strategy The optimization objective function is:

[0060]

[0061] in, is the fusion feature matrix, For the ideal clamping force, For the ideal gripping angle, 、 is the weight coefficient.

[0062] The gripping force given by the drive mechanism according to the gripping strategy , grab angle , driving the grasping mechanism to grasp the objects to be sorted.

[0063] Embodiment 2 of the present invention relates to an industrial robot grasping control method for the industrial robot grasping control system of embodiment 1, the method specifically comprising the following steps:

[0064] S1. The visual acquisition unit obtains a multi-view two-dimensional depth map of the object to be sorted and sends it to the processing unit. The flexible tactile array obtains contact force matrix data and sends it to the processing unit.

[0065] S2. The processing unit processes the multi-view depth map and contact force matrix data to extract visual features and tactile features.

[0066] Extracting visual features includes:

[0067] S211. Depth map for each view , extract features through residual blocks, set The input of the residual block is , the output is , then the residual block formula is:

[0068]

[0069]

[0070] in, is the residual function, For the The trainable weight parameters of the first convolutional layer in the residual block, For the The trainable weight parameters of the second convolutional layer in the residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function.

[0071] S212. Passing After the residual blocks, the high-level feature map is obtained:

[0072]

[0073] in, Indicates that Residual network with residual blocks.

[0074] S213. For each feature map Apply adaptive pooling, adjusted to a fixed size ,get:

[0075]

[0076] in, For the The feature map of each viewpoint, Represents adaptive pooling. Adaptive pooling ensures consistent output size by dynamically adjusting the pooling window size and stride.

[0077] S214. Extracting visual features :

[0078]

[0079] in, For the The feature map of each viewpoint, is the number of viewing angles of the visual acquisition unit, For the The weight of each perspective.

[0080] Extracting tactile features includes:

[0081] S221. Contact Force Matrix Data for Matrix, calculate the feature dimension after pooling:

[0082]

[0083]

[0084] in, represents the feature dimension after pooling, represents the pooling window size, is the step length in the row direction, is the step size in the column direction.

[0085] S222. Calculation

[0086]

[0087] In the formula,

[0088]

[0089]

[0090]

[0091]

[0092] in, Representing tactile features Each output position of 、 is the independent variable.

[0093] S3. The processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fusion feature matrix.

[0094] Get fusion features The specific process includes:

[0095]

[0096]

[0097]

[0098] in, For visual features, For tactile characteristics, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after visual feature projection, is the key vector after tactile feature projection, is the key vector The transpose of is the scaling factor parameter, is the tactile feature weighted parameter matrix, is the fused feature matrix after fusion.

[0099] S4. The operation processing unit uses the fused feature matrix and generates a grasping strategy for the objects to be sorted based on the two-layer GRU model.

[0100] The specific process includes:

[0101]

[0102] in, For the The grasping force of the candidate grasping strategies, For the The grasping angles in the candidate grasping strategies, Indicates double layer Recurrent Neural Networks, Indicates the number of candidate crawling strategies.

[0103] Setting the acquisition and crawling strategy The optimization objective function is:

[0104]

[0105] in, is the fusion feature matrix, For the ideal clamping force, For the ideal gripping angle, 、 is the weight coefficient.

[0106] S5. The gripping force given by the driving mechanism according to the gripping strategy , grab angle , driving the grasping mechanism to grasp the objects to be sorted.

[0107] Embodiment 3 of the present invention relates to a computer program product, which includes a computer program. The computer program is executed by a processor to execute the industrial robot grasping control method of embodiment 2.

[0108] Embodiment 4 of the present invention relates to a computer-readable storage medium, which is used to store a computer program. The computer program is executed by a processor to execute the industrial robot grasping control method of embodiment 2.

[0109] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. An industrial robot grasping control system, characterized in that: The system includes a grasping mechanism, a driving mechanism, a visual acquisition unit, a flexible tactile array, and an operation processing unit; The visual acquisition unit is used to obtain multi-view depth maps of objects to be sorted; The flexible tactile array is arranged on the grasping mechanism and contacts the object to be sorted during grasping, so as to obtain contact force matrix data; The gripping mechanism is used to grip objects to be sorted; The driving mechanism is used to drive the grasping mechanism to grasp the objects to be sorted according to the grasping strategy; The grasping mechanism, driving mechanism, visual acquisition unit, and flexible tactile array are all connected to the computing processing unit; The processing unit is used to receive multi-view depth maps and contact force matrix data, extract visual features and tactile features, perform cross-modal attention fusion on the extracted visual features and tactile features, obtain a fused feature matrix, and use the fused feature matrix to generate a grasping strategy for the object to be sorted based on a two-layer GRU model; The extraction of visual features specifically includes: The processing unit generates a depth map for each view , extract features through residual blocks, set The input of the residual block is , the output is , then the residual block formula is: ； ； in, is the residual function, For the The trainable weight parameters of the first convolutional layer in the residual block, For the The trainable weight parameters of the second convolutional layer in the residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function; go through After the residual blocks, the high-level feature map is obtained: ； in, Indicates that Residual network with residual blocks; For each feature map Apply adaptive pooling, adjusted to a fixed size ,get: ； in, For the The feature map of each viewpoint, Represents adaptive pooling; adaptive pooling ensures consistent output size by dynamically adjusting the pooling window size and step size; Extracting visual features : ； in, For the The feature map of each viewpoint, is the number of viewing angles of the visual acquisition unit, For the The weight of each perspective.

2. The industrial robot grasping control system according to claim 1, characterized in that: The extraction of tactile features specifically includes: Received contact force matrix data for Matrix, get the feature dimension after pooling, ； ； in, represents the feature dimension after pooling, represents the pooling window size, is the step length in the row direction, is the step size in the column direction; Extracting visual features ,in ； In the formula, ； ； ； ； in, Representing tactile features Each output position of 、 is the independent variable.

3. The industrial robot grasping control system according to claim 1, characterized in that: Get fusion features Specifically include: ； ； ； in, For visual features, For tactile characteristics, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after visual feature projection, is the key vector after tactile feature projection, is the key vector The transpose of is the scaling factor parameter, is the tactile feature weighted parameter matrix, is the fused feature matrix after fusion.

4. The industrial robot grasping control system according to claim 1, characterized in that: The grasping strategy for generating objects to be sorted includes: ； in, For the The grasping force of the candidate grasping strategies, For the The grasping angles in the candidate grasping strategies, Indicates double layer Recurrent Neural Networks, represents the number of candidate grasping strategies; Setting the acquisition and crawling strategy The optimization objective function is: ； in, is the fusion feature matrix, For the ideal clamping force, For the ideal gripping angle, 、 is the weight coefficient.

5. The industrial robot grasping control system according to claim 4, characterized in that: The driving mechanism drives the grasping mechanism to grasp the objects to be sorted according to the grasping strategy, specifically including: the grasping force given by the driving mechanism according to the grasping strategy , grab angle , driving the grasping mechanism to grasp the objects to be sorted.

6. An industrial robot grasping control method for the industrial robot grasping control system according to any one of claims 1 to 5, characterized in that: The method specifically comprises the following steps: S1. The visual acquisition unit obtains a multi-view two-dimensional depth map of the object to be sorted and sends it to the processing unit. The flexible tactile array obtains the contact force matrix data and sends it to the processing unit. S2. The processing unit processes the multi-view depth map and contact force matrix data to extract visual features and tactile features; S3. The processing unit performs cross-modal attention fusion on the extracted visual features and tactile features to obtain a fusion feature matrix; S4. The processing unit uses the fused feature matrix and a two-layer GRU model to generate a grasping strategy for the object to be sorted. S5. The gripping force given by the driving mechanism according to the gripping strategy , grab angle , driving the grasping mechanism to grasp the objects to be sorted; In step S2, extracting visual features includes: S211. Depth map for each view , extract features through residual blocks, set The input of the residual block is , the output is , then the residual block formula is: ； ； in, is the residual function, For the The trainable weight parameters of the first convolutional layer in the residual block, For the The trainable weight parameters of the second convolutional layer in the residual block, is the first convolutional layer, is the second convolutional layer, is the normalization layer, is the activation function; S212. Passing After the residual blocks, the high-level feature map is obtained: ； in, Indicates that Residual network with residual blocks; S213. For each feature map Apply adaptive pooling, adjusted to a fixed size ,get: ； in, For the The feature map of each viewpoint, Represents adaptive pooling; adaptive pooling ensures consistent output size by dynamically adjusting the pooling window size and step size; S214. Extracting visual features : ； in, For the The feature map of each viewpoint, is the number of viewing angles of the visual acquisition unit, For the The weight of each perspective.

7. The industrial robot grasping control method according to claim 6, characterized in that: In step S2, Extracting tactile features specifically includes: S221. Contact Force Matrix Data for Matrix, calculate the feature dimension after pooling: ； ； in, represents the feature dimension after pooling, represents the pooling window size, is the step length in the row direction, is the step size in the column direction; S222. Calculation ； In the formula, ； ； ； ； in, Representing tactile features Each output position of 、 is the independent variable; Get fusion features The specific process includes: ； ； ； in, For visual features, For tactile characteristics, is the projection matrix parameter of the visual feature, is the projection matrix parameter of the tactile feature, is the key vector after visual feature projection, is the key vector after tactile feature projection, is the key vector The transpose of is the scaling factor parameter, is the tactile feature weighted parameter matrix, is the fusion feature matrix after fusion; The specific process of generating a grasping strategy for objects to be sorted includes: ； in, For the The grasping force of the candidate grasping strategies, For the The grasping angles in the candidate grasping strategies, Indicates double layer Recurrent Neural Networks, represents the number of candidate grasping strategies; Setting the acquisition and crawling strategy The optimization objective function is: ； in, is the fusion feature matrix, For the ideal clamping force, For the ideal gripping angle, 、 is the weight coefficient.

8. A computer program product, characterized in that The computer program product comprises a computer program, which is executed by a processor to perform the control method according to claim 6 or 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is executed by a processor to perform the control method according to claim 6 or 7.

Images