Object grabbing method based on multi-modal fusion

Abstract

The invention discloses an object grabbing method based on multi-modal fusion. The method comprises the steps that a grabbing reasoning network is constructed, wherein the grabbing reasoning network comprises a multi-modal fusion module, a residual block and a transposition convolution module; the multi-modal fusion module performs multi-branch feature extraction on the depth image, the color image and the fusion information, and performs hierarchical fusion processing on the extracted features by using an attention mechanism; training a capture reasoning network by using the marked data set of the first scene, and selecting an optimal capture detection model as an initialization reference of a teacher model and a student model of a next scene; part of training data is taken from the marked data set of the trained scene, all marked data of the current scene are taken, and a knowledge distillation method is adopted and a teacher model is used as guidance to guide and train the currently initialized student model; and performing grabbing detection on an object in the trained scene by using the currently obtained grabbing detection model. The stability and plasticity of the robot on grabbing learning can be effectively balanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent grasping detection, and in particular relates to an object grasping method based on multimodal fusion. Background Art

[0002] Robot grasping technology involves multiple key areas such as grasping posture detection, robot control and motion planning. Among them, in the robot grasping process, the detection of grasping posture plays a vital role, providing guidance for the subsequent control and motion planning of the robot to grasp, and is the first step of robot grasping. At present, robot grasping detection technology is mainly divided into analysis-based methods and data-driven methods. Data-driven grasping methods are further divided into discriminative grasping algorithms and generative grasping algorithms. Generative grasping algorithms do not rely on the sampling of grasping candidate objects, but directly generate grasping postures based on the pixel level, so they have good real-time performance and are suitable for closed-loop grasping. The discriminative type is divided into two steps: generating candidate grasping boxes and selecting the optimal grasping box. Generative grasping algorithms currently generally use color images and depth images as input, but the existing methods often fuse color image information and depth image information in the channel dimension, which will increase the amount of calculation.

[0003] On the other hand, in the real world, robots often face different working environments or different grasping objects. When the robot switches to a completely new environment for grasping operations, it is obviously unrealistic to re-train the robot using all the data due to many problems such as data loss and waste of computing power to re-use all the data to train the model. If the robot is trained using only extremely limited data in the new task scenario, although the robot may show a certain degree of outstanding performance in the new scenario, this local optimization strategy hides huge risks. When there is a large difference between the new scene and the old scene, the robot is prone to catastrophic forgetting in the old scene. Therefore, it is particularly important to explore a training method that can enable the robot to perform well in the new task without excessively forgetting the old task, that is, to balance the stability and plasticity of the robot's grasping learning, so that the robot maintains its original grasping ability and has the ability to learn new grasping skills. Summary of the invention

[0004] The present invention provides an object grasping method based on multimodal fusion, which can not only enable the robot to perform well in grasping detection tasks in new scenes, but also not excessively forget grasping detection tasks in old scenes, and effectively balance the stability and plasticity of the robot's grasping learning.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: An object grasping method based on multimodal fusion, comprising: S1, build m identical grasping inference networks, with color images and depth feature maps as input and feature maps of grasping postures as output; The crawling inference network includes a multimodal fusion module, a residual block and a transposed convolution module; the multimodal fusion module includes three feature extraction modules and two attention layers; the inputs of the first, second and third feature extraction modules are respectively: a depth image, a color image, a dark image and a depth image; the first attention layer is used to fuse the output features of the first and second feature extraction modules, and the second attention layer is used to fuse the output features of the first attention layer and the third feature extraction module; S2, obtaining training data sets of different scenes, and marking the grasping posture of each training data to obtain a marked data set of the corresponding scene; S3, using the labeled data set of the first scenario to train each grasping inference network to obtain m grasping detection models; Use the trained grasping detection models to perform grasping detection on the first scene and generate grasping postures; S4, let i=1; S5, select the best one from all grasp detection models of the i-th scene as the teacher model of the i+1-th scene and the initialization benchmark of the student model; Take part of the training data from the labeled datasets of the first i scenes, as well as the labeled dataset of the i+1th scene, adopt the knowledge distillation method and use the teacher model as a guide to guide the training of the currently initialized student model; Use the current grasping detection models to perform grasping detection on the previous i+1 scenes and generate grasping postures; Update i=i+1 and repeat step S5 until the grasping detection models for all scenes are obtained.

[0006] Furthermore, the operation process of each attention layer is as follows: the two input feature maps are passed through the average pooling layer and the maximum pooling layer; then the four output features are stacked in the channel dimension, and finally the convolution layer is used to output the weights of the two feature maps input to the attention layer; expressed as: ; ; ; ; In the formula, , , Respectively represent the features extracted by the first, second and third feature extraction modules for the depth image, color image, dark image and depth image, respectively. represents the first attention layer, , are the output weights of the first attention layer for its two input feature maps, Represents the feature map after the fusion of color image features and depth image features, represents the second attention layer, , are the output weights of the second attention layer for its two input feature maps, Represents the output of the multimodal fusion module.

[0007] Furthermore, the grasping posture feature maps output by the grasping inference network include: grasping angle sine map, grasping angle cosine map, grasping width map and grasping quality map; all the output grasping posture feature maps are synthesized to obtain the following grasping posture : The position of the center of the gripper in the world coordinate system , the rotation angle of the gripper around the Z axis of the world coordinate system , the gripping width of the clamping jaws , the width of the gripper itself , crawl quality score , expressed as .

[0008] Furthermore, based on the two measurement indicators of object-level grasping accuracy and image-level grasping accuracy, the best model is selected from all grasping detection models; if the two measurement indicators of object-level grasping accuracy and image-level grasping accuracy do not reach the best at the same time, the model with higher image-level grasping accuracy is given priority as the best model; the calculation formulas of object-level grasping accuracy and image-level grasping accuracy are: ; ; In the formula, OL represents the object level, IL represents the image level, represents the object-level grasping accuracy, represents the image-level capture accuracy, The number of objects in the i-th image, Indicates the number of objects successfully captured in the i-th image, The total number of images in the validation set, Indicates the number of images in the validation set where all objects are captured correctly.

[0009] Furthermore, in step S5, a knowledge distillation method is adopted and the teacher model is used as a guide to guide the training of the currently initialized student model, and its loss function includes the distillation loss of the teacher model guiding the student model training and the label loss of the student model training.

[0010] Furthermore, the distillation loss includes the difference between the student model and the teacher model in each dimension of the output grasping pose.

[0011] Furthermore, the distillation loss also includes a spatial feature loss between the output feature maps of the intermediate convolutional layers of the student model and the teacher model; The distillation loss is expressed as: ; In the formula, represents the distillation loss, They represent the cosine distillation loss of the grasping angle, the sine distillation loss of the grasping angle, the distillation loss of the grasping width, and the distillation loss of the grasping quality. Represents the spatial feature loss of the intermediate convolutional layer; ; In the formula, represents the two-dimensional vector after the teacher model output feature map is flattened, Represents the two-dimensional vector of the flattened output feature map of the student model, Indicates cosine similarity calculation; ; ; ; In the formula, It is used to measure the difference in width between the intermediate convolutional layer outputs of the teacher model and the student model. It is used to measure the difference in height between the output of the intermediate convolutional layer of the teacher model and the student model. Represents the feature map output by the intermediate convolutional layer of the student model, Represents the feature map output by the intermediate convolutional layer of the teacher model, Respectively represent the number of channels, height, and width of the feature map, They represent the channel number of the feature map and the pixel number in the height and width directions respectively.

[0012] Furthermore, the label loss includes the sum of the losses in each dimension between the grasping pose output by the student model and the true label, which is expressed as: ; In the formula, represents the label loss, They represent the cosine label loss of the grasping angle, the sine label loss of the grasping angle, the label loss of the grasping width, and the label loss of the grasping quality respectively. The loss of each dimension adopts the SmoothL1 loss: ; In the formula, represents the grasping pose output by the student model, Represents the actual grasp pose label.

[0013] The simple and efficient multimodal fusion module proposed in the present invention is different from the previous grasping and reasoning model in that it fuses color image information and depth image information in the channel dimension. It makes full use of color image information and depth image information. The continuous learning strategy proposed in the present invention greatly reduces the model's forgetting of old tasks and balances plasticity and stability. Compared with the existing technology, it has the following advantages: The present invention is different from the previous method of fusing color image information and depth image information in the channel dimension. It fully utilizes the color image information and depth image information with only a small increase in computational complexity. On the KWG dataset, the object-level grasping accuracy is improved by 0.86% and the image-level grasping accuracy is improved by 0.65% compared with the basic model.

[0014] The model proposed by the present invention is more suitable for continuous learning than the benchmark model. When using the same continuous learning strategy, the model proposed by the present invention performs best. At the same time, the present invention proposes to use the cosine similarity to calculate the distillation loss for the continuous learning of the robot grasping reasoning model, making full use of the rich knowledge reserves and mature experience contained in the teacher model to accurately guide and effectively regulate the learning process of the student model, thereby greatly reducing the student model's forgetting of past knowledge in the process of continuous learning and training strategies, balancing the stability and plasticity of the student model, so that the average accuracy of the model is effectively improved in the continuous learning of five tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the structure of the present invention for performing multimodal fusion in the spatial dimension.

[0016] Figure 2 It is a network flow diagram of the method for generating grasping posture based on modal fusion of the present invention.

[0017] Figure 3 A continuous learning training process is proposed for the present invention.

[0018] Figure 4 Schematic diagram of the teacher-student architecture used for continuous learning in the present invention. DETAILED DESCRIPTION

[0019] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0020] In order to make full use of color image information and depth image information, and give the robot the ability to learn new grasping skills while maintaining the original grasping ability, the object grasping method based on multimodal fusion in this embodiment includes: Step 1: Build m identical grasping inference networks, with color images and depth feature maps as input and the object’s grasping pose as output.

[0021] The crawling reasoning network includes a multimodal fusion module, a residual block and a transposed convolution module.

[0022] The existing grasping and reasoning network currently uses a method of fusing depth image information and color image information, which mainly integrates depth image information and color image information in the channel dimension, but this method greatly increases the amount of calculation. Figure 1 As shown in the figure, it includes 3 feature extraction modules with the same network structure but independent of each other and 2 attention layers.

[0023] The inputs of the first, second and third feature extraction modules are respectively: depth image, color image, and fusion information of color image and depth image.

[0024] In addition, the two attention layers use spatial attention to generate weights to fuse multimodal information in the spatial dimension: the first attention layer is used to fuse the output features of the first and second feature extraction modules, and the second attention layer is used to fuse the output features of the first attention layer and the third feature extraction module.

[0025] The operation process of each attention layer is as follows: the two input feature maps are passed through the average pooling layer and the maximum pooling layer respectively; then the four output features are stacked in the channel dimension, and finally the convolution layer is used to output the weights of the two feature maps input to the attention layer. Then, the two weights generated by the attention mechanism are used to perform weighted summation processing on the corresponding two feature maps. In this way, the attention mechanism can effectively suppress the information carried by the relatively unimportant parts of the color image and the depth image, so that the fused multimodal information can focus more on the key features and improve the accuracy and effectiveness of the overall information processing. It can be expressed as follows: ; ; ; ; In the formula, , , Respectively represent the features extracted by the first, second and third feature extraction modules for the depth image, color image, dark image and depth image, respectively. represents the first attention layer, , are the output weights of the first attention layer for its two input feature maps, Represents the feature map after the fusion of color image features and depth image features, represents the second attention layer, , are the output weights of the second attention layer for its two input feature maps, Represents the output of the multimodal fusion module.

[0026] The features obtained by multiple fusion modules The input residual block is processed by multiple residual blocks and finally upsampled by transposed convolution to output the grasping quality map, grasping width map, grasping angle sine map, and grasping angle cosine map for synthesizing the grasping posture, such as Figure 2 shown.

[0027] The grasping pose finally output by the grasping inference network Includes: The position of the center of the gripper in the world coordinate system , the rotation angle of the gripper around the Z axis of the world coordinate system , the gripping width of the clamping jaws , the width of the gripper itself , crawl quality score , expressed as .

[0028] Step 2: Obtain training data sets for different scenes, and mark the grasping posture of each training data to obtain a marked data set for the corresponding scene.

[0029] Step 3: Use the labeled data set of the first scene to train each grasping inference network to obtain m grasping detection models.

[0030] In step 3 of this embodiment, the existing training method can be used to train each grasping inference network, that is, each grasping detection model obtained by training can be used to perform grasping detection on the first scene to generate a grasping posture.

[0031] Step 4, let i=1.

[0032] Step 5: When performing grasping detection tasks for other new scenes, the existing trained grasping detection model needs to be further trained using the labeled dataset of the new scene.

[0033] The embodiment of the present invention adopts a continuous learning training strategy, and the training process is as follows Figure 3, this strategy constructs a teacher-student architecture model. In this architecture, the teacher model and the student model have the same structure. The teacher model makes full use of the rich knowledge reserves and mature experience contained in the teacher model to accurately guide and effectively regulate the learning process of the student model, thereby greatly reducing the student model's forgetting of past knowledge in the process of continuous learning. When training for the next new task, the model with the best overall performance in the previous task will be selected as the teacher model of the next task and the initialization benchmark of the student model. For example, the best model selected after the training of Task 0 is completed will be set as the teacher model in Task 1, and this model will be used to initialize the student model of Task 1. In the continuous learning process of a new task, the teacher model will be frozen and its internal parameters will no longer be updated.

[0034] Step 5.1, select the best one from all the grasp detection models of the i-th scene as the teacher model of the i+1-th scene and the initialization benchmark of the student model.

[0035] In order to evaluate the network's ability to predict grasp poses, a correct grasp box should satisfy the following two conditions: (1) The area of ​​the intersection between the predicted grasping box and the true grasping box divided by the area of ​​the two grasping boxes combined must be greater than 25%.

[0036] (2) The deviation angle between the predicted grasping box and the actual grasping box should be less than 30°.

[0037] In order to evaluate the effect of the trained grasping detection model, the embodiment of the present invention selects the best model from all grasping detection models based on the two measurement indicators of object-level grasping accuracy and image-level grasping accuracy. If the two measurement indicators of object-level grasping accuracy and image-level grasping accuracy do not reach the best at the same time, the model with higher image-level grasping accuracy is given priority as the best model.

[0038] Object-level grasping accuracy indicates the proportion of objects that can be grasped to all objects, while image-level grasping accuracy refers to the proportion of images that are considered correct to all images. If there is only one valid grasp for each object in the image, the image is considered correct, otherwise it is incorrect.

[0039] Among them, the calculation formulas for object-level grasping accuracy and image-level grasping accuracy are: ; ; In the formula, OL represents the object level, IL represents the image level, represents the object-level grasping accuracy, represents the image-level capture accuracy, The number of objects in the i-th image, Indicates the number of objects successfully captured in the i-th image, Represents the total number of images in the validation set, Indicates the number of images in the validation set where all objects are captured correctly.

[0040] In step 5.2, part of the training data is taken from the labeled data sets of the first i scenes, as well as the labeled data set of the i+1th scene, and the knowledge distillation method is adopted and the teacher model is used as a guide to guide the training of the currently initialized student model.

[0041] This embodiment adopts a knowledge distillation method and uses a teacher model as a guide to guide the training of the currently initialized student model. Its loss function includes the distillation loss of the teacher model guiding the student model training and the label loss of the student model training.

[0042] (1) Distillation loss.

[0043] In the embodiment of the present invention, the distillation loss includes the difference between the student model and the teacher model in each dimension of the output grasping posture, and further includes the spatial feature loss between the output feature maps of the intermediate convolutional layers of the student model and the teacher model. It is expressed as: ; In the formula, represents the distillation loss, They represent the cosine distillation loss of the grasping angle, the sine distillation loss of the grasping angle, the distillation loss of the grasping width, and the distillation loss of the grasping quality. Represents the spatial feature loss of the intermediate convolutional layer.

[0044] like Figure 4 As shown, the output form of the generative machine grasping point generation network is the grasping angle sine graph, the grasping angle cosine graph and the grasping width graph, which is significantly different from the traditional classification model. Therefore, if the teacher model output is conventionally softened and the KL divergence or cross entropy loss is used to calculate the distillation loss, it is not suitable for the robot grasping point generation network. Based on this, the present invention proposes to use cosine similarity to calculate the distillation loss of grasping quality, grasping width, grasping angle sine value, and grasping angle cosine value. The size of the feature map output by the grasping inference network in the invention is The calculation step is to first transform the output shape of the student model and the teacher model into ( ), and then calculate the cosine similarity as follows: ; ; ; ; In the formula, represents the flattened two-dimensional vector output by the teacher model, represents the flattened two-dimensional vector output by the student model, Represents cosine similarity calculation.

[0045] The embodiment of the present invention also uses spatial feature loss as part of the distillation loss, which is applied to the middle layer of the network. The formula is as follows: ; ; In the formula, It is used to measure the difference in width between the intermediate convolutional layer outputs of the teacher model and the student model. It is used to measure the difference in height between the output of the intermediate convolutional layer of the teacher model and the student model. Represents the feature map output by the intermediate convolutional layer of the student model, Represents the feature map output by the intermediate convolutional layer of the teacher model, Respectively represent the number of channels, height, and width of the feature map, They represent the channel number of the feature map and the pixel number in the height and width directions respectively.

[0046] The embodiment of the present invention calculates the difference in width and height between the intermediate convolutional layer outputs of the teacher model and the student model. , Jointly constitute the spatial feature loss , which can be expressed as follows: ; (2) Label loss.

[0047] The label loss includes the sum of the losses in each dimension between the grasping pose output by the student model and the true label, expressed as: ; In the formula, represents the label loss, They represent the cosine label loss of the grasping angle, the sine label loss of the grasping angle, the label loss of the grasping width, and the label loss of the grasping quality respectively.

[0048] In this embodiment, the loss of each dimension adopts SmoothL1 loss: ; In the formula, represents the grasping pose output by the student model, Represents the actual grasp pose label.

[0049] Step 5.3, use the current grasping detection models to perform grasping detection on the previous i+1 scenes and generate grasping postures.

[0050] Step 5.4, update i=i+1 and return to step 5.1 until the grasping detection model for all scenes is obtained.

[0051] In the continuous learning of new scene tasks, the goal is to balance stability and plasticity. Backward transfer (BF) measures the degree of forgetting. If the degree of forgetting is high, it means that the model tends to adapt to new tasks and lacks stability. The backward transfer after training on the i-th scene task is defined as follows. The lower the BF value, the stronger the model's ability to overcome forgetting.

[0052] ; In order to evaluate the overall performance of the model on all trained tasks during the continuous learning process, the average accuracy is usually used as a performance measurement indicator. It is expressed as follows.

[0053] ;

[0054] Where t represents the number of tasks currently trained. It represents the grasping accuracy of the model for the jth task when training the tth task, and s represents the number of all new tasks.

[0055] Examples: Step 1: Construct the robot grasping continuous learning dataset KWG-CL. The dataset is for conveyor belt grasping dataset, shot by Realsense D455 camera, with 10 categories, including paper cups, paper boxes, plastic bottles, plastic boxes, glass bottles, metal bottles, wood blocks, ceramic cups, foam, and others. According to different scenarios, the dataset is divided into five tasks. Task 1 is a simple background, including 209 RGB-D images; Task 2 is a background with debris, including 242 RGB-D images; Task 3 is a background with plastic bags on the conveyor belt, including 256 RGB-D images; Task 4 is a background with plastic bags and non-oily impurities, including 217 RGB-D images; Task 5 is a background with plastic bags and oily impurities, including 240 RGB-D images. All color images are aligned with depth images.

[0056] Step 2: GR-ConvNet is used as the baseline network, and a three-branch multimodal fusion module is added. Figure 1, the model is trained on the KWG2024 dataset, and the dataset is divided into training set and test set in a ratio of 8:2. In this example, the training requires Ubuntu 18.04.5 LTS and above, and the system environment requires Python3.8.5, Pytroch1.10.1, and above. The hardware platform needs to meet the requirements of NVIDIA GeForce RTX4090 (24G) graphics card, 16G or more memory and a hard disk with a capacity of not less than 256G. When training the model, a total of 150 epochs are trained, and the Adam optimizer is used for parameter update. The learning rate adjustment strategy uses the OneCycleLR learning rate scheduler. The main purpose of OneCycleLR is to dynamically change the learning rate during the training process to train the neural network more efficiently. Its feature is that within a training cycle, the learning rate first rises and then falls according to a specific pattern. This change pattern is similar to the shape of a "triangular wave" or a "periodic wave". This scheduling strategy helps the model converge faster and can improve the performance of the model to a certain extent. Random seeds are used during training so that the same random splits are obtained when running the training at different times, so that the experimental results can be reproduced. The accuracy comparison of the basic model and the improved model of the method of the present invention is shown in Table 1.

[0057] ; Step 3: The grasping scene included in the KWG2024 dataset is regarded as Task 0. The model trained on the KWG2024 dataset is continuously learned on the KWG-CL dataset. Each task is trained for 20 rounds. The continuous learning training process is as follows: Figure 3 , the learning rate scheduler is reinitialized when training each new task, and the Adam optimizer is used to update the parameters. The learning rate adjustment strategy adopts OneCycleLR. After each round of training, the model is tested in the test set, and the average accuracy of all previously trained tasks is calculated.

[0058] In this example, continuous learning is divided into five tasks, and the model with the highest average grasping accuracy in the validation set is selected as the basic prediction model for the subsequent steps. The continuous learning method proposed in the present invention is compared with the continuous learning method of memory replay. In the memory replay method, the total training data set of each task is composed of 20 samples of each old task training set and the current task training set. The method based on memory replay is widely regarded as one of the most effective means in the field of continuous learning. As shown in Table 2, the experimental results show that in the first three tasks, the continuous learning strategy proposed in the present invention is better than memory replay, but as the number of tasks increases, the number of samples of memory replay increases. Therefore, in tasks four and five, the continuous learning strategy proposed in the present invention is slightly worse than memory replay. At the same time, the model of the present invention shows more excellent continuous learning adaptability than the benchmark model. Using the same training strategy and continuous learning of the same number of tasks, the average grasping accuracy of the model proposed in the present invention is higher than that of the benchmark model in the first four tasks, and the grasping accuracy in the fifth task is basically the same, which fully demonstrates its advantages in continuous learning tasks and can adapt to various types of tasks. The reason why the forgetting rate in the fifth task is higher than that of the baseline model is that the baseline model exhibits poor learning ability in the first four tasks.

[0059] At the same time, the present invention proposes to use cosine similarity as a method for calculating distillation loss. Compared with using SmoothL1 loss as distillation loss, using cosine similarity to calculate distillation loss results in an average accuracy of 1.16% after continuous learning of five tasks.

[0060] ; The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. An object grasping method based on multimodal fusion, characterized in that: include: S1, build m identical grasping inference networks, with color images and depth feature maps as input and feature maps of grasping postures as output; The crawling inference network includes a multimodal fusion module, a residual block and a transposed convolution module; the multimodal fusion module includes three feature extraction modules and two attention layers; the inputs of the first, second and third feature extraction modules are respectively: a depth image, a color image, a dark image and a depth image; the first attention layer is used to fuse the output features of the first and second feature extraction modules, and the second attention layer is used to fuse the output features of the first attention layer and the third feature extraction module; S2, obtaining training data sets of different scenes, and marking the grasping posture of each training data to obtain a marked data set of the corresponding scene; S3, using the labeled data set of the first scenario to train each grasping inference network to obtain m grasping detection models; Use the trained grasping detection models to perform grasping detection on the first scene and generate grasping postures; S4, let i=1; S5, select the best one from all grasp detection models of the i-th scene as the teacher model of the i+1-th scene and the initialization benchmark of the student model; Take part of the training data from the labeled datasets of the first i scenes, as well as the labeled dataset of the i+1th scene, adopt the knowledge distillation method and use the teacher model as a guide to guide the training of the currently initialized student model; Use the current grasping detection models to perform grasping detection on the previous i+1 scenes and generate grasping postures; Update i=i+1 and repeat step S5 until the grasping detection models for all scenes are obtained.

2. The object grasping method based on multimodal fusion according to claim 1 is characterized in that: The operation process of each attention layer is as follows: both input feature maps are passed through the average pooling layer and the maximum pooling layer; The four output features are then stacked in the channel dimension, and finally the convolutional layer is used to output the weights of the two feature maps input to the attention layer; it is expressed as ； In the formula, , , Respectively represent the features extracted by the first, second and third feature extraction modules for the depth image, color image, dark image and depth image, respectively. represents the first attention layer, , are the output weights of the first attention layer for its two input feature maps, Represents the feature map after the fusion of color image features and depth image features, represents the second attention layer, , are the output weights of the second attention layer for its two input feature maps, Represents the output of the multimodal fusion module.

3. The object grasping method based on multimodal fusion according to claim 1, characterized in that: The grasping posture feature maps output by the grasping inference network include: grasping angle sine map, grasping angle cosine map, grasping width map and grasping quality map; all the output grasping posture feature maps are synthesized to obtain the following grasping posture : The position of the center of the gripper in the world coordinate system , the rotation angle of the gripper around the Z axis of the world coordinate system , the gripping width of the clamping jaws , the width of the gripper itself , crawl quality score , expressed as .

4. The object grasping method based on multimodal fusion according to claim 1 is characterized in that: Based on the two measurement indicators of object-level grasping accuracy and image-level grasping accuracy, the best model is selected from all grasping detection models; if the two measurement indicators of object-level grasping accuracy and image-level grasping accuracy do not reach the best at the same time, the model with higher image-level grasping accuracy is given priority as the best model; the calculation formulas of object-level grasping accuracy and image-level grasping accuracy are: ； ； In the formula, OL represents the object level, IL represents the image level, represents the object-level grasping accuracy, represents the image-level capture accuracy, The number of objects in the i-th image, Indicates the number of objects successfully captured in the i-th image, Represents the total number of images in the validation set, Indicates the number of images in the validation set where all objects are captured correctly.

5. The object grasping method based on multimodal fusion according to claim 1, characterized in that: In step S5, a knowledge distillation method is adopted and the teacher model is used as a guide to guide the training of the currently initialized student model. The loss function includes the distillation loss of the teacher model guiding the student model training and the label loss of the student model training.

6. The object grasping method based on multimodal fusion according to claim 5 is characterized in that: The distillation loss includes the difference between the student model and the teacher model in each dimension of the output grasp pose.

7. The object grasping method based on multimodal fusion according to claim 6, characterized in that: The distillation loss also includes the spatial feature loss between the output feature maps of the intermediate convolutional layers of the student model and the teacher model; The distillation loss is expressed as: ； In the formula, represents the distillation loss, They represent the cosine distillation loss of the grasping angle, the sine distillation loss of the grasping angle, the distillation loss of the grasping width, and the distillation loss of the grasping quality. Represents the spatial feature loss of the intermediate convolutional layer; ； In the formula, represents the two-dimensional vector after the teacher model output feature map is flattened, Represents the two-dimensional vector of the flattened output feature map of the student model, Indicates cosine similarity calculation; ； ； ； In the formula, It is used to measure the difference in width between the intermediate convolutional layer outputs of the teacher model and the student model. It is used to measure the difference in height between the output of the intermediate convolutional layer of the teacher model and the student model. Represents the feature map output by the intermediate convolutional layer of the student model, Represents the feature map output by the intermediate convolutional layer of the teacher model, Respectively represent the number of channels, height, and width of the feature map, They represent the channel number of the feature map and the pixel number in the height and width directions respectively.

8. The object grasping method based on multimodal fusion according to claim 5, characterized in that: The label loss includes the sum of the losses in each dimension between the grasping pose output by the student model and the true label, expressed as: ； In the formula, represents the label loss, They represent the cosine label loss of the grasping angle, the sine label loss of the grasping angle, the label loss of the grasping width, and the label loss of the grasping quality respectively. The loss of each dimension adopts the SmoothL1 loss: ； In the formula, represents the grasping pose output by the student model, Represents the actual grasp pose label.

Images