A method, device, equipment and storage medium for predicting the grasping position of a robotic arm

Abstract

The present application provides a method, device, equipment and storage medium for predicting the grasping position of a robotic arm. The method comprises: performing feature extraction of a target grasping area image at multiple depths to obtain feature maps of the target grasping area image corresponding to the multiple depths; wherein the feature maps of different depths represent different feature information. Performing fusion encoding of multiple channels on the feature maps of the target grasping area image corresponding to the multiple depths to obtain a fused feature map of the target grasping area image corresponding to the multiple channels; wherein an attention mechanism is used between the multiple channels, and the fused feature maps of different channels correspond to different image sizes. Based on the fused feature maps of the target grasping area image corresponding to the multiple channels, the grasping position of the target grasping area image is predicted. The present application can improve the prediction accuracy of the grasping position of the robotic arm.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for predicting a grasping position of a robot arm. Background Art

[0002] The robotic arm is the most widely used automated mechanical device in the field of robotics technology. It has been widely used in industrial manufacturing, medical treatment, entertainment services, military, semiconductor manufacturing, and space exploration.

[0003] The current mainstream robotic arm solution uses visual technology to predict the grasping position. How to improve the prediction accuracy of the robotic arm's grasping position is a technical problem to be solved by this application. Summary of the invention

[0004] The purpose of this application is to provide a method, device, equipment and storage medium for predicting the grasping position of a robot arm, which can improve the prediction accuracy of the grasping position of the robot arm.

[0005] In order to achieve the above purpose, the embodiment of the present application is implemented as follows:

[0006] In a first aspect, a method for predicting a grasping position of a robot arm is provided, comprising:

[0007] Performing feature extraction of the target capture area image at multiple depths to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein the feature maps at different depths represent different feature information;

[0008] Performing fusion encoding of multiple channels on the feature maps of the target capture area image corresponding to the multiple depths to obtain fusion feature maps of the target capture area image corresponding to the multiple channels; wherein an attention mechanism is used between the multiple channels, and fusion feature maps of different channels correspond to different image sizes;

[0009] Based on the fusion feature maps of the target grabbing area image corresponding to the multiple channels, the grabbing position of the target grabbing area image is predicted.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the method described in the first aspect.

[0011] According to a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the method described in the first aspect when executed by a processor.

[0012] When predicting the position of the robot arm, the present application first extracts features of multiple depths on the current target grasping area image of the robot arm, and the feature maps of different depths represent different feature information. Next, based on the different channels of the attention mechanism, the feature maps of the target grasping area image corresponding to the multiple depths are fused and encoded in multiple channels, and the fused feature maps of each channel correspond to different image sizes, so as to analyze the importance of the fused feature maps of each channel based on the attention mechanism; finally, based on the fused feature maps of each channel and the corresponding importance, the grasping position of the target grasping area image is analyzed and predicted with multiple depth dimensions and multiple visual attentions, thereby improving the prediction accuracy of the grasping position. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 Schematic diagram of the process of predicting the grasping position of a robot arm according to an embodiment of the present application.

[0015] Figure 2 Schematic diagram of the model structure of the device for predicting the grasping position of the robotic arm according to an embodiment of the present application.

[0016] Figure 3 A schematic diagram of the grasping of the device for predicting the grasping position of the robot arm according to an embodiment of the present application.

[0017] Figure 4 Schematic diagram of the structure of the device for predicting the grasping position of the robot arm according to an embodiment of the present application.

[0018] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0020] On the one hand, an embodiment of the present application proposes a method for predicting the grasping position of a robotic arm. Figure 1The following is a flow chart of the prediction method according to the embodiment of the present application, including:

[0021] S102, performing feature extraction at various depths on the target capture area image to obtain feature maps corresponding to the various depths of the target capture area image.

[0022] Among them, feature maps of different depths represent different feature information. For example, the feature map of the bottom layer retains more image details (such as texture, color, etc.) of the target capture area image, while the feature map of the high layer presents semantic information such as contour and shape.

[0023] Specifically, this embodiment can input the target capture area image into a deep residual network (Residual Network, ResNet) to perform feature extraction on the target capture area image based on convolutional layers of various depths in the deep residual network, such as Conv3_4 convolutional layer, Conv4_23 convolutional layer and Conv5_3 convolutional layer.

[0024] S104, performing fusion encoding of multiple channels on the feature maps corresponding to multiple depths of the target capture area image to obtain a fusion feature map of the target capture area image corresponding to the multiple channels.

[0025] Among them, the attention mechanism is adopted between multiple channels, and the fused feature maps of different channels correspond to different image sizes.

[0026] The visual attention mechanism is a brain signal processing mechanism unique to human vision. Visual attention enables humans to focus on a certain area with "high resolution", that is, the area to be focused on, that is, the area where attention is focused; at the same time, perceive the surrounding images with "low resolution", generally irrelevant or low-relevant areas. Then adjust the focus and combine the information to make inferences accordingly.

[0027] In this embodiment, each channel is used to fuse and encode feature maps of multiple depths into a fused feature map. By determining the correlation between the channels, the fused feature maps of each channel are weighted accordingly, so that the fused feature maps output by the channels with higher correlation correspond to higher weights to improve their role in subsequent predictions; similarly, the fused feature maps output by the channels with lower correlation correspond to lower weights to reduce their role in subsequent predictions.

[0028] The above-mentioned weighted process is the attention mechanism. As an exemplary introduction, the attention mechanism of this embodiment is specifically used to: perform average pooling processing on the fused feature maps of the multiple channels, and perform one-dimensional convolution on the average pooling processing results to determine the correlation between the multiple channels; then, based on the correlation between the multiple channels, assign an attention weight (hereinafter referred to as weight) to each channel; finally, perform weighted processing on the fused feature maps of each channel according to the corresponding attention weight.

[0029] Attention mechanisms can be divided into two types: hard attention and soft attention, depending on whether the attention mechanism accesses the entire image or the region. The former only focuses on a small area at a time, and adjusts the area that needs to be paid attention to each time through the algorithm. Therefore, this is a non-differentiable method. It is generally based on reinforcement learning, and rewards are used to motivate the agent to adjust the area that needs to be paid attention to each time. The latter aligns weights and places weights on all pixels of the image. It can calculate weights through regression, so it is generally done by gradient optimization and other methods.

[0030] Here, the Soft Attention method in this embodiment has three reasons:

[0031] First, the control of the robot arm is a continuous process, not a zero-one problem. It is a differentiable process, so random prediction cannot be used;

[0032] Second, the soft attention mechanism can extract global information at once. In the field of image detection, the goal of the attention mechanism is to learn an attention allocation weight and apply the weight to the original features to make the model more effective after the weight is applied. The weight learned here can act on multiple dimensions, such as space, channel, and even the original image. By paying more attention to the area of ​​interest, the role of the area of ​​no interest in image segmentation can be suppressed as much as possible.

[0033] Third, the robotic arm faces small target objects, and the soft attention method can achieve better results for small target detection.

[0034] Specifically, the attention mechanism of this embodiment is used to:

[0035] The fused feature maps of multiple channels are average pooled, and the average pooling results are convolved in one dimension to determine the correlation between multiple channels. Next, based on the correlation between multiple channels, attention weights are assigned to each channel. After that, the fused feature maps of each channel are weighted according to the corresponding attention weights.

[0036] S106, predicting a capture position of the target capture area image based on the fused feature maps of the target capture area image corresponding to the multiple channels.

[0037] This embodiment uses a target detection algorithm to predict the capture position of the target capture area image.

[0038] The task of object detection is to find all the targets (objects) of interest in an image and determine their categories and locations. It is one of the core issues in the field of computer vision.

[0039] Specifically, this embodiment inputs the fused feature map of the target grab area image corresponding to the multiple channels into the target detection model to predict the detection frame of the target grab area image. The detection frame is used to characterize the grab position, that is, the grab position is determined by the detection frame position information of the five-tuple of center point coordinates, center point coordinate correction value, angle, width and height.

[0040] Among them, the target detection model can be trained by taking the fusion feature map of the sample capture area image corresponding to the multiple channels as input and taking the detection box position information (five-tuple) annotated by the sample capture area image as label.

[0041] During the training process, the annotated detection box position information label is the true value. After the fused feature map of multiple channels corresponding to the sample capture area image is input into the target detection model, the target detection model can try to predict the detection box corresponding to the sample capture area image. After that, the loss function is used to calculate the loss (error) between the predicted detection box position information and the true value shown by the label, and the parameters of the target detection model are adjusted with the purpose of reducing the loss, so that the detection box (detection box position information) predicted by the target detection model can be more and more accurate. After multiple rounds of iterations of the above process, the target detection model can have the ability to predict the detection box.

[0042] In summary, when predicting the position of the robotic arm, the method of this embodiment first extracts features at multiple depths on the current target grasping area image of the robotic arm, and feature maps of different depths represent different feature information. Next, based on different channels of the attention mechanism, multiple channels of fusion encoding are performed on the feature maps of the target grasping area image corresponding to the multiple depths, and the fused feature maps of each channel correspond to different image sizes, so as to analyze the importance of the fused feature maps of each channel based on the attention mechanism; finally, based on the fused feature maps of each channel and the corresponding importance, multiple depth dimensions and multiple visual attentions are used to analyze and predict the grasping position of the target grasping area image, thereby improving the prediction accuracy of the grasping position.

[0043] The method of this embodiment is described in detail below.

[0044] refer to Figure 2 As shown, the method of this embodiment is executed by a large model. The overall structure of the large model consists of ResNet, K-neighbor Channel Attention Module (KCNA) and CenterNet. Among them:

[0045] 1. ResNet

[0046] The Conv3_4 convolutional layer, Conv4_23 convolutional layer, and Conv5_3 convolutional layer of ResNet are used to extract features of the target grasping area image at various depths, respectively, and obtain feature maps of three sizes: 64×64, 32×32, and 16×16.

[0047] 2. K-neighbor Channel Attention Module (KCNA)

[0048] KCNA consists of channels of six image encoding sizes: 64×64, 32×32, 16×16, 8×8, 4×4, and 2×2. The K-neighbor attention mechanism is used between these six channels, and each channel is used to fuse the feature maps output by the ResNetConv3_4 convolutional layer, the Conv4_23 convolutional layer, and the Conv5_3 convolutional layer to obtain the fused feature maps of the six image encoding sizes.

[0049] In the K-neighborhood attention mechanism, KCNA performs global pooling on each channel of the input feature map without dimensionality reduction, which increases computational efficiency and outputs a feature map of dimension 1×1×C, which reflects the global receptive field. First, Sigmoid is used as the activation function to output a weight of 1×1×C. The weight ω is calculated as follows S2-1:

[0050] ω=σ(f{ {w} (g(x))) S2-1

[0051] Where σ is the Sigmoid function, g(x) is the global average pooling operation [3], g(x) is the interaction function between k channel neighborhoods, and w and h represent the width and height of the feature map, respectively, as shown in S2-2:

[0052]

[0053] The f function represents the point multiplication of the input feature map and the obtained weights in order to redistribute the weights of each channel. W represents a C×C parameter matrix, that is:

[0054] f {W}(x) = Wx S2-3

[0055] The x in S2-3 represents the independent variable. Substituting S2-3 and S2-2 into S2-1, we can get the following derivation process:

[0056]

[0057] y = g(x), that is

[0058] Take y i The set of k adjacent channels of this channel is denoted as Available

[0059]

[0060] This embodiment obtains the mutual information between channels and k channel neighborhoods through a one-dimensional convolution operation with a convolution kernel of k (two k values ​​are equal), redistributes the weights of the features of each channel, and multiplies the input feature map with the obtained weights. This module makes each channel no longer independent, strengthens the connection between each channel, and promotes the contribution of the overall features to the detection results, so it will also significantly improve the final detection results. This solution sets all W i The same as above, and implemented with a one-dimensional convolution with a convolution kernel of k, S2-4 is simplified to:

[0061] B i =σ(Conv1d k (y)) S2-5

[0062] In Notice S2-5, the general calculation method of the convolution kernel k is:

[0063] k=N in +2P+1-SN out S2-6

[0064] Among them, N in With N out are the sizes of the input feature map and the output feature map respectively. In this scheme, the two are equal. S is 1, p is the padding size, which is 2 in this scheme. Therefore, k is 5 in this experiment (which is also the value of neighborhood k).

[0065] 3. CenterNet

[0066] CenterNet is an object detection model that can identify the grasped objects in the grasped area image and frame them with a detection box.

[0067] In this embodiment, the center point coordinates G(x g ,y g)、center point coordinate correction value offset(o x ,o y ), angle, width and height, the five-tuple information can determine the exact grasping position of the robotic arm.

[0068] In summary, the specific process of the large model is:

[0069] 1) Input image P1, extracted by ResNet, and the feature maps of Conv3_4 convolution layer, Conv4_23 convolution layer and Conv5_3 convolution layer are marked as f1, f2, f3.

[0070] 2) Concat f1, f2, and f3 (a universal function for fusion features) to obtain a multi-channel fusion feature map Z t

[0071] 3) t=0, Repeat:

[0072] Z t ReLU is used to increase the nonlinear relationship, and Dropout is used to prevent overfitting to obtain Z t ';

[0073] Z t 'Get feature map Z through KNCA attention mechanism t ”;

[0074] t+=1;

[0075] Until t=6.

[0076] 4) For the set {Z t ″|t=1,2,3,4,5,6}, perform upsampling and Concat series operation to obtain the final attention weight distribution Y

[0077] 5) Input the Y-weighted fusion feature map into CenterNet to predict the detection box and determine the five-tuple parameters corresponding to the detection box.

[0078] refer to Figure 3 As shown, the method of this embodiment can accurately predict the grasping position in the actual application scenarios of grasping boxes, bananas and bottles, thereby assisting the robot arm to grasp successfully.

[0079] On the other hand, an embodiment of the present application also provides a device for predicting the grasping position of a robotic arm. Figure 4 It is a schematic diagram of the structure, including:

[0080] The extraction module 410 is used to extract features of the target capture area image at multiple depths to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein feature maps of different depths represent different feature information.

[0081] The encoding module 420 is used to perform multi-channel fusion encoding on the feature maps of the target capture area image corresponding to the multiple depths to obtain the fused feature maps of the target capture area image corresponding to the multiple channels; wherein an attention mechanism is adopted between the multiple channels, and the fused feature maps of different channels correspond to different image sizes.

[0082] The prediction module 430 is used to predict the capture position of the target capture area image based on the fusion feature map of the target capture area image corresponding to the multiple channels.

[0083] When predicting the position of the robot arm, the device of this embodiment first extracts features of multiple depths on the current target grasping area image of the robot arm, and the feature maps of different depths represent different feature information. Next, based on different channels of the attention mechanism, the feature maps of the target grasping area image corresponding to the multiple depths are fused and encoded in multiple channels, and the fused feature maps of each channel correspond to different image sizes, so as to analyze the importance of the fused feature maps of each channel based on the attention mechanism; finally, based on the fused feature maps of each channel and the corresponding importance, the grasping position of the target grasping area image is analyzed and predicted with multiple depth dimensions and multiple visual attentions, thereby improving the prediction accuracy of the grasping position.

[0084] Optionally, the attention mechanism is used to: perform average pooling processing on the fused feature maps of the multiple channels, and perform one-dimensional convolution on the average pooling processing results to determine the correlation between the multiple channels; assign an attention weight to each of the channels based on the correlation between the multiple channels; and weight the fused feature maps of each channel according to the corresponding attention weight.

[0085] Optionally, the prediction module 430 predicts the grasping position of the target grasping area image based on the fused feature map of the target grasping area image corresponding to the multiple channels, including: inputting the fused feature map of the target grasping area image corresponding to the multiple channels into the target detection model to predict the detection box of the target grasping area image; wherein the detection box is used to characterize the grasping position.

[0086] Optionally, the target detection model is trained by taking the fusion feature map of the sample capture area image corresponding to the multiple channels as input and using the detection box position information annotated by the sample capture area image as label.

[0087] Optionally, the detection frame position information includes: center point coordinates, center point coordinate correction value, angle, width and height.

[0088] Optionally, the extraction module 410 performs feature extraction at multiple depths on the target capture area image, including: inputting the target capture area image into a deep residual network to perform feature extraction on the target capture area image based on convolutional layers of multiple depths in the deep residual network.

[0089] Optionally, the convolutional layers of multiple depths include a Conv3_4 convolutional layer, a Conv4_23 convolutional layer, and a Conv5_3 convolutional layer of the deep residual network.

[0090] It should be understood that the prediction device of this embodiment can be used as Figure 1 The execution subject of the method shown can thus realize Figure 1 The steps and functions in the method shown.

[0091] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this specification. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0092] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0093] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to perform the following operations:

[0094] Feature extraction at multiple depths is performed on the target capture area image to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein feature maps at different depths represent different feature information.

[0095] The feature maps of the target capture area image corresponding to the multiple depths are fused and encoded in multiple channels to obtain fused feature maps of the target capture area image corresponding to the multiple channels; wherein an attention mechanism is adopted between the multiple channels, and fused feature maps of different channels correspond to different image sizes.

[0096] Based on the fusion feature maps of the target grabbing area image corresponding to the multiple channels, the grabbing position of the target grabbing area image is predicted.

[0097] When predicting the position of the robotic arm, the electronic device of this embodiment first extracts features of multiple depths on the current target grasping area image of the robotic arm, and feature maps of different depths represent different feature information. Next, based on different channels of the attention mechanism, multiple channels of fusion encoding are performed on the feature maps of the target grasping area image corresponding to the multiple depths, and the fusion feature maps of each channel correspond to different image sizes, so as to analyze the importance of the fusion feature maps of each channel based on the attention mechanism; finally, based on the fusion feature maps of each channel and the corresponding importance, the grasping position of the target grasping area image is analyzed and predicted with multiple depth dimensions and multiple visual attentions, thereby improving the prediction accuracy of the grasping position.

[0098] The above-mentioned method for predicting the grasping position of the robot arm disclosed in the embodiment shown in this specification can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0099] Of course, in addition to software implementation, the electronic device of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0100] In addition, the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions. When the instructions are executed by a portable electronic device including multiple application programs, the portable electronic device can execute Figure 1 The steps of the method shown include:

[0101] Feature extraction at multiple depths is performed on the target capture area image to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein feature maps at different depths represent different feature information.

[0102] The feature maps of the target capture area image corresponding to the multiple depths are fused and encoded in multiple channels to obtain fused feature maps of the target capture area image corresponding to the multiple channels; wherein an attention mechanism is adopted between the multiple channels, and fused feature maps of different channels correspond to different image sizes.

[0103] Based on the fusion feature maps of the target grabbing area image corresponding to the multiple channels, the grabbing position of the target grabbing area image is predicted.

[0104] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] The above are only embodiments of this specification and are not intended to limit this specification. For those skilled in the art, this specification may be subject to various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims of this specification. In addition, all other embodiments obtained by ordinary technicians in this field without creative work shall fall within the scope of protection of this document.

Claims

1. A method for predicting the grasping position of a robot arm, characterized in that: include: Performing feature extraction at multiple depths on the target capture area image to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein the feature maps of different depths represent different feature information, the multiple depths include a depth representing a bottom layer and a depth representing a high layer, the feature map representing the depth of the bottom layer retains the image details of the target capture area image; the feature map representing the depth of the high layer retains the semantic information of the contour and / or shape; The feature maps corresponding to the multiple depths of the target capture area image are fused and encoded in multiple channels to obtain fused feature maps corresponding to the multiple channels of the target capture area image; wherein an attention mechanism is used between the multiple channels, and fused feature maps of different channels correspond to different image sizes; the attention mechanism is used to: perform average pooling processing on the fused feature maps of the multiple channels, and perform one-dimensional convolution on the average pooling processing results to determine the correlation between the multiple channels; based on the correlation between the multiple channels, assign an attention weight to each channel; and perform weighted processing on the fused feature map of each channel according to the corresponding attention weight; Based on the fused feature map of the target grab area image corresponding to the multiple channels, predicting the grab position of the target grab area image, including: inputting the fused feature map of the target grab area image corresponding to the multiple channels into a target detection model to predict a detection frame of the target grab area image; wherein the detection frame is used to characterize the grab position; the target detection model is trained by taking the fused feature map of the sample grab area image corresponding to the multiple channels as input and the detection frame position information annotated by the sample grab area image as a label.

2. The method according to claim 1, characterized in that The detection frame position information includes: center point coordinates, center point coordinate correction value, angle, width and height.

3. The method according to claim 1, characterized in that Perform multiple depth feature extraction on the target capture area image, including: The target grab area image is input into a deep residual network to perform feature extraction on the target grab area image based on convolutional layers of various depths in the deep residual network.

4. The method according to claim 3, characterized in that: The convolutional layers of various depths include the Conv3_4 convolutional layer, the Conv4_23 convolutional layer and the Conv5_3 convolutional layer of the deep residual network.

5. A device for predicting the grasping position of a robot arm, characterized in that: include: An extraction module is used to perform feature extraction of multiple depths on the target capture area image to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein the feature maps of different depths represent different feature information, the multiple depths include a depth representing a bottom layer and a depth representing a high layer, the feature map representing the depth of the bottom layer retains the image details of the target capture area image; the feature map representing the depth of the high layer retains the semantic information of the contour and / or shape; The encoding module is used to perform fusion encoding of multiple channels on the feature maps corresponding to the multiple depths of the target capture area image, so as to obtain the fusion feature maps of the target capture area image corresponding to the multiple channels; wherein an attention mechanism is adopted between the multiple channels, and the fusion feature maps of different channels correspond to different image sizes; the attention mechanism is used to: perform average pooling processing on the fusion feature maps of the multiple channels, and perform one-dimensional convolution on the average pooling processing results to determine the correlation between the multiple channels; based on the correlation between the multiple channels, assign an attention weight to each channel; and perform weighted processing on the fusion feature map of each channel according to the corresponding attention weight; A prediction module is used to predict the grasping position of the target grasping area image based on the fused feature map of the target grasping area image corresponding to the multiple channels, including: inputting the fused feature map of the target grasping area image corresponding to the multiple channels into the target detection model to predict the detection frame of the target grasping area image; wherein the detection frame is used to characterize the grasping position; the target detection model is trained with the fused feature map of the sample grasping area image corresponding to the multiple channels as input and the detection frame position information annotated by the sample grasping area image as label.

6. An electronic device comprising: processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the following operations: Performing feature extraction at multiple depths on the target capture area image to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein the feature maps of different depths represent different feature information, the multiple depths include a depth representing a bottom layer and a depth representing a high layer, the feature map representing the depth of the bottom layer retains the image details of the target capture area image; the feature map representing the depth of the high layer retains the semantic information of the contour and / or shape; The feature maps corresponding to the multiple depths of the target capture area image are fused and encoded in multiple channels to obtain fused feature maps corresponding to the multiple channels of the target capture area image; wherein an attention mechanism is used between the multiple channels, and fused feature maps of different channels correspond to different image sizes; the attention mechanism is used to: perform average pooling processing on the fused feature maps of the multiple channels, and perform one-dimensional convolution on the average pooling processing results to determine the correlation between the multiple channels; based on the correlation between the multiple channels, assign an attention weight to each channel; and perform weighted processing on the fused feature map of each channel according to the corresponding attention weight; Based on the fused feature map of the target grab area image corresponding to the multiple channels, predicting the grab position of the target grab area image, including: inputting the fused feature map of the target grab area image corresponding to the multiple channels into a target detection model to predict a detection frame of the target grab area image; wherein the detection frame is used to characterize the grab position; the target detection model is trained by taking the fused feature map of the sample grab area image corresponding to the multiple channels as input and the detection frame position information annotated by the sample grab area image as a label.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the following steps are implemented: Performing feature extraction at multiple depths on the target capture area image to obtain feature maps of the target capture area image corresponding to the multiple depths; wherein the feature maps of different depths represent different feature information, the multiple depths include a depth representing a bottom layer and a depth representing a high layer, the feature map representing the depth of the bottom layer retains the image details of the target capture area image; the feature map representing the depth of the high layer retains the semantic information of the contour and / or shape; The feature maps corresponding to the multiple depths of the target capture area image are fused and encoded in multiple channels to obtain fused feature maps corresponding to the multiple channels of the target capture area image; wherein an attention mechanism is used between the multiple channels, and fused feature maps of different channels correspond to different image sizes; the attention mechanism is used to: perform average pooling processing on the fused feature maps of the multiple channels, and perform one-dimensional convolution on the average pooling processing results to determine the correlation between the multiple channels; based on the correlation between the multiple channels, assign an attention weight to each channel; and perform weighted processing on the fused feature map of each channel according to the corresponding attention weight; Based on the fused feature map of the target grab area image corresponding to the multiple channels, predicting the grab position of the target grab area image, including: inputting the fused feature map of the target grab area image corresponding to the multiple channels into a target detection model to predict a detection frame of the target grab area image; wherein the detection frame is used to characterize the grab position; the target detection model is trained by taking the fused feature map of the sample grab area image corresponding to the multiple channels as input and the detection frame position information annotated by the sample grab area image as a label.

Images