A multi-scale component grasping posture estimation method and system

Abstract

The invention discloses a multi-scale component grasping posture estimation method and system, which includes acquiring scene images and annotating them to obtain a data set; building a posture estimation neural network model; inputting the data set into an image block encoding module to output a coding feature sequence; inputting the coding feature sequence into a sliding window attention retention feature encoding module to perform sequence segmentation with perception windows of different sizes and respectively apply the attention retention mechanism to map to high-dimensional feature output; inputting the high-dimensional feature into a multi-scale space pyramid perception module and optimizing and outputting the optimized feature sequence; a window attention retention feature decoding module receives the optimized feature sequence and outputs the grasping posture; setting a loss function and updating the network parameters according to the gradient situation to obtain a trained posture estimation neural network model, completing the grasping posture estimation of the industrial real grasping operation scene image, and completing the grasping operation task. The grasping efficiency and accuracy are improved, and the hardware cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of robot vision technology, and in particular relates to a multi-scale component grasping posture estimation method and system. Background Art

[0002] Grasping posture estimation is an important research topic of robot vision system. Its main purpose is to estimate the effective grasping posture of target parts from the scene image perceived by the robot vision system, and transmit it to the robot execution unit to complete the grasping task. It has a wide range of applications in intelligent manufacturing fields such as aircraft manufacturing and automobile assembly. In the face of boring and repetitive tasks such as loading and unloading, assembly, and sorting of parts, the effective application of robot automation to replace manual work and improve the intelligence of production lines is the main trend of current industrial development.

[0003] In actual operation scenarios, target parts are of various types, sizes, and positions. Therefore, how to accurately locate the position of parts in the scene and screen the graspable area to generate an effective grasping posture is still a difficult problem in this field. The existing industrial application grasping detection algorithm is based on the classification and position detection of target parts, and determines the effective grasping posture by screening the pre-saved grasping posture template. Since the grasping task is simplified and a complete effective grasping posture is preset, this application method can achieve the purpose of rapid detection. However, for weak textures or occlusions, it cannot effectively handle the spatial interaction and detection positioning problems between parts in the scene, resulting in a low success rate of grasping parts. Therefore, how to quickly and efficiently complete grasping detection and overcome the problems of various types, sizes, and unstable placement of parts is still a challenge. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a multi-scale component grasping posture estimation method and system.

[0005] The technical solution adopted by the present invention to solve the technical problem is:

[0006] A multi-scale component grasping posture estimation method, the method comprising the following steps:

[0007] S100: Acquire a plurality of scene images containing target parts and respectively mark each pixel coordinate as the grasping quality, gripper width, and rotation numerical matrix of the effective grasping posture when the grasping rectangle center is grasped, to obtain a data set;

[0008] S200: Building a posture estimation neural network model, including an image block encoding module, a sliding window retention attention feature encoding module, a multi-scale spatial pyramid perception module, and a window retention attention feature decoding module, which are connected in sequence; wherein the window retention attention feature decoding module and the sliding window retention attention feature encoding module are also connected in a jump form;

[0009] S300: input the data set into the image block encoding module for convolution encoding, and output the encoding feature sequence;

[0010] S400: The sliding window attention-retaining feature encoding module receives the encoding feature sequence, and divides the sequence into perception windows of different sizes. The distance between adjacent sliding windows is smaller than the size of the perception window. The cut encoding feature sequence is respectively applied with the attention-retaining mechanism in units of windows, and the processed feature sequence is restored to the position before the segmentation. The overlapping part is processed in the form of weighted average to obtain the feature information perceived by different windows.

[0011] S500: The multi-scale spatial pyramid perception module receives and fuses the feature information perceived by multiple windows. The feature information of different perception fields complement each other and outputs an optimized feature sequence that is both global and local.

[0012] S600: The window-retained attention feature decoding module receives the optimized feature sequence for decoding and outputs a grasping posture, wherein the grasping posture includes grasping quality, grasping angle and grasping width;

[0013] S700: Setting the loss function to calculate the training posture estimation loss value and updating the network parameters according to the gradient, obtaining a trained posture estimation neural network model, completing the grasping posture estimation of the industrial real grasping operation scene image, and completing the grasping operation task according to the estimated grasping posture.

[0014] Preferably, the grasping posture is expressed as the grasping width as the length, the clamping jaw fixed width as the width, and the image coordinate system The minimum acute angle between the axis and the long side of the grasping rectangle is in the form of a rotation angle, where the fixed width of the gripper depends on the physical properties of the gripper and is a fixed value. It does not participate in the posture estimation. The grasping posture is specifically expressed by the formula:

[0015] ;

[0016] ;

[0017] In the formula is the set of grasping postures of the target objects in the scene, They are respectively the grasping quality, grasping width and grasping angle of the effective grasping posture when the pixel coordinate of the point is the center of the grasping rectangle, and Represent the height and width of the input image respectively.

[0018] Preferably, the image block encoding module includes a two-dimensional convolution layer, the convolution layer input size is the input image channel, and the encoding output size is , the convolution kernel size and step size are the image block size , S300 includes:

[0019] Select an image from the data set at random, use a convolution kernel of a fixed size to convolutionally encode the image, and obtain a coded feature sequence with a sequence dimension of one. The convolution kernel size can be divided by the image size, and the convolution kernel advance step length is consistent with the convolution kernel size, specifically:

[0020] ;

[0021] In the formula, represents the input image data, represents a convolution operation, represents the encoding feature sequence, Represents the encoding feature dimension.

[0022] Preferably, the sliding window attention-retaining feature encoding module includes a first normalization layer, a sliding window multi-head attention-retaining layer, a second normalization layer and a multi-layer perceptron, and the sliding window multi-head attention-retaining layer input size is , the encoding output size is , the convolution kernel size is , the convolution kernel step size is , S400 includes:

[0023] S410: The sliding window attention-retaining feature encoding module receives the encoding feature sequence, splits the encoding feature sequence into a two-dimensional sequence, passes through the first normalization layer, and inputs it into the sliding window multi-head attention-retaining layer, and The window size cuts the two-dimensional coding feature sequence, and the spacing between adjacent windows is ,The cut coded feature sequence is applied with the retention attention mechanism in units of windows, and the processed feature sequence is restored to the position before splitting. The overlapping part is processed in the form of weighted average and expanded to a one-dimensional sequence form. The expanded sequence is input to the second normalization layer for normalization. The normalized feature sequence is input to the multi-layer perceptron for feature optimization, specifically:

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] In the formula represents the sliding window cutting operation, Indicates the number of windows after cutting. For the cutting window size, represents the retention of the attention mechanism, represents a normalization operation. represents a multi-layer perceptron, represents the high-dimensional feature output of the sliding window-retaining attention feature encoding module, , , All of them are learnable parameter matrices generated during the attention calculation process. represents the normalized feature sequence, represents the feature sequence after the sliding window is cut, represents the feature sequence after attention-retaining optimization, is the weight matrix of attention scores, and Represent the rows and columns of the feature matrix respectively;

[0032] S420: Repeat the above steps to change the cutting window size , so that it can generate feature information of different perception window sizes, and perform zero-padding alignment when the generated sequence becomes shorter due to increasing the perception window, so as to build a multi-scale pyramid feature sequence.

[0033] Preferably, the multi-scale spatial pyramid perception module includes an attention-reserving calculation layer, and S500 includes:

[0034] Attention retention is applied to the constructed multi-scale pyramid feature sequence, and channel attention retention is applied to the sequence of the same layer, that is, the sequence generated by the same perception window size; thereafter, the sequences of different layers, that is, the sequences generated by different perception window sizes, are concatenated and spatial attention retention is applied to obtain the optimized feature sequence, specifically:

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] In the formula Indicates that attention is retained between sequence channels, which is used to calculate feature dimensions The weight coefficients between them and optimize the feature representation, Indicates The window size changes. Indicates that attention is retained between feature sequences and is used to calculate sequence channels The weight coefficients between them and optimize the feature representation, represents the feature sequence concatenation operation, represents the output of the multi-scale pyramid perception module, that is, the optimized feature sequence, represents the feature sequence after applying channel-preserving attention, Represents the concatenated sequence of feature sequences generated by multiple perception windows.

[0040] Preferably, the window-retained attention feature decoding module includes a first normalization layer, a window multi-head retained attention layer, a second normalization layer, a multi-layer perceptron and a feature mapping layer. The input generated by the multi-scale pyramid perception module is normalized by the first normalization layer and then input into the window multi-head retained attention layer for cutting and retained attention optimization respectively. The feature sequence generated after optimization is input into the second normalization layer for normalization. The normalized feature sequence is weighted with the output of the previous window attention layer and the output of the symmetrical sliding window attention layer and then refined by the multi-layer perceptron. The refined feature input mapping layer maps out the grasping posture. S600 includes:

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] In the formula Indicates the window cutting operation, Indicates Layer window attention layer output, Indicates Layer sliding window attention layer output, represents the feature map layer, and Respectively represent the height and width of the window, Indicates the number of windows, represents the normalized feature sequence, represents the feature sequence generated after the window cutting operation, Represents the feature sequence after retaining attention optimization, represents a multi-layer perceptron, Represents the output of a multilayer perceptron.

[0050] Preferably, the loss function in S400 is a mean square error loss function, specifically:

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] In the formula , , Respectively represent the loss between the detector's predicted grasp score, grasp width, grasp angle and the true value, represents the predicted value, represents the true value, They represent weighting coefficients respectively.

[0058] A multi-scale component grasping posture estimation system, comprising a data set acquisition module, a posture estimation neural network model building module, an image block encoding module, a sliding window attention retention feature encoding module, a multi-scale spatial pyramid perception module, a window attention retention feature decoding module, and a model training and grasping posture estimation module;

[0059] A data set acquisition module is used to acquire a number of scene images containing target parts and respectively mark each pixel coordinate as the grasping quality, gripper width and rotation numerical matrix of the effective grasping posture when the grasping rectangle center is grasped to obtain a data set;

[0060] A posture estimation neural network model building module is used to build a posture estimation neural network model, including an image block encoding module, a sliding window retention attention feature encoding module, a multi-scale spatial pyramid perception module, and a window retention attention feature decoding module connected in sequence; wherein the window retention attention feature decoding module and the sliding window retention attention feature encoding module are also connected in a jump form;

[0061] The image block encoding module is used to receive the data set and perform convolution encoding, and output the encoding feature sequence;

[0062] The sliding window attention-retaining feature encoding module receives the coded feature sequence and divides the sequence into perception windows of different sizes. The distance between adjacent sliding windows is smaller than the size of the perception window. The cut coded feature sequence is applied with attention-retaining mechanism in units of windows, and the processed feature sequence is restored to the position before segmentation. The overlapping part is processed in the form of weighted average to obtain the feature information perceived by different windows.

[0063] The multi-scale spatial pyramid perception module receives and fuses the feature information perceived by multiple windows. The feature information of different perception fields complement each other and outputs an optimized feature sequence that is both global and local.

[0064] The window-preserving attention feature decoding module receives the optimized feature sequence for decoding and outputs a grasping posture, wherein the grasping posture includes grasping quality, grasping angle and grasping width;

[0065] The model training and grasping posture estimation module is used to set the loss function to calculate the training posture estimation loss value and update the network parameters according to the gradient situation, so as to obtain the trained posture estimation neural network model, complete the grasping posture estimation of the industrial real grasping operation scene image, and complete the grasping operation task according to the estimated grasping posture.

[0066] The above-mentioned multi-scale component grasping posture estimation method and system propose a sliding window retained attention feature encoding method and a window retained attention feature decoding method. Under the premise of ensuring the grasping detector's ability to perceive local detail information of the scene, it fully utilizes the GPU performance in a high-parallel form to achieve efficient training and reasoning, and solves the problem that existing detection methods cannot take into account both scene detail perception and training and reasoning efficiency; a multi-scale spatial pyramid perception module is proposed, which introduces multi-scale information perception capabilities, establishes spatial correlations of multi-receptive field perception information, and improves the detector's ability to extract universal and effective feature information for grasping detection, so that it can cope with the problem of poor universality in grasping and detecting components of many types, different sizes, and placement positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1It is a flow chart of a multi-scale component grasping posture estimation method in one embodiment of the present invention;

[0068] Figure 2 It is a schematic diagram of the principle of a multi-scale component grasping posture estimation method in one embodiment of the present invention;

[0069] Figure 3 FIG. 1 is a schematic diagram of a partial refinement of a posture estimation neural network model in one embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0071] In one embodiment, Figure 1 and Figure 2 As shown, a multi-scale component grasping posture estimation method comprises the following steps:

[0072] S100: obtaining a plurality of scene images containing target parts and respectively marking each pixel coordinate as the grasping quality, gripper width and rotation numerical matrix of the effective grasping posture when the grasping rectangle center is grasped, to obtain a data set; further, the data set is divided into a training set and a test set in a ratio of 9:1;

[0073] S200: Building a posture estimation neural network model, including an image block encoding module, a sliding window retention attention feature encoding module, a multi-scale spatial pyramid perception module, and a window retention attention feature decoding module, which are connected in sequence; wherein the window retention attention feature decoding module and the sliding window retention attention feature encoding module are also connected in a jump form;

[0074] S300: input the data set into the image block encoding module for convolution encoding, and output the encoding feature sequence;

[0075] S400: The sliding window attention-retaining feature encoding module receives the encoding feature sequence, and divides the sequence into perception windows of different sizes. The distance between adjacent sliding windows is smaller than the size of the perception window. The cut encoding feature sequence is respectively applied with the attention-retaining mechanism in units of windows, and the processed feature sequence is restored to the position before the segmentation. The overlapping part is processed in the form of weighted average to obtain the feature information perceived by different windows.

[0076] S500: The multi-scale spatial pyramid perception module receives and fuses the feature information perceived by multiple windows. The feature information of different perception fields complement each other and outputs an optimized feature sequence that is both global and local.

[0077] S600: The window-retained attention feature decoding module receives the optimized feature sequence for decoding and outputs a grasping posture, wherein the grasping posture includes grasping quality, grasping angle and grasping width;

[0078] S700: Setting the loss function to calculate the training posture estimation loss value and updating the network parameters according to the gradient, obtaining a trained posture estimation neural network model, completing the grasping posture estimation of the industrial real grasping operation scene image, and completing the grasping operation task according to the estimated grasping posture.

[0079] The present invention aims to propose a multi-scale component grasping posture estimation method, which aims to solve the practical problems existing in the application tasks of loading and unloading, assembling and sorting components in the context of intelligent manufacturing such as aircraft manufacturing and automobile assembly, such as (1) the existing detection methods cannot take into account the perception of scene details and the efficiency of training and reasoning; (2) the target components are of many types, different sizes and non-fixed placement, resulting in poor universality of the grasping detection method. By (1) proposing a sliding window attention-retaining feature extraction method, while ensuring the grasping detector's ability to perceive local detail information of the scene, the GPU performance is fully utilized in a highly parallel form to achieve efficient training and reasoning, thereby solving the problem that the existing detection method cannot take into account the perception of scene details and the efficiency of training and reasoning; (2) proposing a multi-scale spatial pyramid perception module, by introducing multi-scale information perception capabilities, establishing the spatial correlation of multi-receptive field perception information, and improving the detector's ability to extract universal and effective feature information for grasping detection, so that it can cope with the problem of poor universality of grasping detection of components of many types, different sizes and placement positions.

[0080] In one embodiment, the grasping posture is represented by the grasping width as the length, the clamping width as the width, and the image coordinate system. The minimum acute angle between the axis and the long side of the grasping rectangle is in the form of a rotation angle, where the fixed width of the gripper depends on the physical properties of the gripper and is a fixed value. It does not participate in the posture estimation. The grasping posture is specifically expressed by the formula:

[0081] ;

[0082] ;

[0083] In the formula is the set of grasping postures of the target objects in the scene, They are respectively the grasping quality, grasping width and grasping angle of the effective grasping posture when the pixel coordinate of the point is the center of the grasping rectangle, and Represent the height and width of the input image respectively.

[0084] In one embodiment, the image block encoding module includes a two-dimensional convolution layer, the convolution layer input size is the input image channel, and the encoding output size is , the convolution kernel size and step size are the image block size , S300 includes:

[0085] Select an image from the data set at random, use a convolution kernel of a fixed size to convolutionally encode the image, and obtain a coded feature sequence with a sequence dimension of one. The convolution kernel size can be divided by the image size, and the convolution kernel advance step length is consistent with the convolution kernel size, specifically:

[0086] ;

[0087] In the formula, represents the input image data, represents a convolution operation, represents the encoding feature sequence, Represents the encoding feature dimension.

[0088] Specifically, the image block encoding module contains a two-dimensional convolutional layer with an input size of the input image channel In this embodiment , the encoding output size is In this embodiment , the convolution kernel size is the image block size In this embodiment is 4, and the convolution kernel step size is consistent with the convolution kernel size. .

[0089] In one embodiment, Figure 3 As shown, the sliding window attention feature encoding module includes the first normalization layer, the sliding window multi-head attention layer, the second normalization layer and the multi-layer perceptron. The input size of the sliding window multi-head attention layer is , the encoding output size is , the convolution kernel size is , the convolution kernel step size is , S400 includes:

[0090] S410: The sliding window attention-retaining feature encoding module receives the encoding feature sequence, splits the encoding feature sequence into a two-dimensional sequence, passes through the first normalization layer, and inputs it into the sliding window multi-head attention-retaining layer, and The window size cuts the two-dimensional coding feature sequence, and the spacing between adjacent windows is ,The cut coded feature sequence is applied with the retention attention mechanism in units of windows, and the processed feature sequence is restored to the position before splitting. The overlapping part is processed in the form of weighted average and expanded to a one-dimensional sequence form. The expanded sequence is input to the second normalization layer for normalization. The normalized feature sequence is input to the multi-layer perceptron for feature optimization, specifically:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] In the formula represents the sliding window cutting operation, Indicates the number of windows after cutting. For the cutting window size, represents the retention of the attention mechanism, represents a normalization operation. represents a multi-layer perceptron, represents the high-dimensional feature output of the sliding window-retaining attention feature encoding module, , , All of them are learnable parameter matrices generated during the attention calculation process. represents the normalized feature sequence, represents the feature sequence after the sliding window is cut, represents the feature sequence after attention-retaining optimization, is the weight matrix of attention scores, and Represent the rows and columns of the feature matrix respectively;

[0099] S420: Repeat the above steps to change the cutting window size , so that it can generate feature information of different perception window sizes, and perform zero-padding alignment when the generated sequence becomes shorter due to increasing the perception window, so as to build a multi-scale pyramid feature sequence.

[0100] Specifically, the sliding window attention feature encoding module contains two normalization layers, a multi-layer perceptron, and a sliding window multi-head attention layer. The input size of the sliding window multi-head attention layer is , the encoding output size is , the convolution kernel size is , in this example , the convolution kernel step size is A depth of The feature encoding module, in this case , each layer of the network has a different number of attention heads The attention retention calculation layer, in this case .

[0101] In one embodiment, the multi-scale spatial pyramid perception module includes a reserved attention calculation layer, S500 includes:

[0102] Attention retention is applied to the constructed multi-scale pyramid feature sequence, and channel attention retention is applied to the sequence of the same layer, that is, the sequence generated by the same perception window size; thereafter, the sequences of different layers, that is, the sequences generated by different perception window sizes, are concatenated and spatial attention retention is applied to obtain the optimized feature sequence, specifically:

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] In the formula Indicates that attention is retained between sequence channels, which is used to calculate feature dimensions The weight coefficients between them and optimize the feature representation, Indicates The window size changes. Indicates that attention is retained between feature sequences and is used to calculate sequence channels The weight coefficients between them and optimize the feature representation, represents the feature sequence concatenation operation, represents the output of the multi-scale pyramid perception module, that is, the optimized feature sequence, represents the feature sequence after applying channel-preserving attention, Represents the concatenated sequence of feature sequences generated by multiple perception windows.

[0108] Specifically, the multi-scale spatial pyramid perception module contains a depth of , the number of heads is The attention retention calculation layer, in this case , , the input size is a multi-scale sequence , the output size is ; The depth is , the number of heads is The attention retention calculation layer, in this case , , the input size is a multi-scale sequence , the output size is the same as the input size.

[0109] In order to solve the problem of size differences of target objects in perceived scene data, the present invention introduces a sliding window attention-retaining feature encoding module, a multi-scale spatial pyramid perception module and a window attention-retaining feature decoding module, and completes the block encoding of the scene image through windows of different sizes. This processing method outputs multiple image data sequences that need to be subsequently processed according to different windows, which increases the difficulty of data processing. The attention calculation in the traditional transformer will greatly increase the memory usage and calculation time in the calculation process as the sequence increases. In order to solve the problem of increased sequence length, the present invention optimizes the attention calculation method, realizes the linearization of the attention calculation through a retention mechanism, and then ensures that the relevant calculation content can be completed in the form of loop iteration, thereby reducing the calculation cost.

[0110] For example, when the perceived image size is 224*224, assuming that the scene contains a 20*20 object A and a 70*70 object B, when cutting with a fixed window, assuming that the window is 50*50, the global features of object A can be fully obtained, but the local features are not extracted in detail, while all the features of object B are distributed on at least two windows, which can obtain more detailed local features, and the extraction of global features is not as complete as that of object A. In addition, block coding will also be performed in blank areas without objects, and the sequence of this part of the area is invalid for subsequent analysis. Based on this situation, the present invention solves three problems: (1) the window is too small, resulting in incomplete global feature capture and generating too many useless sequences, making it difficult to locate the object; (2) the window is too large, resulting in insufficient capture of local detail features, and the inability to effectively estimate the capture width and angle; (3) the problem of high computational cost of multi-scale sequences. The present invention adopts a multi-scale perception window. The large-scale window perceives the position of the object and passes it to the small window for local detail extraction. The large window perceives that there are no valid features in the current area, and passes it to the small window to inform it that this is invalid and there is no need for small window perception, thereby reducing computing expenses, and the large and small fields of view complement each other and learn from each other's information.

[0111] In one embodiment, Figure 3As shown, the window-retained attention feature decoding module includes a first normalization layer, a window multi-head retained attention layer, a second normalization layer, a multi-layer perceptron and a feature mapping layer. The input generated by the multi-scale pyramid perception module is normalized by the first normalization layer and then input into the window multi-head retained attention layer for cutting and retained attention optimization. The feature sequence generated after optimization is input into the second normalization layer for normalization. The normalized feature sequence is weighted with the output of the previous window attention layer and the output of the symmetrical sliding window attention layer and then refined by the multi-layer perceptron. The refined feature input mapping layer maps out the grasping posture. S600 includes:

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] In the formula Indicates the window cutting operation, Indicates Layer window attention layer output, Indicates Layer sliding window attention layer output, represents the feature map layer, and Respectively represent the height and width of the window, Indicates the number of windows, represents the normalized feature sequence, represents the feature sequence generated after the window cutting operation, Represents the feature sequence after retaining attention optimization, represents a multi-layer perceptron, Represents the output of a multilayer perceptron.

[0121] Specifically, the window-preserving attention feature decoding module contains two normalization layers; a multi-layer perceptron; a window multi-head preserving attention layer, and the input size of the window multi-head preserving attention layer is , the encoding output size is , the convolution kernel size is , the convolution kernel step size is A depth of The feature encoding module, in this case , each layer of the network has a different number of attention heads Keep the attention calculation layer, in this case ; A feature mapping module, the input data format is , the output data format is .

[0122] In one embodiment, the loss function in S400 is a mean square error loss function, specifically:

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] In the formula , , Respectively represent the loss between the detector's predicted grasp score, grasp width, grasp angle and the true value, represents the predicted value, represents the true value, They represent weighting coefficients respectively.

[0130] A multi-scale component grasping posture estimation system, comprising a data set acquisition module, a posture estimation neural network model building module, an image block encoding module, a sliding window attention retention feature encoding module, a multi-scale spatial pyramid perception module, a window attention retention feature decoding module, and a model training and grasping posture estimation module;

[0131] A data set acquisition module is used to acquire a number of scene images containing target parts and respectively mark each pixel coordinate as the grasping quality, gripper width and rotation numerical matrix of the effective grasping posture when the grasping rectangle center is grasped to obtain a data set;

[0132] A posture estimation neural network model building module is used to build a posture estimation neural network model, including an image block encoding module, a sliding window retention attention feature encoding module, a multi-scale spatial pyramid perception module, and a window retention attention feature decoding module connected in sequence; wherein the window retention attention feature decoding module and the sliding window retention attention feature encoding module are also connected in a jump form;

[0133] The image block encoding module is used to receive the data set and perform convolution encoding, and output the encoding feature sequence;

[0134] The sliding window attention-retaining feature encoding module receives the coded feature sequence and divides the sequence into perception windows of different sizes. The distance between adjacent sliding windows is smaller than the size of the perception window. The cut coded feature sequence is applied with attention-retaining mechanism in units of windows, and the processed feature sequence is restored to the position before segmentation. The overlapping part is processed in the form of weighted average to obtain the feature information perceived by different windows.

[0135] The multi-scale spatial pyramid perception module receives and fuses the feature information perceived by multiple windows. The feature information of different perception fields complement each other and outputs an optimized feature sequence that is both global and local.

[0136] The window-preserving attention feature decoding module receives the optimized feature sequence for decoding and outputs a grasping posture, wherein the grasping posture includes grasping quality, grasping angle and grasping width;

[0137] The model training and grasping posture estimation module is used to set the loss function to calculate the training posture estimation loss value and update the network parameters according to the gradient situation, so as to obtain the trained posture estimation neural network model, complete the grasping posture estimation of the industrial real grasping operation scene image, and complete the grasping operation task according to the estimated grasping posture.

[0138] For the specific definition of a multi-scale component grasping posture estimation system, please refer to the definition of a multi-scale component grasping posture estimation method mentioned above, which will not be repeated here. Each module in the above-mentioned multi-scale component grasping posture estimation system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] Compared with the prior art, the advantages of the present invention are:

[0140] (1) High efficiency of grasping and detection: The present invention uses weight coefficients instead of normalization functions in the attention calculation link, which solves the problem of nonlinearity in the attention score calculation link caused by the normalization function in the conventional attention calculation method, improves the parallelism of the attention calculation link, and combines the calculation mode of window attention to improve the degree of parallelization in the model reasoning process, fully utilizes the parallel computing performance of GPU, realizes efficient model reasoning, and reduces the time cost of grasping and detection.

[0141] (2) High grasping detection accuracy: Existing grasping detection methods ignore the difference in perceived content under different model perception fields, resulting in the grasping detection accuracy being affected by the different sizes and placements of the target objects. The present invention perceives scene image feature information through different perception windows, and enhances the spatial correlation of sequence information under different perception fields by fusing channel retention attention and spatial retention attention, thus realizing a perception process from coarse to fine, and improving the grasping detection accuracy.

[0142] (3) Low hardware requirements for grasping detection: The present invention solves the nonlinear problem in the attention calculation link, avoids the memory usage caused by excessive matrix parameter transfer in the attention score calculation link, and can adjust the size of intermediate variables according to actual hardware conditions, thereby reducing the hardware cost of industrial application of learning-based grasping detection methods.

[0143] The above is a detailed introduction to a multi-scale component grasping posture estimation method and system provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-scale component grasping posture estimation method, characterized in that: The method comprises the following steps: S100: Acquire a plurality of scene images containing target parts and respectively mark each pixel coordinate as the grasping quality, gripper width, and rotation numerical matrix of the effective grasping posture when the grasping rectangle center is grasped, to obtain a data set; S200: Building a posture estimation neural network model, including an image block encoding module, a sliding window retention attention feature encoding module, a multi-scale spatial pyramid perception module, and a window retention attention feature decoding module, which are connected in sequence; wherein the window retention attention feature decoding module and the sliding window retention attention feature encoding module are also connected in a jump form; S300: input the data set into the image block encoding module for convolution encoding, and output the encoding feature sequence; S400: The sliding window attention-retaining feature encoding module receives the encoding feature sequence, and divides the sequence into perception windows of different sizes. The distance between adjacent sliding windows is smaller than the size of the perception window. The cut encoding feature sequence is respectively applied with the attention-retaining mechanism in units of windows, and the processed feature sequence is restored to the position before the segmentation. The overlapping part is processed in the form of weighted average to obtain the feature information perceived by different windows. The sliding window attention feature encoding module includes the first normalization layer, the sliding window multi-head attention layer, the second normalization layer and the multi-layer perceptron. The input size of the sliding window multi-head attention layer is , the encoding output size is , the convolution kernel size is , the convolution kernel step size is , S400 includes: S410: The sliding window attention-retaining feature encoding module receives the encoding feature sequence, splits the encoding feature sequence into a two-dimensional sequence, passes through the first normalization layer, and inputs it into the sliding window multi-head attention-retaining layer, and The window size cuts the two-dimensional coding feature sequence, and the spacing between adjacent windows is ,The cut coded feature sequence is applied with the retention attention mechanism in units of windows, and the processed feature sequence is restored to the position before splitting. The overlapping part is processed in the form of weighted average and expanded to a one-dimensional sequence form. The expanded sequence is input to the second normalization layer for normalization. The normalized feature sequence is input to the multi-layer perceptron for feature optimization, specifically: ； ； ； ； ； ； ； In the formula represents the sliding window cutting operation, Indicates the number of windows after cutting. For the cutting window size, represents the retention of the attention mechanism, represents a normalization operation. represents a multi-layer perceptron, represents the high-dimensional feature output of the sliding window-retaining attention feature encoding module, , , All of them are learnable parameter matrices generated during the attention calculation process. represents the normalized feature sequence, represents the feature sequence after the sliding window is cut, represents the feature sequence after attention-retaining optimization, is the weight matrix of attention scores, and Represent the rows and columns of the feature matrix respectively, represents the encoding feature sequence, represents the encoding feature dimension, Represents input image data; S420: Repeat the above steps to change the cutting window size , so that it can generate feature information of different perception window sizes, and perform zero padding alignment when the sequence becomes shorter due to increasing the perception window, so as to build a multi-scale pyramid feature sequence; S500: The multi-scale spatial pyramid perception module receives and fuses the feature information perceived by multiple windows. The feature information of different perception fields complement each other and outputs an optimized feature sequence that is both global and local. S600: The window-retained attention feature decoding module receives the optimized feature sequence for decoding and outputs a grasping posture, wherein the grasping posture includes grasping quality, grasping angle and grasping width; S700: Setting the loss function to calculate the training posture estimation loss value and updating the network parameters according to the gradient, obtaining a trained posture estimation neural network model, completing the grasping posture estimation of the industrial real grasping operation scene image, and completing the grasping operation task according to the estimated grasping posture.

2. The method according to claim 1, characterized in that The grasping posture is expressed as the grasping width as the length, the clamping width as the width, and the image coordinate system The minimum acute angle between the axis and the long side of the grasping rectangle is in the form of a rotation angle, where the fixed width of the gripper depends on the physical properties of the gripper and is a fixed value. It does not participate in the posture estimation. The grasping posture is specifically expressed by the formula: ； ； In the formula Grasp the pose set of the target object in the scene, They are respectively the grasping quality, grasping width and grasping angle of the effective grasping posture when each marked pixel coordinate is the center of the grasping rectangle, and Represent the height and width of the input image respectively.

3. The method according to claim 2, characterized in that The image block encoding module includes a two-dimensional convolution layer, the convolution layer input size is the input image channel, and the encoding output size is , the convolution kernel size and step size are the image block size , S300 includes: Select an image from the data set at random, use a convolution kernel of a fixed size to convolutionally encode the image, and obtain a coded feature sequence with a sequence dimension of one. The convolution kernel size can be divided by the image size, and the convolution kernel advance step length is consistent with the convolution kernel size, specifically: ； In the formula, represents the input image data, represents a convolution operation, represents the encoding feature sequence, Represents the encoding feature dimension.

4. The method according to claim 3, characterized in that The multi-scale spatial pyramid perception module includes a reserved attention calculation layer, S500 includes: Attention retention is applied to the constructed multi-scale pyramid feature sequence, and channel attention retention is applied to the sequence of the same layer, that is, the sequence generated by the same perception window size; thereafter, the sequences of different layers, that is, the sequences generated by different perception window sizes, are concatenated and spatial attention retention is applied to obtain the optimized feature sequence, specifically: ； ； ； ； In the formula Indicates that attention is retained between sequence channels, which is used to calculate feature dimensions The weight coefficients between them and optimize the feature representation, Indicates The window size changes. Indicates that attention is retained between feature sequences and is used to calculate sequence channels The weight coefficients between them and optimize the feature representation, represents the feature sequence concatenation operation, represents the output of the multi-scale pyramid perception module, that is, the optimized feature sequence, represents the feature sequence after applying channel-preserving attention, Represents the concatenated sequence of feature sequences generated by multiple perception windows.

5. The method according to claim 4, characterized in that The window-retained attention feature decoding module includes a first normalization layer, a window multi-head retained attention layer, a second normalization layer, a multi-layer perceptron and a feature mapping layer. The input generated by the multi-scale pyramid perception module is normalized by the first normalization layer and then input into the window multi-head retained attention layer for cutting and retained attention optimization. The feature sequence generated after optimization is input into the second normalization layer for normalization. The normalized feature sequence is weighted with the output of the previous window attention layer and the output of the symmetrical sliding window attention layer and then refined by the multi-layer perceptron. The refined feature input mapping layer maps out the grasping posture. S600 includes: ； ； ； ； ； ； ； ； In the formula Indicates the window cutting operation, Indicates Layer window attention layer output, Indicates Layer sliding window attention layer output, represents the feature map layer, and Respectively represent the height and width of the window, Indicates the number of windows, represents the normalized feature sequence, represents the feature sequence generated after the window cutting operation, Represents the feature sequence after retaining attention optimization, represents a multi-layer perceptron, Represents the output of a multilayer perceptron.

6. The method according to claim 5, characterized in that The loss function in S400 is the mean square error loss function, specifically: ； ； ； ； ； ； In the formula , , Respectively represent the loss between the detector's predicted grasp score, grasp width, grasp angle and the true value, represents the predicted value, represents the true value, They represent weighting coefficients respectively.

7. A multi-scale component grasping posture estimation system, characterized in that: It includes a data set acquisition module, a posture estimation neural network model building module, an image block encoding module, a sliding window attention feature encoding module, a multi-scale spatial pyramid perception module, a window attention feature decoding module, and a model training and grasping posture estimation module; A data set acquisition module is used to acquire a number of scene images containing target parts and respectively mark each pixel coordinate as the grasping quality, gripper width and rotation numerical matrix of the effective grasping posture when the grasping rectangle center is grasped to obtain a data set; A posture estimation neural network model building module is used to build a posture estimation neural network model, including an image block encoding module, a sliding window retention attention feature encoding module, a multi-scale spatial pyramid perception module, and a window retention attention feature decoding module connected in sequence; wherein the window retention attention feature decoding module and the sliding window retention attention feature encoding module are also connected in a jump form; The image block encoding module is used to receive the data set and perform convolution encoding, and output the encoding feature sequence; The sliding window attention-retaining feature encoding module receives the coded feature sequence and divides the sequence into perception windows of different sizes. The distance between adjacent sliding windows is smaller than the size of the perception window. The cut coded feature sequence is applied with the attention-retaining mechanism in units of windows, and the processed feature sequence is restored to the position before segmentation. The overlapping part is processed in the form of weighted average to obtain the feature information perceived by different windows. The sliding window attention-retaining feature encoding module includes the first normalization layer, the sliding window multi-head attention-retaining layer, the second normalization layer and the multi-layer perceptron. The input size of the sliding window multi-head attention-retaining layer is , the encoding output size is , the convolution kernel size is , the convolution kernel step size is , the sliding window retaining attention feature encoding module includes: Receive the encoded feature sequence, split the encoded feature sequence into a two-dimensional sequence, pass through the first normalization layer, input it into the sliding window multi-head retention attention layer, and use The window size cuts the two-dimensional coding feature sequence, and the spacing between adjacent windows is ,The cut coded feature sequence is applied with the retention attention mechanism in units of windows, and the processed feature sequence is restored to the position before splitting. The overlapping part is processed in the form of weighted average and expanded to a one-dimensional sequence form. The expanded sequence is input to the second normalization layer for normalization. The normalized feature sequence is input to the multi-layer perceptron for feature optimization, specifically: ； ； ； ； ； ； ； In the formula represents the sliding window cutting operation, Indicates the number of windows after cutting. For the cutting window size, represents the retention of the attention mechanism, represents a normalization operation. represents a multi-layer perceptron, represents the high-dimensional feature output of the sliding window-retaining attention feature encoding module, , , All of them are learnable parameter matrices generated during the attention calculation process. represents the normalized feature sequence, represents the feature sequence after the sliding window is cut, represents the feature sequence after attention-retaining optimization, is the weight matrix of attention scores, and Represent the rows and columns of the feature matrix respectively, represents the encoding feature sequence, represents the encoding feature dimension, Represents input image data; Repeat the above steps to change the cutting window size , so that it can generate feature information of different perception window sizes, and perform zero padding alignment when the sequence becomes shorter due to increasing the perception window, so as to build a multi-scale pyramid feature sequence; The multi-scale spatial pyramid perception module receives and fuses the feature information perceived by multiple windows. The feature information of different perception fields complement each other and outputs an optimized feature sequence that is both global and local. The window-preserving attention feature decoding module receives the optimized feature sequence for decoding and outputs a grasping posture, wherein the grasping posture includes grasping quality, grasping angle and grasping width; The model training and grasping posture estimation module is used to set the loss function to calculate the training posture estimation loss value and update the network parameters according to the gradient situation, so as to obtain the trained posture estimation neural network model, complete the grasping posture estimation of the industrial real grasping operation scene image, and complete the grasping operation task according to the estimated grasping posture.

Images