Method and device for directly regressing object 6D pose based on image joint attention

Abstract

The invention discloses a direct regression object 6D pose method and device based on image joint attention, and the method comprises the steps: firstly obtaining a color image containing a target object based on an RGB camera, feeding the color image to a constructed direct regression pose network which comprises a feature extraction module, a joint attention module, an up-sampling module and a PnP-like module, and finally, directly outputting a pose result. After an input image is preprocessed, a mainstream feature extraction backbone network is combined with network model features to embed feature attention modules at different parts so as to improve the feature extraction and expression ability of the network to different dimensions of channels, spaces and the like; and aiming at the obtained intermediate features, designing a PnP-like module to replace the existing PnP solving process, and enabling the network model to directly predict the pose result of the target object. Compared with a mainstream two-stage method, the method provided by the invention can overcome the problem of error increase caused by proxy loss, and further improves the accuracy of an object 6D pose estimation result.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and pose estimation, and in particular to a method and device for directly regressing the 6D pose of an object based on image joint attention. Background Art

[0002] The 6D pose estimation problem of an object refers to calculating the position transformation relationship of an object relative to a reference coordinate system (such as a camera coordinate system or a world coordinate system) through two-dimensional images or three-dimensional point cloud information, including the rotation matrix R and the translation vector t, each with three degrees of freedom, hence the name 6D pose. In recent years, this task has received increasing attention in the field of computer vision because it has important application value in many practical scenarios, such as augmented reality, autonomous driving, and robot grasping. At the same time, this task also faces many difficulties and challenges, and is affected by a variety of external factors, such as complex scenes, occlusions between objects, objects with few textures (with unclear features), and environmental factors (such as lighting changes).

[0003] Traditional pose estimation methods often use object point clouds or color images, based on manually designed features, to construct template libraries to complete matching, thereby calculating poses. However, such methods have high requirements for feature design, large amount of calculation, insufficient real-time performance, and are difficult to meet the needs of dynamic scene tasks. Based on RGB images, combined with the powerful feature extraction capabilities of convolutional neural networks, object pose estimation tasks can be achieved. By sacrificing the time of training data in the offline stage, higher computational efficiency can be obtained in the online stage than traditional methods.

[0004] Among the existing deep learning pose estimation methods, some adopt a two-stage method. In the first stage, the corresponding relationship of key point coordinates is predicted through the network, and in the second stage, the PnP problem is constructed to solve the pose. Some adopt a one-stage method, which does not include an explicit intermediate feature extraction process and directly regresses the pose of the object. However, these methods have their limitations. The loss function of the former does not directly fit the final pose result, which may amplify the error in the PnP solution stage; the latter does not make good use of the semantic information of the intermediate high-dimensional features, resulting in a waste of data information. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a method and device for directly regressing the 6D pose of an object based on image joint attention, which fully utilizes the advantages of the two-stage and one-stage methods to jointly realize the object pose estimation task, and can effectively improve the image feature extraction capability and reduce the prediction error.

[0006] The objective of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a method for directly regressing the 6D pose of an object based on image joint attention, the method comprising the following steps:

[0007] (1) Obtain the original image for which pose estimation is required and perform preprocessing;

[0008] (2) The pre-processed image is used as the input feature. In the process of extracting high-dimensional features of the image, feature enhancement is performed based on the bottleneck attention module and the efficient channel attention module. Then, after deconvolution and upsampling operations, the intermediate features containing the correspondence relationship between the key points of the object in the image and the object mask information are obtained.

[0009] (3) The intermediate features are input into a PnP-like module, which includes multiple layers of strided convolution and group normalization operations. The stride and number of layers of the strided convolution are determined based on the dimension of the intermediate features. Finally, the predicted 6D pose is output through a fully connected layer.

[0010] Furthermore, in step (1), the preprocessing is specifically to obtain the target object from the original image through a target detection algorithm, crop it and then scale it to a uniform size to obtain a pre-processed cropped image.

[0011] Furthermore, in step (2), a feature extraction backbone network is used to extract high-dimensional features of the image. The feature extraction backbone network includes 4 layers of residual blocks. A bottleneck attention module is added between the residual blocks. Channel attention features are obtained based on average pooling operations, multi-layer perceptrons and batch normalization. Spatial attention features are obtained based on convolution operations and batch normalization, and are superimposed on the input features for feature enhancement. An efficient channel attention module is added inside the first 3 layers of residual blocks. Attention features are obtained based on average pooling operations and convolution operations, and are superimposed on the input features for feature enhancement.

[0012] Furthermore, in the convolution operation of the efficient channel attention module, the size of the convolution kernel is adaptively determined by the number of channels, and the specific calculation is as follows:

[0013]

[0014] Among them, k represents the size of the convolution kernel, C represents the number of channels, odd represents the odd number closest to the number of channels, η, λ, and δ are hyperparameters. η and λ are used to control the linear relationship between the logarithm of C and k, and δ is used to expand the receptive field of the convolution kernel.

[0015] Furthermore, in step (2), the intermediate features include 5-dimensional channels, the feature maps corresponding to the first 3-dimensional channels represent the 3D point coordinate prediction feature maps corresponding to the pixels of the object on the image, and the feature maps corresponding to the last 2-dimensional channels represent the mask prediction feature maps of the target object.

[0016] Furthermore, in step (3), after each layer of strided convolution and group normalization operation, a RELU activation function is added, and through two fully connected layers, the rotation matrix and translation vector of the predicted posture are output respectively.

[0017] Furthermore, the final pose estimation loss and intermediate feature loss are calculated separately, and the weighted sum is used as the final loss function for training; the intermediate feature loss The definition is as follows:

[0018]

[0019] in, Predict a feature map for the mask of the target object, The feature map of the three-dimensional point coordinates corresponding to the pixels of the object on the image is predicted. The first item of the intermediate feature loss represents the mask prediction loss of the target object, and the second item represents the prediction loss of the three-dimensional coordinate points. Since the three-dimensional coordinates of the background object are not the prediction targets, Multiply to filter the effect of background information.

[0020] In a second aspect, the present invention also provides a device for directly regressing an object's 6D pose based on image joint attention, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the method for directly regressing an object's 6D pose based on image joint attention is implemented.

[0021] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for directly regressing the 6D pose of an object based on image joint attention.

[0022] In a fourth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for directly regressing the 6D pose of an object based on image joint attention.

[0023] Beneficial effects of the present invention: After preprocessing the input image, the present invention embeds feature attention modules in different parts based on the mainstream feature extraction backbone network and the characteristics of the network model to enhance the network's feature extraction and expression capabilities for different dimensions such as channels and spaces; for the intermediate features obtained, a PnP-like module is designed to replace the existing PnP solution process, allowing the network model to directly predict the pose result of the target object. Compared with the mainstream two-stage method, the present invention can overcome the problem of increased error caused by proxy loss and further improve the accuracy of the object's 6D pose estimation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 It is the input and output flow chart of the present invention;

[0026] Figure 2 Schematic diagram of the direct regression pose network model based on joint attention adopted by the present invention;

[0027] Figure 3 The figure shows the visualization results of different pose estimations of the implementation example of the present invention.

[0028] Figure 4 It is a structural diagram of a device for directly regressing the 6D pose of an object based on image joint attention according to the present invention. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation examples disclosed below.

[0031] The samples for training the network in this implementation example use the public dataset LINEMOD, which contains 13 different types of objects, including objects with weak textures, symmetry and other features, which are challenging for pose estimation tasks. The setting conditions are only examples and are not limited to this. The method for directly regressing the 6D pose of an object based on joint attention of an image specifically includes the following steps:

[0032] 1. Dataset and preprocessing

[0033] Select a dataset for pose estimation tasks and preprocess the images, that is, obtain the bounding box of the target object through the target detection algorithm, crop it and scale it to a uniform size as the initialization input of the network. The specific process is as follows:

[0034] 1) The LINEMOD dataset contains information such as original images and object CAD models, as well as labels such as poses, masks, key point coordinates, etc. of different target objects. The training sample images provided by it are selected as the input of the training process of this implementation example.

[0035] 2) The input image I first detects the target object based on the mainstream target detection algorithm Faster-RCNN, obtains the bounding box containing the target object, crops it, and then scales it to a uniform size of 256×256 through bilinear interpolation. The cropped image I′ after this preprocessing step is used as the initialization input image of the direct regression network.

[0036] 2. Direct regression pose network model

[0037] Based on the mainstream feature extraction backbone network, the high-dimensional features of the preprocessed image I' are extracted, and the feature map size is expanded through several upsampling modules to obtain intermediate features containing the correspondence between key points and the object mask information. Finally, a PnP-like module is used to replace the existing PnP solution method to directly output the pose. This PnP-like module consists of several strided convolutions, activation functions, and fully connected layers. The number of the aforementioned network structures depends on the backbone network used. The PnP-like module enables the neural network to directly regress the pose of the object, thereby overcoming the error increase problem caused by non-direct regression methods such as predicting key points. The specific process is as follows:

[0038] 1) For the initialization image I′, after the feature extraction backbone network ResNet-34, the feature size is reduced to 8×8, which is 1 / 32 of the original; the number of feature channels C increases from 3 to 512. Figure 2 As shown in the feature extraction module section, the front part of ResNet-34 consists of a convolutional layer and a maximum pooling layer. After these two layers, the feature size is reduced to 1 / 4 of the original; after 4 layers of residual blocks, high-dimensional features are further extracted, and the feature size is reduced to 1 / 8 of the original.

[0039] 2) If Figure 2 As shown in the upsampling module in the figure, the high-dimensional features extracted by the backbone network undergo one deconvolution operation and two upsampling operations, in which two layers of convolution layers and activation functions are connected in series, and the feature size is enlarged by 8 times, and the size is restored to 64×64, with 5 channels. The intermediate features are obtained by integrating different channels, where the feature map corresponding to the first 3-dimensional channels represents the 3D point coordinate prediction feature map corresponding to the pixel of the object on the image. The feature map corresponding to the last 2-dimensional channel represents the mask prediction feature map of the target object. Designing intermediate feature output is conducive to making full use of the semantic information in high-dimensional features and improving the training effect of the network model.

[0040] 3) If Figure 2 As shown in the PnP-like module in the figure, the intermediate feature map obtained and It is further sent to the PnP-like module, which contains 3 layers of strided convolution and group normalization operations, and adds a RELU activation function after each layer. Finally, it passes through 2 layers of fully connected layers to output the predicted pose, which includes the rotation matrix R and the translation vector t, and obtains the 6D pose estimation result of the object. In previous algorithms, a two-stage method is usually used to implement the position estimation task, that is, first predict the two-dimensional key points of the object on the image, combine the predefined three-dimensional key points, construct the PnP problem solution to obtain the pose, but this type of method does not directly predict the pose, and the prediction error of the key points will be amplified in the PnP solution process. The PnP-like module designed in the present invention can overcome this problem.

[0041] 3. Joint Attention Module

[0042] In the feature extraction module, combined with the characteristics of the network model, a joint attention module containing two different attention modules, bottleneck attention and efficient channel attention, is embedded to help the network better learn the abstract feature representation of different parts of the image, thereby improving the effect of 6D pose estimation of the object. The specific process is as follows:

[0043] 1) If Figure 2 As shown in the joint attention module in Figure 1, a bottleneck attention module is added between the residual blocks of the feature extraction backbone network. In this example, a three-layer bottleneck attention module is inserted, which is divided into channel attention M C and spatial attention M S The attention feature maps obtained from these two parts are superimposed on the original input to obtain the output after feature enhancement, which is beneficial to improve the accuracy of network regression. Channel Attention M C and spatial attention M S The processing process is as follows:

[0044]

[0045] AvgPool stands for average pooling, MLP stands for multi-layer perceptron, BN stands for batch normalization, and f k×k Represents a convolution kernel of size k×k. The bottleneck attention module obtains the output feature map after feature enhancement for:

[0046]

[0047] Where σ represents the activation function, Represents the multiplication of elements at corresponding positions in the feature map.

[0048] 2) An efficient channel attention module is embedded in the first three residual blocks to detect the input features. The output feature map is The processing of this module is as follows:

[0049]

[0050] The size of the convolution kernel k is adaptively determined by the number of channels C. This design can improve the robustness of the system. The calculation method is as follows:

[0051]

[0052] Where odd means taking the odd number closest to the number, η, λ and δ are hyperparameters, η and λ are used to control the linear relationship between the logarithm of C and k, and adding a δ is used to expand the receptive field of the convolution kernel to prevent the convolution kernel from having insufficient feature extraction ability when the number of channels is small. In this implementation example, η is set to 0.5, λ is set to 1.5, and δ is set to 1.

[0053] 4. Network model training and testing

[0054] According to the label value in the data set, the label value of the intermediate feature is obtained by combining the camera internal parameters. Based on the designed network model, the loss function is defined, which includes the intermediate feature loss and the pose loss. Including more loss functions can further improve the model training effect. After setting the training parameters, start training the network model. After the network model parameters are optimized, for the image whose pose needs to be predicted, input it into the network to obtain the pose estimation result. The specific process is as follows:

[0055] 1) After the introduction of the PnP-like module, the network loss function consists of two parts. One part is the final pose estimation loss. The other part is designed for the intermediate features of network fitting, which is recorded as the intermediate feature loss Loss function of the overall network training process It is the weighted sum of these two parts, which can make full use of the feature losses of different parts of the network model to improve the training effect, thereby increasing the accuracy of pose estimation.

[0056] 2) For the final task goal, that is, the pose obtained by network regression, set the pose estimation loss for:

[0057]

[0058] in, Predict a feature map for the mask of the target object, The feature map of the three-dimensional point coordinates corresponding to the pixels of the object on the image is predicted. The first item of the intermediate feature loss represents the mask prediction loss of the target object, and the second item represents the prediction loss of the three-dimensional coordinate points. Since the three-dimensional coordinates of the background object are not the prediction targets, Multiply to filter the influence of background information, M represents the CAD model of the object, x represents the three-dimensional coordinates of the point cloud, m represents the total number of points in the model point cloud; R represents the true value of the rotation matrix, represents the predicted value of the rotation matrix; (t x , t y , t z ) represents the true value of the translation vector, Represents the predicted translation vector.

[0059] 3) In addition, the intermediate features of the network fitting are also added to the construction of the loss function, recorded as intermediate feature loss Its definition is as follows:

[0060]

[0061] The first item represents the mask prediction loss of the target object, and the second item represents the prediction loss of the 3D coordinate point. Since the 3D coordinates of the background object are not the prediction target, Multiply to filter the effect of background information.

[0062] Loss function of the overall network training process Defined as:

[0063]

[0064] The two parameters λ1 and λ2 are used to adjust the weights of the influence of different loss function parts on network training. In this example, λ1 is set to 1 and λ2 is set to 0.5.

[0065] 4) Set the network training interval to 80 and the number of training images per batch to 24 to train the network model. When the loss function drops to a certain threshold range, the model parameters are optimized and the network model for the object 6D pose estimation task is obtained.

[0066] 5) By inputting the test set in the LINEMOD dataset, or the scene image containing the target object taken by the RGB camera, the position and posture information of the object can be directly output. Figure 3 As shown, the target object is in different postures, and its position changes dynamically with the conveyor belt. Good posture estimation results can be obtained using the method proposed in the present invention.

[0067] According to the method of the present invention, target detection is performed on the input image, and after feature extraction, attention map generation, upsampling recovery, the object pose is finally directly regressed, which can reduce the error caused by proxy loss and improve the accuracy of pose estimation.

[0068] Corresponding to the aforementioned embodiment of a method for directly regressing the 6D pose of an object based on image joint attention, the present invention also provides an embodiment of a device for directly regressing the 6D pose of an object based on image joint attention.

[0069] See also Figure 4 An embodiment of the present invention provides a device for directly regressing the 6D pose of an object based on image joint attention, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a method for directly regressing the 6D pose of an object based on image joint attention in the above embodiment.

[0070] An embodiment of a device for directly regressing the 6D pose of an object based on joint attention of an image provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 4 As shown, it is a hardware structure diagram of a device for directly regressing the 6D pose of an object based on image joint attention provided by the present invention, in which any device with data processing capability is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0071] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0072] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0073] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for directly regressing the 6D pose of an object based on image joint attention in the above embodiment is implemented.

[0074] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0075] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method of directly regressing the 6D pose of an object based on image joint attention.

[0076] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the invention should be included in the protection scope of the invention.

Claims

1. A method for directly regressing the 6D pose of an object based on image joint attention, characterized in that: The method comprises the following steps: (1) Obtain the original image for which pose estimation is required and perform preprocessing; (2) The pre-processed image is used as the input feature. In the process of extracting high-dimensional features of the image, feature enhancement is performed based on the bottleneck attention module and the efficient channel attention module. Then, after deconvolution and upsampling operations, the intermediate features containing the correspondence relationship between the key points of the object in the image and the object mask information are obtained. (3) The intermediate features are input into a PnP-like module, which includes multiple layers of strided convolution and group normalization operations. The stride and number of layers of the strided convolution are determined based on the dimension of the intermediate features. Finally, the predicted 6D pose is output through a fully connected layer.

2. The method for directly regressing the 6D pose of an object based on image joint attention according to claim 1, characterized in that: In step (1), the preprocessing is specifically to obtain the target object from the original image through a target detection algorithm, crop it and then scale it to a uniform size to obtain a pre-processed cropped image.

3. The method for directly regressing the 6D pose of an object based on image joint attention according to claim 1, characterized in that: In step (2), a feature extraction backbone network is used to extract high-dimensional features of the image. The feature extraction backbone network includes 4 layers of residual blocks. A bottleneck attention module is added between the residual blocks. Channel attention features are obtained based on average pooling operation, multi-layer perceptron and batch normalization. Spatial attention features are obtained based on convolution operation and batch normalization, and superimposed on the input features for feature enhancement. An efficient channel attention module is added to the first three layers of residual blocks to obtain attention features based on average pooling and convolution operations, and then superimposed on the input features for feature enhancement.

4. The method for directly regressing 6D pose of an object based on image joint attention according to claim 1, characterized in that: In the convolution operation of the efficient channel attention module, the size of the convolution kernel is adaptively determined by the number of channels, and the specific calculation is as follows: Among them, k represents the size of the convolution kernel, C represents the number of channels, odd represents the odd number closest to the number of channels, η, λ, and δ are hyperparameters. η and λ are used to control the linear relationship between the logarithm of C and k, and δ is used to expand the receptive field of the convolution kernel.

5. The method for directly regressing 6D pose of an object based on image joint attention according to claim 1, characterized in that: In step (2), the intermediate features include 5-dimensional channels. The feature maps corresponding to the first 3-dimensional channels represent the 3D point coordinate prediction feature maps corresponding to the pixels of the object on the image, and the feature maps corresponding to the last 2-dimensional channels represent the mask prediction feature maps of the target object.

6. The method for directly regressing 6D pose of an object based on image joint attention according to claim 1, characterized in that: In step (3), after each layer of strided convolution and group normalization operation, a RELU activation function is added, and through two fully connected layers, the rotation matrix and translation vector of the predicted posture are output respectively.

7. The method for directly regressing 6D pose of an object based on image joint attention according to claim 1, characterized in that: The final pose estimation loss and intermediate feature loss are calculated separately, and the weighted sum is used as the final loss function for training; Intermediate feature loss The definition is as follows: in, Predict a feature map for the mask of the target object, The feature map of the three-dimensional point coordinates corresponding to the pixels of the object on the image is predicted. The first item of the intermediate feature loss represents the mask prediction loss of the target object, and the second item represents the prediction loss of the three-dimensional coordinate points. Since the three-dimensional coordinates of the background object are not the prediction targets, Multiply to filter the effect of background information.

8. A device for directly regressing an object's 6D pose based on image joint attention, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a method for directly regressing an object's 6D pose based on image joint attention is implemented as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a method for directly regressing an object's 6D pose based on image joint attention as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a method for directly regressing an object's 6D pose based on image joint attention is implemented as described in any one of claims 1 to 7.

Images