An intelligent vision system
By combining semantic segmentation and pose estimation units in an intelligent vision system, using multi-scale feature fusion and attention mechanism, the accuracy of object segmentation and pose estimation in the prior art in occlusion, lighting and messy environments is solved, and the accuracy of object segmentation and pose estimation in the prior art is achieved. Efficient object recognition and pose estimation in these environments.
Patent Information
- Application Number
- CN202011378142.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-30
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2040-11-30
AI Technical Summary
The prior art is difficult to achieve accurate effects of object segmentation and posture estimation in real environments of occlusion, lighting and messyness.
An intelligent vision system is designed, combining semantic segmentation unit and pose estimation unit to extract and fuse the color and geometric information of the image through a multi-scale feature fusion module and attention mechanism for object recognition and 6D pose estimation.
In occlusion and messy environments, the accuracy of object recognition and pose estimation is significantly improved, providing more effective feature representation and pose prediction, and optimizing system performance.
Smart Images
Figure CN112651423B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of electronic information technology, and in particular, relates to an intelligent vision system. Background Art
[0002] Intelligent visual analysis refers to computer image visual analysis technology, which analyzes and tracks the targets appearing in the camera scene by separating the background and the target in the scene. In recent years, with the rapid development of artificial intelligence and robotics, more and more intelligent industrial robots have been deployed on traditional assembly lines to replace manual operations. These robots are usually equipped with intelligent vision systems, which enable them to not only detect parts in the workspace, but also estimate their postures before taking further actions (such as grasping, rotating, moving, assembling, etc.). In addition to industrial robots, more and more medical robots and home service robots are gradually appearing in the public eye. For these intelligent robots, accurate object recognition and 6D pose estimation are important prerequisites for their interaction with the real world. In addition, object recognition and pose estimation also play an irreplaceable role in other important topics in the field of computer vision, such as virtual reality, augmented reality, and autonomous driving. With the vigorous development of deep learning technology and its good performance in visual tasks, convolutional neural networks have made major breakthroughs in all directions in the field of computer vision.
[0003] Existing technologies find it difficult to achieve good object segmentation and pose estimation results under the challenges of real-world environments such as occlusion, lighting, and clutter. Summary of the invention
[0004] 1. Technical problems to be solved
[0005] Based on the problem that the existing technology is difficult to achieve good object segmentation and posture estimation effects under the challenges of real-world environments such as occlusion, lighting, and clutter, the present application provides an intelligent vision system.
[0006] 2. Technical solution
[0007] In order to achieve the above-mentioned objectives, the present application provides an intelligent vision system, including a semantic segmentation unit and a posture estimation unit, wherein the semantic segmentation unit is communicatively connected to the posture estimation unit; the semantic segmentation unit is used to crop the color image and depth image of the corresponding object from the scene after determining the segmentation mask, and then transmit the cropped target image to the posture estimation unit; the posture estimation unit is used to extract features from the target image, and then embed and fuse the features together, obtain a more effective feature representation after processing, and obtain the output object posture.
[0008] Another implementation manner provided by the present application is: the semantic segmentation unit includes an encoder subunit and a decoder subunit, the encoder subunit is used to extract features of different scales; the decoder subunit is used to fuse multi-scale features and output pixel-by-pixel classification.
[0009] Another implementation manner provided by the present application is: the encoder subunit is composed of 5 maximum pooling layers and 16 convolutional layers, and the decoder subunit is composed of 3 multi-scale feature fusion modules, 2 upsampling layers and 12 convolutional layers.
[0010] Another implementation manner provided by the present application is: the multi-scale feature fusion module includes an upsampling layer and a convolution layer, and the multi-scale feature fusion module aggregates low-level and high-level features through dense connections to improve the ability to understand the image.
[0011] Another implementation manner provided by the present application is: the posture estimation unit includes a feature extraction subunit, a feature fusion subunit and a posture regression subunit, the feature extraction subunit is used to extract feature information, the feature fusion subunit is used to fuse the feature information, and the posture regression subunit is used to perform regression prediction and posture optimization adjustment on 6D posture parameters and confidence.
[0012] Another implementation manner provided by the present application is: the feature extraction subunit includes a color information extraction module and a geometric information extraction module; the color information extraction module adopts a fully convolutional network, and the geometric information extraction module adopts a PointNet-based network.
[0013] Another implementation method provided by the present application is: the feature fusion subunit includes a position attention module, a channel attention module, and a global feature extraction module, and the output features of the position attention module, the channel attention module, and the global feature extraction module are fused and input into the posture regression subunit.
[0014] Another implementation manner provided by the present application is: the position attention module includes 3 convolutional layers and 1 normalization layer, the channel attention module includes 1 normalization layer, and the global feature extraction module includes a convolutional layer and an average pooling layer.
[0015] Another implementation manner provided by the present application is: the posture regression subunit includes a posture regression module and a posture optimization module, which are used to perform regression prediction and posture optimization adjustment on 6D posture parameters and confidence levels.
[0016] Another implementation method provided by the present application is: the posture regression module is a posture predictor, which includes several one-dimensional convolutional layers, used to estimate a posture and confidence for each pixel, and select the posture with the highest confidence as the 6D posture of the object; the posture optimization module is an iterative optimization network, used to optimize the 6D posture of the object.
[0017] 3. Beneficial effects
[0018] Compared with the prior art, the intelligent visual system provided by this application has the following beneficial effects:
[0019] The intelligent vision system provided in this application is an intelligent vision system for object recognition and 6D posture estimation based on deep neural networks.
[0020] The intelligent vision system provided in this application is designed to meet the needs of application scenarios such as intelligent manufacturing and autonomous driving, and is an intelligent vision system that can accurately and efficiently perform object recognition and posture estimation in occluded and cluttered real environments.
[0021] The intelligent vision system provided in this application can effectively improve the accuracy of posture estimation.
[0022] The intelligent vision system provided in this application proposes a new segmentation network, which uses dense connections to aggregate multi-scale features of different layers in the network to provide rich semantic information for pixel-by-pixel classification.
[0023] The intelligent vision system provided in this application introduces a position attention module and a channel attention module in the posture estimation unit, which can propose more useful features and effectively improve the system performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a schematic diagram of the principle of the intelligent vision system of this application;
[0025] Figure 2 It is a schematic diagram of the semantic segmentation unit of the present application;
[0026] Figure 3 It is a front view schematic diagram of the posture estimation unit of the present application;
[0027] Figure 4 is a schematic diagram of the position attention module of the present application;
[0028] Figure 5 is a schematic diagram of the channel attention module of this application;
[0029] Figure 6 is a schematic diagram of the qualitative analysis results of this application;
[0030] In the figure: 1-semantic segmentation unit, 2-posture estimation unit, 3-position attention module, 4-channel attention module, 5-global feature extraction module, 6-posture regression sub-unit. DETAILED DESCRIPTION
[0031] Hereinafter, the specific embodiments of the present application will be described in detail with reference to the accompanying drawings. According to these detailed descriptions, those skilled in the art can clearly understand the present application and can implement the present application. Without violating the principles of the present application, the features in the various embodiments can be combined to obtain new implementations, or certain features in certain embodiments can be replaced to obtain other preferred implementations.
[0032] See also Figures 1 to 6 The present application provides an intelligent visual system, including a semantic segmentation unit 1 and a posture estimation unit 2, wherein the semantic segmentation unit 1 is communicatively connected to the posture estimation unit 2; the semantic segmentation unit 1 is used to crop a color image and a depth image of a corresponding object from a scene after determining a segmentation mask, and then transmit the cropped target image to the posture estimation unit 2; the posture estimation unit 2 is used to extract features from the target image, and then embed and fuse the features together, obtain a more effective feature representation after processing, and obtain the output object posture.
[0033] The present application is an object recognition and 6D posture estimation method based on deep learning. It includes a semantic segmentation unit 1 and a posture estimation unit 2. First, a multi-scale information fusion module is applied in the semantic segmentation unit 1 to extract features with richer semantic information and obtain better segmentation results, and then the bounding box of the obtained segmentation result and the corresponding point cloud (obtained by depth map conversion) are input into the posture estimation network. In the posture estimation unit 2, color features and geometric features are first extracted using two backbone networks, and then these two features are spliced together and sent to the position attention module 3, the channel attention module 4 and the global information extraction module 5 respectively. Then the features obtained by these three modules are fused and sent to a posture predictor 6, i.e., a posture regressor, to estimate a posture and confidence for each pixel, and select the posture with the highest confidence as the global posture.
[0034] Furthermore, the semantic segmentation unit 1 includes an encoder subunit and a decoder subunit, the encoder subunit is used to extract features of different scales; the decoder subunit is used to fuse multi-scale features and output pixel-by-pixel classification.
[0035] Furthermore, the encoder subunit consists of 5 maximum pooling layers and 16 convolutional layers, and the decoder subunit consists of 3 multi-scale feature fusion modules, 2 upsampling layers and 12 convolutional layers.
[0036] Furthermore, the multi-scale feature fusion module includes an upsampling layer and a convolution layer. The multi-scale feature fusion module aggregates low-level and high-level features in a densely connected manner to improve the ability to understand the image.
[0037] The semantic segmentation unit 1 framework is as follows Figure 2 As shown. The entire network consists of two parts: encoder subunit Figure 2 (A left) and the decoder in the unit Figure 2 (A right). The encoder subunit consists of 5 max pooling layers and 16 convolutional layers to extract features of different scales. After each pooling, the feature size is halved and the number of channels is increased. In the decoder subunit, the multi-scale feature fusion module (MFFM) implements multi-scale feature fusion and outputs pixel-by-pixel classification through the final convolution and normalization layers. The decoding network consists of 3 MFFMs, 2 upsampling layers, and 12 convolutional layers.
[0038] In convolutional neural networks, feature maps of different sizes not only have different receptive fields, but also usually contain complementary information. Therefore, fusing features of different scales is an important means to improve network performance. In theory, low-level features contain more geometric details and less semantic information. In contrast, high-level feature maps discard some geometric details and retain more semantic information. Therefore, a new multi-scale fusion module is adopted to effectively aggregate low-level and high-level features through dense connections, thereby improving the network's ability to understand images.
[0039] like Figure 2 As shown in (B), each MFFM layer in the decoder subunit obtains feature input from two data sources: (1) the encoder subunit with the same or lower resolution as the current MFFM layer; (2) the previous layer of the current MFFM layer in the decoder subunit. First, all feature inputs with a smaller size than the current layer will be upsampled to the same resolution as the current layer, and then each feature will pass through a separate convolutional layer and finally be aggregated and output. For the input from the encoder subunit, the number of convolution output channels is set to 64 to reduce computational complexity. For the input from the previous layer, the number of convolution output channels remains unchanged to retain as much information from the previous layer as possible. Figure 2 (A) Contains the inputs of different MFFM layers. It can be seen that different MFFM layers usually have different numbers of input layers. COCN is the number of convolution output channels, and CICIN is the number of convolution input channels.
[0040] Furthermore, the posture estimation unit 2 includes a feature extraction subunit, a feature fusion subunit and a posture regression subunit. The feature extraction subunit is used to extract feature information, the feature fusion subunit is used to fuse the feature information, and the posture regression subunit is used to perform regression prediction and posture optimization adjustment on 6D posture parameters and confidence.
[0041] Furthermore, the feature extraction subunit includes a color information extraction module and a geometric information extraction module; the color information extraction module adopts a fully convolutional network, and the geometric information extraction module adopts a PointNet-based network.
[0042] Furthermore, the feature fusion subunit includes a position attention module 3, a channel attention module 4, and a global feature extraction module 5, and fuses the output features of the position attention module 3, the channel attention module 4, and the global feature extraction module 5, and inputs them into the posture regression subunit 6.
[0043] Furthermore, the position attention module includes 3 convolutional layers and 1 normalization layer, the channel attention module includes 1 normalization layer, and the global feature extraction module includes a convolutional layer and an average pooling layer.
[0044] Furthermore, the posture regression subunit 6 includes a posture regression module and a posture optimization module, which are used to perform regression prediction and posture optimization adjustment on 6D posture parameters and confidence levels.
[0045] Furthermore, the posture regression module is a posture predictor, which includes several one-dimensional convolutional layers for estimating a posture and confidence for each pixel, and selecting the posture with the largest confidence as the 6D posture of the object; the posture optimization module is an iterative optimization network for optimizing the 6D posture of the object.
[0046] like Figure 3 As shown in (A) Feature extraction stage: color feature embedding is extracted using a fully convolutional network, and geometric feature embedding is extracted using a dotnet-based network. (B) Feature fusion stage: the two feature embeddings are fused together, and then three types of features are generated respectively through the spatial attention module, channel attention module, and global feature extraction module, and these features are fused and input into the pose regression network. (C) Pose regression stage: The 6D pose parameters and confidence are predicted using a pose regression network composed of multiple one-dimensional convolutions.
[0047] The architecture of the posture estimation unit 2, i.e., the 6D posture estimation network, is as follows Figure 3 As shown. The entire posture estimation unit 2 consists of three stages: feature extraction stage, Figure 3 (A); Feature fusion stage, Figure 2 (B); and posture return phase, Figure 3 (C).
[0048] RGB images and depth images contain the color information and geometric information of objects respectively. Although the storage formats of RGB images and depth images are similar, their physical meanings and distribution spaces are quite different. Therefore, in order to make full use of these two types of information, we use two convolutional neural networks to extract color and geometric features respectively, such as Figure 3 (A) shown.
[0049] Ordinary neural networks usually treat all features equally. However, in reality, some features contain more critical information and should receive more attention. In order to highlight important features and suppress unnecessary features, three modules are adopted: Position Attention Module 3 (PAM), Channel Attention Module 4 (CAM) and Global Feature Extraction Module 5 (GFEM). In the feature fusion stage, color features and geometric features are concatenated and fed back to these modules, so that the network can adaptively capture the correlation between local features and global features, thereby providing better features for the pose predictor.
[0050] Position attention module 3: For an input feature at a specified position, it will be weighted and updated by the features at all positions, with the weight being the similarity between the corresponding two features. Because it updates the weight of the feature by the similarity of all features with the feature, the more similar features there are, the greater the impact on the input feature. Figure 4 The process is shown.
[0051] The input feature matrix of size (C×N) passes through two convolutional layers to obtain two new feature matrices. The dimensions of these two new matrices change from C to C' after convolution. Then the first matrix is transposed and multiplied with the second matrix, and then normalized to obtain the spatial attention matrix (N×N). At the same time, the input matrix passes through the third convolutional layer to obtain a new matrix (C×N), and then this matrix is multiplied by the feature similarity matrix to aggregate the global features (C×N). Finally, the original features are added to the global features to obtain the output features (C×N).
[0052] Channel attention module 4: For any two channel features, the attention mechanism can be used to capture the dependency between channels. The calculation of channel attention is similar to that of position attention. The difference is that before calculating the relationship between the two channels, we do not use a convolutional layer but directly calculate the spatial attention matrix (C×C) through the input matrix. Figure 5 The calculation process of the channel attention module is shown.
[0053] Global Feature Extraction Module 5: The global features of the object are very important for the pose estimation task. Here, a convolutional layer is used to adjust the features, and an average pooling layer is applied to obtain the global features.
[0054] Finally, the output features of the position attention module 3, the channel attention module 4 and the global feature extraction module 5 are connected in series and input into a pose predictor 6, which consists of several one-dimensional convolutional layers for regressing 6D pose parameters and confidence.
[0055] Furthermore, it also includes an iterative optimization unit, which is in communication connection with the posture estimation unit; the iterative optimization unit is used to optimize the output image. In order to further improve the accuracy of posture estimation, the iterative optimization unit is used to adjust the posture.
[0056] Currently, object recognition is mainly divided into several categories: image classification, target detection, and semantic segmentation. The use of semantic segmentation technology that can classify images at the pixel level can better handle occlusion problems.
[0057] The 6D posture of an object is also called 6D position, which represents the rotation transformation matrix R and translation transformation vector t between the coordinate system of the object and the camera coordinate system. From the perspective of input data type, posture estimation can be divided into three categories: posture estimation based on RGB images, posture estimation based on point clouds / depth maps, and posture estimation based on RGB-D data.
[0058] like Figure 1 As shown, the method proposed in this application is mainly divided into two processes: a semantic segmentation unit 1 and a posture estimation unit 2. First, features of different scales are aggregated using dense connections, and an encoder-decoder based segmentation network is established. After determining the segmentation mask, the RGB and depth images of the corresponding object are cropped from the scene. The cropped target image is then sent to the posture estimation unit 2, i.e., the 6D posture prediction network, which uses two backbone networks to extract color and geometric features respectively. The two are then fused together, and a more effective feature representation is obtained through a channel attention module 4, a position attention module 3, and a global feature extraction module 5. Finally, an iterative optimization unit is used to optimize the output of the posture predictor.
[0059] We conducted comparative experiments with the best existing methods on two benchmark datasets for posture estimation, YCB-Video and LineMOD. The quantitative results are shown in Table 1, and the percentage of ADD (average distance) < 2 cm was used for evaluation and verification. It can be seen that thanks to the new network structure we proposed, we achieved better results than existing methods on the two benchmark datasets.
[0060] Table 1 Quantitative results on pose estimation dataset (%)
[0061] Existing methods This method YCB-Video(ADD<2cm) 73.7 78.0 LineMOD(ADD<2cm) 94.3 96.9
[0062] Qualitative analysis
[0063] For a more direct comparison, some results on the YCB-Video dataset are visualized. Figure 6 As shown, (A) is the original image in the dataset, (B) and (D) are the segmentation results of the literature [4] and this method, respectively, (C) and (E) are the pose estimation results of the existing method and this application, respectively, and different colors represent different object categories. After obtaining the segmentation mask, the pixel area of each object in the image is extracted. If the number of valid pixels in the object depth map is less than a certain threshold, it is identified as an invalid object and its pose is not estimated. The point cloud of all valid objects is calculated, and the pose transformation is performed on each point. Then its projection in the 2D image is superimposed on the object area, as shown in Figure 6 As shown in (C) and (E).
[0064] As shown in the second column of the left figure, the existing method predicts the bowl's posture far from the actual direction, while the present application gives a more correct prediction, showing its advantage in handling symmetrical objects. For some objects with poor textures, such as the bananas in the first and fourth columns, it can be seen that the existing method has some obvious errors, but the present application has no visually obvious errors.
[0065] Although the present application is described above with reference to specific embodiments, it should be understood by those skilled in the art that many modifications may be made to the configurations and details disclosed in the present application within the principles and scope disclosed in the present application. The scope of protection of the present application is determined by the appended claims, and the claims are intended to cover all modifications included in the technical features or scope of the claims.
Claims
1. An intelligent visual system, characterized in that: It includes a semantic segmentation unit and a posture estimation unit, wherein the semantic segmentation unit is communicatively connected with the posture estimation unit; The semantic segmentation unit is used to crop the color image and the depth image of the corresponding object from the scene after determining the segmentation mask, and then transmit the cropped target image to the posture estimation unit; The posture estimation unit is used to extract features from the target image, then embed and fuse the features together, obtain a more effective feature representation through processing, and obtain the output object posture; The posture estimation unit includes a feature extraction subunit, a feature fusion subunit and a posture regression subunit, wherein the feature extraction subunit is used to extract feature information, the feature fusion subunit is used to fuse the feature information, and the posture regression subunit is used to perform regression prediction and posture optimization adjustment on 6D posture parameters and confidence levels; The feature fusion subunit includes a position attention module, a channel attention module, and a global feature extraction module. The output features of the position attention module, the channel attention module, and the global feature extraction module are fused and input into the posture regression subunit.
2. The intelligent visual system according to claim 1, characterized in that: The semantic segmentation unit includes an encoder subunit and a decoder subunit, wherein the encoder subunit is used to extract features of different scales; The decoder subunit is used for multi-scale feature fusion and outputs pixel-by-pixel classification.
3. The intelligent visual system according to claim 2, characterized in that: The encoder subunit consists of 5 maximum pooling layers and 16 convolutional layers, and the decoder subunit consists of 3 multi-scale feature fusion modules, 2 upsampling layers and 12 convolutional layers.
4. The intelligent visual system according to claim 3, characterized in that: The multi-scale feature fusion module includes an upsampling layer and a convolution layer. The multi-scale feature fusion module aggregates low-level and high-level features in a densely connected manner to improve the ability to understand the image.
5. The intelligent visual system according to claim 1, wherein: The feature extraction subunit includes a color information extraction module and a geometric information extraction module; the color information extraction module adopts a fully convolutional network, and the geometric information extraction module adopts a PointNet-based network.
6. The intelligent visual system according to claim 1, wherein: The position attention module includes 3 convolutional layers and 1 normalization layer, the channel attention module includes 1 normalization layer, and the global feature extraction module includes a convolutional layer and an average pooling layer.
7. The intelligent visual system according to claim 1, wherein: The posture regression subunit includes a posture regression module and a posture optimization module, which are used to perform regression prediction and posture optimization adjustment on 6D posture parameters and confidence.
8. The intelligent visual system according to claim 7, characterized in that: The posture regression module is a posture predictor, which includes several one-dimensional convolutional layers for estimating a posture and confidence for each pixel, and selecting the posture with the largest confidence as the 6D posture of the object; the posture optimization module is an iterative optimization network for optimizing the 6D posture of the object.