Method for identifying moving target by mobile robot

Abstract

The invention relates to the technical field of robot control, and particularly provides a method for a mobile robot to identify a moving target, which comprises the following steps: loading a convolution attention mechanism in a jump connection layer of an encoder of a depth estimation network, inputting an image into the improved depth estimation network, obtaining a corresponding depth map, carrying out point cloud processing on the depth map, and obtaining a point cloud processing result. Constructing a color image feature fusion module based on a convolutional neural network; inputting an image acquired by a camera into the color image feature fusion module; obtaining a priori frame of a moving target in the image; inputting the point cloud data and the priori frame into a three-dimensional detection model; according to the method, the foreground and the background are accurately segmented, the spatial position information of the target object is combined, and the influence of the motion of the robot end effector on the feature extraction of the moving target can be effectively eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and particularly provides a method for identifying a moving target by a mobile robot. BACKGROUND

[0002] A mobile robot is a robot capable of moving in an environment and performing tasks. With the continuous development of artificial intelligence technology, mobile robots have been widely used in industries, services, education, entertainment and other fields. For example, mobile robots can realize material handling, cleaning, customer service and other functions. They can execute pre-set program logic to perform tasks planned by humans in advance, and also can learn and improve their own behavior in the process of interacting with humans. Mobile robots can free people from dangerous, harmful, repetitive and heavy work. Robots bring a lot of convenience and influence to human society in the process of performing tasks. However, once a robot collides, it may cause serious consequences and damage to human safety and property. In reality, the movement of a robot is affected by various factors. Traditional robot obstacle detection problems are only applicable to specific ideal conditions. However, when a robot is in a complex obstacle environment, the movement of the robot must have the ability to avoid obstacles. Therefore, it is crucial to study the autonomous obstacle detection technology of a mobile robot to ensure the safety of the robot and its surrounding environment.

[0003] Under the trend of diversification and complication of mobile robot application scenarios, people's requirements for the universality and reliability of mobile robot obstacle detection ability are also increasing. According to the different motion states of the robot motion space environment and obstacles, the robot obstacle avoidance planning can be divided into: obstacle static and robot motion space environment known obstacle avoidance planning; obstacle moving and robot motion space environment known obstacle avoidance planning; obstacle static and robot motion space environment unknown obstacle avoidance planning; obstacle moving and robot motion space environment unknown obstacle avoidance planning.

[0004] When the robot is in motion at the same time as the obstacle, the background segmentation and feature extraction performance of the detection model have higher requirements. According to the different data acquisition methods, the robot obstacle avoidance system can be divided into methods based on laser radar sensors, methods based on sonar sensors and methods based on monocular cameras. The method of using laser radar or sonar as a ranging sensor can only capture limited information, and is expensive, too heavy for some small mobile robot platforms, and has high power consumption. Traditional robot vision obstacle detection usually uses a method based on a ranging sensor to solve the problem, and in some small mobile robot platforms, lightweight is one of the main considerations. Therefore, the selection of a lightweight and low-power monocular camera is the first choice for mobile robot vision obstacle detection equipment. Compared with sensors, monocular cameras can provide more rich scene information, and are low in cost, light in weight and low in power consumption, and are suitable for various platforms. In the environment of the continuous development of mobile robot technology and the rapid development of artificial intelligence algorithms, with the update of processors, the application of various advanced algorithms and the rapid development of deep learning in recent years, it is completely possible to use neural network models to locate and identify obstacles in monocular images in an embedded platform. The method of two-dimensional image mobile robot obstacle avoidance uses visual information to infer the traversable space and the surrounding obstacles, and uses the robot vision information system to identify and detect the spatial obstacles, so that the robot can detect the obstacles and correct the planned work path of the robot in different environments, and complete the obstacle avoidance task planning. Most of these methods obtain environmental features through two-dimensional color images, which simplifies the information source from three-dimensional space to two-dimensional image plane, eliminating the correspondence between pixels and distance, which makes it very difficult for mobile robots to determine the spatial position of obstacles.

[0005] Most current robot obstacle detection uses reinforcement learning methods. The reinforcement learning method uses a deep reinforcement learning algorithm to directly map the robot's action control from the original image. This method performs well in simple or simulated environments, but it takes a lot of time to adjust and optimize for actual robot applications, and the reinforcement learning method requires a large amount of training data set to learn complex environments. In the method of mobile robot obstacle detection based on monocular cameras, the depth and surface normal are predicted to obtain information closely related to objects in three-dimensional environments, and then color image data and point cloud data are fused to input into a three-dimensional detection model to detect obstacles. The final performance of this method directly depends on the quality of the depth map, and distinguishing between the front and back of the monocular image in a complex environment is also a big difficulty in the depth mapping process. SUMMARY

[0006] The application provides a method for identifying a moving target by a mobile robot, and the method comprises the following steps: inputting an image collected by the robot into a depth estimation network, outputting a depth map, converting depth data in the depth map into point cloud data, and inputting the point cloud data into a convolutional neural network for moving target detection.

[0007] The application provides a method for identifying a moving target by a mobile robot, and the method comprises the following steps: inputting an image collected by the robot into a depth estimation network, outputting a depth map, converting depth data in the depth map into point cloud data, and inputting the point cloud data into a convolutional neural network for moving target detection. S1: improving the depth estimation network by loading a convolutional attention mechanism in a skip connection layer of an encoder of the depth estimation network to obtain an improved depth estimation network; S2: inputting the image into the improved depth estimation network to obtain a corresponding depth map; S3: performing point cloud processing on the depth map to obtain corresponding point cloud data; S4: constructing a color image feature fusion module based on the convolutional neural network, inputting the image into the color image feature fusion module to obtain a prior box of a moving target in the image; S5: inputting the point cloud data and the prior box into a three-dimensional detection model to generate a three-dimensional bounding box, taking the position of the center of the three-dimensional bounding box in space as the position of the moving target, and taking the size of the three-dimensional bounding box as the size of the moving target, so as to complete the identification of the moving target.

[0008] Preferably, the convolutional attention mechanism comprises a channel attention module and a spatial attention module.

[0009] Preferably, the processing procedure of the channel attention module is as follows: the image is subjected to global average pooling and global maximum pooling based on width and height respectively to obtain two 1x1xC feature maps, wherein C represents the number of channels; the two 1x1xC feature maps are subjected to nonlinear transformation through a shared fully connected layer to generate channel weights after being combined.

[0010] Preferably, the processing procedure of the spatial attention module is as follows: the result output by the channel attention module is subjected to channel dimension global average pooling and global maximum pooling, and the two kinds of pooling results are spliced in the channel dimension to generate spatial weights.

[0011] Preferably, in S2, the image is subjected to image enhancement before being input into the improved depth estimation network.

[0012] Preferably, the point cloud processing procedure is as follows: the parameters of the depth value in the image coordinate system are converted into point cloud data in the world coordinate system through coordinate transformation.

[0013] Preferably, the method for generating the three-dimensional bounding box is: generating a two-dimensional candidate region according to the prior box of the moving target extracted by the color image feature fusion module, and generating the three-dimensional bounding box according to the point cloud data and the two-dimensional candidate region.

[0014] Compared with the prior art, the present application can achieve the following beneficial effects: The present application proposes a method for moving robot to identify moving targets based on images collected by a monocular camera, loads a convolution attention mechanism in a depth estimation network, can more accurately segment foreground and background in a graph, and can effectively eliminate the influence of the movement of the end effector of the robot on the feature extraction of the moving obstacle by combining the spatial position information of the moving target. The convolution attention mechanism is used to fuse depth data and color image data, which maximizes the reduction of local information loss in the feature extraction process and effectively segments the moving target area and the background area.

[0015] In the depth estimation network, the convolution layer of the shallow feature can be introduced into the next stage through the way of skip connection, the features of the corresponding positions of the encoder are fused in the channel by using deconvolution for step-by-step upsampling, the high-resolution information in the high-level feature map is retained by the fusion of the bottom features and the high-level features, thereby enhancing the segmentation accuracy of the image and maximizing the utilization rate of the feature information in each layer of the network.

[0016] The prior box can be obtained through the color image feature fusion module, which can avoid using a large number of inaccurate two-dimensional candidate regions in three-dimensional space and significantly limit the search space of the three-dimensional bounding box, thereby accelerating the detection process. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flow chart of a moving target identification method according to an embodiment of the present application; Figure 2 is a schematic diagram of the combination of a depth estimation network and a convolution attention mechanism according to an embodiment of the present application; Figure 3 is a flow chart of a three-dimensional detection model detection process according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] For the purpose of making the object, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not constitute a limitation on the present application. In different embodiments, similar elements are associated with similar element labels. In the following embodiments, many details are described in order to make the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different cases, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification in order to avoid the core part of the present application being overwhelmed by too much description, and it is not necessary for those skilled in the art to describe these related operations in detail according to the description in the specification and general technical knowledge in the art.

[0019] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other to form various embodiments without conflict. At the same time, each step or action in the method description can also be sequentially adjusted or adjusted in a manner obvious to those skilled in the art. Therefore, the various sequences in the specification and drawings are only for the purpose of clearly describing a certain embodiment, and do not mean a necessary sequence, unless otherwise stated that a certain sequence must be followed.

[0020] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0021] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0022] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0023] During the movement of the robot, the surrounding environment is observed in real time, obstacles are identified and avoided to ensure the smooth execution of the task. In general, a camera is arranged at a fixed position on the robot, and the surrounding environment can be collected through the camera. The moving target in the collected image is identified, and then the obstacle avoidance is performed. The present embodiment studies the robot which is equipped with a monocular camera and avoids moving obstacles, and the moving obstacle to be avoided is referred to as a moving target. It should be noted that the execution of the task by the robot is a continuous movement process, and the detection and identification of the moving target is also a continuous process. A series of images are collected by the camera, and the collected images are identified frame by frame to avoid the moving target.

[0024] As shown in Figure 1 , the present embodiment provides a method for identifying a moving target by a mobile robot, and the specific content is as follows: S1: improving the depth estimation network: loading the convolution attention mechanism in the skip connection layer of the encoder of the depth estimation network.

[0025] As shown in Figure 2 , the traditional depth estimation network structure gradually loses some edge information under the step-by-step pooling, and the shallow features of the convolution layer are introduced into the next stage through the way of skip connection, and the inverse convolution is used for step-by-step upsampling. The feature maps at the corresponding positions of the encoder are fused in the channel. Through the fusion of the bottom features and the high-level features, the depth estimation network can retain more high-resolution information in the high-level feature maps, thereby enhancing the segmentation accuracy of the image and maximizing the utilization rate of the feature information in each layer of the network.

[0026] In the depth estimation network, the skip connection layer of the encoder can retain high-level information in the final output depth information. The embodiment of the application improves the depth estimation network based on the residual network model, reconstructs the depth estimation network in the skip connection mode, and loads the convolution attention mechanism in the skip connection layer of the encoder. The convolution attention mechanism is a technology used to enhance the feature expression capability of the network in the deep learning model. The core idea of this mechanism is to give different weights to different channels in the network, so that the deep learning model can focus on the feature channels of effective information and suppress the channels with less information. The convolution attention mechanism can aggregate the depth features and extract the correlation between the object and the surrounding environment. The skip connection layer of the encoder can retain high-level information in the final output depth information. Therefore, adding the convolution attention mechanism to the skip connection layer of the encoder can significantly improve the prediction performance of the depth estimation network.

[0027] The convolution attention mechanism includes a channel attention module and a spatial attention module. The processing procedure of the channel attention module is as follows: the image is subjected to global average pooling and global maximum pooling based on the width and height respectively to obtain two 1x1xC feature maps, wherein C represents the number of channels. The two 1x1xC feature maps are subjected to nonlinear transformation by a shared fully connected layer (MLP), and the channel weights are generated after merging to enhance the feature response of important channels. The channel attention module focuses on the importance of the channel. For example, in the weak texture area (such as the sky and white wall), the channel attention module can enhance the dependence of global features and reduce the influence of insufficient local features. The feature map output by the channel attention module is called a channel attention map, and the channel attention map is input into the spatial attention module. The spatial attention module performs channel dimension pooling on the input channel attention map, that is, global average pooling and global maximum pooling are performed on the channel dimension (C) at each spatial position (height of the feature and width of the feature) to generate two HxWx1 feature maps, wherein H represents the height of the feature map and W represents the width of the feature map. The two kinds of pooling results are spliced in the channel dimension (HxWx2), the spatial context information is integrated through a 7x7 convolution layer to generate the spatial weight for enhancing the features of important spatial positions. The spatial attention module focuses on the importance of the space. For example, in terms of distinguishing repeated textures (such as tiles and dense vegetation), the spatial attention module can focus on the key spatial positions to improve the sensitivity of the depth estimation network to the edges of the object and the depth mutation area.

[0028] The channel attention module focuses on the channel weights, and by dynamically adjusting the weights of each channel in the feature map, it highlights the channel features that are more important for identifying moving targets. The spatial attention module focuses on the spatial weights, and by focusing on the importance of the spatial position in the feature map, it highlights the target area or key structure. Compared with the traditional depth estimation network, loading the convolutional attention mechanism that combines the channel attention module and the spatial attention module in the depth estimation network can enable the depth estimation network to achieve better recognition results. After adding the convolutional attention mechanism, the convolutional layer will first pass through the channel attention module to obtain the weighted result during image recognition, and then pass through the spatial attention module for weighting to obtain the final output result. The depth estimation network after adding the convolutional attention mechanism outputs the pixel disparity rate of multiple disparity layers, and then obtains the discrete disparity amount. The selective kernel unit feature (SK feature) is input into a two-dimensional convolutional layer of size 3×3, and then the disparity probability of K channels is output. and K disparity layers ,in, is the minimum disparity value, is the parallax interval.

[0029] The depth drift and texture duplication problems commonly encountered in depth estimation tasks are caused by the depth estimation network's over-reliance on local texture features and its failure to effectively incorporate global context information. Integrating the convolutional attention mechanism into the depth estimation network can effectively address these issues. The present invention proposes the following solutions: The input features are normalized to range to unify the output distribution of global average pooling and global maximum pooling in the channel attention module and spatial attention module. The input features here refer to the features output by the channel attention and spatial attention in the convolutional attention mechanism, that is, the features to be input to the Softplus activation function. Normalization can make the feature distribution more consistent, help the depth estimation network better balance features of different scales, and improve the stability of training. Use the Softplus activation function instead of the ReLU activation function. When the input of the ReLU activation function is negative, the gradient is zero, which will cause these neurons to be unable to be activated again. The Softplus activation function is a smoothed version of the ReLU activation function, which has a non-zero gradient for all inputs. This can avoid excessive neuron death during training and maintain the expressive power of the depth estimation network.

[0030] S2: Input the image captured by the camera into the improved depth estimation network to obtain the corresponding depth map.

[0031] The image captured by the camera is input frame by frame into the depth estimation network loaded with the convolution attention mechanism to obtain the corresponding depth map. In particular, after the camera captures the image, image enhancement can be performed first, and then the enhanced image is input into the improved depth estimation network to obtain the depth map.

[0032] S3: The depth map is subjected to point cloud processing to obtain corresponding point cloud data.

[0033] The parameters of the depth value in the image coordinate system are converted into point cloud data in the world coordinate system through coordinate transformation.

[0034] S4: A color image feature fusion module is constructed based on a convolutional neural network, the image is input into the color image feature fusion module, and the prior box of the moving target in the image is obtained.

[0035] In order to better utilize the color image information, the color image feature fusion module is added in the moving target recognition process in the embodiment of the application, which uses a two-dimensional target detection model based on a convolutional neural network to extract the corresponding prior box from the input image, and uses the extracted prior box as the input data of the subsequent step to enhance the ability to distinguish point clouds.

[0036] S5: The point cloud data and the prior box are input into a three-dimensional detection model to generate a three-dimensional bounding box, the position of the center of the three-dimensional bounding box in space is taken as the position of the moving target, and the size of the three-dimensional bounding box is taken as the size of the moving target, thereby completing the moving target recognition.

[0037] As shown in Figure 3 , the effect of the three-dimensional detection model mainly depends on the accuracy of the depth map, and after obtaining the accurate depth data in the depth map, the trained three-dimensional detection model is relied on to generate the three-dimensional bounding box of the moving target.

[0038] Before generating the three-dimensional bounding box of the moving target, it is necessary to assume that the camera matrix is known, to convert the three-dimensional bounding box output parameters into seven parameters, i.e., the moving target center coordinates , the size of the moving target , and the heading angle , divide the seven parameter data into a group of data, and use the group of data to predict the three-dimensional bounding box. The existing eight-corner point method needs 12 parameters to predict the three-dimensional bounding box, and the existing four-corner point method needs 9 parameters to predict the three-dimensional bounding box, and the method of the embodiment of the application only needs 7 parameters to predict the three-dimensional bounding box, thereby realizing the prediction of the three-dimensional bounding box with fewer parameters.

[0039] The pseudo laser radar point cloud directly derives the three-dimensional position of each pixel from the depth estimation value and the camera matrix , wherein represents rotation, represents translation. The pseudo-lidar point cloud is a key step from depth estimation to 3D object detection, which maps the 2D depth map output from depth estimation into 3D point cloud data through a mathematical expression. Because the pseudo-lidar point cloud is different from the real lidar point cloud obtained by the lidar sensor, the depth map is estimated from the 2D image, and the data is not real, so it is called pseudo-lidar point cloud. The pseudo-lidar point cloud is a common data flow of monocular 3D object detection.

[0040] By projecting the coarse depth map and the pose transformation matrix into the camera with the intrinsic matrix, the reconstructed target depth map is obtained, the depth value is mapped to the pseudo-lidar point cloud, and the depth estimation network can better learn the nonlinear mapping from the input to the output. A two-stage detection model based on point cloud data is used as the backbone network to extract the feature vector of each point in the point cloud data. Given the depth coordinate in the two-dimensional image space (the coordinate of each pixel point in the depth map output by the depth estimation network) , the three-dimensional coordinate in the camera coordinate system can be represented as: , wherein, represents the focal length of the camera, represents the projection center of the camera, represents the depth value of each pixel in the depth map, represents the coordinate of each pixel point in the depth map.

[0041] The three-dimensional detection model is a two-stage object detection network based on point cloud data. The first stage of the three-dimensional detection model is to generate a two-dimensional candidate region according to the prior box extracted by the color image feature fusion module. The second stage is to pool the extracted features according to the point cloud data based on the two-dimensional candidate region, obtain more detailed local features, and aggregate the image features and local features. With the help of the segmentation branch method, the foreground points are estimated, and the three-dimensional bounding box is generated from the segmented foreground points. Finally, based on the distance, the overlapping bounding boxes are removed, and the three-dimensional bounding box of the moving target is generated, i.e. the coordinates of the three-dimensional bounding box are calculated combined with the geometric constraints of the prior box. Based on this strategy, a large number of inaccurate two-dimensional candidate regions in three-dimensional space can be avoided, the search space of the three-dimensional bounding box is limited, and the detection process is accelerated.

[0042] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0043] The above detailed description of the application is not intended to limit the scope of the application. Various other changes and modifications of the application can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A method for a mobile robot to identify a moving target, wherein the robot is provided with a camera and an image of the moving target is captured by the camera, characterized in that: include: S1: Improve the depth estimation network: Add a convolutional attention mechanism to the skip connection layer of the encoder of the depth estimation network to obtain an improved depth estimation network; S2: Input the image into the improved depth estimation network to obtain a corresponding depth map; S3: performing point cloud processing on the depth map to obtain corresponding point cloud data; S4: constructing a color image feature fusion module based on a convolutional neural network, inputting the image into the color image feature fusion module, and obtaining a priori frames of moving objects in the image; S5: Input the point cloud data and the prior frame into the three-dimensional detection model to generate a three-dimensional bounding box, use the position of the center of mass of the three-dimensional bounding box in space as the position of the moving target, and use the size of the three-dimensional bounding box as the size of the moving target to complete the moving target recognition.

2. The method for a mobile robot to identify a moving target according to claim 1, wherein: The convolutional attention mechanism includes a channel attention module and a spatial attention module.

3. The method for mobile robot to identify moving targets according to claim 2, characterized in that: The processing flow of the channel attention module is as follows: the image is subjected to global average pooling and global maximum pooling based on width and height respectively to obtain two 1×1×C feature maps, where C represents the number of channels; the two 1×1×C feature maps are nonlinearly transformed through a shared fully connected layer and merged to generate channel weights.

4. The method for a mobile robot to identify a moving target according to claim 3, wherein: The processing flow of the spatial attention module is as follows: global average pooling and global maximum pooling are performed on the results output by the channel attention module in the channel dimension, and the two pooling results are spliced ​​in the channel dimension to generate spatial weights.

5. The method for a mobile robot to identify a moving target according to claim 1, wherein: In S2, before inputting the image into the improved depth estimation network, the image is further enhanced.

6. The method for a mobile robot to identify a moving target according to claim 1, wherein: The point cloud processing process is: converting the depth value parameters in the image coordinate system into point cloud data in the world coordinate system through coordinate transformation.

7. The method for a mobile robot to identify a moving target according to claim 1, wherein: The method for generating a three-dimensional bounding box is as follows: extracting a priori frame of a moving target based on a color image feature fusion module to generate a two-dimensional candidate area, and generating a three-dimensional bounding box based on point cloud data and the two-dimensional candidate area.

Images