Perception method based on lightweight real-time instance segmentation and semantic mapping

Abstract

The invention belongs to the technical field of computer vision, provides a real-time semantic perception system running on equipment, performs real-time instance segmentation and semantic mapping on a working environment, and provides a perception method based on lightweight real-time instance segmentation and semantic mapping. In the first part, firstly, a color image containing color and texture information is used for object classification and bounding box regression; cutting the original depth image into a depth image with an accurate geometric edge, and inputting the depth image into the truncation network to generate a mask of the instance; a second part: (1) calculating the attitude and size of a target object in the view of the camera through principal component analysis; (2) obtaining the category of the object in the view obtained by segmentation; (3) creating a point cloud of the depth image projection of the target object in a map; and then cross-frame tracking and fusion are carried out by using data association to realize the purpose of sensing. The method is mainly applied to automatic sensing occasions.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a lightweight instance segmentation method based on RGB-D (Red, Green, Blue-Depth red, green and blue color image + depth image) and a semantic mapping method based on object parameterization. Background Art

[0002] Image-based object detection algorithm is a challenging research problem in the field of computer vision, which can be divided into the traditional object detection algorithm period and the deep learning-based object detection algorithm period. Traditional object detection technology has problems such as poor robustness, high time complexity and redundant detection windows, which limits the performance improvement of object detection and makes the development of traditional object detection technology enter a bottleneck period. In 2014, with the good performance of the convolutional neural network AlexNet in computer vision tasks, image-based object detection research began to develop in the direction of deep learning. R-CNN (Region-based Convolutional Neural Network) used convolutional neural networks to realize the two-stage object detection method based on deep learning for the first time. Although the detection performance has been significantly improved compared with traditional object detection methods, the training process of this method is relatively complicated and has high time and space complexity. The subsequent Fast R-CNN (Fast Region-based Convolutional Neural Network) and Faster R-CNN (Faster Region-based Convolutional Neural Network) methods have improved the above problems one by one. Fast R-CNN proposed a multi-task loss function to make bounding box regression trainable. It solved the problem of multi-stage training by designing an end-to-end trainable model, greatly improving the real-time performance of detection. However, the selective search method used by Fast R-CNN to generate candidate boxes is slow. Faster R-CNN proposed a fully convolutional network structure as a region proposal network (RPN), which solves the limitations of selective search and effectively improves the speed of generating candidate boxes. On this basis, algorithms such as FPN (Feature Pyramid Networks) and Mask R-CNN (Mask Region-based Convolutional Neural Network) further enriched the components of Faster R-CNN to improve the performance of the two-stage target detection method.

[0003] Real-time perception of the surrounding environment is also crucial for the robot to perform trajectory planning, target tracking, human-computer interaction, etc. Simultaneous Localization and Mapping (SLAM) solves the problem of positioning and mapping in an unknown environment of the Global Positioning System (GPS), enabling the robot to obtain positioning information in a GPS-denied environment and thus have the ability to perceive the scene. In early studies, mapping was usually used as an additional part of SLAM, directly projecting point clouds or sparse feature points onto the map based on the positioning results. In fields such as trajectory planning and human-computer interaction, raster maps are usually used for operations. However, when dealing with complex tasks, ordinary raster maps are difficult to meet the needs of robots interacting with the environment. In response to this problem, relevant scholars have carried out research on semantic mapping in recent years. In 2017, McCormac et al. combined the geometric information and semantic segmentation in SLAM to incrementally construct dense semantically annotated three-dimensional maps. In 2018, Runz et al. tracked and reconstructed dynamic objects, enabling robots to maintain scene descriptions at the object level and interact with dynamic environments. In 2019, Grinvald not only projected semantic labels for single 3D points, but also maintained objects as central entities in the map. In 2020, Chen et al. proposed RepPointsv2, which added object corner information predicted by network-extracted features to the original prediction frame, improving the prediction accuracy of object positions and achieving further performance improvements.

[0004] In summary, domestic and foreign scholars have conducted fruitful research on target detection and real-time perception and achieved rich research results, but most of the related research is conducted on desktop computing devices. Real-time instance segmentation and semantic mapping on small onboard devices equipped with robots is still a challenging task. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention aims to propose a real-time semantic perception system running on a lightweight airborne device to perform real-time instance segmentation and semantic mapping of the robot's working environment. To this end, the technical solution adopted by the present invention is a perception method based on lightweight real-time instance segmentation and semantic mapping. The first part: using red, green, and blue color images and depth images as input, first use the color image containing color and texture information to perform object classification and bounding box regression; then, according to the obtained bounding box, the corresponding input original depth image is cropped into a depth image with precise geometric edges, and it is input into the truncation network to generate an instance mask; the second part: (1) calculate the posture and size of the target object in the camera's field of view through principal component analysis; (2) obtain the category of the object in the field of view obtained by the instance segmentation in the first part; (3) create a point cloud of the target object depth image projection in the map; through the above three operations, each object in the camera's field of view is parameterized, and then data association is used for cross-frame tracking and fusion to achieve the purpose of perception.

[0006] The detailed steps of the first part are as follows:

[0007] Model architecture: In the first stage, a lightweight object detection network YOLO is used to process images with color and texture information to obtain the category and bounding box of the predicted object; then, the original depth image is cropped into blocks according to the predicted bounding box and used as the RoI alignment layer; in the second stage, the mask generation problem is restated as a binary decision about depth through threshold regression: the depth values ​​of the object and background in each frame image follow independent normal distributions, then each object in the image can be separated from the background in depth. First, the effective depth value D(u,v) of the pixel (u,v) in the bounding box is processed by the following formula, and then the depth histogram H is obtained by statistics:

[0008]

[0009] Among them, H i is the number of pixels at each depth level, Dim represents the resolution of the depth map, i represents the depth value, and d min ,d max are the minimum and maximum distances represented by the depth histogram, respectively, and then the maximum truncation threshold T is predicted by the depth histogram max , use the maximum cutoff threshold to separate the target and background in each frame image;

[0010] Due to the different sizes of images, in order to ensure stability during the learning process, the depth histogram needs to be normalized before the depth map is input into the truncated network. Then the processed depth histogram is input into the truncated network with the following structure: three one-dimensional convolutional layers with 64, 32 and 8 cores are used to traverse the sliding window and extract distinguishable local features; then, the obtained feature map is input into two linear layers consisting of 1024 cores and 128 cores respectively, and then a softmax layer is used to output a 128-dimensional vector, which represents the probability value of the best threshold in each depth layer; finally, the depth value with the largest probability value is used as the truncation threshold T max ; Then, according to the predicted cutoff threshold, the depth below T max Pixels with α are considered as target objects, otherwise they are considered as background.

[0011] The first part also includes the following steps:

[0012] Truncated network training strategy: Use the inter-class variance to measure the separability of the threshold and use it to guide supervised learning. First, normalize the depth histogram and pass Treat the depth map as a probability distribution and then calculate the between-class variance σ 2 (t):

[0013]

[0014] in, is the total average depth value within the cropped bounding box, is the zero-order cumulative moment of the depth histogram, and t is the depth level; then the inter-class variance is converted into the probability distribution of the optimal threshold through the softmax kernel function:

[0015]

[0016] The loss function L is the cross entropy between the predicted value and the true value, where y(t), Represent the true value and predicted value respectively:

[0017]

[0018] In order to enrich the distribution of samples, the depth histogram is translated and random noise is added for data enhancement;

[0019] Occlusion elimination: Use a simplified method to find the minimum truncation threshold T min ∈[d min ,T max ] makes the depth histogram located at [T min ,T max] is closest to the normal distribution: For a normal distribution, 68%, 95%, and 99.7% of the values ​​are within one, two, or three standard deviations around the mean, respectively. First, calculate H∈[d min ,T max ], and then remove the outliers below μ1-λσ1 by the following formula:

[0020] T min =μ1-λσ1 (6)

[0021] In the above formula, λ is an adjustable parameter. For the case without occlusion, due to the noise of the depth camera, a minimum truncation threshold is also required for image segmentation. In the area cropped by each bounding box, the binary mask of each instance is obtained by the following formula:

[0022]

[0023] In order to better visualize, statistical outlier processing and other methods are used to eliminate sensor noise and [T min ,T max ] background misclassified by depth values.

[0024] The detailed steps of the second part are as follows:

[0025] First, the unknown space is divided into voxels with occupancy and category attributes and initialized using uniform distribution. When a new observation is obtained, the obtained pose, instance mask and depth image are aligned by timestamp, and the points in the pixel coordinate system are mapped to the world coordinate system by the following formula:

[0026] p w =T wc K -1 D uv p uv (8)

[0027] Among them, p uv =[u,v,1] T and p w =[x,y,z] T are the coordinate values ​​of the point in the pixel coordinate system and the world coordinate system, respectively, uv is the depth value of the point, K=[f x ,0,c x ; 0,f y ,c y ; 0,0,1] is the camera’s intrinsic parameter matrix, T wc is the transformation matrix from the camera coordinate system to the world coordinate system, p w The state of the voxel is set to occupied state, from the origin to p wThe voxels between are set to unoccupied state, and the semantic information of the object obtained by the first part of the target detection network, that is, the category of the object, is projected to the initialized voxels through the mask using the aligned pose, mask and depth image. Each prediction uses recursive Bayesian update to integrate the state and category of the voxels in the map, and its probability distribution in the k-1 frame is P(SI1,...,k-1), I 1,...,k-1 represents the observations from the first frame to the k-1th frame, S is an N+1-dimensional vector representing the occupancy state and N categories, and the probability distribution P(SI) of the first k observations is 1,...,k ) is updated by:

[0028] P(SI 1,..,k )=ηP(SI 1,...,k-1 )P(SI k ) (9)

[0029] Where η is used for normalization to obtain a suitable distribution, P(SI k ) represents the probability value of the occupancy state and category observed at the current moment; for any Si∈S, the distribution P(S i ) is independent, i.e. Therefore, in the case of uniform initialization, by eliminating η, the probability distribution of the kth frame is calculated as follows:

[0030]

[0031] Assign voxel labels to the maximum probability greater than a threshold, otherwise it is defined as the background, so that the grid map composed of voxel labels and background can be directly adapted to perception and motion planning;

[0032] Object modeling: Each object is modeled as a cuboid ο = {T, d, c, pcl}, where T∈SE(3) represents the six-degree-of-freedom pose, d = (l, w, h) represents the half-length, half-width and half-height of the cuboid, c represents the category, and pcl represents the point cloud of the object. All objects are placed on a horizontal plane, and the covariance matrix of the point cloud pcl is expressed as:

[0033]

[0034] Where cov(x,y) represents the covariance of the point cloud projected onto the xy plane, and then the normalized eigenvector ν1=[x1,y1] representing the main direction of the point cloud in the xy plane is solved. T and ν2=[x2,y2] T , get the rotation state of the object through the following matrix:

[0035]

[0036] In order to track the minimum and maximum coordinates in the main direction, the object point cloud p is transformed into w Transform to local frame p w ′:

[0037] p w '=R -1 p w =R T p w (13)

[0038] Afterwards, the vertices of the directed bounding box are denoted as p′ max =(x′ max ,y′ max ) and p′ min =(x′ min ,y′ min ), x′ max ,x′ min ,y′ max and y′ min are the maximum and minimum coordinates of the transformation point respectively. Finally, through (p′ max -p′ min ) / 2 to obtain the half length and half height of the object, and through R(p′ max +p′ min ) / 2 Obtain the position information of the object center in the world coordinate system;

[0039] Data association: Using the target association method based on Intersection over Union (IoU), the current prediction result is matched with the landmarks in the map, that is, the landmarks with object models and semantic information in the map. For the i-th object in the k-th frame, Select a group of candidate landmarks with the same category. In order to avoid the wrong association caused by directly using 3-D IoU matching, the sum of the intersection ratios C ij To measure spatial correlation:

[0040]

[0041] Among them, V i ,V j ,V I Represent objects and the volume of the intersection of the two objects, if C ij If the threshold exceeds 1.4, the corresponding landmark can be considered Match, otherwise it is added to the map as a new object; in order to improve data efficiency, the point cloud of the matching object in the current frame is fused into the corresponding landmark and downsampled to the size of a voxel.

[0042] To eliminate the influence of outliers, all landmarks in the field of view are checked for validity, their occupancy and category information are updated and integrated into the map, and voxels with free status or background category are removed from the landmark point cloud. The updated landmark parameters are then recalculated and the removed points are removed from the map.

[0043] The characteristics and beneficial effects of the present invention are:

[0044] The present invention is of great significance to the research on real-time instance segmentation and semantic mapping in the field of computer vision. In the face of unknown multi-obstacle environments, the present invention can realize the efficient operation of target detection algorithms on small airborne devices to complete the tasks of instance segmentation and semantic mapping. The present invention has high theoretical and engineering value, and the advantages of the present invention are summarized as follows:

[0045] (1) This paper proposes an instance segmentation method based on RGB-D input. Different from traditional methods, this method uses both color images and depth images to capture the three-dimensional structure in the scene and improve the accuracy of instance segmentation, especially in the case of complex backgrounds or occlusions. This method proposes a simplified method to find the minimum truncation threshold, which can better understand the spatial position and shape of the object and reduce the impact of noise from the depth camera, which helps to achieve more accurate target detection and segmentation.

[0046] (2) The present invention designs a semantic mapping method based on object parameterization. Traditional methods use points or polygons to store map information, which has low storage efficiency and high computational complexity, and cannot meet the requirements of storing large amounts of data in complex scenes and real-time computing. The present invention uses a grid map to store the occupancy rate and category probability of voxels, parameterizes objects, and can more accurately represent and understand objects and their attributes in the scene, improve the accuracy of semantic mapping, and support robots to complete complex tasks such as human-computer interaction.

[0047] (3) The lightweight semantic perception system proposed in the present invention can run on small airborne devices, reducing the demand for hardware and being suitable for robot operating platforms. The algorithms and models are simplified and can quickly process data to complete real-time perception tasks. It is also easier to integrate into the existing system architecture and work in conjunction with other modules or sensors to achieve efficient operation of complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Attached Figure 1 Overall structure diagram of lightweight real-time instance segmentation and semantic mapping,

[0049] Attached Figure 2 Depth maps of objects and background in the image,

[0050] Attached Figure 3 Schematic diagram of the depth value histogram of objects and background in the image,

[0051] Attached Figure 4 Schematic diagram of target detection and semantic segmentation of keyboard,

[0052] Attached Figure 5 Schematic diagram of target detection and semantic segmentation for keyboard and mouse,

[0053] Attached Figure 6 Schematic diagram of target detection and semantic segmentation for mobile phones, DETAILED DESCRIPTION

[0054] The present invention is based on a lightweight object detection model and transforms the mask generation problem into a threshold regression problem through a newly designed truncation network, and then realizes the semantic mapping of object parameterization to perform complex tasks such as trajectory planning.

[0055] In order to obtain accurate segmentation contours, the present invention proposes a real-time instance segmentation network running on an onboard small computing device, using red, green, blue and depth images as input, and performing target detection and instance segmentation in sequence. First, a color image containing rich color and texture information is used for classification and bounding box regression. Then, the depth image with precise geometric edges is cropped according to the bounding box and input into the truncation network to generate an instance mask. In addition, for the complex tasks performed by the robot, the present invention proposes a semantic mapping method based on object parameterization, which creates and tracks each object represented by parameterized information such as pose, range, category and point cloud in the map and tracks it across frames, so that the robot can query the target position and the distance to the obstacle to perform complex tasks such as human-computer interaction.

[0056] The detailed steps of the first part are as follows:

[0057] Model architecture: In the first stage, a lightweight object detection network YOLO is used to process images with color and texture information to obtain the category and bounding box of the predicted object; then, the original depth image is cropped into blocks according to the predicted bounding box and used as the RoI (Region of Interest) alignment layer; in the second stage, the mask generation problem is restated as a binary decision about depth through threshold regression: the depth values ​​of the object and background in each frame image follow independent normal distributions, then each object in the image can be separated from the background in depth. First, the effective depth value D(u,v) of the pixel (u,v) in the bounding box is processed by the following formula, and then the depth histogram H is obtained by statistics:

[0058]

[0059] Among them, H i is the number of pixels at each depth level, Dim represents the resolution of the depth map, i represents the depth value, and dmin ,d max are the minimum and maximum distances represented by the depth histogram, respectively, and then the maximum truncation threshold T is predicted by the depth histogram max , use the maximum cutoff threshold to separate the target and background in each frame image;

[0060] Due to the different sizes of images, in order to ensure stability during the learning process, the depth histogram needs to be normalized before the depth map is input into the truncated network. Then the processed depth histogram is input into the truncated network with the following structure: three one-dimensional convolutional layers with 64, 32 and 8 cores are used to traverse the sliding window and extract distinguishable local features; then, the obtained feature map is input into two linear layers consisting of 1024 cores and 128 cores respectively, and then a softmax layer is used to output a 128-dimensional vector, which represents the probability value of the best threshold in each depth layer; finally, the depth value with the largest probability value is used as the truncation threshold T max ; Then, according to the predicted cutoff threshold, the depth below T max Pixels with α are considered as target objects, otherwise they are considered as background.

[0061] The first part also includes the following steps:

[0062] Truncated network training strategy: Use the inter-class variance to measure the separability of the threshold and use it to guide supervised learning. First, normalize the depth histogram and pass Treat the depth map as a probability distribution and then calculate the between-class variance σ 2 (t):

[0063]

[0064] in, is the total average depth value within the cropped bounding box, is the zero-order cumulative moment of the depth histogram, and t is the depth level; then the inter-class variance is converted into the probability distribution of the optimal threshold through the softmax kernel function:

[0065]

[0066] The loss function L is the cross entropy between the predicted value and the true value, where Represent the true value and predicted value respectively:

[0067]

[0068] In order to enrich the distribution of samples, the depth histogram is translated and random noise is added for data enhancement;

[0069] Occlusion elimination: Use a simplified method to find the minimum truncation threshold Tmin ∈[d min ,T max ] makes the depth histogram located at [T min ,T max ] is closest to the normal distribution: For a normal distribution, 68%, 95%, and 99.7% of the values ​​are within one, two, or three standard deviations around the mean, respectively. First, calculate H∈[d min ,T max ], and then remove the outliers below μ1-λσ1 by the following formula:

[0070] T min =μ1-λσ1 (20)

[0071] In the above formula, λ is an adjustable parameter. For the case without occlusion, due to the noise of the depth camera, a minimum truncation threshold is also required for image segmentation. In the area cropped by each bounding box, the binary mask of each instance is obtained by the following formula:

[0072]

[0073] In order to better visualize, statistical outlier processing and other methods are used to eliminate sensor noise and [T min ,T max ] background misclassified by depth values.

[0074] The detailed steps of the second part are as follows:

[0075] First, the unknown space is divided into voxels with occupancy and category attributes and initialized using uniform distribution. When a new observation is obtained, the obtained pose, instance mask and depth image are aligned by timestamp, and the points in the pixel coordinate system are mapped to the world coordinate system by the following formula:

[0076] p w =T wc K -1 D uv p uv (twenty two)

[0077] Among them, p uv =[u,v,1] T and p w =[x,y,z] T are the coordinate values ​​of the point in the pixel coordinate system and the world coordinate system, respectively, uv is the depth value of the point, K=[f x ,0,c x ; 0,f y ,c y ; 0,0,1] is the camera’s intrinsic parameter matrix, Twc is the transformation matrix from the camera coordinate system to the world coordinate system, p w The state of the voxel is set to occupied state, from the origin to p w The voxels between are set to non-occupied state, and the object semantic information (object category) obtained by the first part of the target detection network is projected to the initialized voxels through the mask using the aligned pose, mask and depth image. Each prediction uses recursive Bayesian update to integrate the state and category of the voxels in the map, and its probability distribution in the k-1 frame is P(SI1,...,k-1), I 1,...,k-1 represents the observations from the first frame to the k-1th frame, S is an N+1-dimensional vector representing the occupancy state and N categories, and the probability distribution P(SI) of the first k observations is 1,...,k ) is updated by:

[0078] P(SI 1,..,k )=ηP(SI 1,...,k-1 )P(SI k ) (twenty three)

[0079] Where η is used for normalization to obtain a suitable distribution, P(SI k ) represents the probability value of the occupancy state and category observed at the current moment; for any Si∈S, the distribution P(S i ) is independent, i.e. Therefore, in the case of uniform initialization, by eliminating η, the probability distribution of the kth frame is calculated as follows:

[0080]

[0081] Assign voxel labels to the maximum probability greater than the threshold, otherwise it is defined as the background, thus forming a grid map (voxel labels and background constitute a grid map). The constructed grid map can be directly adapted to the robot system to complete the purpose of motion planning;

[0082] Object modeling: Each object is modeled as a cuboid ο = {T, d, c, pcl}, where T∈SE(3) represents the six-degree-of-freedom pose, d = (l, w, h) represents the half-length, half-width and half-height of the cuboid, c represents the category, and pcl represents the point cloud of the object. All objects are placed on a horizontal plane, and the covariance matrix of the point cloud pcl is expressed as:

[0083]

[0084] Where cov(x,y) represents the covariance of the point cloud projected onto the xy plane, and then the normalized eigenvector ν1=[x1,y1] representing the main direction of the point cloud in the xy plane is solved. T and ν2=[x2,y2] T, get the rotation state of the object through the following matrix:

[0085]

[0086] In order to track the minimum and maximum coordinates in the main direction, the object point cloud p is transformed into w Transform to local frame p w ′:

[0087] p w '=R -1 p w =R T p w (27)

[0088] Afterwards, the vertices of the directed bounding box are denoted as p′ max =(x′ max ,y′ max ) and p′ min =(x′ min ,y′ min ), x′ max ,x′ min ,y′ max and y′ min are the maximum and minimum coordinates of the transformation point respectively. Finally, through (p′ max -p′ min ) / 2 to obtain the half length and half height of the object, and through R(p′ max +p′ min ) / 2 Obtain the position information of the object center in the world coordinate system;

[0089] Data association: Using the target association method based on Intersection over Union (IoU), the current prediction result is matched with the landmarks in the map (the landmarks with object models and semantic information in the map). For the i-th object in the k-th frame, Select a group of candidate landmarks with the same category. In order to avoid the wrong association caused by directly using 3-D IoU matching, the sum of the intersection ratios C ij To measure spatial correlation:

[0090]

[0091] Among them, V i ,V j ,V I Represent objects and the volume of the intersection of the two objects, if C ij If the threshold exceeds 1.4, the corresponding landmark can be considered Match, otherwise it is added to the map as a new object; in order to improve data efficiency, the point cloud of the matching object in the current frame is fused into the corresponding landmark and downsampled to the size of a voxel.

[0092] To eliminate the influence of outliers, all landmarks in the field of view are checked for validity, their occupancy and category information are updated and integrated into the map, and voxels with free status or background category are removed from the landmark point cloud. The updated landmark parameters are then recalculated and the removed points are removed from the map.

[0093] The present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0094] The present invention belongs to the field of computer vision technology. It requires the use of a color camera and a depth camera to perceive and identify the target. The algorithm part mainly consists of a lightweight instance segmentation method based on RGB-D (color image + depth image) and an object-based parameterized semantic mapping method. The following is a detailed introduction:

[0095] The overall structure diagram of the lightweight real-time instance segmentation and semantic mapping method is shown in the attached figure. Figure 1 All algorithms in the present invention are written in Python and developed under the Linux operating system. The method designed by the present invention mainly includes the following two parts:

[0096] Part I, Instance Segmentation Method Based on RGB--D Input: This part proposes a real-time instance segmentation method that runs on a small airborne device. RGB color images and depth images are used for segmentation at the same time. RGB images encode rich visual appearance features, and depth images represent the precise geometric edges of objects. This is explained with examples. The example is a small airborne device, so it is better to keep this section.

[0097] The detailed steps of the first part are as follows:

[0098] Model architecture: In the first stage, a lightweight object detection network YOLO (You Only LookOnce) is used to process images with color and texture information to obtain the category and bounding box of the predicted object; then, the original depth image is cropped into blocks according to the predicted bounding box and used as the RoI (Region of Interest) alignment layer; in the second stage, the mask generation problem is reformulated as a binary decision about depth through threshold regression: the depth values ​​of the object and background in each frame follow independent normal distributions, as shown in the attached figure. Figure 2 and 3As shown, each object in the image can be separated from the background in depth. First, the effective depth value D(u,v) of the pixel (u,v) in the bounding box is processed by the following formula, and then the depth histogram H is obtained by statistics:

[0099]

[0100] Among them, H i is the number of pixels at each depth level, Dim represents the resolution of the depth map, i represents the depth value, and d min ,d max are the minimum and maximum distances represented by the depth histogram, respectively, and then the maximum truncation threshold T is predicted by the depth histogram max , use the maximum cutoff threshold to separate the target and background in each frame image;

[0101] Due to the complexity of the object structure in the viewing cone and the diversity of background distribution, the traditional adaptive histogram segmentation method cannot be applied to all situations, especially when the depth histogram is multimodal, it is very time-consuming to predict the optimal threshold by traversing each depth layer. Therefore, the present invention proposes a truncated network to learn the depth distribution of different objects and scenes. Due to the different sizes of images, in order to ensure the stability during the learning process, the depth histogram needs to be normalized before the depth map is input into the truncated network, and then the processed depth histogram is input into the truncated network with the following structure: 3 one-dimensional convolutional layers with 64, 32 and 8 cores are used to traverse the sliding window and extract distinguishable local features; then, the obtained feature map is input into two linear layers consisting of 1024 cores and 128 cores respectively, and then a 128-dimensional vector is output through a softmax layer, indicating the probability value of the optimal threshold in each depth layer; finally, the depth value with the largest probability value is used as the truncation threshold T max ; Then, according to the predicted cutoff threshold, the depth below T max Pixels with α are considered as target objects, otherwise they are considered as background.

[0102] Training strategy: Taking the normalized depth histogram as input, the truncated network transforms the generation of instance masks into an optimal depth threshold prediction problem. Since the separable threshold is not unique, the expected prediction should be a probability distribution around the optimal solution.

[0103] Truncated network training strategy: Use the inter-class variance to measure the separability of the threshold and use it to guide supervised learning. First, normalize the depth histogram and pass Treat the depth map as a probability distribution and then calculate the between-class variance σ 2 (t):

[0104]

[0105] in, is the total average depth value within the cropped bounding box, is the zero-order cumulative moment of the depth histogram, and t is the depth level; then the inter-class variance is converted into the probability distribution of the optimal threshold through the softmax kernel function:

[0106]

[0107] The loss function L is the cross entropy between the predicted value and the true value, where y(t), Represent the true value and predicted value respectively:

[0108]

[0109] In order to enrich the distribution of samples, the depth histogram is shifted and random noise is added for data enhancement; thanks to the abstract features and enhanced data, the truncated network proposed in the present invention is robust in noisy, multimodal and unbalanced distributions where classical methods fail.

[0110] Occlusion elimination: In some specific cases, the detected target will inevitably have occluders, but the bounding box predicted by the target detection network tends to select the visible part of the occluded object, which will cause a large error in the threshold segmentation. Assuming that the occluded part occupies a small part of the bounding box cropping area, this paper proposes a simplified method to find the minimum truncation threshold T min ∈[d min ,T max ] makes the depth histogram located at [T min ,T max ] is closest to the normal distribution: For a normal distribution, 68%, 95%, and 99.7% of the values ​​are within one, two, or three standard deviations around the mean, respectively. First, calculate H∈[d min ,T max ], and then remove the outliers below μ1-λσ1 by the following formula:

[0111] T min =μ1-λσ1 (34)

[0112] In the above formula, λ is an adjustable parameter. For the case without occlusion, due to the noise of the depth camera, a minimum truncation threshold is also required for image segmentation. In the area cropped by each bounding box, the binary mask of each instance is obtained by the following formula:

[0113]

[0114] In order to better visualize, statistical outlier processing and other methods are used to eliminate sensor noise and [Tmin ,T max ] background misclassified by depth values.

[0115] The second part is the semantic mapping method based on object parameterization:

[0116] Semantic Mapping: With limited computing resources, this paper proposes a semantic mapping method that can be used for robot motion planning and exploration. Compared with the traditional method of using points or polygons to store map information, this paper uses a grid map to store the occupancy rate and category probability of voxels to save computational effort. The following operations are performed on each object: (1) calculate the posture and size through principal component analysis; (2) obtain the object category obtained by instance segmentation; (3) create a point cloud projected by the depth image in the map. Each object is parameterized through the above three operations, and then data association is used for cross-frame tracking and fusion.

[0117] First, the unknown space is divided into voxels with occupancy and category attributes and initialized using uniform distribution. When a new observation is obtained, the obtained pose, instance mask and depth image are aligned by timestamp, and the points in the pixel coordinate system are mapped to the world coordinate system by the following formula:

[0118] p w =T wc K -1 D uv p uv (36)

[0119] Among them, p uv =[u,v,1] T and p w =[x,y,z] T are the coordinate values ​​of the point in the pixel coordinate system and the world coordinate system, respectively, uv is the depth value of the point, K=[f x ,0,c x ; 0,f y ,c y ; 0,0,1] is the camera’s intrinsic parameter matrix, T wc is the transformation matrix from the camera coordinate system to the world coordinate system, p w The state of the voxel is set to occupied state, from the origin to p w The voxels between are set to non-occupied state, and the object semantic information (object category) obtained by the first part of the target detection network is projected to the initialized voxels through the mask using the aligned pose, mask and depth image. Each prediction uses recursive Bayesian update to integrate the state and category of the voxels in the map, and its probability distribution in the k-1 frame is P(SI1,...,k-1), I 1,...,k-1represents the observations from the first frame to the k-1th frame, S is an N+1-dimensional vector representing the occupancy state and N categories, and the probability distribution P(SI) of the first k observations is 1,...,k ) is updated by:

[0120] P(SI 1,..,k )=ηP(SI 1,...,k-1 )P(SI k ) (37)

[0121] Where η is used for normalization to obtain a suitable distribution, P(SI k ) represents the probability value of the occupancy state and category observed at the current moment; for any Si∈S, the distribution P(S i ) is independent, i.e. Therefore, in the case of uniform initialization, by eliminating η, the probability distribution of the kth frame is calculated as follows:

[0122]

[0123] Assign voxel labels to the maximum probability greater than the threshold, otherwise it is defined as the background, thus forming a grid map (voxel labels and background constitute a grid map). The constructed grid map can be directly adapted to the robot system to complete the purpose of motion planning;

[0124] Object modeling: Each object is modeled as a cuboid ο = {T, d, c, pcl}, where T∈SE(3) represents the six-degree-of-freedom pose, d = (l, w, h) represents the half-length, half-width and half-height of the cuboid, c represents the category, and pcl represents the point cloud of the object. All objects are placed on a horizontal plane, and the covariance matrix of the point cloud pcl is expressed as:

[0125]

[0126] Where cov(x,y) represents the covariance of the point cloud projected onto the xy plane, and then the normalized eigenvector ν1=[x1,y1] representing the main direction of the point cloud in the xy plane is solved. T and ν2=[x2,y2] T , get the rotation state of the object through the following matrix:

[0127]

[0128] In order to track the minimum and maximum coordinates in the main direction, the object point cloud p is transformed into w Transform to local frame p w ′:

[0129] p w '=R -1 p w =RT p w (41)

[0130] Afterwards, the vertices of the directed bounding box are denoted as p′ max =(x′ max ,y′ max ) and p′ min =(x′ min ,y′ min ), x′ max ,x′ min ,y′ max and y′ min are the maximum and minimum coordinates of the transformation point respectively. Finally, through (p′ max -p′ min ) / 2 to obtain the half length and half height of the object, and through R(p′ max +p′ min ) / 2 Obtain the position information of the object center in the world coordinate system;

[0131] Data association: Using the target association method based on Intersection over Union (IoU), the current prediction result is matched with the landmarks in the map (the landmarks with object models and semantic information in the map). For the i-th object in the k-th frame, Select a group of candidate landmarks with the same category. In order to avoid the wrong association caused by directly using 3-D IoU matching, the sum of the intersection ratios C ij To measure spatial correlation:

[0132]

[0133] Among them, V i ,V j ,V I Represent objects and the volume of the intersection of the two objects, if C ij If the threshold exceeds 1.4, the corresponding landmark can be considered Match, otherwise it is added to the map as a new object; in order to improve data efficiency, the point cloud of the matching object in the current frame is fused into the corresponding landmark and downsampled to the size of a voxel.

[0134] To eliminate the influence of outliers, all landmarks in the field of view are checked for validity, their occupancy and category information are updated and integrated into the map, and voxels with free status or background category are removed from the landmark point cloud. The updated landmark parameters are then recalculated and the removed points are removed from the map.

[0135] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A perception method based on lightweight real-time instance segmentation and semantic mapping, characterized by: Part 1: Using red, green, and blue color images and depth images as input, first use the color image containing color and texture information to perform object classification and bounding box regression; then, according to the obtained bounding box, the corresponding input original depth image is cropped into a depth image with precise geometric edges, and it is input into the truncation network to generate the instance mask; Part 2: (1) Calculate the posture and size of the target object in the camera field of view through principal component analysis; (2) Obtain the category of the object in the field of view obtained by instance segmentation in the first part; (3) Create a point cloud of the target object’s depth image projection in the map; parameterize each object in the camera’s field of view through the above three operations, and then use data association to perform cross-frame tracking and fusion to achieve the purpose of perception.

2. The perception method based on lightweight real-time instance segmentation and semantic mapping as claimed in claim 1, characterized in that: The detailed steps of the first part are as follows: Model architecture: In the first stage, a lightweight object detection network YOLO is used to process images with color and texture information to obtain the category and bounding box of the predicted object; Then, the original depth image is cropped into blocks according to the predicted bounding boxes and used as the RoI alignment layer; In the second stage, the mask generation problem is reformulated as a binary decision about depth through threshold regression: the depth values ​​of the object and the background in each frame follow independent normal distributions, so each object in the image can be separated from the background in depth. First, the effective depth value D(u,v) of the pixel (u,v) in the bounding box is processed by the following formula, and then the depth histogram H is obtained by statistics: Among them, H i is the number of pixels at each depth level, Dim represents the resolution of the depth map, i represents the depth value, and d min ,d max are the minimum and maximum distances represented by the depth histogram, respectively, and then the maximum truncation threshold T is predicted by the depth histogram max , use the maximum cutoff threshold to separate the target and background in each frame image; Due to the different sizes of images, in order to ensure stability during the learning process, the depth histogram needs to be normalized before the depth map is input into the truncated network. Then the processed depth histogram is input into the truncated network with the following structure: three one-dimensional convolutional layers with 64, 32 and 8 cores are used to traverse the sliding window and extract distinguishable local features; then, the obtained feature map is input into two linear layers consisting of 1024 cores and 128 cores respectively, and then a softmax layer is used to output a 128-dimensional vector, which represents the probability value of the best threshold in each depth layer; finally, the depth value with the largest probability value is used as the truncation threshold T max ; Then, according to the predicted cutoff threshold, the depth below T max Pixels with α are considered as target objects, otherwise they are considered as background.

3. The perception method based on lightweight real-time instance segmentation and semantic mapping as claimed in claim 1, characterized in that: The first part also includes the following steps: Truncated network training strategy: Use the inter-class variance to measure the separability of the threshold and use it to guide supervised learning. First, normalize the depth histogram and pass Treat the depth map as a probability distribution and then calculate the between-class variance σ 2 (t): in, is the total average depth value within the cropped bounding box, is the zero-order cumulative moment of the depth histogram, and t is the depth level; then the inter-class variance is converted into the probability distribution of the optimal threshold through the softmax kernel function: The loss function L is the cross entropy between the predicted value and the true value, where y(t), Represent the true value and predicted value respectively: In order to enrich the distribution of samples, the depth histogram is translated and random noise is added for data enhancement; Occlusion elimination: Use a simplified method to find the minimum truncation threshold T min ∈[d min ,T max ] makes the depth histogram located at [T min ,T max ] is closest to the normal distribution: For a normal distribution, 68%, 95%, and 99.7% of the values ​​are within one, two, or three standard deviations around the mean, respectively. First, calculate H∈[d min ,T max ], and then remove the outliers below μ1-λσ1 by the following formula: T min =μ1-λσ1 (6) In the above formula, λ is an adjustable parameter. For the case without occlusion, due to the noise of the depth camera, a minimum truncation threshold is also required for image segmentation. In the area cropped by each bounding box, the binary mask of each instance is obtained by the following formula: In order to better visualize, statistical outlier processing and other methods are used to eliminate sensor noise and [T min ,T max ] background misclassified by depth values.

4. The perception method based on lightweight real-time instance segmentation and semantic mapping as claimed in claim 1, characterized in that: The detailed steps of the second part are as follows: First, the unknown space is divided into voxels with occupancy and category attributes and initialized using uniform distribution. When a new observation is obtained, the obtained pose, instance mask and depth image are aligned by timestamp, and the points in the pixel coordinate system are mapped to the world coordinate system by the following formula: p w =T wc K -1 D uv p uv (8) Among them, p uv =[u,v,1] T and p w =[x,y,z] T are the coordinate values ​​of the point in the pixel coordinate system and the world coordinate system, respectively, uv is the depth value of the point, K=[f x ,0,c x ; 0,f y ,c y ; 0,0,1] is the camera’s intrinsic parameter matrix, T wc is the transformation matrix from the camera coordinate system to the world coordinate system, p w The state of the voxel is set to occupied state, from the origin to p w The voxels between are set to unoccupied state, and the semantic information of the object obtained by the first part of the target detection network, that is, the category of the object, is projected to the initialized voxels through the mask using the aligned pose, mask and depth image. Each prediction uses recursive Bayesian update to integrate the state and category of the voxels in the map, and its probability distribution in the k-1th frame is P(S|I 1,...,k-1 ), I 1,...,k-1 represents the observations from the first frame to the k-1th frame, S is an N+1-dimensional vector representing the occupancy state and N categories, and the probability distribution of the first k observations is P(S|I 1,...,k ) is updated by: P(S|I 1,..,k )=ηP(S|I 1,...,k-1 )P(S|I k ) (9) Where η is used for normalization to obtain a suitable distribution, P(S|I k ) represents the probability value of the occupancy state and category observed at the current moment; for any S i ∈S, distribution P(S i ) is independent, i.e. Therefore, in the case of uniform initialization, by eliminating η, the probability distribution of the kth frame is calculated as follows: Assign voxel labels to the maximum probability greater than a threshold, otherwise it is defined as the background, so that the grid map composed of voxel labels and background can be directly adapted to perception and motion planning; Object modeling: Each object is modeled as a cuboid ο = {T, d, c, pcl}, where T∈SE(3) represents the six-degree-of-freedom pose, d = (l, w, h) represents the half-length, half-width and half-height of the cuboid, c represents the category, and pcl represents the point cloud of the object. All objects are placed on a horizontal plane, and the covariance matrix of the point cloud pcl is expressed as: Where cov(x,y) represents the covariance of the point cloud projected onto the xy plane, and then the normalized eigenvector ν1 = [x1, y1] representing the main direction of the point cloud in the xy plane is solved. T and ν2=[x2,y2] T , get the rotation state of the object through the following matrix: In order to track the minimum and maximum coordinates in the main direction, the object point cloud p is transformed into w Transform to local frame p w ′: p w '=R -1 p w =R T p w (13) Afterwards, the vertices of the directed bounding box are denoted as p′ max =(x′ max ,y′ max ) and p′ min =(x′ min ,y′ min ), x′ max ,x′ min ,y′ max and y′ min are the maximum and minimum coordinates of the transformation point respectively. Finally, through (p′ max -p′ min ) / 2 to obtain the half length and half height of the object, and through R(p′ max +p′ min ) / 2 Obtain the position information of the object center in the world coordinate system; Data association: Using the target association method based on Intersection over Union (IoU), the current prediction result is matched with the landmarks in the map, that is, the landmarks with object models and semantic information in the map. For the i-th object in the k-th frame, Select a group of candidate landmarks with the same category. In order to avoid the wrong association caused by directly using 3-D IoU matching, the sum of the intersection ratios C ij To measure spatial correlation: Among them, V i ,V j ,V I Represent objects and the volume of the intersection of the two objects, if C ij If the threshold exceeds 1.4, the corresponding landmark can be considered Match, otherwise it is added to the map as a new object; in order to improve data efficiency, the point cloud of the matching object in the current frame is fused into the corresponding landmark and downsampled to the size of a voxel.

5. The perception method based on lightweight real-time instance segmentation and semantic mapping as claimed in claim 1, characterized in that: The following steps are also included: In order to eliminate the influence of outliers, all landmarks in the field of view are checked for validity, and the occupancy and category information are updated and integrated into the map. Voxels with free status or background category are removed from the landmark point cloud. The updated landmark parameters are then recalculated and the eliminated points are removed from the map.

Images