An obstacle detection method for industrial AGV vehicles through point cloud detection

Abstract

The present invention discloses a method for detecting obstacles in an industrial AGV (Automated Guided Vehicle) through point cloud detection. The method acquires point cloud data surrounding the industrial AGV; uses a single-stage obstacle avoidance detector based on two point cloud scales to detect the locations of obstacles around the vehicle using this point cloud data; introduces a learnable foreground point downsampling module to more effectively preserve foreground points; utilizes a multi-abstract scale feature extraction module to extract complex target feature information from BEV feature maps, resulting in a feature map with enhanced representation of target information; and utilizes a multi-distance scale feature aggregation module to effectively acquire foreground point structural information, thereby obtaining an accurate bounding box.

Description

Technical Field

[0001] The present invention belongs to the technical field of scene text detection and relates to an obstacle detection method for industrial AGV vehicles through point cloud detection, which is used for automatic path finding and obstacle avoidance of AGV vehicles. Point cloud data is acquired through a laser radar, and a multi-scale feature extraction module voxelizes the point cloud to generate a BEV. 2D convolution is used for feature extraction, and finally a heat map is output to predict the positioning information of the target. Background Art

[0002] Automated Guided Vehicles (AGVs) are widely used in modern factories due to their improved handling efficiency. AGVs possess functions such as obstacle avoidance and pathfinding. In the area of obstacle avoidance, conventional 2D object detection based on monocular images lacks depth information, making it difficult to accurately locate obstacles in space. Current 3D object detection methods are categorized by data acquisition method, primarily into binocular vision, millimeter-wave radar point clouds, and lidar point clouds. Binocular vision is a mature technology with low-cost sensors. However, binocular vision suffers from inherent image issues such as optical distortion, image reflections, inconsistent content between left and right views, and depth errors. The point clouds generated by millimeter-wave radar are sparse and contain little information, making them difficult to extract features and perform deep learning. Therefore, recent research is integrating them with vision or lidar for detection. LiDAR offers advantages in terms of high accuracy and the richness of the point cloud it produces. However, its penetration is poor, it is highly susceptible to environmental influences, and its sensors are often expensive. The fusion of lidar and vision is also a hot topic of research. However, different data multimodal fusion methods must ensure the synchronization of multiple sensors during data collection. At the same time, point clouds have characteristics such as sparseness, irregularity, and transformation invariance, while images are compact and regular. The two types of data are qualitatively different, making fusion difficult.

[0003] 3D point cloud object detection is currently divided into two main categories: single-stage detectors and dual-stage detectors. Single-stage detectors directly generate bounding boxes after feature extraction, so single-stage detectors have a simple structure and high efficiency. After generating a proposal through the RPN (Region Proposal Network), the dual-stage detector has another stage to refine the proposal and correct the confidence, giving it higher detection accuracy. Since single-stage detectors cannot extract feature information around the proposal in the second stage like dual-stage detectors, current single-stage networks mostly rely on linear networks to directly regress the bounding box confidence. With the development of computer vision and pattern recognition, the performance of single-stage detectors has been significantly improved, and their accuracy is close to that of dual-stage detectors. Summary of the Invention

[0004] The first object of the present invention is to address the deficiencies in the prior art and provide an industrial AGV obstacle detection method through point cloud detection.

[0005] The present invention provides an industrial AGV obstacle detection method through point cloud detection, comprising the following steps:

[0006] Step (1) using radar to obtain point cloud data around the industrial AGV in operation;

[0007] Step (2) uses the industrial AGV obstacle avoidance single-stage detector TPC-SSD (Two Point Cloud Scales Single-stage Detector) based on two point cloud scales to detect obstacles around the vehicle on the above point cloud data;

[0008] The single-stage obstacle avoidance detector for industrial AGVs based on two point cloud scales includes a learnable foreground point downsampling module, a multi-abstraction scale feature extraction module, a multi-distance scale feature aggregation module, and a detection head. The detector first obtains foreground points from the original point cloud through the learnable foreground point downsampling module, then generates a feature map through the multi-abstraction scale feature extraction module, uses the feature map to predict a heat map, then converts the heat map into center points, aggregates the foreground points in the multi-distance scale feature aggregation module to generate a semantic feature vector, and finally combines the semantic feature vector and the heat map to predict a bounding box.

[0009] 1) The learnable foreground point downsampling module is used to extract semantic features of the input point cloud through a multi-layer perceptron (MLPS) operation, thereby screening out foreground points with rich target information, which includes multiple serial downsampling operations;

[0010] In each downsampling, each point in the point cloud uses MLPS to obtain the foreground score of the current point, and then selects the top K points as the input of the next downsampling based on the foreground score.

[0011] Each downsampling is optimized using the focal loss function:

[0012]

[0013]

[0014] Where: L seg represents the focal loss, N represents the number of foreground points, and p i Indicates the one-hot label that distinguishes foreground and background points, represents the average foreground point probability, represents the probability of foreground points predicted by the sigmoid function, and λ and ω represent hyperparameters;

[0015] 2) Multi-scale abstract feature extraction module:

[0016] ① Voxelize the foreground points extracted by the learnable foreground point downsampling module to obtain a voxel space;

[0017] ② Each voxel with a point is feature-encoded through the voxel feature encoding layer to obtain non-empty voxel features;

[0018] ③ Connect the voxelized points in step ① along the z-axis to obtain a denser bird's eye view (BEV).

[0019] ④ Perform multi-scale feature extraction on the bird's-eye view BEV to obtain the BEV feature map; specifically:

[0020] The bird's-eye view (BEV) passes through three convolutional layers in the multi-scale feature separation network. Each convolutional layer produces a feature map of a different abstract level. The middle-level features obtained by the second convolution and the deep-level features obtained by the third convolution are then deconvolved to keep their dimensions consistent with the shallow-level features. The shallow-level features obtained by the first convolution are then element-wise added to the middle-level features and the deep-level features. Finally, the attention mechanism is used to fuse the two added feature maps.

[0021] The feature fusion is to compress the number of channels of the two added feature maps to 1 using a convolution layer, and then splice them to obtain an attention feature map; normalize the two channels of the attention feature map through the softmax function to obtain two connected channels; then use the two connected channels as weights to weight the corresponding original input added feature maps, that is, perform a multiplication operation; finally, add the two weighted feature maps element by element to achieve the purpose of adaptive fusion.

[0022] ⑤ Predict the thermal map based on the BEV characteristic map; specifically:

[0023] For any BEV feature map of size W*H, a heatmap of size W / R*H / R*K is generated, where W is the image height, H is the image width, K is the number of detected categories, and R is the radius of the Gaussian circle. The values of the elements in the heatmap are either 0 or 1. If the heatmap value is 1, the point in the feature map is a foreground point, and if it is 0, the point in the feature map is background.

[0024] The heat map retains only one channel, and at the same time retains a single-channel ground height map for three-dimensional positioning of the center point, and a dual-channel subvoxel positioning refinement map for reducing the positioning error of the center point.

[0025] 3) Multi-distance scale feature aggregation module:

[0026] ① Obtain all center points from the heat map output by the multi-scale feature extraction module, and then mark all voxels containing the center point in the voxel space as the center voxel c n , record to the set H={c1,c2,…,c n}.

[0027] ② Since only the point cloud of foreground points is voxelized, each non-empty voxel contains only the feature information of the foreground point. The coordinates of the non-empty voxels are expressed as P = {p1, p2, ..., p l}, its characteristics are expressed as Considering the regularity of voxel arrangement, voxel c i ,c j The distance between them is expressed as Manhattan distance:

[0028] That is, D(c i ,c j )=|a x -b x |+|a y -b y |+|a z -b z Formula (3)

[0029] where c i =(a x , a y , a z ), c j =(b x , b y , b z ), a x , a y , a z Indicates the coordinates of the voxel on the xyz axis.

[0030] Query the center voxel c i ∈H in the neighborhood of all non-empty voxels within r Manhattan distances, and list them in the non-empty voxel feature set:

[0031]

[0032] in Represents the inverse distance between the center of the non-empty voxel and the center of the central voxel, which is the same as p j -c i Together they constitute the local spatial structure of the voxel, p j -c i Denotes the coordinates of non-empty voxels relative to the central voxel. D(·) denotes the Manhattan distance.

[0033] ③ Since the feature vector of the foreground point only contains the global coordinate value, the inverse distance between the foreground point and the center point is added to the feature vector to expand the local information to ensure that the foreground point contains as many spatial features as possible. The non-empty voxels and the central voxel c in i The Manhattan distance is classified into m groups, which are expressed as: The Manhattan distance of all non-empty voxels in the k-th group category to the central voxel is:

[0034]

[0035] in Represents the coordinates of a non-empty voxel in the k-th group.

[0036] right Aggregate to obtain a set of single-scale feature vectors:

[0037]

[0038] Where R(·) represents the random downsampling function. PointNet k (·) represents the network loaded with the kth set of weights.

[0039] ④ For the central voxel c i The m groups of single-scale feature vectors in the neighborhood are sequentially concatenated to obtain multi-scale semantic features: The set of multi-scale semantic features of all central voxels is expressed as It preserves the spatial structure at different distance scales, thus providing accurate spatial structure information for each target.

[0040] 4) The detection head receives the multi-scale semantic features output by the multi-distance scale feature aggregation module and predicts the bounding box size, orientation angle and confidence of each target.

[0041] A second object of the present invention is to provide an industrial AGV obstacle avoidance detection system, comprising:

[0042] Solid-state area array radar module to obtain point cloud information around the industrial AGV;

[0043] The obstacle detection module uses the trained industrial AGV obstacle avoidance single-stage detector TPC-SSD based on two point cloud scales to predict the obstacle locations in the point cloud information around the vehicle.

[0044] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0045] A fourth object of the present invention is to provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0046] The beneficial effects of the present invention are as follows:

[0047] The present invention introduces a learnable foreground point downsampling module in the detector, which can more effectively retain the foreground points; the present invention adopts a multi-abstract scale feature extraction module to extract target feature information with complex structure from the BEV feature map, and the obtained feature map has stronger expressiveness for the target information; the multi-distance scale feature aggregation module used in the present invention can effectively obtain the foreground point structure information, thereby obtaining an accurate bounding box.

[0048] In actual factory environments, the new single-stage detector TPC-SSD proposed in this paper outperforms other detectors in terms of real-time and accuracy in detecting surrounding obstacles, solving the obstacle avoidance detection problem of AGVs in actual industrial scenarios and achieving considerable accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a diagram of the industrial AGV obstacle avoidance network model based on two point cloud scales;

[0050] Figure 2 It is the downsampling process diagram;

[0051] Figure 3 Figure 2 is a multi-scale abstract feature extraction method;

[0052] Figure 4 Figure 2 is a diagram of the multi-distance scale sampling and encoding method. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and examples.

[0054] A method for detecting obstacles in an industrial AGV using point cloud detection includes the following steps:

[0055] Step (1), obtaining point cloud data around the industrial AGV vehicle in the running state;

[0056] Step (2), using the industrial AGV vehicle obstacle avoidance single-stage detector based on two point cloud scales to detect obstacles around the vehicle on the above point cloud data;

[0057] The industrial AGV obstacle avoidance single-stage detector based on two point cloud scales is shown in the attached Figure 1 It includes a learnable foreground point downsampling module, a multi-abstraction scale feature extraction module, and a multi-distance scale feature aggregation module;

[0058] 1) Learnable foreground point downsampling module:

[0059] The learnable foreground point downsampling module has an accurate learnable downsampling method. The input point cloud can be used to extract its semantic features through MLPS operation, and then filter out foreground points with rich target information. Figure 2 As shown in the figure, during each downsampling, MLPS is used to obtain a foreground score for each point. Based on the foreground scores, the top K points are selected as input for the next downsampling. This foreground point downsampling method can learn the latent semantic features in the point cloud, reduce computational effort, and ultimately obtain as many foreground points as possible. Downsampling foreground points is a point cloud segmentation task and can be optimized using the focal loss function:

[0060]

[0061] in:

[0062]

[0063] N is the number of foreground points, is the probability of the foreground point predicted by the sigmoid function, p i is the one-hot label that distinguishes foreground and background points, and λ and ω are hyperparameters;

[0064] 2) Multi-scale abstract feature extraction module:

[0065] To generate accurate and interference-free feature maps, this module retains only the foreground points obtained by the learnable foreground point downsampling module and filters out the remaining background points. This prevents irrelevant pixels from interfering with detection in the subsequently generated feature maps. The foreground points extracted by the learnable foreground point downsampling module are voxelized to obtain a voxel space.

[0066] Each voxel with a point is feature-encoded through the voxel feature encoding layer to obtain non-empty voxel features; in a factory environment, since the collected point cloud is relatively dense and the collection range of the solid-state array laser radar is small, the target is large in the point cloud space. After screening out enough background points, the foreground points still occupy a lot of positions in the point cloud space.

[0067] By connecting the voxelized point cloud along the z-axis, a denser BEV can be obtained. At the same time, each foreground point has obtained a receptive field in the MLPS, and there is no need to perform feature extraction operations on the voxel space through sparse convolution, thus avoiding the time consumption caused by 3D convolution.

[0068] It is crucial to effectively extract features of BEV in order to reduce the information loss of BEV in the z-axis. Figure 3As shown in the figure, BEV first passes through three convolutional layers in a multi-scale feature separation network, and obtains a feature map of an abstract level after each convolution layer. Then, the middle-level features obtained by the second convolution and the deep-level features obtained by the third convolution are deconvolutionized to keep their dimensions consistent with the shallow-level features. Then, the shallow-level features are element-wise added to the middle-level features and the deep-level features respectively. Finally, the two added features are fused using the attention mechanism. The feature fusion method is to first compress the number of channels of the two feature maps to 1 using the convolution layer, and then splice the two feature maps to obtain the attention feature map. The two channels of the attention feature map are normalized by the softmax function. Then, the two connected channels are used as weights to weight the corresponding feature maps, that is, multiplication operation is performed. Finally, the two weighted feature maps are added element-by-element to achieve the purpose of adaptive fusion.

[0069] CenterNet proposed the concept of heat map, whose pixel values show Gaussian distribution, where each Gaussian peak pixel point greater than the threshold μ is the center point of the target. The heat map is inferred from the BEV feature map, and the target center point is obtained to realize the positioning of the target. Where L is the length and W is the width. Compared with CenterPoint, since this module does not consider the classification problem, the present invention only retains one channel in the heat map. At the same time, it also retains a single-channel ground height map for the three-dimensional positioning of the center point, and a dual-channel subvoxel positioning refinement map for reducing the positioning error of the center point.

[0070] 3) Multi-distance scale feature aggregation module:

[0071] First, all the center points are obtained from the heat map, and then all the voxels containing the center points in the voxel space are marked and called center voxels. The coordinates are expressed as H = {c1, c2, ..., c n}.

[0072] Since only the point cloud of foreground points is voxelized, each non-empty voxel contains only the feature information of the foreground point. The coordinates of the non-empty voxels are expressed as P = {p1, p2, ..., p l}, its characteristics are expressed as Considering the regularity of voxel arrangement, the distance between voxels a and b is expressed by Manhattan distance, that is, D(a,b)=|a x -b x |+|a y -b y |+|a z -b z |, where (x,y,z) represents the index of the voxel on the xyz coordinate axis respectively.

[0073] like Figure 4 As shown, query the central voxel c i ∈H in the neighborhood of all non-empty voxels within r Manhattan distances, and list them in the non-empty voxel feature set:

[0074]

[0075] Represents the inverse distance between the center of the non-empty voxel and the center of the central voxel, which is the same as p j -c i Together they constitute the local spatial structure of the voxel, and D(·) represents the Manhattan distance. Since the feature vector of the foreground point only contains the global coordinate value, the inverse distance between the foreground point and the center point is added to the feature vector to expand the local information to ensure that the foreground point contains as many spatial features as possible. The non-empty voxels and the central voxel c in i The Manhattan distance is classified into m groups, which are expressed as: The Manhattan distance of all non-empty voxels in the k-th group category to the central voxel is:

[0076]

[0077] in Represents the coordinates of a non-empty voxel in the kth group. Aggregate to obtain a set of single-scale feature vectors:

[0078]

[0079] Where R(·) represents a random downsampling function, which randomly selects at most V voxels in the set. This paper sets m groups of weights to load PointNet respectively. k (·) indicates that the network with the kth group of weights is loaded. Finally, the central voxel c i The m groups of feature vectors in the neighborhood are sequentially concatenated to obtain multi-scale semantic features: The set of multi-scale semantic features of all central voxels is expressed as It preserves the spatial structure at different distance scales, thus providing accurate spatial structure information for each target. In the detection head, a multi-scale semantic feature set predicts the bounding box size, orientation angle, and confidence of each target.

Claims

1. A method for industrial AGV obstacle detection through point cloud detection, characterized in that The following steps are involved: Step (1) using radar to obtain point cloud data around the industrial AGV in operation; Step (2) using the industrial AGV obstacle avoidance single-stage detector TPC-SSD based on two point cloud scales to detect obstacles around the vehicle on the above point cloud data; The industrial AGV obstacle avoidance single-stage detector TPC-SSD based on two point cloud scales includes a learnable foreground point downsampling module, a multi-abstraction scale feature extraction module, a multi-distance scale feature aggregation module, and a detection head; The learnable foreground point downsampling module is used to extract semantic features of the input point cloud through a multi-layer perceptron (MLPS) operation, thereby screening out foreground points with rich target information. The multi-scale feature extraction module is specifically: Voxelize the foreground points extracted by the learnable foreground point downsampling module to obtain a voxel space; Each voxel with a point is feature-encoded through the voxel feature encoding layer to obtain non-empty voxel features; Connect the voxelized points along the z-axis to obtain a bird's-eye view BEV; Perform multi-scale feature extraction on the bird's-eye view BEV to obtain the BEV feature map; Predict the heat map based on the BEV feature map; The multi-distance scale feature aggregation module is specifically: Get all the center points from the heat map output by the multi-scale feature extraction module, and then mark all the voxels containing the center points in the voxel space as the center voxel c n , record to the set H={c1,c2,…,c n }; Since only the point cloud of foreground points is voxelized, each non-empty voxel contains only the feature information of the foreground point. The coordinates of the non-empty voxels are expressed as P = {p1, p2, ..., p l }, its characteristics are expressed as Considering the regularity of voxel arrangement, voxel c i ,c j The distance between them is expressed as Manhattan distance: That is, D(c i ,c j )=|a x -b x |+|a y -b y |+|a z -b z Formula (1) where c i =(a x , a y , a z ), c j =(b x , b y , b z ), a x , a y , a z Represents the coordinates of the voxel on the xyz axis; Query the center voxel c i ∈H in the neighborhood of all non-empty voxels within r Manhattan distances, and list them in the non-empty voxel feature set: in Represents the inverse distance between the center of the non-empty voxel and the center of the central voxel, which is the same as p j -c i Together they constitute the local spatial structure of the voxel, p j -c i represents the coordinates of the non-empty voxel relative to the central voxel. D(·) represents the Manhattan distance; Since the feature vector of the foreground point only contains the global coordinate value, the inverse distance between the foreground point and the center point is added to the feature vector to expand the local information to ensure that the foreground point contains as many spatial features as possible; The non-empty voxels and the central voxel c in i The Manhattan distance is classified into m groups, which are expressed as: The Manhattan distance of all non-empty voxels in the k-th group category to the central voxel is: in represents the coordinates of a non-empty voxel in the kth group; right Aggregate to obtain a set of single-scale feature vectors: Where R(·) represents the random downsampling function; PointNet k (·) indicates the network loaded with the kth set of weights; For the central voxel c i The m groups of single-scale feature vectors in the neighborhood are sequentially concatenated to obtain multi-scale semantic features: The set of multi-scale semantic features of all central voxels is expressed as It preserves the spatial structure at different distance scales, thus providing accurate spatial structure information for each target; The detection head receives the multi-scale semantic features output by the multi-distance scale feature aggregation module and predicts the bounding box size, orientation angle and confidence of each target; The multi-scale feature extraction module performs multi-scale feature extraction on the bird's-eye view image BEV as follows: The bird's-eye view BEV passes through three convolution layers in the multi-abstract scale feature separation network, and obtains a feature map of an abstract level after each convolution layer. Then, the middle-level features obtained by the second convolution and the deep-level features obtained by the third convolution are deconvolutionized to make their dimensions consistent with the shallow-level features. Then, the shallow-level features obtained by the first convolution are added element-by-element with the middle-level features and the deep-level features respectively. Finally, the attention mechanism is used to fuse the two added feature maps.

2. The method according to claim 1, characterized in that The learnable foreground point downsampling module in step (2) includes multiple serial downsamplings; in each downsampling, each point in the point cloud uses MLPS to obtain the foreground score of the current point, and then selects the top K points as the input for the next downsampling based on the foreground score.

3. The method according to claim 2, characterized in that Each downsampling in the learnable foreground point downsampling module in step (2) is optimized using the focal loss function: Where: L seg represents the focal loss, N represents the number of foreground points, and p i Indicates the one-hot label that distinguishes foreground and background points, represents the average foreground point probability, represents the probability of the foreground point predicted by the sigmoid function, and λ and ω represent hyperparameters.

4. The method according to claim 1, characterized in that The feature fusion in the multi-scale feature extraction is to compress the number of channels of the two added feature maps to 1 using a convolution layer, and then splice them to obtain an attention feature map; normalize the two channels of the attention feature map through the softmax function to obtain two connected channels; then use the two connected channels as weights to weight the corresponding original input added feature maps, that is, perform a multiplication operation; finally, add the two weighted feature maps element by element to achieve the purpose of adaptive fusion.

5. The method according to claim 1, characterized in that The multi-scale feature extraction is specifically to predict the heat map based on the BEV feature map: For any BEV feature map of size W*H, a heat map of size W / R*H / R*K is generated, where W is the height of the image, H is the width of the image, K is the number of detected categories, and R is the radius of the Gaussian circle; the value of the elements in the heat map is 0 or 1, where if the point in the heat map is 1, then the point in the feature map is a foreground point, and if it is 0, then the point in the feature map is background.

6. An industrial AGV obstacle avoidance detection system implementing the method according to any one of claims 1 to 5, characterized in that include: Solid-state area array radar module to obtain point cloud information around the industrial AGV; The obstacle detection module uses the trained industrial AGV obstacle avoidance single-stage detector TPC-SSD based on two point cloud scales to predict the obstacle locations in the point cloud information around the vehicle.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

8. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 5 is implemented.

Images