Stacker collision avoidance method and device based on three-dimensional perception

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a stacker collision avoidance method and device based on three-dimensional perception, wherein the method comprises: obtaining the storage environment point cloud data of the current frame; using the kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and using the distribution probability as a supplement to the point cloud description; inputting the expanded point cloud data into the feature extraction network to extract the Bev feature map of the current frame; inputting the Bev feature map of the current frame into the detection head to obtain the category and positioning of the prediction frame; after the prediction frame of the current frame is subjected to non-maximum suppression, a fast clustering algorithm along the z-axis direction is used for it, and then the prediction frame is filtered according to the number of points in the prediction frame and the confidence level; according to the prediction result after filtering the current frame, a feasible path is planned using an obstacle avoidance algorithm for a restricted receptive field. Thus, the problem that the stacker cannot adapt to complex environments, resulting in reduced operating efficiency, is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a stacker collision avoidance method and device based on three-dimensional perception. Background Art

[0002] As the scale of manufacturing industry expands, the logistics industry is faced with faster and smarter challenges. As an automated device for warehouse cargo entry and exit, the intelligence level of stackers determines the warehouse throughput. Improving the intelligence level of stackers can speed up warehouse operation efficiency, thereby improving the competitiveness of enterprises.

[0003] In order to solve the common problems in warehouses such as chaotic placement of items and protruding cargo boxes, which affect the operation of stackers, reduce logistics efficiency, and even cause collisions, causing irreparable damage to both the stackers and goods, in related technologies, the stacker uses sensor ranging and other technologies to detect obstacles in front, brake to stop and alarm, and wait for manual resolution of the obstacles before continuing to work.

[0004] However, since sensors are greatly affected by environmental factors, traditional methods are prone to false alarms and missed alarms and cannot adapt to complex environments, which reduces the operating efficiency of the stacker. Summary of the invention

[0005] The present application provides a stacker collision avoidance method, device, electronic device, storage medium and program product based on three-dimensional perception to solve the problem that the stacker cannot adapt to complex environments, resulting in reduced operating efficiency.

[0006] The first aspect of the present application provides a stacker collision avoidance method based on three-dimensional perception, including the following steps: obtaining storage environment point cloud data of the current frame; using the kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and using the distribution probability as a supplement to the point cloud description; inputting the expanded point cloud data into the feature extraction network to extract the Bev feature map of the current frame; inputting the Bev feature map of the current frame into the detection head to obtain the category and positioning of the prediction box; after non-maximum suppression of the prediction box of the current frame, using a fast clustering algorithm along the z-axis direction for the prediction box, and then filtering the prediction box according to the number of points in the prediction box and the confidence level; according to the prediction result after filtering the current frame, using an obstacle avoidance algorithm for a restricted receptive field to plan a feasible path.

[0007] Optionally, in an embodiment of the present application, before using a kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and using the distribution probability as a supplement to the point cloud description, it also includes: obtaining historical warehouse environment point cloud data, and marking pedestrians and protruding cargo boxes in the historical warehouse environment point cloud data; dividing the historical warehouse environment point cloud data into a training set and a validation set, and performing data enhancement on the point cloud data in the training set; using a kernel density estimation method to estimate the distribution probability of the point cloud in each pillar for the data-enhanced point cloud data as a supplement to the point cloud description; iteratively training the feature extraction network and the detection head based on the training data set, and during the iterative training process, calculating the training loss value based on the output value of the detection head and the true label, and optimizing the weights of the feature extraction network and the detection head by back propagation of the loss value; stopping the iterative training until the iterative training of the feature extraction network and the detection head reaches the target number of iterations; and using the validation set to perform performance verification on the trained feature extraction network and the detection head.

[0008] Optionally, in an embodiment of the present application, a kernel density estimation method is used to estimate the distribution probability of each point in the point cloud located in the same pillar as a supplement to the point cloud description, including: dividing the point cloud space according to square columns on the target plane; selecting a Gaussian kernel function to estimate the point cloud density distribution function in each square column; estimating the point cloud distribution probability of each point in the square column based on the point cloud density distribution function, and using it as a supplement to the point cloud description.

[0009] Optionally, in an embodiment of the present application, the expanded point cloud data is input into the feature extraction network to extract the Bev feature map of the current frame, including: using a multi-layer perceptron and maximum pooling to extract features from the point cloud in each square column to obtain a preliminary Bev feature map; based on the preliminary Bev feature map, a two-dimensional convolutional neural network is used to further extract a multi-scale feature map, wherein the two-dimensional convolutional neural network is a ResNet network or a VoVNet network; based on the multi-scale feature map, an FPN or BiFPN network is used to fuse the multi-scale feature map to obtain the final Bev feature map

[0010] Optionally, in an embodiment of the present application, the Bev feature of the current frame is input into the detection head to obtain the category and positioning of the prediction box, including: the category of the prediction box is divided into two categories: pedestrians and protruding cargo boxes in the shelf; each feature in the Bev feature map of the current frame corresponds to multiple pre-defined aiming point boxes; based on the aiming point box, the detection head uses the feature corresponding to the aiming point box to predict the category score and coordinates, size and heading angle offset.

[0011] Optionally, in the embodiment of the present application, the fast clustering algorithm along the z-axis direction includes: for each category of the prediction box (xb ,y b ,z b ,l b ,w b ,h b ,θ b ), take out the current frame located at (x b ,y b ,l b ,w b ,θ b ) are the points in the bottom pillar, forming the point set P ′ ; Statistical point set P ′ The histogram of the z coordinate in the histogram; the central value and width of the continuous non-zero part in the histogram are used as the z coordinate and height of the new prediction box to obtain the new prediction box.

[0012] Optionally, in the embodiment of the present application, the prediction box is filtered according to the number of points in the prediction box and the confidence level, including: counting the number of point clouds N in the prediction box; dividing the prediction box into three categories: near, medium and far according to the x-axis coordinate of the center point of the prediction box; and pre-setting a threshold N for each category. min 、N posmin and min If N <N min , then filter out; if N>N posmin , then keep it; otherwise, further judge according to the confidence level s. If s min , it is still filtered out.

[0013] Optionally, in an embodiment of the present application, the prediction result after filtering the current frame is used to plan a feasible path using an obstacle avoidance algorithm for a restricted receptive field, including: modeling the stacker motion scene as a two-dimensional plane grid map; marking the obstacle position on the grid map and setting the obstacle position and the surrounding grids as unreachable based on the prediction result after filtering the current frame; and finding a feasible path for the stacker based on the two-dimensional plane grid map information using an A* search algorithm or a Dijkstra search algorithm.

[0014] ​The second aspect of the present application provides a stacker collision avoidance device based on three-dimensional perception, including: an acquisition module, used to obtain the storage environment point cloud data of the current frame; an estimation module, used to use the kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and use the distribution probability as a supplement to the point cloud description; an extraction module, used to input the expanded point cloud data into the feature extraction network, and extract the Bev feature map of the current frame; a prediction module, used to input the Bev feature map of the current frame into the detection head, and obtain the category and positioning of the prediction box; a filtering module, used to non-maximum suppress the prediction box of the current frame, and then use a fast clustering algorithm along the z-axis direction to the prediction box, and then filter the prediction box according to the number of points in the prediction box and the confidence level; an obstacle avoidance module, used to plan a feasible path based on the prediction result after filtering the current frame using an obstacle avoidance algorithm for a restricted receptive field.

[0015] Optionally, in an embodiment of the present application, it also includes: a training module, which is used to estimate the distribution probability of each point in the point cloud located in the same pillar using a kernel density estimation method, and use the distribution probability as a supplement to the point cloud description, including: obtaining historical warehouse environment point cloud data, and marking pedestrians and protruding cargo boxes in the shelves in the historical warehouse environment point cloud data; dividing the historical warehouse environment point cloud data into a training set and a validation set, and performing data enhancement on the point cloud data in the training set; using a kernel density estimation method to estimate the distribution probability of the point cloud in each pillar for the data-enhanced point cloud data as a supplement to the point cloud description; iteratively training the feature extraction network and the detection head based on the training data set, and during the iterative training process, calculating the training loss value based on the output value of the detection head and the true label, and optimizing the weights of the feature extraction network and the detection head by back propagation of the loss value; stopping the iterative training until the iterative training of the feature extraction network and the detection head reaches the target number of iterations; and using the validation set to perform performance verification on the trained feature extraction network and the detection head.

[0016] Optionally, in an embodiment of the present application, the estimation module is further used to divide the point cloud space according to square columns on the target plane; select a Gaussian kernel function to estimate the point cloud density distribution function in each square column; estimate the point cloud distribution probability of each point in the square column based on the point cloud density distribution function, and use it as a supplement to the point cloud description.

[0017] Optionally, in an embodiment of the present application, the extraction module is further used to use a multi-layer perceptron and maximum pooling to extract features from the point cloud in each square column to obtain a preliminary Bev feature map; based on the preliminary Bev feature map, a two-dimensional convolutional neural network is used to further extract a multi-scale feature map, wherein the two-dimensional convolutional neural network is a ResNet network or a VoVNet network; based on the multi-scale feature map, an FPN or BiFPN network is used to fuse the multi-scale feature map to obtain the final Bev feature map.

[0018] Optionally, in an embodiment of the present application, the prediction module is further used to divide the categories of the prediction boxes into two categories: pedestrians and protruding cargo boxes in the shelves; each feature in the Bev feature map of the current frame corresponds to multiple pre-defined aiming point boxes; based on the aiming point boxes, the detection head uses the features corresponding to the aiming point boxes to predict category scores and coordinates, sizes, and heading angle biases.

[0019] Optionally, in the embodiment of the present application, the filtering module is further used to predict the box (x b ,y b ,z b ,l b ,w b ,h b ,θ b ), take out the current frame located at (x b ,y b ,l b ,w b ,θ b ) are the points in the bottom pillar, forming the point set P ′ ; Statistical point set P ′ The histogram of the z coordinate in the histogram; the central value and width of the continuous non-zero part in the histogram are used as the z coordinate and height of the new prediction box to obtain the new prediction box.

[0020] Optionally, in the embodiment of the present application, the filtering module is further used to count the number of point clouds N in the prediction box; the prediction box is divided into three categories: near, medium and far according to the x-axis coordinate of the center point of the prediction box; for each category, a threshold N is pre-set min 、N posmin and min , if N <N min , then filter out; if N>N posmin , then keep it; otherwise, further judge according to the confidence level s. If s min , it is still filtered out.

[0021] ​Optionally, in an embodiment of the present application, the obstacle avoidance module is further used to model the stacker motion scene as a two-dimensional plane grid map; based on the prediction result after filtering the current frame, mark the obstacle position on the grid map and set the obstacle position and the surrounding grids as unreachable; based on the two-dimensional plane grid map information, use the A* search algorithm or the Dijkstra search algorithm to find a feasible path for the stacker.

[0022] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a stacker collision avoidance method based on three-dimensional perception as in the above-mentioned embodiment.

[0023] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a stacker collision avoidance method based on three-dimensional perception as described in the above embodiment.

[0024] The fifth aspect of the present application provides a computer program product, including a computer program or instructions. When the computer program is executed, it is used to implement the stacker collision avoidance method based on three-dimensional perception as in the above embodiment.

[0025] Therefore, this application includes the following beneficial effects:

[0026] The embodiment of the present application obtains the storage environment point cloud data of the current frame; uses the kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and uses the distribution probability as a supplement to the point cloud description; inputs the expanded point cloud data into the feature extraction network to extract the Bev feature map of the current frame; inputs the Bev feature map of the current frame into the detection head to obtain the category and positioning of the prediction frame; after the prediction frame of the current frame is subjected to non-maximum suppression, it is subjected to a fast clustering algorithm along the z-axis direction, and then the prediction frame is filtered according to the number of points in the prediction frame and the confidence level; according to the prediction result after filtering the current frame, an obstacle avoidance algorithm for a restricted receptive field is used to plan a feasible path, so as to achieve autonomous obstacle avoidance of the stacker, effectively save labor, and improve the efficiency of the stacker operation. Thus, the technical problems that the stacker cannot adapt to complex environments, resulting in reduced operating efficiency, etc. are solved.

[0027] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A flowchart of a stacker collision avoidance method based on three-dimensional perception provided according to an embodiment of the present application;

[0030] Figure 2 It is a diagram showing the overall network structure of target detection of a stacker collision avoidance method based on three-dimensional perception according to a specific embodiment of the present invention;

[0031] Figure 3 It is a partial structural diagram of a target detection feature extraction network of a stacker collision avoidance method based on three-dimensional perception according to a specific embodiment of the present invention;

[0032] Figure 4 It is a flow chart of a stacker collision avoidance method based on three-dimensional perception according to a specific embodiment of the present invention;

[0033] Figure 5 The figure is a schematic structural diagram of a stacker collision avoidance device based on three-dimensional perception according to an embodiment of the present invention.

[0034] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0035] Explanation of the reference numerals: acquisition module-100, estimation module-200, extraction module-300, prediction module-400, filtering module-500, obstacle avoidance module-600, memory-601, processor-602 and communication interface-603. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the stacker collision avoidance method and device based on three-dimensional perception of the embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides a stacker collision avoidance method based on three-dimensional perception, in which the storage environment point cloud data of the current frame is obtained; the distribution probability of each point is estimated by the kernel density estimation method for the point cloud located in the same pillar, and the distribution probability is used as a supplement to the point cloud description; the expanded point cloud data is input into the feature extraction network to extract the Bev feature map of the current frame; the Bev feature map of the current frame is input into the detection head to obtain the category and positioning of the prediction frame; the prediction frame of the current frame is subjected to non-maximum suppression, and then a fast clustering algorithm along the z-axis direction is used for it, and then the prediction frame is filtered according to the number of points in the prediction frame and the confidence; according to the prediction result after filtering the current frame, an obstacle avoidance algorithm for a restricted receptive field is used to plan a feasible path, so as to realize the autonomous obstacle avoidance of the stacker, effectively save labor and improve the efficiency of the stacker operation. Thus, the technical problems such as the stacker cannot adapt to complex environments and the reduced operating efficiency are solved.

[0038] Specifically, Figure 1 A schematic flow chart of a stacker collision avoidance method based on three-dimensional perception provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the stacker collision avoidance method based on three-dimensional perception includes the following steps:

[0040] In step S101, the storage environment point cloud data of the current frame is obtained.

[0041] It can be understood that the embodiment of the present application can obtain the point cloud data recorded by the current laser radar, wherein the embodiment of the present application can use at least one method to obtain the point cloud data, and there is no specific limitation on this.

[0042] In step S102, the distribution probability of each point is estimated using a kernel density estimation method for the point cloud located in the same pillar, and the distribution probability is used as a supplement to the point cloud description.

[0043] It can be understood that the embodiment of the present application uses a kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and uses the distribution probability as a supplement to the point cloud description.

[0044] Kernel density estimation means generating a kernel function for each point and then performing weighted averaging of all kernel functions to obtain a probability estimate of data distribution. Kernel density estimation starts from the sample data itself and uses kernel functions to fit density distribution. It is a non-parametric method. Commonly used kernel functions include Gaussian kernel function, Epanechnikov kernel function and rectangular kernel function.

[0045] In an embodiment of the present application, the distribution probability of each point cloud of point cloud data located on the same pillar in a storage environment is calculated, including: dividing the point cloud space according to square columns on the target plane; estimating the point cloud density distribution function in each square column; and estimating the distribution probability of each point cloud in the square column based on the point cloud density distribution function.

[0046] Specifically, a Gaussian kernel function is selected to fit the density distribution of the point cloud within the pillar.

[0047] Let the points inside the pillar be x i ,y i ,z i is the xyz coordinate, r i is the intensity, and N is the number of points taken out from each pillar. Therefore, the density distribution function obtained by fitting these N points is:

[0048]

[0049] Where σ is taken as 0.25, and v = (x, y, z) is a point in space.

[0050] For the distribution probability p of the jth point j have:

[0051]

[0052] The p j The four-dimensional feature (x j ,y j ,z j ,r j ) to form a five-dimensional feature vector, which is input into the subsequent part of the 3D object detection model.

[0053] In an embodiment of the present application, before using a kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and using the distribution probability as a supplement to the point cloud description, it also includes: acquiring historical warehouse environment point cloud data, and marking pedestrians and protruding cargo boxes in the shelves; dividing the historical warehouse environment point cloud data into a training set and a validation set, and performing data enhancement on the point cloud data in the training set; using a kernel density estimation method to estimate the distribution probability of the point cloud in each pillar for the data-enhanced point cloud data as a supplement to the point cloud description; iteratively training the feature extraction network and the detection head based on the training data set, during the iterative training process, calculating the training loss value based on the output value of the detection head and the true label, and optimizing the weights of the feature extraction network and the detection head by back propagation of the loss value; stopping the iterative training until the iterative training of the feature extraction network and the detection head reaches the target number of iterations; and performing performance verification on the trained feature extraction network and the detection head using the validation set.

[0054] The target number of iterations can be set according to actual conditions and is not specifically limited.

[0055] It can be understood that the embodiment of the present application can divide the training set and the validation set into batches of equal size, and calculate the loss of the output value and the label through forward propagation, and optimize the feature extraction network and the detection head weights through loss back propagation until the maximum number of training times is reached, and the final feature extraction network and detection head can be obtained; thereby, reliable data results can be obtained through iterative training to improve the accuracy of the solution.

[0056] Further, the specific contents in the embodiments of the present application are described:

[0057] (1) Dataset Division

[0058] After acquiring the historical warehouse environment point cloud data, the embodiment of the present application can mark the pedestrians and protruding cargo boxes on the shelves, and then divide the historical warehouse environment point cloud data into a training set and a verification set.

[0059] (2) Data enhancement

[0060] The embodiment of the present application performs data enhancement on the input point cloud data, and its specific operations are to randomly copy and paste the target detection objects in other point cloud frames into the current frame; randomly add noise point clouds to the current frame; rotate the current frame by a random angle; and scale the current frame by a random ratio.

[0061] (3) Model training

[0062] This embodiment of the application uses a feature extraction network to extract Bev feature f bev , using Bev feature f bev The type, coordinates, size and heading angle of the aiming point frame are obtained through the detection head.

[0063] (4) Calculation of loss value

[0064] The model training in the embodiment of the present application adopts the weighted sum of category loss, positioning loss and heading angle category loss as the loss function.

[0065]

[0066] Among them, L cls For category loss, focal loss is used, L reg For the positioning loss, Smooth L1 loss is used, L dir is the heading angle category loss, using cross entropy loss. In the embodiment of the present application, λ is set to 2, β is set to 0.2, N pos is the number of positive samples.

[0067] (5) Iterative training

[0068] The embodiment of the present application repeats the above steps until the maximum number of training times is reached. For example, the iterative training of the embodiment of the present application iterates 400 training rounds in total, and the initial learning rate is set to 0.001; thus, when the number of iterations reaches the target number of iterations, the iterative training is stopped to obtain the final feature extraction network and detection head.

[0069] In step S103, the expanded point cloud data is input into the feature extraction network to extract the Bev feature map of the current frame.

[0070] In one embodiment of the present invention, the specific internal structure of the feature extraction network is as follows: Figure 3 Specifically, the process of obtaining the Bev characteristic graph in the embodiment of the present invention is as follows:

[0071] In the xy plane, the point cloud space [X min ,X max ]、[Y min ,Y max ]、[Z min ,Z max ] is divided according to the square column. Assuming the length, width and height of the square column are L, W and H respectively, then the division results in U×V square columns, where H=Z max -Z min ,L=W,

[0072] For each non-empty square column, a fixed number of N points are extracted. If the number of point clouds in the square column is greater than N, N points are randomly selected from it. Otherwise, the random selection is repeated until N points are reached, thus obtaining an array of size (P, N, 4), where P represents the number of non-empty square columns, N represents the number of points selected from each non-empty square column, and 4 represents the four-dimensional feature of the point.

[0073] Use multi-layer perceptron and maximum pooling to extract features f∈R for each point cloud in the non-empty square column P×C After that, it is returned to the original image of size [U, V] to obtain the preliminary Bev feature map;

[0074] Use a two-dimensional convolutional neural network to further extract features from the preliminary Bev feature map to obtain a multi-scale feature map;

[0075] Use FPN or BiFPN network to fuse multi-scale feature maps to obtain the final Bev feature map.

[0076] In step S104, the Bev feature map of the current frame is input into the detection head to obtain the category and location of the predicted box.

[0077] The detection head consists of a classification part and a localization part. Both the classification part and the localization part are convolution layers with a convolution kernel of one. The classification part predicts the category of the aiming point box, and the localization part predicts the coordinates, size, and heading angle of the aiming point box.

[0078] Specifically, the process of obtaining the category and positioning of the prediction box in the embodiment of the present invention is as follows:

[0079] For any of the categories of pedestrians and protruding boxes, each feature in the Bev feature map corresponds to two pre-set aiming point boxes with different heading angles but the same size;

[0080] For each aiming point box, the corresponding Bev feature passes through the detection head to obtain the category score and coordinates, size and heading angle offset.

[0081] In step S105, the prediction box of the current frame is subjected to non-maximum suppression, and then a fast clustering algorithm along the z-axis direction is used for it, and then the prediction box is filtered according to the number of points in the prediction box and the confidence level.

[0082] After obtaining the detection result output by the detection head, that is, the classification and positioning result of the prediction box, it is necessary to further correct the detection result through post-processing operations. Specifically, the post-processing process of the embodiment of the present invention is as follows:

[0083] First, use the non-maximum suppression method to remove some redundant prediction boxes.

[0084] Then use the fast clustering method along the z-axis to solve the problem of boxes protruding in the same column. The specific steps are as follows: Set the prediction box set For each box prediction box in the set S (x b ,y b ,z b ,l b ,w b ,h b ,θ b ), first take out the point cloud P located at (x b ,y b ,l b ,w b ,θ b ) The boxes are the points in the pillar at the bottom, forming the point set P ′ ; Then count the point set P ′ Finally, the center value and width of the continuous non-zero part of the histogram are used as the z coordinate and height of the new prediction box to obtain a new prediction box set.

[0085]

[0086] Then, the prediction box is filtered according to the number of points and confidence in the prediction box. The specific steps are as follows: According to the x-axis coordinate of the center point of the prediction box, the set S ′ The prediction boxes in are divided into three categories: near, medium and far. For each category, a threshold N is pre-set. min 、N posmin and min ; For the set S ′ Each box prediction box (x b ,y b ,z b ,l b ,w b ,h b ,θ b ), count the number of points N in the current point cloud frame; if N <N min , then filter out; if N>N posmin , then retain; otherwise, according to the confidence s b Further judgment, that is, s b min , still filtered out.

[0087] In step S106, according to the prediction result after filtering the current frame, a feasible path is planned using an obstacle avoidance algorithm for a limited receptive field.

[0088] After obtaining the prediction result after filtering the current frame, it is necessary to search for a feasible path for the stacker using an obstacle avoidance algorithm for a limited receptive field. Specifically, the process of planning a feasible path using an obstacle avoidance algorithm for a limited receptive field in an embodiment of the present invention is as follows:

[0089] Model the stacker movement scene as a two-dimensional plane grid map;

[0090] According to the prediction result after filtering the current frame, marking the obstacle position on the grid map and setting it and the surrounding grids as unreachable;

[0091] According to the two-dimensional plane grid map information, a search algorithm such as A* or Dijkstra is used to find a feasible path for the stacker. In the embodiment of the present invention, the A* algorithm is used, and the specific steps are as follows: let s be the initial point of the stacker, g be the target working point of the stacker, o1,…,o M is the detected obstacle, P is an open set, C is a closed set; calculate the heuristic function value of the initial point s h(s) = g(s) + f(s), where g(s) is the Manhattan distance from point s to the target point, f(s) is the shortest distance from point s to the starting point, and then add point s to the open set P; take out the point t with the smallest heuristic function value in the open set P and add it to the closed set C; consider the set of points adjacent to point t For each point u∈N t ​, first determine whether point u can be reached; if it can be reached, calculate the heuristic function value of point u, and then check whether it decreases; if it decreases, update the heuristic function value of point u and add it to the open set P, and record point t as the father node of point u; iterate until the target working point g is taken out from the closed set C; according to the father node of the target working point g, backtrack and iterate to get the final path.

[0092] Therefore, the method of the embodiment of the present application can effectively solve the problem of autonomous obstacle avoidance of stackers in a warehousing environment. Pedestrians in the warehouse environment and protruding cargo boxes in the shelves are identified and located through a 3D target detection model, and accurate detection results are obtained using post-processing methods such as non-maximum suppression and rapid clustering along the z-axis. Based on the detection results, an obstacle avoidance algorithm for a restricted receptive field is used to search for a feasible path for the stacker. Real-time collision avoidance of the stacker can be completely achieved using only a laser radar installed on the stacker, ultimately achieving the goal of autonomous obstacle avoidance.

[0093] Furthermore, the stacker crane collision avoidance method based on three-dimensional perception can be divided into two parts: offline and online, such as Figure 4 As shown, the details are as follows:

[0094] Offline stage: Collect the storage environment point cloud data required for training, and divide it into training samples and verification samples; construct Figure 2 The deep target detection neural network model shown in FIG. 1 is used to preprocess the training samples; the neural network model is forward propagated using the training set, and the training error is back-propagated until the preset training step is reached;

[0095] Online stage: The warehouse environment point cloud obtained in real time by the laser radar on the warehouse stacker is Figure 2 After the deep target detection neural network model shown in the figure is applied, the prediction box result is obtained through post-processing operations. According to the prediction result after filtering the current frame, an obstacle avoidance algorithm for restricted receptive field is used to search for a feasible path for the stacker.

[0096] According to the stacker collision avoidance method based on three-dimensional perception proposed in the embodiment of the present application, accurate detection of pedestrians and protruding cargo boxes in the storage environment can be fully realized with only one laser radar and sufficient data support. No additional staff is required to protect the stacker to prevent collisions, which can effectively save labor and improve the efficiency of the stacker operation. The kernel density estimation method can be used to supplement the description of the point cloud to help the 3D target detection model extract the Bev features of the point cloud data, thereby improving the detection accuracy. The post-processing algorithm also has good correction results for missed detection and wrong detection of the detection frame, and a result with a high accuracy rate can be obtained. At the same time, since the kernel density estimation method and the post-processing method used in the embodiment of the present application are implemented using matrix operations, the operation speed is relatively fast, which meets the needs of the stacker for real-time collision avoidance and realizes the stacker's autonomous obstacle avoidance. As a result, the problem that the stacker cannot adapt to complex environments, resulting in reduced operating efficiency, is solved.

[0097] Next, a stacker collision avoidance device based on three-dimensional perception proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0098] Figure 5 It is a block diagram of a stacker collision avoidance device based on three-dimensional perception according to an embodiment of the present application.

[0099] like Figure 5 As shown, the stacker collision avoidance device 10 based on three-dimensional perception includes: an acquisition module 100 , an estimation module 200 , an extraction module 300 , a prediction module 400 , a filtering module 500 and an obstacle avoidance module 600 .

[0100] Among them, the acquisition module 100 is used to obtain the warehouse environment point cloud data of the current frame; the estimation module 200 is used to use the kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and use the distribution probability as a supplement to the point cloud description; the extraction module 300 is used to input the expanded point cloud data into the feature extraction network to extract the Bev feature map of the current frame; the prediction module 400 is used to input the Bev feature map of the current frame into the detection head to obtain the category and positioning of the prediction box; the filtering module 500 is used to non-maximum suppress the prediction box of the current frame, and then use the fast clustering algorithm along the z-axis direction on the prediction box, and then filter the prediction box according to the number of points in the prediction box and the confidence level; the obstacle avoidance module 600 is used to plan a feasible path based on the prediction result after filtering the current frame using an obstacle avoidance algorithm for a restricted receptive field.

[0101] In an embodiment of the present application, it also includes: a training module, which is used to estimate the distribution probability of each point in the point cloud located in the same pillar using a kernel density estimation method, and before using the distribution probability as a supplement to the point cloud description, it includes: obtaining historical warehouse environment point cloud data, and marking pedestrians and protruding cargo boxes in the shelf in the historical warehouse environment point cloud data; dividing the historical warehouse environment point cloud data into a training set and a validation set, and performing data enhancement on the point cloud data in the training set; using a kernel density estimation method to estimate the distribution probability of the point cloud in each pillar for the data-enhanced point cloud data as a supplement to the point cloud description; iteratively training the feature extraction network and the detection head based on the training data set, during the iterative training process, calculating the training loss value based on the output value of the detection head and the true label, and optimizing the weights of the feature extraction network and the detection head by back propagation of the loss value; stopping the iterative training until the iterative training of the feature extraction network and the detection head reaches the target number of iterations; and using the validation set to perform performance verification on the trained feature extraction network and the detection head.

[0102] Optionally, in an embodiment of the present application, the estimation module 200 is further used to divide the point cloud space according to square columns on the target plane; select a Gaussian kernel function to estimate the point cloud density distribution function in each square column; estimate the point cloud distribution probability of each point in the square column based on the point cloud density distribution function, and use it as a supplement to the point cloud description.

[0103] Optionally, in an embodiment of the present application, the extraction module 300 is further used to use a multi-layer perceptron and maximum pooling to extract features from the point cloud in each square column to obtain a preliminary Bev feature map; based on the preliminary Bev feature map, a two-dimensional convolutional neural network is used to further extract a multi-scale feature map, wherein the two-dimensional convolutional neural network is a ResNet network or a VoVNet network; based on the multi-scale feature map, an FPN or BiFPN network is used to fuse the multi-scale feature map to obtain the final Bev feature map.

[0104] Optionally, in an embodiment of the present application, the prediction module 400 is further used to classify the prediction boxes into two categories: pedestrians and protruding boxes in the shelves; each feature in the Bev feature map of the current frame corresponds to multiple pre-defined aiming point boxes; based on the aiming point boxes, the detection head uses the features corresponding to the aiming point boxes to predict category scores and coordinates, sizes, and heading angle biases.

[0105] Optionally, in the embodiment of the present application, the filtering module 500 is further used to predict the box (x b ,y b ,z b ,l b ,w b ,hb ,θ b ), take out the current frame located at (x b ,y b ,l b ,w b ,θ b ) are the points in the bottom pillar, forming the point set P ′ ; Statistical point set P ′ The histogram of the z coordinate in the histogram; the central value and width of the continuous non-zero part in the histogram are used as the z coordinate and height of the new prediction box to obtain the new prediction box.

[0106] Optionally, in the embodiment of the present application, the filtering module 500 is further used to count the number of point clouds N in the prediction box; the prediction box is divided into three categories: near, medium and far according to the x-axis coordinate of the center point of the prediction box; for each category, a threshold N is pre-set min 、N posmin and min ; if N <N min , then filter out; if N>N posmin , then keep it; otherwise, according to the confidence s b Further judgment, if s b min , it is still filtered out.

[0107] Optionally, in an embodiment of the present application, the obstacle avoidance module 600 is further used to model the stacker motion scene as a two-dimensional plane grid map; based on the prediction result after filtering the current frame, mark the obstacle position on the grid map and set the obstacle position and the surrounding grids as unreachable; based on the two-dimensional plane grid map information, use the A* search algorithm or the Dijkstra search algorithm to find a feasible path for the stacker.

[0108] According to the stacker collision avoidance device based on three-dimensional perception proposed in the embodiment of the present application, accurate detection of pedestrians and protruding cargo boxes in the storage environment can be fully realized with only one laser radar and sufficient data support. No additional staff is required to protect the stacker to prevent collisions, which can effectively save labor and improve the efficiency of the stacker operation. The kernel density estimation method can be used to supplement the description of the point cloud to help the 3D target detection model extract the Bev features of the point cloud data, thereby improving the detection accuracy. The post-processing algorithm also has good correction results for missed detection and wrong detection of the detection frame, and a result with a high accuracy rate can be obtained. At the same time, since the kernel density estimation method and the post-processing method used in the embodiment of the present application are implemented using matrix operations, the operation speed is relatively fast, which meets the needs of the stacker to avoid collisions in real time and realizes the autonomous obstacle avoidance of the stacker. As a result, the problem that the stacker cannot adapt to complex environments, resulting in reduced operating efficiency, etc., is solved.

[0109] ​ Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0110] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0111] When the processor 602 executes the program, the stacker collision avoidance method based on three-dimensional perception provided in the above embodiment is implemented.

[0112] Furthermore, the electronic device further comprises:

[0113] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0114] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0115] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0116] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0117] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0118] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0119] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned stacker collision avoidance method based on three-dimensional perception.

[0120] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed, is used to implement a stacker collision avoidance method based on three-dimensional perception.

[0121] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0124] It should be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, the steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0125] A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0126] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A stacker collision avoidance method based on three-dimensional perception, characterized in that: The following steps are involved: Step S101: obtaining storage environment point cloud data of the current frame; Step S102: using a kernel density estimation method to estimate the distribution probability of each point in the point cloud located in the same pillar, and using the distribution probability as a supplement to the point cloud description; Step S103: input the expanded point cloud data into the feature extraction network to extract the Bev feature map of the current frame; Step S104: input the Bev feature map of the current frame into the detection head to obtain the category and location of the predicted box; Step S105: After the prediction box of the current frame is subjected to non-maximum suppression, a fast clustering algorithm along the z-axis direction is applied to the prediction box, and then the prediction box is filtered according to the number of points in the prediction box and the confidence level; Step S106: According to the prediction result after filtering the current frame, a feasible path is planned using an obstacle avoidance algorithm for a limited receptive field.

2. The stacker collision avoidance method based on three-dimensional perception according to claim 1 is characterized in that: Before estimating the distribution probability of each point using the kernel density estimation method for the point cloud located in the same pillar and using the distribution probability as a supplement to the point cloud description, it also includes: Acquire historical warehouse environment point cloud data, and mark pedestrians and protruding cargo boxes on shelves in the historical warehouse environment point cloud data; Dividing the historical storage environment point cloud data into a training set and a validation set, and performing data enhancement on the point cloud data in the training set; Using a kernel density estimation method on the data-enhanced point cloud data to estimate the distribution probability of the point cloud in each pillar as a supplement to the point cloud description; Iteratively training the feature extraction network and the detection head based on the training set, calculating the training loss value based on the output value of the detection head and the true label during the iterative training, and optimizing the weights of the feature extraction network and the detection head by back propagation of the loss value; The iterative training is stopped until the iterative training of the feature extraction network and the detection head reaches the target number of iterations; The verification set is used to perform performance verification on the trained feature extraction network and the detection head.

3. The stacker collision avoidance method based on three-dimensional perception according to claim 1 is characterized in that: The point cloud located in the same pillar uses a kernel density estimation method to estimate the distribution probability of each point as a supplement to the point cloud description, including: Divide the point cloud space into square columns on the target plane; Select the Gaussian kernel function to estimate the point cloud density distribution function in each square column; The point cloud distribution probability of each point in the square column is estimated based on the point cloud density distribution function, and is used as a supplement to the point cloud description.

4. The stacker crane collision avoidance method based on three-dimensional perception according to claim 1, characterized in that: The expanded point cloud data is input into the feature extraction network to extract the Bev feature map of the current frame, including: Use multi-layer perceptron and maximum pooling to extract features from the point cloud in each square column to obtain a preliminary Bev feature map; Based on the preliminary Bev feature map, a two-dimensional convolutional neural network is used to further extract a multi-scale feature map, wherein the two-dimensional convolutional neural network is a ResNet network or a VoVNet network; Based on the multi-scale feature map, the multi-scale feature map is fused using an FPN or BiFPN network to obtain the final Bev feature map.

5. The stacker collision avoidance method based on three-dimensional perception according to claim 1 is characterized in that: The Bev feature of the current frame is input into the detection head to obtain the category and location of the predicted frame, including: The categories of the prediction boxes are divided into two categories: pedestrians and prominent boxes on the shelves; Each feature in the Bev feature map of the current frame corresponds to a plurality of predefined aiming point frames; Based on the aiming point frame, the detection head predicts the category score and coordinates, size, and heading angle offset using features corresponding to the aiming point frame.

6. The stacker collision avoidance method based on three-dimensional perception according to claim 1 is characterized in that: The fast clustering algorithm along the z-axis direction includes: For each category, the prediction box (x b ,y b ,z b ,l b ,w b ,h b ,θ b ), take out the current frame located at (x b ,y b ,l b ,w b ,θ b ) are the points in the bottom pillar, forming the point set P ′ ; Statistical point set P ′ Histogram of z coordinates in ; The center value and width of the continuous non-zero part in the histogram are used as the z coordinate and height of the new prediction box to obtain the new prediction box.

7. The stacker crane collision avoidance method based on three-dimensional perception according to claim 1, characterized in that: The filtering of the prediction box according to the number of points in the prediction box and the confidence level includes: Count the number of point clouds N in the prediction box; According to the x-axis coordinate of the center point of the prediction box, the prediction box is divided into three categories: near, medium, and far; For each type of situation, a threshold N is pre-set min 、N posmin and min , if N <N min , then filter out; if N>N posmin , then keep it; otherwise, further judge according to the confidence level s. If s min , it is still filtered out.​ 8. The stacker crane collision avoidance method based on three-dimensional perception according to claim 1, characterized in that: The method of planning a feasible path based on the prediction result after filtering the current frame using an obstacle avoidance algorithm for a limited receptive field includes: Model the stacker movement scene as a two-dimensional plane grid map; According to the prediction result after filtering the current frame, marking the obstacle position on the grid map and setting the obstacle position and surrounding grids as unreachable; According to the two-dimensional plane grid map information, an A* search algorithm or a Dijkstra search algorithm is used to find a feasible path for the stacker.

9. A stacker collision avoidance device based on three-dimensional perception, characterized in that: include: The acquisition module is used to obtain the storage environment point cloud data of the current frame; An estimation module, for estimating the distribution probability of each point using a kernel density estimation method for the point cloud located in the same pillar, and using the distribution probability as a supplement to the point cloud description; The extraction module is used to input the expanded point cloud data into the feature extraction network to extract the Bev feature map of the current frame; The prediction module is used to input the Bev feature map of the current frame into the detection head to obtain the category and location of the predicted box; A filtering module is used to perform non-maximum suppression on the prediction box of the current frame, then use a fast clustering algorithm along the z-axis direction on the prediction box, and then filter the prediction box according to the number of points in the prediction box and the confidence level; The obstacle avoidance module is used to plan a feasible path based on the prediction results after filtering the current frame using an obstacle avoidance algorithm for restricted receptive fields.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the stacker collision avoidance method based on three-dimensional perception as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the stacker collision avoidance method based on three-dimensional perception as described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the stacker collision avoidance method based on three-dimensional perception as described in any one of claims 1 to 8 is implemented.

Images