Dynamic target detection and tracking method, device and equipment for unmanned excavator

Abstract

The application relates to the field of target tracking of engineering machinery, and discloses a dynamic target detection and tracking method, device and equipment for unmanned excavators, which comprises the following steps: collecting point cloud data of the unmanned excavator through a pre-constructed data acquisition platform; performing voxel feature coding on the point cloud data according to a columnar unit coding mode to obtain a first dense tensor, generating a pseudo image for feature extraction, and obtaining a dense point cloud feature map; adopting a center point positioning method based on a heat map, determining the position of the center point by combining a minimum Gaussian radius value with a Gaussian distribution function, and then performing one-stage feature regression to obtain feature information of a target detection bounding box; and performing target detection and tracking on the unmanned excavator according to the feature information. The application adopts a voxel division method with better real-time performance and uses Mahalanobis distance as a measure, solves the problem of time consumption caused by point cloud down-sampling, better tracks small target objects, and is suitable for the detection and tracking of unstructured roads of engineering machinery.

Description

Technical Field

[0001] This invention relates to the field of target tracking in engineering machinery, and specifically to a method, device, and system for dynamic target detection and tracking of unmanned excavators. Background Technology

[0002] Autonomous driving technology is a synthesis of multiple cutting-edge disciplines, and in recent years it has made continuous breakthroughs in many aspects and has been applied to various industries. In the passenger vehicle sector, autonomous driving technology has achieved rapid development and received increasingly widespread attention, achieving a certain degree of maturity and implementation. However, in the field of autonomous driving in construction machinery, it is still in an early stage of development. Tracked excavators are representative products in construction machinery, widely used in building construction, mining, and disaster relief, but the implementation of autonomous driving technology in this field faces challenges such as harsh and complex working conditions, unstructured roads, and the need for all-weather operation in special operation scenarios.

[0003] In autonomous driving technology, the environmental perception layer mainly consists of target detection, tracking, segmentation, and simultaneous localization and mapping (SMR). Sensors such as cameras, LiDAR, millimeter-wave radar, and integrated navigation systems acquire information about the vehicle and its surrounding environment to ensure the effectiveness and safety of subsequent planning and control layers. Mature environmental perception technologies in the passenger vehicle sector primarily target three aspects: road surface, static objects, and dynamic objects. These technologies involve road boundary detection, obstacle detection, vehicle and pedestrian detection, and trajectory tracking, mainly employing cameras combined with LiDAR for environmental perception. This relies on algorithmic logic, using deep learning neural networks to perform pixel segmentation, object classification, model calibration, and target tracking on images captured by cameras. This enables obstacle identification and matching, and the LiDAR point cloud is then mapped onto the image. Based on the LiDAR scan of surrounding obstacles and the map, the vehicle's direction and speed are controlled. While passenger vehicle environmental perception technology is primarily designed for multi-reference, structured road environments, it is difficult to apply to engineering machinery.

[0004] This is primarily due to the application of target detection technology on unstructured roads for construction machinery. The main focus is on obstacle detection, identifying static or dynamic targets such as earthwork sites, pedestrians, construction machinery, carts, and obstacles. Previously, target detection typically used cameras as sensing sensors to acquire information. However, detection is affected by lighting conditions, and the complex ground conditions on unstructured roads make depth information in the vertical direction crucial. Therefore, the image data acquired by cameras suffers from insufficient scene depth information, leading to the adoption of LiDAR as the primary sensing sensor for construction machinery. However, due to the large cardinality, disorder, and sparsity of point cloud data, downsampling preprocessing is essential. Traditional LiDAR detection data preprocessing typically employs voxel downsampling by dividing the data into cubic units.

[0005] Furthermore, current target tracking technology applied to unstructured roads for engineering machinery primarily focuses on maintaining the identity of targets detected by sensors. By tracking the position of each category frame by frame to provide semantic information for system decision-making, this technology is a prerequisite for behavioral decision-making and is mainly used to realize autonomous driving functions such as automatic following and automatic emergency braking. The most significant challenges in target tracking technology are changes in the target background, changes in the object itself, and changes in light intensity. To overcome the impact of changes in light intensity, the strong anti-interference capability and all-weather operation of lidar (LiDAR) make it a good solution as a perception sensor. Mainstream multi-target tracking technologies primarily employ a detection-based tracking mode, consisting of three parts: data association, state prediction and update, and trajectory generation and deletion. Typically, the Hungarian algorithm is used for data association, followed by association matching based on a three-dimensional IOU metric, and a Kalman filter is used for state prediction and update.

[0006] Image-based detection and tracking methods offer advantages such as low cost, rich information content, and good real-time performance. However, due to the unique operating environments of construction machinery, cameras are susceptible to light interference, making them unsuitable for the environmental perception requirements of excavator operations. Currently, LiDAR-based environmental perception systems are widely used. LiDAR leverages its extremely high resolution, strong anti-interference capabilities, rich scene depth information, all-weather operation, and long detection range to provide the necessary dynamic environmental target pose, size, velocity, and trajectory information for subsequent planning and control in unstructured road environments. Traditional LiDAR-based target detection technologies often employ voxel-based downsampling for point cloud data preprocessing, which suffers from time consumption and inability to guarantee real-time performance, thus compromising the effectiveness and safety of subsequent tasks. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for detecting and tracking dynamic targets by an unmanned excavator based on lidar, so as to solve the problems existing in the prior art.

[0008] This invention provides a dynamic target detection and tracking method for unmanned excavators, which includes the following steps:

[0009] Point cloud data of unmanned excavators is collected through a pre-built data acquisition platform, and a target detection and tracking dataset that conforms to the operation scenario of construction machinery is created according to the point cloud annotation specifications.

[0010] The point cloud data is encoded using a columnar unit encoding method to obtain a first dense tensor, and a pseudo-image of a two-dimensional bird's-eye view is generated based on the first dense tensor.

[0011] The pseudo-image is used to extract features by two-dimensional convolution to obtain a dense point cloud feature map;

[0012] For the dense point cloud feature map, a center point localization method based on heatmap is adopted. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box.

[0013] The unmanned excavator performs target detection and tracking based on the feature information of the target detection bounding box.

[0014] Preferably, the data acquisition platform includes a horizontal support bracket equipped with a lidar, an on-board computing platform, and a combined inertial navigation system. The horizontal support bracket is installed in the center of the front grille of the excavator and has reserved holes for installing camera sensors. The on-board computing platform and the combined inertial navigation system are installed on the top of the excavator.

[0015] Preferably, point cloud data of the unmanned excavator is collected through a pre-built data acquisition platform, and a target detection and tracking dataset conforming to the construction machinery operation scenario is created according to the point cloud annotation specifications, specifically including:

[0016] In a specified work scenario, an unmanned excavator is driven along different routes to collect multiple data segments. Frame extraction is then used to process the data. Based on the point cloud annotation specifications, corresponding semantic categories are set according to the engineering work scenario to create the target detection and tracking dataset required for network training. When annotating according to the point cloud annotation specifications, corresponding semantic categories are set according to the engineering work scenario, namely: construction machinery, pedestrians, handcarts, piles of soil, and obstacles.

[0017] Preferably, the point cloud data is voxel-encoded according to the columnar unit encoding method to obtain a first dense tensor, and a two-dimensional bird's-eye view pseudo-image is generated based on the first dense tensor, specifically including:

[0018] The large-scale irregular 3D point cloud data is divided into various columnar cells according to the columnar cell encoding method. A sampling threshold is set for the columnar cells. Points with a value greater than the sampling threshold are sampled using a random sampling method. Points with a value less than the sampling threshold are padded with zeros to obtain downsampled columnar 3D point cloud data.

[0019] Data augmentation is performed on the columnar 3D point cloud data, expanding the original four-dimensional information {x, y, z, i} of the 3D point cloud data to nine-dimensional information {x, y, z, i, x}. c y c , z c x p y p}, where {x, y, z} are the spatial coordinates of the point, i is the reflectance, and {x c y c , z c} represents the offset from the point to the average coordinates of all points in the column cell containing that point, {x p y p} represents the offset in X and Y coordinates from the center point of the column cell containing that point;

[0020] The augmented columnar 3D point cloud data is feature-encoded frame by frame into a first dense tensor with dimension D, number of columnar units P, and number of point clouds in columnar units N.

[0021] The first dense tensor is activated using an activation function to generate a second dense tensor with C channels, P bars, and N points in each bar. Then, max pooling is performed on each bar cell to obtain a third dense tensor with dimensions (C, P). Finally, a pseudo-image (C, H, W) with width W, height H, and number of categories C is generated from a bird's-eye view using a stacking method.

[0022] Preferably, for the dense point cloud feature map, a center point localization method based on heatmaps is used. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box, specifically including:

[0023] A heatmap-based localization method is used to locate the center point. Feature mapping is performed on the dense point cloud feature map to expand the Gaussian peak value of the center rendering of each ground real object, reset the Gaussian radius, and obtain the center point location based on the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box. The feature information includes center heatmap information, center point offset, target size, ground height, target orientation, and target speed.

[0024] Preferably, it further includes:

[0025] A second-stage feature regression is performed on the characteristic information of the first stage to obtain the refined features of the target detection bounding box and the confidence of the detected target.

[0026] The current target detection result and the previous tracking result are input into a trainable distance combination module to obtain the depth feature distance, which is then combined with the Mahalanobis distance. Finally, a greedy matching algorithm is applied to the combined distance of data association.

[0027] The target state is estimated by using the Kalman filter based on the tracking result of the previous time step, and then the target is tracked by combining the greedy matching algorithm for pairing and updating, so as to obtain the tracking result at the current time step.

[0028] Based on the tracking initialization module, tracking initialization is performed on input unmatched detection results whose confidence scores are greater than a threshold.

[0029] Preferably, a two-stage feature regression is performed on the characteristic information from the first stage to obtain the refined features of the target detection bounding box and the confidence score of the detected target, specifically including:

[0030] Based on the feature information of the target detection bounding box, for the three-dimensional center point of the selected surface of the prediction box, the corresponding features are extracted from the bird's-eye view using bilinear interpolation, and then connected to obtain the feature vector of the target detection bounding box.

[0031] The data is refined through a multilayer perceptron, resulting in a refined bounding box size and orientation prediction parameter for the first-stage target detection boundary, as well as a final confidence score prediction.

[0032] This invention also provides a dynamic target detection and tracking device for an unmanned excavator, comprising:

[0033] The data production unit is used to collect point cloud data of unmanned excavators through a pre-built data acquisition platform, and to produce target detection and tracking datasets that conform to the operation scenarios of construction machinery according to point cloud annotation specifications.

[0034] The encoding unit is used to encode the point cloud data using a columnar unit encoding method to obtain a first dense tensor, and to generate a pseudo image with a two-dimensional bird's-eye view based on the first dense tensor.

[0035] The feature extraction unit is used to extract features from the pseudo image using two-dimensional convolution to obtain a dense point cloud feature map.

[0036] The first-stage feature regression unit is used to locate the center point of the dense point cloud feature map using a heatmap-based center point localization method. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, the first-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box.

[0037] The target detection and tracking unit is used to perform target detection and tracking on the unmanned excavator based on the feature information of the target detection bounding box.

[0038] This invention also provides a dynamic target detection and tracking device for an unmanned excavator, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement the dynamic target detection and tracking method for an unmanned excavator as described above.

[0039] The embodiments of the present invention have at least one of the following beneficial effects:

[0040] 1. Design and Deployment of Data Acquisition Platform: To address the perception of surrounding environmental information by unmanned excavators, this invention designs a point cloud data acquisition platform. Unlike previous methods where LiDAR was installed on the top of the cab, this platform features a sensor bracket installed at the front of the excavator. This bracket is well-suited for tracked and wheeled excavators, replacing the existing radar installation scheme. This allows for better scanning of the more crucial environmental information ahead and the excavator bucket's operational information. Due to the complex operating environment of construction machinery, a 128-line LiDAR is selected to capture richer object details, improving the reliability of environmental perception and enhancing the effectiveness and safety of subsequent planning and control.

[0041] 2. Creation of a dedicated dataset: The environmental semantic information of construction machinery operation scenarios differs from that of passenger vehicles. Therefore, the creation of a dataset is essential for the detection and tracking of dynamic objects using deep learning. A detection and tracking dataset containing categories such as construction machinery, pedestrians, handcarts, piles of dirt, and roadblocks was created for unstructured road scenarios of construction machinery, ensuring the reliability and robustness of network training.

[0042] 3. Target Detection and Tracking: Addressing the limitation of cameras as sensors in engineering machinery operation environments, this invention utilizes LiDAR as an environmental perception sensor. This leverages LiDAR's advantages of being unaffected by light and its stable operation, meeting the requirements for 24-hour operation of unmanned excavators. Secondly, recognizing the significant workload and slow execution time of conventional voxelization processing, which cannot guarantee real-time performance in real-world excavator deployment experiments, this invention employs columnar voxelization feature encoding. The 3D point cloud features are then mapped onto a 2D bird's-eye view before 2D convolutional feature extraction, improving the algorithm's real-time performance. To address the issue of detecting dynamic targets with varying poses, this invention uses a two-stage feature regression based on the center point, improving the detection of orientation information and ground clearance, facilitating subsequent trajectory tracking. Finally, addressing the poor performance of most algorithms using 3D intersection-union ratio (IoU) as a metric, this invention uses Mahalanobis distance as the metric. This is combined with depth feature distance obtained during training and applied to a greedy matching algorithm for data association. Finally, a Kalman filter is used for state estimation and updating, achieving accurate tracking of multiple dynamic targets. Attached Figure Description

[0043] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0044] Figure 1 This is a flowchart illustrating a dynamic target detection and tracking method for an unmanned excavator provided in the first embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the overall working principle of a dynamic target detection and tracking method for an unmanned excavator provided in the first embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram showing the deployment of the data acquisition platform on the unmanned excavator.

[0047] Figure 4 This is a schematic diagram of the structure of the dynamic target detection and tracking device for unmanned excavators provided in the second embodiment of the present invention. Detailed Implementation

[0048] To make the technical solutions of the embodiments of the present invention clearer, the technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described are only some embodiments of the present invention, not all embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and should not be used to limit the scope of protection of the present invention.

[0049] Please see Figure 1 and Figure 2 The first embodiment of the present invention provides a dynamic target detection and tracking method for an unmanned excavator, which includes the following steps:

[0050] S101 collects point cloud data of unmanned excavators through a pre-built data acquisition platform, and creates a target detection and tracking dataset that conforms to the operation scenario of construction machinery according to the point cloud annotation specifications.

[0051] Because excavators have a different structure than conventional passenger vehicles, with a higher body and a boom, stick, and bucket, if their lidar is installed on the top of the cab in the same way as passenger vehicles, the limited vertical field of view during lidar scanning will result in large blind spots around the excavator, reducing the effectiveness and safety of environmental perception.

[0052] In this embodiment, the data acquisition platform includes a horizontal support 10 equipped with a lidar 20, an on-board computing platform 30, and a combined inertial navigation system 40. The horizontal support 10 is installed in the center of the front grille of the excavator and has reserved holes for installing camera sensors. The on-board computing platform 30 and the combined inertial navigation system 40 are installed on the top of the excavator.

[0053] Specifically, to obtain richer feature information, this embodiment uses a single 128-line LiDAR sensor. To address the aforementioned blind spot issue, a horizontal support was designed for the LiDAR sensor. As shown in Figure 3, the horizontal support 10 is installed in the center of the front side of the excavator's cab, 1.6 meters above the ground. This horizontal support 10 is well-suited for tracked and wheeled excavators, is easy to disassemble, and effectively avoids blind spots when collecting point clouds for environmental perception. Replacing the existing LiDAR installation scheme with this method allows for better acquisition of the excavator's more needed forward environmental information and excavator bucket operation information, improving the reliability of environmental perception and enhancing the effectiveness and safety of subsequent planning and control.

[0054] The horizontal support is as follows Figure 3 As shown, the mounting area is divided into a lidar mounting area and a camera mounting area, which can meet the requirements of subsequent multi-sensor fusion. It can also be adjusted according to the thickness of the excavator's front grille to ensure the lidar mounting surface is horizontal. The bracket houses a 128-line lidar with a vertical field of view of -25° to +15°, capable of scanning areas beyond a two-meter radius. Since the excavator's bucket has an operating range of approximately 2-3 meters, this effectively solves the blind spot problem compared to mounting the lidar on the roof.

[0055] In this embodiment, the environmental semantic information of the construction machinery operation scenario is different from that of passenger vehicles. Therefore, when using deep learning to detect and track dynamic objects, the creation of a dataset is indispensable. Thus, a single lidar target detection and tracking dataset that conforms to the construction machinery operation scenario was created based on point cloud collection platform and point cloud annotation specifications, which facilitates the subsequent training and evaluation of the network.

[0056] Specifically, firstly, in a defined operating scenario, the excavator travels along different routes, and point cloud data is collected segment by segment during the travel process, resulting in 40 scene segments. Each scene lasts approximately 120-180 seconds, and the LiDAR uses a scanning frequency of 10Hz for data acquisition. Due to frame loss during the acquisition process, each scene segment yields 1000-1500 frames of data. Then, the raw point clouds are collected and managed as the total source data, divided into 30 segments of data to be labeled and 10 segments of data that do not require labeling. Next, the data to be labeled is further divided into training data... Fifteen training data segments and 15 validation data segments were prepared, with 10 unlabeled data segments used as test data. To ensure better training data performance, the unlabeled data was processed by frame extraction, dividing it into 30 segments of 100 or 150 frames each. The 30 segments were then labeled according to point cloud annotation standards, with corresponding semantic categories set based on engineering scenarios: construction machinery, pedestrians, handcarts, piles of earth, and obstacles. Finally, to meet the needs of detection and tracking, additional frame and ID information was added during the annotation process, following the requirements for tracking datasets.

[0057] S102, the point cloud data is encoded using voxel features according to the columnar unit encoding method to obtain a first dense tensor, and a pseudo image of a two-dimensional bird's-eye view is generated based on the first dense tensor.

[0058] In this embodiment, although the 3D point cloud data contains a wealth of information, its data volume is also enormous. When a 128-line LiDAR scans and acquires point cloud data, each frame contains over 160,000 points. Directly performing convolution processing on the 3D point cloud data after inputting it into the network places high demands on the onboard computing platform's hardware and results in very slow processing speed and a lack of real-time performance. Therefore, the acquired point cloud data undergoes downsampling first. A common 3D point cloud downsampling method is voxelization, but conventional voxelization distributes the 3D point cloud into equal-sized 3D cells, resulting in slow processing speed. Therefore, this embodiment uses columnar cell features to distribute the 3D point cloud data into equal-sized columnar cells in the XY plane of the LiDAR coordinate system. This method does not consider the Z-axis size information of the cells, significantly reducing the data downsampling time and ensuring real-time performance.

[0059] Specifically, in this embodiment, the large-scale irregular three-dimensional point cloud data is first divided into various columnar cells according to the columnar cell encoding method. A sampling threshold is set for the columnar cells. Points greater than the threshold are sampled using a random sampling method, and points less than the threshold are padded with zeros, thereby obtaining downsampled three-dimensional point cloud data.

[0060] Then, data augmentation is performed on the obtained columnar 3D point cloud data, expanding the original four-dimensional information {x, y, z, i} of the 3D point cloud data to nine-dimensional information {x, y, z, i, x}. c y c , z c x p y p}, where {x, y, z} are the spatial coordinates of the point, i is the reflectance, and {x c y c , z c} represents the offset from the point to the average coordinates of all points in the point column, {x p y p} represents the offset of the X, Y coordinates from the center point of the column. Due to the irregularity and sparsity of the point cloud, some columns contain a large number of points, while others contain few or none. Therefore, a threshold is set for the number of points sampled within a column. If the actual number of points is greater than the threshold, random sampling is used to sample the number of points within that threshold. If the actual number of points is less than the threshold, zero-padding is performed. Finally, the point cloud is feature-encoded frame by frame into a first dense tensor (D, P, N) with dimension D, number of column units P, and number of points in each column unit N.

[0061] In addition, directly performing convolution processing on 3D point clouds would greatly increase the difficulty and time consumption. Therefore, after downsampling and feature encoding the data, in order to avoid this situation, this embodiment first maps the preprocessed 3D point cloud data onto a 2D bird's-eye view, and then uses 2D convolution operation to extract features, which greatly saves time, ensures real-time performance, and is suitable for complex and ever-changing working conditions of engineering machinery.

[0062] The specific operation is as follows: First, the first dense tensor (D, P, N) obtained after the above feature encoding is processed by the activation function to generate a second dense tensor (C, P, N) with C channels, P columnar units, and N points in the columnar units. Then, max pooling is performed on each columnar unit to obtain a third dense tensor with dimension (C, P). Finally, a pseudo-image (C, H, W) with width W, height H, and number of categories C is generated on the bird's-eye view using a stacking method to meet the requirements of the two-dimensional convolution operation of point cloud.

[0063] S103, use two-dimensional convolution to extract features from the pseudo image to obtain a dense point cloud feature map.

[0064] S104, For the dense point cloud feature map, a center point localization method based on heatmap is adopted. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box.

[0065] In this embodiment, a heatmap-based center point localization method is used for the dense point cloud feature map obtained above. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function, and then the preliminary feature information of the target detection bounding box is obtained through center point feature regression. The feature information mainly includes: center point heatmap information, target size, height above the ground, rotation orientation, and target velocity.

[0066] Specifically:

[0067] First, the dense point cloud features obtained above are mapped to the dense tensor (C, H, W) of the bird's-eye view and input into the object detection network, where C is the number of channels, and H and W are the height and width of the two-dimensional mapping.

[0068] Then, when a center-point-based detection head detects an object, it generates a heatmap peak at the center location. This detection head generates heatmaps with a corresponding number of channels based on the number of categories K. During training, the 3D centers of the labeled object detection bounding boxes are projected onto a bird's-eye view to generate a 2D Gaussian image, and a focus loss is applied. Objects in the bird's-eye view are sparser than those in the pseudo-image. Distances in the bird's-eye view are absolute, while distances in the pseudo-image view are distorted due to mapping. Simultaneously, the depth dimension is compressed during projection, making object centers in the pseudo-image view closer together, thus most locations are treated as background.

[0069] To avoid this problem, positive supervision of the target heatmap is increased by amplifying the Gaussian peaks presented at the centers of each real-world object. The Gaussian radius is set to... ,in It is the minimum Gaussian radius, and f is the radius function, which ensures the simplicity of target assignment based on the center point.

[0070] The target's center point features include multiple object attributes: sub-voxel position refinement, ground clearance, 3D dimensions, and yaw rotation angle. Sub-voxel position refinement reduces quantization errors from voxelization and step size in the backbone network; ground clearance helps locate the target in 3D space and adds missing height information by projecting it in a bird's-eye view; target orientation prediction uses the sine and cosine of the yaw angle as continuous regression targets, combined with the bounding box size, these regression heads provide feature information for the 3D target detection bounding box.

[0071] Furthermore, during the training of the regression head, an L1 regression loss is used to supervise the actual centering of the target. The regression yields a logarithmic size to better handle target detection boxes of different shapes. During inference, all attributes are extracted by indexing the dense regression head output at each target peak location.

[0072] S105, the unmanned excavator performs target detection and tracking based on the feature information of the target detection bounding box.

[0073] Specifically,

[0074] S1051, perform two-stage feature regression on the characteristic information of the first stage to obtain the refined features of the target detection bounding box and obtain the confidence of the detected target.

[0075] In this study, the number of point cloud data points input into the detector after feature mapping is relatively small. Predicting the overall shape and orientation of an object using these few points lacks sufficient feature information to support accurate predictions. Therefore, to address this issue, a two-stage feature regression is used after the first-stage feature regression to further refine the features of the target detection bounding box and generate confidence predictions. The two-stage feature regression extracts additional point features from the output point cloud after feature mapping. Specifically, it extracts one point from the 3D center of each face of the predicted target detection bounding box (excluding the top and bottom faces). These five center points, including the previously extracted target center point, are then subjected to bilinear interpolation to extract their corresponding features from the dense point cloud feature mapping output of the bird's-eye view. The extracted point features are then concatenated to obtain a vector containing the overall 3D target detection bounding box features. This vector is then passed to the multilayer perceptron to refine the bounding box size and orientation prediction parameters and generate the final confidence score prediction.

[0076] For the prediction of the confidence score, I is determined by the 3D Intersection over Union (IOU) ratio between the target detection box and the actual target bounding box.

[0077]

[0078] in It is the intersection-union ratio (IoU) between the target detection box and the actual target bounding box, and a binary cross-entropy loss is used to supervise training:

[0079]

[0080] in, This involves predicting the two-stage confidence score. In the inference process, this embodiment directly uses the category prediction from the one-stage feature regression. The geometric mean is calculated together with the predictions of the confidence scores: ,Will As the final confidence prediction, the regression of the detection box is based on the L1 loss model trained on the basis of the one-stage feature regression.

[0081] S1052, the target detection result at the current moment and the tracking result at the previous moment are input into the trainable distance combination module to obtain the depth feature distance, which is then combined with the Mahalanobis distance, and then the greedy matching algorithm is applied to the combined distance of data association.

[0082] In the data association section, this embodiment employs a greedy matching algorithm and a combined distance obtained through training. The combined distance module provides a learning metric between N detection results and M trajectory predictions. This metric combines state estimation with appearance and geometric feature information, linearly combining Mahalanobis distance and depth feature distance.

[0083]

[0084] in This represents the Mahalanobis distance matrix, where each element contains the distance between each detection and prediction state for each trajectory; The feature distance matrix represents the feature dissimilarity between each detection and each trajectory; (α, β) is the combination coefficient matrix with shape N×M; (symbols omitted) This represents the element-wise multiplication operator; the constant 0.5 is used as the initial bias term for the linear combination to help the model converge faster during training.

[0085] Each element is calculated using the following formula:

[0086]

[0087] in The detection result is defined in Equation 4, where H is the linear observation model. Let S be the average state of the m-th trajectory prediction, and S be the state update covariance matrix defined in Equation 6. We employ a two-stage training method on the neural network, first learning the deep feature distance... Then, the coefficient matrix (α, β) is learned to generate the final combined distance metric D.

[0088] (1) Deep feature distance: An N×M distance map is obtained by training the network from N detection features and M trajectory features:

[0089]

[0090] in Indicates the detection features Trajectory features First, a convolutional layer with a 3×3 kernel and 256 output channels is passed through the input, then activated by the ReLU function, and finally passed through an MLP layer to obtain the depth feature distance. The network is trained using binary cross-entropy loss.

[0091]

[0092] Where K is a matrix used to supervise the matching, where 0 indicates that the detection result matches the tracking trajectory feature, and 1 indicates that the detection result does not match the tracking trajectory feature. Since each trajectory detection pair does not have a target actual position annotation, if the nearest actual position of the tracking box in the previous frame and the nearest actual position of the detection box in the current frame belong to the same target identifier, and their two-dimensional center Euclidean distances to the nearest target actual position are both less than 2 meters, then the pair is considered a match.

[0093] (2) Combination coefficient: First, the depth feature distance is combined. Keeping the coefficient matrix constant, the remaining part of the distance combination module is then trained to learn the coefficient matrix. and This allows them to adjust the final distance D based on the importance of the distance to each depth feature.

[0094]

[0095] in Indicates and Similar convolutional layers, activation functions, and MLP layers are used. During testing, a greedy matching algorithm is employed to associate the data once the combined distance is calculated.

[0096] S1053: The target state is estimated by using the Kalman filter to evaluate the tracking result of the previous time step, and then the target is tracked by combining the greedy matching algorithm to perform pairing update and obtain the tracking result of the current time step.

[0097] Based on the detection, a Kalman filter is used to estimate the target state, with the state of each target represented by 11 variables: ,in It is the center position of the three-dimensional bounding box of the target object; It is the angle between the target's orientation and the x-axis, i.e., the orientation angle; This represents the current frame t and the previous frame t-1. The difference between them. Since construction machinery is low-speed machinery, a linear motion model is selected for tracking dynamic targets, assuming that the linear velocity, angular velocity, and target size are constant, that is, they will not change in the prediction step. According to the standard Kalman filter formula, this embodiment defines the prediction step as:

[0098]

[0099]

[0100] in It is the estimated average value of the actual state s at time t. This is the average predicted state at time t+1. Matrix A is the state transition matrix of the process model. It is the state covariance at time t. It is the predicted state covariance at time t+1. Matrix Q is the process model noise covariance.

[0101] A 3D object detector is used to feed observations into the Kalman filter. Each frame of 3D object detection results consists of a set of 3D bounding boxes, each represented by 9 variables: ,in These are the center position, orientation, and size of the bounding box; the remaining two variables... Indicates the time between the current frame and the previous frame. The difference. These two values ​​can be derived by multiplying the detector's estimated center velocity by the duration between two consecutive frames. We use a linear observation model H with additive Gaussian noise having zero mean and noise covariance R, and use this observation model and the predicted object state. We can predict the next measurement that represents the uncertainty in predicting object detection. and innovation covariance :

[0102]

[0103]

[0104] The noise covariance matrices Q and R of the process model and the observation model are estimated from the statistical data of the training set.

[0105] S1054, based on the tracking initialization module, performs tracking initialization on input unmatched detection results whose confidence scores are greater than a threshold.

[0106] Tracking trajectory initialization management is a crucial component of multi-target tracking systems. Most existing methods initialize a new trajectory for each mismatched detection or create a temporary trajectory and wait for a constant number of consecutive matches to convert the temporary trajectory into a complete one. Unlike previous heuristic methods, this invention transforms the trajectory initialization task into a binary classification task. The trajectory initialization module takes the unmatched detection result F as input, passes it through a convolutional layer, a multilayer perceptron, and a sigmoid activation function, and outputs a confidence score P. Whether trajectory initialization should proceed is determined based on whether the confidence score exceeds a set threshold of 0.5. This trajectory initialization module helps the tracking system reduce the number of false positive trajectories.

[0107] In summary, the embodiments of the present invention have at least one of the following beneficial effects:

[0108] 1. Design and Deployment of Data Acquisition Platform: To address the perception of surrounding environmental information by unmanned excavators, this invention designs a point cloud data acquisition platform. Unlike previous methods where LiDAR was installed on the top of the cab, this platform features a sensor bracket installed at the front of the excavator. This bracket is well-suited for tracked and wheeled excavators, replacing the existing radar installation scheme. This allows for better scanning of the more crucial environmental information ahead and the excavator bucket's operational information. Due to the complex operating environment of construction machinery, a 128-line LiDAR is selected to capture richer object details, improving the reliability of environmental perception and enhancing the effectiveness and safety of subsequent planning and control.

[0109] 2. Creation of a dedicated dataset: The environmental semantic information of construction machinery operation scenarios differs from that of passenger vehicles. Therefore, the creation of a dataset is essential for the detection and tracking of dynamic objects using deep learning. A detection and tracking dataset containing categories such as construction machinery, pedestrians, handcarts, piles of dirt, and roadblocks was created for unstructured road scenarios of construction machinery, ensuring the reliability and robustness of network training.

[0110] 3. Target Detection and Tracking: Addressing the limitation of cameras as sensors in engineering machinery operation environments, this invention utilizes LiDAR as an environmental perception sensor. This leverages LiDAR's advantages of being unaffected by light and its stable operation, meeting the requirements for 24-hour operation of unmanned excavators. Secondly, recognizing the significant workload and slow execution time of conventional voxelization processing, which cannot guarantee real-time performance in real-world excavator deployment experiments, this invention employs columnar voxelization feature encoding. The 3D point cloud features are then mapped onto a 2D bird's-eye view before 2D convolutional feature extraction, improving the algorithm's real-time performance. To address the issue of detecting dynamic targets with varying poses, this invention uses a two-stage feature regression based on the center point, improving the detection of orientation information and ground clearance, facilitating subsequent trajectory tracking. Finally, addressing the poor performance of most algorithms using 3D intersection-union ratio (IoU) as a metric, this invention uses Mahalanobis distance as the metric. This is combined with depth feature distance obtained during training and applied to a greedy matching algorithm for data association. Finally, a Kalman filter is used for state estimation and updating, achieving accurate tracking of multiple dynamic targets.

[0111] Please see Figure 4 The second embodiment of the present invention also provides a dynamic target detection and tracking device for an unmanned excavator, comprising:

[0112] The data production unit 210 is used to collect point cloud data of unmanned excavators through a pre-built data acquisition platform, and to produce a target detection and tracking dataset that conforms to the operation scenario of construction machinery according to the point cloud annotation specifications.

[0113] The encoding unit 220 is used to encode the point cloud data using a columnar unit encoding method to obtain a first dense tensor, and to generate a pseudo image of a two-dimensional bird's-eye view based on the first dense tensor.

[0114] Feature extraction unit 230 is used to extract features from the pseudo image using two-dimensional convolution to obtain a dense point cloud feature map;

[0115] The first-stage feature regression unit 240 is used to perform a heatmap-based center point localization method on the dense point cloud feature map. The center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, the first-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box.

[0116] The target detection and tracking unit 250 is used to perform target detection and tracking on the unmanned excavator based on the feature information of the target detection bounding box.

[0117] The third embodiment of the present invention also provides a dynamic target detection and tracking device for an unmanned excavator, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the dynamic target detection and tracking method for an unmanned excavator as described above.

[0118] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dynamic target detection and tracking method for an unmanned excavator, characterized in that, Includes the following steps: Point cloud data of unmanned excavators is collected through a pre-built data acquisition platform, and a target detection and tracking dataset that conforms to the operation scenario of construction machinery is created according to the point cloud annotation specifications. The point cloud data is encoded using a columnar unit encoding method to obtain a first dense tensor, and a pseudo-image of a two-dimensional bird's-eye view is generated based on the first dense tensor. The pseudo-image is used to extract features by two-dimensional convolution to obtain a dense point cloud feature map; For the dense point cloud feature map, a center point localization method based on heatmap is adopted. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box. The unmanned excavator performs target detection and tracking based on the feature information of the target detection bounding box; specifically, this includes: A second-stage feature regression is performed on the characteristic information of the first stage to obtain the refined features of the target detection bounding box and the confidence of the detected target. The current target detection result and the previous tracking result are input into a trainable distance combination module to obtain the depth feature distance, which is then combined with the Mahalanobis distance. Finally, a greedy matching algorithm is applied to the combined distance of data association. The target state is estimated by using the Kalman filter based on the tracking result of the previous time step, and then the target is tracked by combining the greedy matching algorithm for pairing and updating, so as to obtain the tracking result at the current time step. Based on the tracking initialization module, tracking initialization is performed on input unmatched detection results whose confidence scores are greater than a threshold; Specifically, a second-stage feature regression is performed on the characteristic information from the first stage to obtain the refined features of the target detection bounding box and the confidence score of the detected target. This includes: Based on the feature information of the target detection bounding box, for the three-dimensional center point of the selected surface of the prediction box, the corresponding features are extracted from the bird's-eye view using bilinear interpolation, and then connected to obtain the feature vector of the target detection bounding box. The data is refined through a multilayer perceptron, resulting in a refined bounding box size and orientation prediction parameter for the first-stage target detection boundary, as well as a final confidence score prediction.

2. The dynamic target detection and tracking method for unmanned excavators according to claim 1, characterized in that, The data acquisition platform includes a horizontal support bracket equipped with a lidar, an on-board computing platform, and a combined inertial navigation system. The horizontal support bracket is installed in the center of the front grille of the excavator and has reserved holes for installing camera sensors. The on-board computing platform and the combined inertial navigation system are installed on the top of the excavator.

3. The dynamic target detection and tracking method for unmanned excavators according to claim 1, characterized in that, Point cloud data of unmanned excavators is collected through a pre-built data acquisition platform, and a target detection and tracking dataset conforming to the operation scenario of construction machinery is created according to the point cloud annotation specifications. Specifically, it includes: In a specified work scenario, an unmanned excavator is driven along different routes to collect multiple data segments. Frame extraction is then used to process the data. Based on the point cloud annotation specifications, corresponding semantic categories are set according to the engineering work scenario to create the target detection and tracking dataset required for network training. When annotating according to the point cloud annotation specifications, corresponding semantic categories are set according to the engineering work scenario, namely: construction machinery, pedestrians, handcarts, piles of soil, and obstacles.

4. The dynamic target detection and tracking method for unmanned excavators according to claim 2, characterized in that, The point cloud data is voxel-encoded according to the columnar unit encoding method to obtain a first dense tensor, and a two-dimensional bird's-eye view pseudo-image is generated based on the first dense tensor, specifically including: The large-scale irregular 3D point cloud data is divided into various columnar cells according to the columnar cell encoding method. A sampling threshold is set for the columnar cells. Points with a value greater than the sampling threshold are sampled using a random sampling method. Points with a value less than the sampling threshold are padded with zeros to obtain downsampled columnar 3D point cloud data. Data augmentation is performed on the columnar 3D point cloud data, expanding the original four-dimensional information {x, y, z, i} of the 3D point cloud data to nine-dimensional information {x, y, z, i, x}. c y c , z c x p y p }, where {x, y, z} are the spatial coordinates of the point, i is the reflectance, and {x c y c , z c } represents the offset from the point to the average coordinates of all points in the column cell containing that point, {x p y p } represents the offset in X and Y coordinates from the center point of the column cell containing that point; The augmented columnar 3D point cloud data is feature-encoded frame by frame into a first dense tensor with dimension D, number of columnar units P, and number of point clouds in columnar units N. The first dense tensor is activated using an activation function to generate a second dense tensor with C channels, P bars, and N points in each bar. Then, max pooling is performed on each bar cell to obtain a third dense tensor with dimensions (C, P). Finally, a pseudo-image (C, H, W) with width W, height H, and number of categories C is generated from a bird's-eye view using a stacking method.

5. The dynamic target detection and tracking method for unmanned excavators according to claim 1, characterized in that, For the dense point cloud feature map, a heatmap-based center point localization method is used. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box, specifically including: A heatmap-based localization method is used to locate the center point. Feature mapping is performed on the dense point cloud feature map to expand the Gaussian peak value of the center rendering of each ground real object, reset the Gaussian radius, and obtain the center point location based on the Gaussian distribution function. Then, a one-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box. The feature information includes center heatmap information, center point offset, target size, ground height, target orientation, and target speed.

6. A dynamic target detection and tracking device for an unmanned excavator, characterized in that, include: The data production unit is used to collect point cloud data of unmanned excavators through a pre-built data acquisition platform, and to produce target detection and tracking datasets that conform to the operation scenarios of construction machinery according to point cloud annotation specifications. The encoding unit is used to encode the point cloud data using a columnar unit encoding method to obtain a first dense tensor, and to generate a pseudo image with a two-dimensional bird's-eye view based on the first dense tensor. The feature extraction unit is used to extract features from the pseudo image using two-dimensional convolution to obtain a dense point cloud feature map. The first-stage feature regression unit is used to locate the center point of the dense point cloud feature map using a heatmap-based center point localization method. The position of the center point is determined by combining the minimum Gaussian radius value with the Gaussian distribution function. Then, the first-stage feature regression is performed on the center point to obtain the feature information of the target detection bounding box. The target detection and tracking unit is used to detect and track targets on the unmanned excavator based on the feature information of the target detection bounding box; specifically, the target detection and tracking unit is used to perform two-stage feature regression on the feature information of the first stage to obtain the refined features of the target detection bounding box and obtain the confidence of the detected target. The current target detection result and the previous tracking result are input into a trainable distance combination module to obtain the depth feature distance, which is then combined with the Mahalanobis distance. Finally, a greedy matching algorithm is applied to the combined distance of data association. The target state is estimated by using the Kalman filter based on the tracking result of the previous time step, and then the target is tracked by combining the greedy matching algorithm for pairing and updating, so as to obtain the tracking result at the current time step. Based on the tracking initialization module, tracking initialization is performed on input unmatched detection results whose confidence scores are greater than a threshold; Specifically, a second-stage feature regression is performed on the characteristic information from the first stage to obtain the refined features of the target detection bounding box and the confidence score of the detected target. This includes: Based on the feature information of the target detection bounding box, for the three-dimensional center point of the selected surface of the prediction box, the corresponding features are extracted from the bird's-eye view using bilinear interpolation, and then connected to obtain the feature vector of the target detection bounding box. The data is refined through a multilayer perceptron, resulting in a refined bounding box size and orientation prediction parameter for the first-stage target detection boundary, as well as a final confidence score prediction.

7. A dynamic target detection and tracking device for an unmanned excavator, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the dynamic target detection and tracking method for unmanned excavators as described in any one of claims 1 to 5.

Images