Mine multi-target tracking method based on multi-scale attention and graph neural network

Abstract

This invention discloses a multi-target tracking method for mining areas based on multi-scale attention and graph neural networks, belonging to the field of environmental perception for unmanned driving in mining areas. The method includes: multimodal perception and feature enhancement, employing a multi-scale attention mechanism for spatial, channel, and multi-scale adaptive weighting; target detection and re-identification feature extraction; temporal-spatial joint modeling, constructing a cross-frame target spatiotemporal map, and obtaining a spatiotemporal correlation probability matrix through graph neural network inference; obtaining a comprehensive correlation cost matrix based on mining area scene knowledge constraints, and using a knowledge base including static geography, dynamic operations, and equipment characteristics for hard constraint filtering and soft constraint optimization; trajectory lifecycle management, outputting a stable trajectory. This invention enhances feature robustness through multi-scale attention, achieves global spatiotemporal correlation through graph neural networks, and improves decision rationality through scene knowledge constraints, effectively solving the problem of target loss due to occlusion in complex mining environments.

Description

Technical Field

[0001] This invention belongs to the field of environmental perception technology for unmanned driving in open-pit mines. It is adapted to multi-target tracking scenarios of unmanned mining vehicles and solves the technical challenge of stable tracking of various targets such as mining trucks, personnel, falling rocks, and small instruments in mining operation environments under conditions of dust, variable lighting, and unstructured roads. It involves technologies such as multi-sensor fusion, deep learning, graph neural networks, and scene knowledge engineering. It can be directly applied to the target tracking module of unmanned driving systems in mining areas, providing reliable target trajectory support for vehicle decision-making and control. Background Technology

[0002] The multi-target tracking component of unmanned driving systems in open-pit mines is directly related to driving safety and operational efficiency. Its core task is to continuously, stably, and accurately maintain the identities and track the trajectories of various targets, such as dynamic mining vehicles, workers, and small equipment. However, the unique characteristics of mining areas, such as high dust levels, strong vibrations, dramatic changes in lighting, and unstructured road conditions, make it difficult to directly adapt general multi-target tracking technologies. This is mainly reflected in the following three bottlenecks: First, feature extraction lacks robustness in harsh environments. Existing methods rely on feature extraction networks typically designed for general scenarios and are not optimized for conditions such as dust, strong light, and low illumination in mining areas. This leads to distortion of target appearance features, decreased reliability of re-identification, and a tendency for target loss during tracking. Furthermore, single-scale features struggle to account for the differences between large nearby targets and small distant targets, affecting the balance of multi-scale tracking.

[0003] Second, the spatiotemporal correlation and trajectory continuity modeling are weak. Most mainstream tracking methods adopt a frame-by-frame matching strategy, lacking modeling of global spatiotemporal information. In complex scenarios such as occlusion, intersection, and interaction, this can easily lead to ID identity switching, trajectory breakage, or merging errors. At the same time, the commonly used uniform speed / uniform acceleration model fails to adapt to the dynamic differences between empty and fully loaded mining trucks and the sudden changes in motion on unstructured roads, resulting in inaccurate motion prediction and decreased correlation accuracy.

[0004] Third, there is a lack of utilization of prior knowledge in mining areas. Most general tracking algorithms are purely data-driven and do not incorporate prior information such as high-precision maps of mining areas, work processes, and the physical limits of equipment. This may lead to trajectories that violate physical laws or operational logic, such as traversing restricted areas or speeding. This not only reduces the reliability of the results but may also introduce safety hazards and hinder the implementation of projects.

[0005] In related technologies, existing technologies also have obvious limitations. For example, Chinese patent application CN115792891A, "Target Tracking Method Based on Fusion of Multi-millimeter Wave Radar and LiDAR," although it expands the field of view through multi-radar fusion and improves stability by utilizing identity ID tuples, still mainly relies on traditional filtering and rule matching, lacking deep fusion of visual information. Its ability to distinguish and continuously track targets such as pedestrians is limited in scenarios with changing lighting and missing textures, and it is prone to association failure under severe occlusion. Another Chinese patent application, CN117237887A, "A Multi-Sensor Fusion Target Detection Method and System under Sandstorm Weather," focuses on target detection under sandstorm conditions. It improves detection robustness through multi-modal feature fusion and attention mechanisms, but it does not construct a cross-frame temporal association model and lacks the ability to model the continuity of target motion. Therefore, it cannot be directly applied to multi-target tracking tasks that require maintaining identity consistency and handling occlusion recurrence.

[0006] In summary, existing technologies for multi-target tracking in mining areas generally suffer from insufficient feature robustness, weak spatiotemporal correlation capabilities, and limited utilization of scene knowledge, making it difficult to meet the comprehensive requirements of high precision, high robustness, and high availability. Therefore, there is an urgent need for a dedicated multi-target tracking solution for mining areas that can balance environmental adaptability, trajectory continuity, and decision-making rationality. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a multi-target tracking method for mining areas based on multi-scale attention and graph neural networks, which solves at least the following technical problems in the prior art: it cannot adapt to the complex environment of mining areas, and cannot achieve stable feature extraction of multi-scale targets under conditions of dust and varying lighting; it cannot adapt to multi-target occlusion and identity ID switching problems, resulting in discontinuous trajectory tracking; and the rationality and engineering adaptability of tracking decisions are poor.

[0008] This invention discloses a multi-target tracking method for mining areas based on multi-scale attention and graph neural networks, comprising the following steps: Step S1, Multimodal perception and feature enhancement: Acquire raw data from multiple sensors and perform spatiotemporal alignment. Use a multi-scale attention mechanism to perform adaptive feature weighting on the aligned multimodal data stream at spatial, channel and multi-scale levels, and output the enhanced target feature representation. Step S2, Target Detection and Re-identification Feature Extraction: Based on the enhanced target feature representation, multi-category targets are detected and target detection boxes are generated, while re-identification feature vectors of each target are extracted; Step S3, Temporal-spatial joint modeling: Based on the target detection box and the re-identification feature vector, a cross-frame target spatiotemporal graph is constructed. The nodes of the spatiotemporal graph fuse the target's appearance features, motion state and category information. Message passing and feature updates between nodes are performed through a graph neural network, and the spatiotemporal correlation probability matrix between targets is obtained through inference. Step S4, obtain the comprehensive association cost matrix based on mining area scene knowledge constraints: construct a scene prior knowledge base containing static geographical knowledge, dynamic operation knowledge and equipment characteristic knowledge of the mining area, and use the scene prior knowledge base to perform hard constraint filtering and soft constraint optimization on the spatiotemporal association probability matrix to generate a comprehensive association cost matrix that integrates data association degree and knowledge conformity degree. Step S5, Trajectory Lifecycle Management: Based on the comprehensive correlation cost matrix, detection-trajectory matching is performed, and the matching scheme is verified for new trajectory initialization, updated for existing trajectories, terminated for disappearing trajectories, and finally outputs stable multi-target trajectory information.

[0009] Optionally, the multi-scale attention mechanism in step S1 includes: For image data, channel attention submodules and spatial attention submodules are integrated in the backbone convolutional network to achieve adaptive feature calibration in a sequential manner; The level of environmental interference is dynamically estimated by analyzing the image's sharpness and brightness information, and the sensitivity of the attention mechanism is adjusted according to the level of environmental interference. After voxelization of the lidar point cloud, contextual information of voxels in the local neighborhood is fused using a self-attention-based feature aggregation network.

[0010] Optionally, the specific steps for constructing the cross-frame target spatiotemporal graph in step S3 include: Define a target node set V, where each node v∈V corresponds to a target detected in a frame of an image. The node features are fused with the target’s appearance features, motion state and position information. Define an edge set E, where the edge e∈E includes intra-frame spatial edges connecting different targets within the same frame, inter-frame temporal edges connecting the same or different targets between adjacent frames, and semantic edges connecting targets with the same category attributes. A graph attention network is used for message passing and node feature updating between target nodes. A multi-head attention mechanism is used to aggregate neighborhood information and infer the spatiotemporal correlation probability matrix between target nodes.

[0011] Optionally, step S3 further includes multi-target motion state estimation: using a constant turning rate and velocity model (CTRV) to describe the motion of the target on an unstructured road, and combining an adaptive Kalman filter to fuse the lidar point cloud position observation and millimeter-wave radar velocity observation of the current frame to estimate and predict the position and velocity state of the target, and using the estimated motion state as the motion feature of the node in the spatiotemporal graph.

[0012] Optionally, the scenario prior knowledge base in step S4 specifically includes: Static geographic knowledge, including high-precision vector maps of mining areas, drivable areas, and dangerous restricted zones polygons; Dynamic operation knowledge, including mining operation process models based on finite state machines; Equipment characteristic knowledge, including the kinematic limit parameters and safety rules for various tracked targets.

[0013] Optionally, the hard constraint filtering in step S4 includes: based on the static geographical knowledge and device characteristic knowledge, directly rejecting any associated assumptions that would cause the trajectory to cross restricted areas or violate physical motion limits; The soft constraint optimization includes: based on the dynamic operation knowledge, verifying whether the trajectory state transition conforms to the operation process logic; if the association result violates the consistency of the operation process, then applying a decay to the corresponding association probability, thereby obtaining a knowledge-enhanced association decision result.

[0014] Optionally, step S5 specifically includes: Using the comprehensive correlation cost matrix as input, the Hungarian algorithm is used to solve the optimal bipartite graph matching between the current detected target and the existing trajectory, and the matching scheme with the minimum cost is found by adjusting the bit matrix; Based on the matching scheme, a new trajectory is initialized for the new unmatched detection target, the state and appearance model of the existing trajectory is updated using the matching detection results, motion prediction and maintenance are performed on the existing unmatched trajectory, and the trajectory that has been unmatched for a long time is terminated.

[0015] Optionally, the re-identification feature extraction in step S2 includes: taking the region corresponding to the target detection box in the image as input, using a re-identification network based on the ResNet backbone and employing a partial convolutional baseline structure to extract re-identification feature vectors, and performing L2 normalization on the extracted vectors.

[0016] Optionally, step S5 further includes monitoring trajectory quality indicators, which include ID switching frequency, average trajectory confidence, and trajectory continuity duration.

[0017] The present invention also discloses a multi-target tracking device for mining areas based on multi-scale attention and graph neural networks for implementing the aforementioned method, comprising: The sensor array, including at least a visible light camera, lidar, millimeter-wave radar, and integrated navigation system, is used to collect raw multi-sensor data of the mining area environment. An onboard edge computing unit is connected to the sensor group and has an algorithm software stack deployed thereon, the algorithm software stack including: The feature enhancement module is used for multimodal perception and feature enhancement. The detection and re-identification module is used for target detection and re-identification feature extraction. The spatiotemporal graph modeling module is used to obtain the spatiotemporal correlation probability matrix between targets; The knowledge constraint association module has a built-in scenario prior knowledge base, which is used to obtain a comprehensive association cost matrix based on the mining area scenario knowledge constraints. The trajectory management module is used for trajectory lifecycle management and ultimately outputs stable multi-target trajectory information. The output interface is used to transmit multi-target trajectory information to the main autonomous driving system of the mining truck.

[0018] The present invention revolves around three dimensions: "feature perception enhancement, temporal-spatial joint modeling, and mining area scene knowledge constraints," forming a synergistic and complementary technical system. Compared with existing technologies, the present invention has at least the following beneficial effects: 1. The multi-scale attention-driven feature perception enhancement mechanism of this invention: Based on the traditional feature extraction network, a three-level attention collaborative architecture of space-channel-multi-scale is designed, which can adaptively focus on key areas such as road areas and dynamic target areas, and enhance core feature channels such as edges and textures. At the same time, dust concentration perception and illumination condition adaptive sub-modules are integrated to dynamically adjust attention parameters, specifically solve the feature distortion problem in the harsh environment of mining areas, realize stable feature extraction of multi-scale targets, and improve the robustness of feature perception.

[0019] 2. The graph neural network-enabled global spatiotemporal joint modeling method of this invention transforms multi-target tracking into a spatiotemporal graph optimization problem, defines node attributes including appearance, motion, and spatial features, and constructs multi-dimensional connection relationships of temporal edges, spatial edges, and semantic edges. Through a three-layer network architecture, it respectively realizes intra-frame spatial relationship modeling, short-temporal association modeling, and long-temporal association modeling, and combines association loss and trajectory continuity loss to ensure tracking stability, effectively solving industry pain points such as target occlusion, frequent ID switching, and trajectory breakage.

[0020] 3. The multi-source knowledge fusion mining scene constraint system of this invention: A multi-source knowledge base is constructed, encompassing static geographical knowledge, dynamic operational knowledge, equipment characteristic knowledge, and empirical rule knowledge. A hierarchical system of hard and soft constraints is designed. Hard constraints ensure that trajectories conform to regional restricted areas, physical laws, and equipment limit requirements, while soft constraints optimize the consistency of operational processes and the rationality of behavior. Simultaneously, it possesses online learning and multi-mining area knowledge transfer capabilities, significantly improving scene adaptability and engineering practicality. Attached Figure Description

[0021] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] Figure 1 This is a schematic diagram of the hardware installation of the tracking system of the present invention; Figure 2This is a flowchart of the multi-target tracking method in mining areas based on multi-scale attention and graph neural networks of the present invention; Figure 3 This is a structural diagram of the multi-scale attention feature enhancement module of the present invention; Figure 4 This is a diagram of the spatiotemporal graph neural network modeling structure of the present invention; Figure 5 This is a structural diagram of the mining area scene knowledge constraint module of the present invention. Detailed Implementation

[0023] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0024] This invention uses a closed-loop logic of "multimodal data input → feature enhancement → target detection and feature extraction → spatiotemporal joint modeling → knowledge constraint optimization → tracking result output" to achieve stable tracking of multiple targets in mining areas through five core stages. The input and output of each stage are clear, the modules are independent and coordinated, and the overall process is reproducible and easy to be engineered.

[0025] A specific embodiment of the present invention, such as Figure 1-5 This paper discloses a multi-target tracking method for mining areas based on multi-scale attention and graph neural networks, which specifically includes the following steps: S1 Multimodal Perception and Feature Enhancement: Input raw multi-sensor data from the entire open-pit mine scene (mining face, main transport road, spoil heap, etc.), and complete the timestamp alignment and coordinate system unification of camera images, LiDAR point clouds, millimeter-wave radar data, and GPS / IMU positioning information through the multi-sensor data acquisition and synchronization module, outputting a standardized multimodal data stream; this data stream is then processed by a feature perception enhancement processing module using a multi-scale attention mechanism to perform adaptive feature weighting at the spatial, channel, and multi-scale levels, strengthening key target features and suppressing environmental interference such as dust and low light. Finally, in addition to the spatiotemporally aligned multimodal data stream, a robust enhanced target feature representation is also output.

[0026] S1.1 Multi-sensor data acquisition and synchronization: such as Figure 1As shown, a multimodal sensing array is constructed using a high-resolution visible light camera, a multi-beam lidar, a mid-to-long-range millimeter-wave radar, and a high-precision integrated navigation system, strategically positioned around the mine truck to ensure comprehensive field of view. The system employs a combination of hardware triggering and software timestamps for data synchronization, aligning data streams based on a unified clock source. By constructing a time synchronization buffer, using the multi-beam lidar frame rate as a reference, frame alignment and packaging of multimodal data with time deviations within a preset range are performed. Simultaneously, pre-calibrated sensor extrinsic parameters are used to unify all data to the lidar coordinate system.

[0027] Furthermore, the buffer serves as a data caching mechanism for the multi-sensor data acquisition and synchronization module. It maintains a timestamped data queue for each sensor and aligns and packages the modal data using the LiDAR as a reference.

[0028] Furthermore, the points in the multi-beam lidar point cloud Its projection point on a visible light camera image The expression is:

[0029] in, This is the intrinsic parameter matrix of the visible light camera; The extrinsic parameter matrix for lidar to visible light camera is used to establish a cross-modal spatial correspondence. Similarly, through the pre-calibrated extrinsic parameter matrix, the detection points of millimeter-wave radar (such as position and velocity vectors) and the pose calculated by the high-precision integrated navigation system GPS / IMU are uniformly transformed to the lidar coordinate system, completing the spatial alignment of all modal data.

[0030] S1.2 Feature-aware enhancement processing to obtain robust multimodal feature representations: To improve the robustness of features in the harsh environment of the mining area, adaptive enhancement is performed on image and point cloud data. For image data, a convolutional block attention mechanism is introduced into the backbone convolutional network. This mechanism sequentially integrates channel attention and spatial attention sub-modules, automatically focusing on key regions and channels. Simultaneously, by analyzing the image's sharpness and brightness information, the level of environmental interference is dynamically estimated, and the sensitivity of the attention mechanism is adjusted accordingly.

[0031] For example, the key regions are: the spatial attention module focuses on the image regions where the target may appear, such as roads and mining faces, and suppresses non-key backgrounds; the key channels are: the channel attention module learns autonomously and enhances the weights of feature channels related to mining targets (such as yellow paint on vehicles and reflective stripes on safety clothing).

[0032] like Figure 3As shown, for lidar point clouds, voxelization is first performed, and then a self-attention-based feature aggregation network (Point Transformer) is used to fuse the contextual information of voxels in the local neighborhood (defined by ball query with a query radius of 0.3 meters) to enhance the feature representation of sparse or irregular targets.

[0033] The S2 target detection and re-identification feature extraction module takes the robust multimodal feature representation obtained from S1.2 as input and identifies targets such as mining trucks, engineering vehicles, personnel, and equipment by the multi-class target detection module, generating 2D / 3D target detection boxes and calibrating the confidence level. Then, the re-identification feature extraction module extracts appearance feature vectors that are robust to viewpoint, lighting, and occlusion, completes feature normalization and dimensionality reduction, and finally outputs the target detection results (target detection boxes) including 3D target bounding boxes, class labels, detection confidence levels, and target re-identification feature vectors, providing support for subsequent target association.

[0034] This step receives the enhanced, robust target feature representation, completes the 3D localization of the target and the extraction of fine appearance features, and provides a basic target state for tracking.

[0035] S2.1 Multi-class Target Detection: A 3D target detection framework based on the fusion of vision and LiDAR is adopted. The image semantic information in the robustly enhanced target feature representation is pre-fused with the LiDAR point cloud to form a semantically enhanced point cloud. This enhanced point cloud is then input into a voxel-based 3D detection network (PointPillars) to directly regress the 3D bounding boxes, categories, and confidence scores of various targets (such as mine carts, personnel, and falling rocks) in the open-pit mine scene.

[0036] Furthermore, to improve the reliability of the results, the confidence level of the 3D detection network output is calibrated by temperature scaling.

[0037] S2.2 Re-identification Feature Extraction: To reliably identify the same target during tracking, this invention extracts a depth appearance feature vector robust to viewpoint and occlusion. This depth appearance feature vector is automatically learned by the network through metric learning and can capture the key semantic information of the target. Taking the target's two-dimensional bounding box region detected and projected onto image coordinates in S2.1 as input, a re-identification network based on a ResNet backbone and employing a partial convolutional baseline structure is used to extract the re-identification feature vector; the extracted re-identification feature vector is then L2 normalized and used for subsequent association.

[0038] Furthermore, the network is trained and optimized using metric learning (such as using triplet loss) to make features of the same target as close as possible in the embedding space.

[0039] S3 Temporal-Spatial Joint Modeling: Input the target detection results, re-identified feature vectors, and historical trajectory states obtained in step 2. First, through the multi-target motion state estimation module, an adaptive Kalman filter combining a mining area unstructured road motion model (CTRV) is used to estimate the target's position, velocity, and other states. Then, through the spatiotemporal graph neural network modeling module, a cross-frame target spatiotemporal graph is constructed, and spatiotemporal context information is fused using node message passing to complete global association consistency optimization. Finally, the target motion state estimation results and spatiotemporal association probability matrix are output, effectively improving trajectory continuity.

[0040] Each node represents a target detected in each frame. Each node incorporates the target's appearance features, motion state, and category information.

[0041] like Figure 4 As shown, this step is the core of the tracking system, aiming to build a global correlation between targets across frames and solve the problem of trajectory continuity in complex scenarios such as occlusion and interaction.

[0042] S3.1 Multi-target motion state estimation: The constant steering rate and velocity (CTRV) model, suitable for unstructured roads, is used to describe target motion. This model accurately depicts the motion patterns of typical targets in mining areas, such as mining trucks, workers, and small machinery, through two core parameters: constant steering rate and constant velocity. This overcomes the shortcomings of traditional uniform speed / uniform acceleration models, which cannot adapt to the complex motion characteristics of targets in mining areas. Combined with an adaptive Kalman filter, the predicted target motion state output from the CTRV model is used as a basis. This is achieved by fusing position observations from the current frame's lidar point cloud and velocity observations from millimeter-wave radar. The process noise covariance matrix is ​​then adjusted online based on the prediction error covariance (reflecting the degree of deviation between the predicted and observed values), resulting in stable estimation and prediction of the target's position, velocity, and other states. The estimated state of this multi-target motion state will be used for motion matching in subsequent data association and will serve as the motion features of spatiotemporal graph nodes; it will also serve as trajectory prediction values ​​for data association in the next frame. Furthermore, it will be input into the spatiotemporal graph neural network in S3.2 as part of the node features.

[0043] Furthermore, after the prediction step, an adaptive Kalman filter is used to fuse the position observations from the current frame's lidar point cloud and the velocity observations from the millimeter-wave radar, thus entering the state update process. The inputs are the prior state, Kalman gain, observation values, and observation matrix; the output is the posterior state. The state update follows the standard Kalman filter formula:

[0044] Where K represents the Kth frame; K-1 represents the (K-1)th frame; This represents the posterior state estimate; This represents the prior state estimate; Represents the observation matrix; For Kalman gain, This is the current observation value.

[0045] S3.2 Spatiotemporal correlation probability matrix modeling: The multi-target tracking problem is constructed as a spatiotemporal graph optimization problem.

[0046] Specifically, the spatiotemporal graph is constructed using the following expression:

[0047] in, Represents the set of target nodes, target nodes For each target detected in a frame, node features are fused with appearance, motion, and location information; Denotes the set of edges, edges Define the temporal, spatial, and semantic relationships between targets.

[0048] A graph attention network is used for message passing and feature updating between target nodes. A multi-head attention mechanism is used to aggregate neighborhood information and infer the correlation probability between target nodes at the global level.

[0049] Furthermore, the spatiotemporal correlation probability matrix is ​​constructed from the correlation probabilities between target nodes.

[0050] Furthermore, the model is trained by jointly optimizing appearance matching and motion continuity.

[0051] S4 obtains a comprehensive association cost matrix based on mining area scenario knowledge constraints: Input spatiotemporal association probability matrix, mining area map, operation process rules and historical statistical data, the scenario prior knowledge base module loads knowledge such as mining area road topology, drivable area, operation mode, etc., and outputs scenario constraints and prior probabilities; then, through the knowledge-guided data association module, the association cost is adjusted based on road constraints, the consistency of operation process is verified, abnormal association results are filtered, and finally, knowledge-enhanced association decision results are output to improve the rationality of tracking.

[0052] For example, when the operation process is a loading and unloading process, the operation process rules are: empty mine car - enters the loading area - loads - fully loaded - drives to the unloading area - unloads - empty.

[0053] The association cost is the association hypothesis between any two target nodes (which may belong to different frames) in the spatiotemporal association probability matrix output by S3.2.

[0054] like Figure 5As shown, this step transforms the prior knowledge of the mining area's unique scenario into computable constraint rules, which are used to correct and optimize the association results of purely data-driven (data generated by the S1-S3 algorithms without knowledge correction, i.e., the initial association probability matrix output by S3.2 and the motion state estimate output by S3.1).

[0055] S4.1 Construction of the scenario prior knowledge base.

[0056] Preferably, the scenario prior knowledge base is a structured scenario prior knowledge base module, mainly including: 1) Static geographic knowledge (i.e., road constraints): such as high-precision vector mining area maps, drivable areas, and dangerous restricted area polygons; 2) Dynamic job knowledge: such as job flow models based on finite state machines; 3) Equipment characteristics knowledge: such as the kinematic limits and safety rules of various types of equipment.

[0057] The various types of equipment include the mining equipment being tracked, such as mining trucks, excavators, bulldozers, and water trucks, as well as personnel.

[0058] S4.2 Knowledge-guided data association: Based on the above-mentioned structured mining area scene knowledge base, the initial association probability matrix output by the spatiotemporal graph network is verified and corrected.

[0059] Furthermore, knowledge-guided data association includes hard constraint filtering and soft constraint optimization. Hard constraint filtering directly rejects any association assumptions that would cause the trajectory to cross restricted areas or violate physical limits. Soft constraint optimization applies rule-based probabilistic corrections to the association results, yielding knowledge-enhanced association decisions.

[0060] Understandably, the rule-based probability correction is based on specific rules called from the S4.1 scenario prior knowledge base. For example: prohibiting crossing fence polygons, maximum speed of minecarts is 40km / h, and the minecart must be fully loaded after loading.

[0061] Among them, the mining area map and drivable area in the structured mining area scenario knowledge base are used for hard constraint filtering and soft constraint optimization, the operation process rules are used for soft constraint optimization, and the trajectory state transition is verified to be logical; historical statistical data, such as the average speed range of mining trucks on a certain section, are used to define hard constraint filtering and soft constraints.

[0062] For example, if the association decision result of knowledge enhancement violates the consistency of the workflow, then a decay is applied to the association probability between its target nodes. Finally, a comprehensive association cost matrix that integrates data association degree and knowledge conformity degree is generated as input for trajectory matching.

[0063] S5 Trajectory Lifecycle Management: Inputting a comprehensive association cost matrix and historical trajectory data, the multi-target data association module calculates the cost matrix using a multi-dimensional approach and employs the Hungarian algorithm to complete detection-trajectory matching. The trajectory generation and maintenance module is then responsible for initializing and verifying new trajectories, updating existing trajectories, terminating disappearing trajectories, and simultaneously conducting trajectory quality assessment and confidence management. Finally, it outputs stable and continuous multi-target trajectory information and trajectory quality assessment results.

[0064] S5.1 Multi-target data association: Taking the comprehensive association cost matrix that incorporates knowledge constraints as input, the Hungarian algorithm is used to solve the optimal bipartite graph matching between the current detected target and the existing trajectory. This algorithm finds the matching scheme with the minimum cost by adjusting the bit matrix.

[0065] S5.2 Trajectory Generation and Maintenance: Manage the lifecycle of each trajectory according to the matching scheme; initialize trajectories for new targets; update the state and appearance models of matched trajectories; perform motion prediction and maintenance for unmatched trajectories; and terminate trajectories that have been lost for a long time. The system also monitors trajectory quality indicators (such as ID switching frequency, mean trajectory confidence, and trajectory continuity duration) to ensure the stability of the output trajectory.

[0066] Understandably, when a detected target in the current frame matches an existing trajectory, the system uses this detection result to update the trajectory's state and appearance model, while the original trajectory continues to exist and evolve. A target detected in the current frame that cannot be associated with any existing trajectory indicates the presence of a new target, generally due to the new target entering the scene for the first time or reappearing after a brief occlusion but failing to match. The system initializes a new trajectory for this target and assigns it initial state and appearance features.

[0067] Another embodiment of the present invention discloses a multi-target tracking system for mining areas, used to execute the aforementioned multi-target tracking method for mining areas. The system adopts a hierarchical modular architecture, consisting of a sensor group, an on-board edge computing unit, and an algorithm software stack. The software layer integrates all the above-mentioned algorithm modules, interacts with data through a high-speed on-board network, and communicates with the main autonomous driving system of the mining truck based on the robot operating system, realizing end-to-end real-time processing from multimodal perception to stable trajectory output.

[0068] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-target tracking method for mining areas based on multi-scale attention and graph neural networks, characterized in that, Includes the following steps: Step S1, Multimodal perception and feature enhancement: Acquire raw data from multiple sensors and perform spatiotemporal alignment. Use a multi-scale attention mechanism to perform adaptive feature weighting on the aligned multimodal data stream at spatial, channel and multi-scale levels, and output the enhanced target feature representation. Step S2, Target Detection and Re-identification Feature Extraction: Based on the enhanced target feature representation, multi-category targets are detected and target detection boxes are generated, while re-identification feature vectors of each target are extracted; Step S3, Temporal-spatial joint modeling: Based on the target detection box and the re-identification feature vector, a cross-frame target spatiotemporal graph is constructed. The nodes of the spatiotemporal graph fuse the target's appearance features, motion state and category information. Message passing and feature updates between nodes are performed through a graph neural network, and the spatiotemporal correlation probability matrix between targets is obtained through inference. Step S4, obtain the comprehensive association cost matrix based on mining area scene knowledge constraints: construct a scene prior knowledge base containing static geographical knowledge, dynamic operation knowledge and equipment characteristic knowledge of the mining area, and use the scene prior knowledge base to perform hard constraint filtering and soft constraint optimization on the spatiotemporal association probability matrix to generate a comprehensive association cost matrix that integrates data association degree and knowledge conformity degree. Step S5, Trajectory Lifecycle Management: Based on the comprehensive correlation cost matrix, detection-trajectory matching is performed, and the matching scheme is verified for new trajectory initialization, updated for existing trajectories, terminated for disappearing trajectories, and finally outputs stable multi-target trajectory information.

2. The method according to claim 1, characterized in that, The multi-scale attention mechanism in step S1 includes: For image data, channel attention submodules and spatial attention submodules are integrated in the backbone convolutional network to achieve adaptive feature calibration in a sequential manner; The level of environmental interference is dynamically estimated by analyzing the image's sharpness and brightness information, and the sensitivity of the attention mechanism is adjusted according to the level of environmental interference. After voxelization of the lidar point cloud, contextual information of voxels in the local neighborhood is fused using a self-attention-based feature aggregation network.

3. The method according to claim 1, characterized in that, The specific steps for constructing the cross-frame target spatiotemporal graph in step S3 include: Define a target node set V, where each node v∈V corresponds to a target detected in a frame of an image. The node features are fused with the target’s appearance features, motion state and position information. Define an edge set E, where the edge e∈E includes intra-frame spatial edges connecting different targets within the same frame, inter-frame temporal edges connecting the same or different targets between adjacent frames, and semantic edges connecting targets with the same category attributes. A graph attention network is used for message passing and node feature updating between target nodes. A multi-head attention mechanism is used to aggregate neighborhood information and infer the spatiotemporal correlation probability matrix between target nodes.

4. The method according to claim 1, characterized in that, Step S3 further includes multi-target motion state estimation: the constant turning rate and velocity model CTRV is used to describe the motion of the target on the unstructured road, and the position and velocity state of the target are estimated and predicted by combining the current frame's lidar point cloud position observation and millimeter-wave radar velocity observation with an adaptive Kalman filter, and the estimated motion state is used as the motion feature of the node in the spatiotemporal graph.

5. The method according to claim 1, characterized in that, The scenario prior knowledge base in step S4 specifically includes: Static geographic knowledge, including high-precision vector maps of mining areas, drivable areas, and dangerous restricted zones polygons; Dynamic operation knowledge, including mining operation process models based on finite state machines; Equipment characteristic knowledge, including the kinematic limit parameters and safety rules for various tracked targets.

6. The method according to claim 5, characterized in that, The hard constraint filtering in step S4 includes: based on the static geographical knowledge and equipment characteristic knowledge, directly rejecting any associated assumptions that would cause the trajectory to cross restricted areas or violate physical motion limits; The soft constraint optimization includes: based on the dynamic operation knowledge, verifying whether the trajectory state transition conforms to the operation process logic; if the association result violates the consistency of the operation process, then applying a decay to the corresponding association probability, thereby obtaining a knowledge-enhanced association decision result.

7. The method according to claim 1, characterized in that, Step S5 specifically includes: Using the comprehensive correlation cost matrix as input, the Hungarian algorithm is used to solve the optimal bipartite graph matching between the current detected target and the existing trajectory, and the matching scheme with the minimum cost is found by adjusting the bit matrix; Based on the matching scheme, a new trajectory is initialized for the new unmatched detection target, the state and appearance model of the existing trajectory is updated using the matching detection results, motion prediction and maintenance are performed on the existing unmatched trajectory, and the trajectory that has been unmatched for a long time is terminated.

8. The method according to claim 1, characterized in that, The re-identification feature extraction in step S2 includes: taking the region corresponding to the target detection box in the image as input, using a re-identification network based on the ResNet backbone and employing a partial convolutional baseline structure to extract re-identification feature vectors, and performing L2 normalization on the extracted vectors.

9. The method according to any one of claims 1-8, characterized in that, Step S5 further includes monitoring trajectory quality indicators, which include ID switching frequency, average trajectory confidence score, and trajectory continuity duration.

10. A multi-target tracking device for mining areas based on multi-scale attention and graph neural networks for implementing the method of any one of claims 1-9, characterized in that, include: The sensor array, including at least a visible light camera, lidar, millimeter-wave radar, and integrated navigation system, is used to collect raw multi-sensor data of the mining area environment. An onboard edge computing unit is connected to the sensor group and has an algorithm software stack deployed thereon, the algorithm software stack including: The feature enhancement module is used for multimodal perception and feature enhancement. The detection and re-identification module is used for target detection and re-identification feature extraction. The spatiotemporal graph modeling module is used to obtain the spatiotemporal correlation probability matrix between targets; The knowledge constraint association module has a built-in scenario prior knowledge base, which is used to obtain a comprehensive association cost matrix based on the mining area scenario knowledge constraints. The trajectory management module is used for trajectory lifecycle management and ultimately outputs stable multi-target trajectory information. The output interface is used to transmit multi-target trajectory information to the main autonomous driving system of the mining truck.

Images