Multi-source heterogeneous sensor fusion-based strip mine unmanned mine car positioning method and system

Abstract

The invention discloses a strip mine unmanned mine car positioning method and system based on multi-source heterogeneous sensor fusion, and relates to the technical field of unmanned mine car positioning, and a plurality of multi-source heterogeneous sensors are determined based on detection of a space area; the plurality of multi-source heterogeneous sensors are distributed at the periphery of the unmanned mine car and output corresponding sensing data; and the fusion data combination is determined based on the multiple pieces of sensing data, the corresponding data types and the overall form of the unmanned mine car, so that the accuracy of the fusion data combination is improved. Therefore, a prediction positioning event of the unmanned mine car is determined based on the fusion data combination, the current image of the unmanned mine car and the prediction range of the unmanned mine car relative to the space area; the multiple pieces of positioning information are determined based on detection of the predicted positioning event of the unmanned mine car, the current positioning position of the unmanned mine car is determined according to the multiple pieces of positioning information, the corresponding positioning dimensions and the overall form of the unmanned mine car, and the accuracy of the current positioning position of the unmanned mine car is improved.

Description

Technical Field

[0001] This invention relates to the technical field of unmanned mining truck positioning, and in particular to a method and system for positioning unmanned mining trucks in open-pit mines based on the fusion of multi-source heterogeneous sensors. Background Technology

[0002] With the development of technology, unmanned mining trucks are gradually being applied to mines, moving within the mine's internal transport channels. These unmanned mining trucks are intelligent vehicles capable of automatically completing transportation operations in the mining environment. Current technology involves collecting data on the complex shape of the mine's internal transport channels. While the unmanned mining truck operates autonomously within these channels and collects multiple images to determine its position, it neglects the sensing data from multiple heterogeneous multi-source sensors. This affects the accuracy of the fused data combination and reduces the precision of the unmanned mining truck's current location. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a method and system for positioning unmanned mining vehicles in open-pit mines based on the fusion of multi-source heterogeneous sensors.

[0004] This invention provides a method for locating unmanned mining trucks in open-pit mines based on multi-source heterogeneous sensor fusion, comprising: In the internal transport channels of the mine, the movement route of the unmanned mining truck is collected, multiple moving nodes are identified based on the identification of the unmanned mining truck's movement route, multiple dynamic images of the unmanned mining truck are determined based on the tracing of multiple moving nodes, and the current image of the unmanned mining truck is determined based on multiple dynamic images and the corresponding shooting time. The spatial region of the unmanned mining truck relative to the mine is determined by recognizing the current image of the unmanned mining truck, and multiple multi-source heterogeneous sensors are identified based on the detection of this spatial region. Multiple heterogeneous sensors from multiple sources are distributed around the unmanned mining truck and output corresponding sensing data; the fusion data combination is determined based on multiple sensing data, the corresponding data types, and the overall shape of the unmanned mining truck. Collect the predicted range of the unmanned mining vehicle relative to the spatial area, and determine the predicted positioning event of the unmanned mining vehicle based on the fused data combination, the current image of the unmanned mining vehicle, and the predicted range of the unmanned mining vehicle relative to the spatial area. Multiple location information is determined based on the detection of predictive location events of unmanned mining vehicles. The current location of the unmanned mining vehicle is determined based on the multiple location information, the corresponding location dimensions, and the overall shape of the unmanned mining vehicle.

[0005] This invention provides a positioning system for unmanned mining trucks in open-pit mines based on multi-source heterogeneous sensor fusion. This system is applied to the aforementioned positioning method for unmanned mining trucks in open-pit mines based on multi-source heterogeneous sensor fusion. The positioning system includes: The current image module is used to collect the movement route of unmanned mining trucks in the internal conveying channels of the mine, identify multiple moving nodes based on the identification of the movement route of the unmanned mining trucks, determine multiple dynamic images of the unmanned mining trucks based on the tracing of multiple moving nodes, and determine the current image of the unmanned mining trucks based on multiple dynamic images and the corresponding shooting time. The detection module is used to determine the spatial area of ​​the unmanned mining truck relative to the mine based on the recognition of the current image of the unmanned mining truck, and to determine multiple multi-source heterogeneous sensors based on the detection of the spatial area. The data fusion module is used to distribute multiple multi-source heterogeneous sensors around the unmanned mining vehicle and output corresponding sensing data; the data fusion combination is determined based on multiple sensing data, corresponding data types and the overall shape of the unmanned mining vehicle. The predictive positioning event module is used to collect the predicted range of the unmanned mining vehicle relative to the spatial area, and determine the predicted positioning event of the unmanned mining vehicle based on the fused data combination, the current image of the unmanned mining vehicle, and the predicted range of the unmanned mining vehicle relative to the spatial area. The current location module is used to determine multiple location information based on the detection of predicted location events of the unmanned mining vehicle. Based on the multiple location information, the corresponding location dimensions, and the overall shape of the unmanned mining vehicle, the current location of the unmanned mining vehicle is determined.

[0006] Compared with the prior art, the beneficial effects of the present invention are: In this embodiment of the invention, the spatial region of the unmanned mining truck relative to the mine is determined based on the recognition of the current image of the unmanned mining truck. Multiple multi-source heterogeneous sensors are then identified based on the detection of this spatial region. These sensors are distributed around the unmanned mining truck and output corresponding sensing data. A fusion data combination is determined based on the multiple sensing data, the corresponding data types, and the overall shape of the unmanned mining truck. This approach introduces the spatial region of the unmanned mining truck relative to the mine, manages and controls the multiple multi-source heterogeneous sensors, and considers the overall shape of the unmanned mining truck, thereby improving the accuracy of the fusion data combination.

[0007] Therefore, the predicted range of the unmanned mining truck relative to the spatial area is collected. Based on the fused data combination, the current image of the unmanned mining truck, and the predicted range of the unmanned mining truck relative to the spatial area, the predicted positioning event of the unmanned mining truck is determined. Based on the detection of the predicted positioning event of the unmanned mining truck, multiple positioning information is determined. According to the multiple positioning information, the corresponding positioning dimensions, and the overall shape of the unmanned mining truck, the current positioning position of the unmanned mining truck is determined. The predicted positioning event of the unmanned mining truck is introduced to control multiple positioning information, realize the overall consideration of multiple positioning information, corresponding positioning dimensions, and the overall shape of the unmanned mining truck, and improve the accuracy of the current positioning position of the unmanned mining truck. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the unmanned mining truck positioning method in open-pit mines based on multi-source heterogeneous sensor fusion in an embodiment of the present invention. Figure 2 This is a flowchart illustrating step S11 in the open-pit mine unmanned mining vehicle positioning method based on multi-source heterogeneous sensor fusion in an embodiment of the present invention. Figure 3 This is a flowchart illustrating step S12 in the open-pit mine unmanned mining vehicle positioning method based on multi-source heterogeneous sensor fusion in an embodiment of the present invention. Figure 4 This is a flowchart illustrating step S13 in the open-pit mine unmanned mining vehicle positioning method based on multi-source heterogeneous sensor fusion in an embodiment of the present invention. Figure 5 This is a flowchart illustrating step S14 of the open-pit mine unmanned mining truck positioning method based on multi-source heterogeneous sensor fusion in an embodiment of the present invention. Figure 6 This is a flowchart illustrating step S15 of the open-pit mine unmanned mining vehicle positioning method based on multi-source heterogeneous sensor fusion in an embodiment of the present invention. Figure 7 This is a schematic diagram of the structural composition of the unmanned mining truck positioning system in an open-pit mine based on the fusion of multi-source heterogeneous sensors in an embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0010] Please see Figures 1 to 7 A localization method for unmanned mining trucks in open-pit mines based on multi-source heterogeneous sensor fusion is proposed and applied to unmanned mining truck localization scenarios. The method includes: Step S11: In the internal conveying channel of the mine, collect the movement route of the unmanned mining truck, identify multiple moving nodes based on the identification of the movement route of the unmanned mining truck, determine multiple dynamic images of the unmanned mining truck based on the tracing of multiple moving nodes, and determine the current image of the unmanned mining truck based on multiple dynamic images and the corresponding shooting time. Step S12: Determine the spatial region of the unmanned mining truck relative to the mine based on the recognition of the current image of the unmanned mining truck, and determine multiple multi-source heterogeneous sensors based on the detection of the spatial region; Step S13: Multiple multi-source heterogeneous sensors are distributed around the unmanned mining truck and output corresponding sensing data; the fusion data combination is determined based on the multiple sensing data, the corresponding data types, and the overall shape of the unmanned mining truck. Step S14: Collect the predicted range of the unmanned mining truck relative to the spatial area, and determine the predicted positioning event of the unmanned mining truck based on the fused data combination, the current image of the unmanned mining truck and the predicted range of the unmanned mining truck relative to the spatial area. Step S15: Based on the detection of predicted positioning events of the unmanned mining truck, determine multiple positioning information, and determine the current positioning position of the unmanned mining truck according to the multiple positioning information, the corresponding positioning dimension and the overall shape of the unmanned mining truck.

[0011] refer to Figure 2 In step S11, the specific steps are as follows: S111: Mark the internal transport channels of the mine based on mine detection, determine the transport distribution map of the mine based on the detection of the internal transport channels of the mine, determine the movement route of the unmanned mining vehicle based on the transport distribution map of the mine, the database of unmanned mining vehicles and the corresponding transport tasks, and collect the movement route of the unmanned mining vehicle. S112: Based on the detection of the unmanned mining truck's movement route, determine multiple sub-movement areas; based on the location of the multiple sub-movement areas, the corresponding area shape, and the stopping position of the unmanned mining truck, determine the corresponding moving nodes; and based on the tracing of multiple moving nodes, determine multiple dynamic images of the unmanned mining truck; mark the shooting events of each dynamic image; and determine the current image of the unmanned mining truck based on multiple dynamic images and their corresponding shooting times.

[0012] In the embodiments of this application, a three-dimensional LiDAR or a mobile scanning device equipped with structured light / binocular cameras is used to perform a comprehensive scan of the mine interior; the scanning process is meticulous and covers all potential transport channels, including main roads, branches, loading and unloading points, ramps, etc.

[0013] The collected raw point cloud data contains a lot of noise, such as dust in the air, birds, or interference points generated by equipment. It needs to be further preprocessed, including noise reduction, filtering (smoothing the ground and walls), and downsampling (reducing the amount of data while ensuring accuracy and improving the efficiency of subsequent processing). Through point cloud segmentation algorithms (such as region growing and RANSAC plane fitting), the system can automatically identify the key geometric features of the passage, such as flat ground, vertical sidewalls, and curved roof. Based on these boundary features, the system can automatically or semi-automatically delineate continuous spatial areas that can be safely passed by unmanned mining trucks. These identified continuous spatial areas are abstracted into a topological network. In this network, the edge represents a specific transport channel, while the node represents key locations such as intersections and turning points of the channel, ultimately forming a vectorized map of the internal transport channels of the mine.

[0014] Assign multiple dimensions of attributes to each "edge" (i.e., channel) in the topology network generated in the previous step; physical attributes: channel width, height, slope, radius of curvature, road surface material (asphalt, concrete, gravel); traffic attributes: one-way / two-way, suggested driving speed, height limit / weight limit; functional attributes: main transportation channel, backup channel, maintenance channel; divide the entire mine into different logical areas according to transportation functions and business processes, such as "mining area - crushing station transportation line", "crushing station - stockpile yard transportation line", etc., to facilitate management and task allocation; divide the entire mine into different logical areas according to transportation functions and business processes.

[0015] Taking into account the environment, vehicle capabilities, and mission requirements, a safe and efficient specific driving route is generated; Input: Transport distribution map: provides the "road network" and "traffic rules" for route planning; Unmanned mining truck database: provides the unmanned mining truck's own parameters, such as external dimensions (length, width, and height), minimum turning radius, maximum gradeability, load capacity, etc. These parameters determine which roads vehicle B can and cannot travel on; Transportation mission: provides the "start and end point" and "target" for route planning.

[0016] Classical pathfinding algorithms, such as Dijkstra's algorithm or A algorithm, are employed. A algorithm is particularly suitable because it can significantly accelerate the search process by utilizing heuristic functions (such as Euclidean distance). During the search, the algorithm performs constraint checks in real time: geometric constraints: the size of the minecart is less than the minimum width and height of the passage; kinematic constraints: the turning radius and slope of the passage are within the performance range of the minecart; task constraints: the path connects the start and end points of the task. Among all feasible paths, the system selects the optimal path as the final movement route based on the optimization objective (such as shortest distance, shortest time, and lowest energy consumption). This route is usually composed of a series of precise geographical coordinates.

[0017] Furthermore, multiple sub-movement areas are determined based on the detection of the unmanned mining truck's movement route. Based on the location, corresponding shape, and stopping position of the unmanned mining truck in each sub-movement area, corresponding movement nodes are determined. Multiple dynamic images of the unmanned mining truck are then determined based on the tracing of these movement nodes. The shooting events of each dynamic image are marked, and the current image of the unmanned mining truck is determined based on the multiple dynamic images and their corresponding shooting times. This comprehensive consideration of multiple dynamic images and their corresponding shooting times ensures the accuracy of the current image of the unmanned mining truck.

[0018] At this point, the system employs a hybrid segmentation algorithm based on curvature, rate of change of heading, and distance to intelligently analyze the movement route (a series of dense coordinate points) generated by S111. The segmentation triggering conditions mainly include: geometric features: when the curvature of the route exceeds a preset threshold (i.e., entering a curve), a segmentation is automatically triggered; topological features: when the route passes through intersections, branch points, or slope change points in the transport distribution map, a segmentation is forcibly performed; distance threshold: on long straight road sections, a segmentation is also performed every fixed distance (e.g., 50 meters). Each segmented road section is defined as a "sub-movement area," which has relatively consistent geometric characteristics and functional attributes.

[0019] For each sub-movement area, the system generates one or more node candidate points based on the following information: Area location: The start and end points of the sub-movement area are natural node candidate points; Area morphology: Within the area, unique geometric or visual feature points are identified through algorithms; for example, the location of specific kilometer markers in straight sections, the apex of curves, and unique signs or markers (such as reflective strips, equipment numbers) on both sides of the transport road; Dwelling location: The system identifies locations where unmanned mining trucks or frequently stop, such as loading / unloading points, waiting areas, and checkpoints, from historical operation data or task instructions. These locations are directly defined as high-priority moving nodes; Each determined moving node is assigned a multi-dimensional feature descriptor, including precise coordinates, geometric features, visual features, and semantic labels.

[0020] During operation, the unmanned mining vehicle's positioning module (such as a wheeled odometer + IMU) provides a real-time, low-precision pose estimate. The system continuously matches this real-time pose with the location of moving nodes in a high-precision map. When the unmanned mining vehicle enters the "influence range" of a moving node (for example, a circular area with a radius of 5 meters centered on the node), the system determines that the vehicle is "tracing" the node. Once tracing is triggered, the system immediately instructs the onboard camera to capture an image. The captured image is then defined as a "dynamic image".

[0021] Each captured motion image is packaged by the system into an event data packet. The core of this data packet is the image itself, but more importantly, it carries metadata, namely "capture event markers," which include: a high-precision timestamp: usually from a GPS time clock, with millisecond-level accuracy; associated node ID: explicitly indicating which mobile node the image was captured from; initial vehicle pose values: the vehicle position and attitude estimates from the wheel odometer and IMU at the time of capture; the system maintains a time-series motion image buffer; and the "current image" is defined as the motion image with the latest timestamp at the time of localization calculation.

[0022] refer to Figure 3 In step S12, the specific steps are as follows: S121: Collect the current image of the unmanned mining truck, determine multiple sub-environmental regions based on the detection of the current image of the unmanned mining truck, determine the corresponding environmental features based on the detection of each sub-environmental region, and determine the spatial region of the unmanned mining truck relative to the mine based on the overall shape of the unmanned mining truck, the feature shape of multiple environmental features and the transportation distribution map of the mine. S122: Based on the detection of the spatial area, multiple electronic devices are determined, and a sensing combination is determined according to the devices of the multiple electronic devices. Based on the sensing combination, the spatial position of the unmanned mining vehicle relative to the spatial area, and the response data of the unmanned mining vehicle, multiple multi-source heterogeneous sensors are determined. At this time, the multiple multi-source heterogeneous sensors are located around the unmanned mining vehicle and simultaneously sense the unmanned mining vehicle.

[0023] In the embodiments of this application, the system acquires the "current image" determined by step S112. This image is typically captured by a wide-angle or fisheye camera to obtain the widest possible field of view of the environment, and is often in an uncompressed raw format (such as RAW). This image is input into a pre-trained deep learning semantic segmentation network, which (e.g., based on U-Net, DeepLabV3+, or SegFormer architecture) is specifically trained and optimized for mining environments.

[0024] The network classifies each pixel in an image into a predefined category; each connected region consisting of pixels of the same type is a "sub-environment region"; typical categories include: drivable road surface, slope of a transport road, retaining walls on both sides of a transport road, obstacles (such as other vehicles, falling rocks), structural features (such as ditches, pipes, signs), and background (such as dark areas in the distance); the network output is not a single label, but a segmentation mask of the same size as the original image; on this mask, different sub-environment regions are clearly marked with different colors or grayscale values.

[0025] After semantic segmentation, the system needs to extract quantized features from each identified sub-environment region, transforming abstract pixel regions into numerical descriptors that can be calculated and compared, i.e., "environmental features." Geometric feature extraction includes: for drivable road surfaces, perspective transformation (such as inverse perspective transformation, IPM) is applied to convert the road surface in the image to a bird's-eye view, allowing for accurate calculation of the road width, the curvature of the boundary lines (used to determine the curve radius), and the road slope; for retaining wall areas on transport roads, edge detection and line fitting (such as Hough transform) are used to extract the boundary lines of the retaining walls on both sides, calculating their distance and angle with the vehicle's centerline, thereby determining... Lateral position and heading of the vehicle in the passageway; texture and appearance feature extraction: For retaining walls and structural feature areas of the transportation road: local feature descriptors (such as SIFT, ORB) or deep learning feature extractors (such as SuperPoint) are used to extract key points and their descriptors. These descriptors are robust to changes in lighting and viewpoint and serve as "visual fingerprints" for position recognition; For obstacle areas: by analyzing their contours, sizes, and shapes, the type of obstacle (such as point, area, or volume) can be preliminarily determined; Output: Each sub-environment area corresponds to a feature vector containing its geometric, texture, and appearance information.

[0026] The vehicle is precisely located from "image space" to "map space" to complete the final positioning; Input: The overall shape of the unmanned mining truck: This provides key sensor extrinsic parameter information, such as the camera's installation height, pitch angle, and roll angle. These parameters are used to back-project the two-dimensional features in the image to the three-dimensional world coordinate system, which is a prerequisite for achieving geometric matching; Feature shape of environmental features: These are real-time observations extracted from the current image; The mine's transport distribution map is a pre-constructed high-precision vector map with rich semantic information; Each path in the map is associated with a priori feature description.

[0027] The system compares the road geometric features extracted from the image with the prior geometric information of candidate road segments in the map; only road segments with highly matching geometric features can proceed to the next round of screening. At the same time, the system matches the visual feature descriptors extracted from the current image (such as key points on walls) with the "bag of visual words" or feature point cloud of the area stored in the map database; the more feature points that are successfully matched, the greater the probability that the vehicle is located in that area; the system usually uses particle filtering or probabilistic graphical models to synthesize all matching results; each candidate map location has a confidence score; by fusing evidence from geometric matching and visual matching, the system updates the confidence score of each location; one or more consecutive map road segments with the highest confidence scores are identified as the spatial area where the unmanned mining truck is currently located.

[0028] Specifically, suppose an unmanned mining truck travels to an intersection in the mine, and its front-facing camera captures the current image. After the image is input into a semantic segmentation network, the output segmentation mask clearly marks: a large drivable road surface area at the bottom of the image (this area is Y-shaped in the image); two transport road retaining wall areas on the left and right; at the bottom of the left retaining wall, there is a small, regularly shaped structural feature area; and in the distance at the right fork, there is a blurry obstacle area.

[0029] In the scenario of a mine intersection, the system extracts features from the segmented area: Geometric features: The system performs IPM transformation on the drivable road area, calculating that the main road is 5 meters wide, the left fork road is 4 meters wide, and the radius of curvature of the fork road intersection is approximately 15 meters; Texture features: The system extracts SIFT feature points from the structural feature area of ​​the left retaining wall and matches them with a template to identify it as an "emergency stop" sign; At the same time, a large number of rock texture feature points are extracted from the right retaining wall; At this point, the environmental feature set of the unmanned mining vehicle is: {Y-shaped intersection, main road width 5m, left fork road width 4m, "emergency stop" sign on the left, rock wall on the right}.

[0030] The system began map matching; it queried the mine's transport distribution map and found an area on the map with prior information: "Intersection of the main transport channel and No. 3 connecting roadway, main road width 5 meters, No. 3 connecting roadway width 4 meters, emergency parking point on the left side of the intersection"; geometric matching: the real-time extracted road width and bifurcation shape are completely consistent with the map's prior information; visual matching: the real-time identified "emergency parking point" sign perfectly matches the location and type marked on the map; morphological correction: the system uses the installation height (2.5 meters) and pitch angle (-2 degrees) of the unmanned mining truck's camera in the overall shape to accurately back-project the position of the sign in the image onto the world coordinate system. The calculated position has an error of only 10 centimeters compared with the position on the map, finally confirming: all evidence points to the same conclusion; the system determines with extremely high confidence that the current spatial area of ​​the unmanned mining truck is "intersection of the main transport channel and No. 3 connecting roadway".

[0031] Furthermore, multiple electronic devices are identified based on the detection of the spatial area, and a sensing combination is determined based on the devices of the multiple electronic devices. Based on the sensing combination, the spatial position of the unmanned mining vehicle relative to the spatial area, and the response data of the unmanned mining vehicle, multiple multi-source heterogeneous sensors are determined. At this time, the multiple multi-source heterogeneous sensors are located around the unmanned mining vehicle and simultaneously sense the unmanned mining vehicle. This takes into account the overall consideration of the sensing combination, the spatial position of the unmanned mining vehicle relative to the spatial area, and the response data of the unmanned mining vehicle, ensuring the accuracy of the multiple multi-source heterogeneous sensors.

[0032] At this point, the system maintains an expert rule base or a policy model trained through reinforcement learning. This base defines the mapping relationship between the attributes of different spatial areas and the required electronic devices (sensors). After S121 determines the spatial area where the vehicle is located, the system analyzes the key attributes of that area, such as: geometric attributes: whether it is a long straight road, a sharp bend, or an intersection; topological attributes: whether it is a one-way street or a multi-lane intersection; environmental attributes: based on historical data or real-time image analysis, whether the area has high dust, low light, or strong electromagnetic interference. Based on these attributes, the system queries the rule base and determines a list of candidate electronic devices. For example, the rules may define: if the area is a "sharp bend, multi-lane intersection", then the candidate devices are "main lidar, side blind spot lidar, surround view camera"; if the area is a "long straight road, high dust", then the candidate devices are "millimeter-wave radar, IMU, wheeled odometer".

[0033] The system evaluates the real-time performance status of each candidate electronic device, including its health status (whether there are any faults), data quality (signal-to-noise ratio, frame rate stability), and power consumption and computing power requirements. Next, the system performs complementarity and redundancy analysis, aiming to build a sensor combination that can complement and back up each other. For example, lidar provides high-precision geometric information but is affected by dense fog; millimeter-wave radar has strong penetration but lower accuracy. Combining them can obtain reliable sensing under different weather conditions. The system determines an optimal sensor combination, which includes not only "which sensors to use" but also configuration parameters of "how to use them".

[0034] The system utilizes the precise location of the unmanned mining vehicle within the spatial area (from S121) and response data (such as current vehicle speed and steering wheel angle) to make final fine adjustments to the sensor combination. For example, if the vehicle is very close to the right-side tunnel wall, the system can dynamically reduce the detection threshold of the right-side sensor or increase its scanning frequency. When traveling at high speed, the system will prioritize sensors with long-range detection capabilities. When turning at low speed, the system will increase the data weight of sensors with short range and wide field of view. Combining all the above information, the system finally determines an optimal list of multi-source heterogeneous sensors for the current moment, current position, and current motion state. The system sends activation commands to the ECUs of these sensors and performs strict time synchronization of all activated sensors through a unified hardware clock source.

[0035] Specifically, the unmanned mining truck was just located at the "intersection of the main transport channel and the No. 3 connecting roadway" in the mine by S121; the system detected that the spatial area was an "intersection", and the rule base was immediately triggered; the rule pointed out that the intersection is geometrically complex and has blind spots, requiring precise geometric perception with no blind spots in 360 degrees; therefore, the system pre-selected a list of multiple electronic devices as follows: {main lidar, 4 corner radars, 4 surround-view cameras, high-precision IMU}.

[0036] In the scenario of a mine intersection, the system evaluates the pre-selected equipment list and finds that all equipment is in good working order. Next, a complementarity analysis is performed: the main LiDAR provides high-precision point clouds from the front and sides, but there is a blind spot directly behind the vehicle; four surround-view cameras can compensate for this blind spot and provide rich texture information; the corner radar (millimeter wave) can penetrate any light dust and stably track moving targets (such as other mining vehicles). Based on this, the system generates a specific sensor combination: {Main LiDAR: 10Hz full scan; 4x surround-view cameras: 30Hz synchronous acquisition; 4x corner radar: 20Hz; high-precision IMU: 100Hz}. This combination achieves comprehensive coverage of geometry and texture, optical and electromagnetic waves, and long, medium, and short ranges.

[0037] The unmanned mining truck's spatial position at the mine intersection shows it is located in the center of the intersection, with a speed of 5 km / h, and the response data shows the steering wheel angle is 0 (preparing to proceed straight). Based on this information, the system makes a final confirmation of the sensor combination. Due to the low speed, the system decides to maintain the scanning frequency of the main LiDAR at 10Hz to save computing power, but switches the image processing algorithm of the surround-view camera to a high-sensitivity mode for "pedestrians / small obstacles". The system sends activation commands to all the aforementioned physical sensors around the unmanned mining truck. These multi-source heterogeneous sensors are synchronously awakened and begin to synchronously and in real-time sense the environment around the unmanned mining truck according to preset configuration parameters and a unified clock reference.

[0038] refer to Figure 4 In step S13, the specific steps are as follows: S131: Mark multiple multi-source heterogeneous sensors and determine the multi-dimensional detection space of the unmanned mining truck based on the multiple multi-source heterogeneous sensors and the unmanned mining truck. In the multi-dimensional detection space of the unmanned mining truck, multiple multi-source heterogeneous sensors detect the unmanned mining truck along different directions and output corresponding sensing data. The sensing data are matched in the time dimension. S132: Among multiple sensing data, the corresponding data type is determined based on the identification of each sensing data, and the first data combination is determined according to the multiple sensing data and the overall shape of the unmanned mining vehicle; S133: Determine the second data combination based on the data types of multiple sensing data and the overall shape of the unmanned mining vehicle. Determine the corresponding fused data combination based on the fusion of the first data combination and the first data combination. In the fused data combination, mark the data priority of each sensing data and perform anomaly screening on multiple sensing data.

[0039] In the embodiments of this application, multiple multi-source heterogeneous sensors are marked, and a multi-dimensional detection space of the unmanned mining vehicle is determined based on the multiple multi-source heterogeneous sensors and the unmanned mining vehicle. In the multi-dimensional detection space of the unmanned mining vehicle, multiple multi-source heterogeneous sensors detect the unmanned mining vehicle along different directions and output corresponding sensing data. Each sensing data is matched in the time dimension, which is compatible with the overall consideration of the first data combination and ensures the accuracy of the multi-dimensional detection space of the unmanned mining vehicle.

[0040] At this point, the system assigns a globally unique identifier (ID) to each sensor activated in S122, such as Lidar_Front, Camera_Left, etc. The most critical part is the calibration process, which is divided into intrinsic parameter calibration and extrinsic parameter calibration. Intrinsic parameter calibration determines the sensor's internal characteristics (such as camera focal length and LiDAR scanning frequency). Extrinsic parameter calibration uses professional algorithms to accurately determine the installation position and attitude of each sensor relative to the coordinate system of the unmanned mining truck. This is an offline process completed by professional calibration equipment, and its accuracy is crucial.

[0041] Based on the calibration results of all sensors, the system constructs a unified multi-dimensional detection space. This space is essentially an information field that extends the physical dimensions based on the vehicle coordinate system. It not only contains basic three-dimensional spatial coordinates, but each spatial point can also be associated with multiple data attributes, such as the reflection intensity from LiDAR, the RGB color value from the camera, the radial velocity from the radar, and the inertial information provided by the IMU.

[0042] Different sensors perform detection based on different physical principles, collecting information along different "directions" or "dimensions" in the detection space: LiDAR: By measuring the flight time of laser pulses, it accurately acquires the three-dimensional coordinates of the surface of objects in the environment along the laser beam direction, forming point cloud data. Its detection direction is discrete, a ray direction determined by the scanning mode; Camera: Through lens imaging, it projects light from the three-dimensional world along an optical path onto a two-dimensional image sensor, outputting image data. Its detection direction is a continuous, cone-shaped area limited by the field of view; Millimeter-wave radar: By emitting electromagnetic waves and receiving echoes, it measures the distance, velocity, and angle of targets along the beam direction, outputting a list of target points. Its detection direction is a fan-shaped area with a specific beam width; Inertial Measurement Unit: Through internal accelerometers and gyroscopes, it measures linear acceleration and angular velocity along the three axes of the vehicle coordinate system, outputting inertial data. Its detection direction is strictly bound to the vehicle coordinate system; Each sensor outputs raw sensing data at its own inherent frequency and format, forming multiple asynchronous data streams.

[0043] Because the sampling frequencies and internal processing delays of each sensor are different, their data streams are asynchronous in time; directly fusing these data will produce huge timing errors; the solution usually combines hardware and software: hardware synchronization: ideally, all sensors are connected to a unified hardware clock source, such as the pulses per second (PPS) signal of a GPS receiver; the PPS signal provides a high-precision unified time reference, and all sensors can be configured to trigger data acquisition on the rising edge of the PPS signal, ensuring the synchronization of data acquisition time from the source.

[0044] Based on hardware synchronization, or when hardware synchronization is not possible, software-level time alignment is required. The system maintains a high-precision global clock, and the data stream from each sensor enters a buffer queue with timestamps. When fusion is required, the system extracts the data frame closest to the target time from the buffer of each sensor. If the timestamps are not completely consistent, the system uses an interpolation algorithm to estimate the value at the target time. For high-frequency IMU data, linear interpolation is usually used. For LiDAR point clouds, motion compensation may be performed based on IMU data. For cameras, the most recent frame is usually used directly.

[0045] Specifically, in the scenario at the mine intersection, the system marked four sensors: Lidar_Front (mounted at the front of the vehicle roof), Camera_Front (mounted behind the windshield), Radar_Left (mounted on the left side of the vehicle body), and IMU_Main (mounted at the vehicle's center of gravity). Based on the offline calibration file, the system precisely determined the installation position of Lidar_Front (in the vehicle coordinate system) (X=2.5m, Y=0m, Z=3.0m) and the installation position of Camera_Front (X=2.0m, Y=0m, Z=2.2m). Using this information, the system constructed a multi-dimensional detection space with the rear axle center of the mining truck as the origin, providing a unified spatial reference for all subsequent data processing.

[0046] As the unmanned mining truck travels at the mine intersection, Lidar_Front continuously emits laser beams along its scanning plane, detecting mining truck C 15 meters ahead and returning its three-dimensional coordinate point cloud in a unified detection space; Camera_Front simultaneously captures an image of mining truck C, which is located slightly to the left of the center of the image; Radar_Left does not detect any moving targets along its beam direction; IMU_Main continuously outputs the current acceleration and angular velocity of the mining truck. Although these data all describe the environment at the same moment, their formats and frequencies are different.

[0047] Assuming a data fusion is required at system time T=123456.789s, the system queries the buffers of each sensor: IMU_Main has new data at time T; the latest frame image timestamp of Camera_Front is T-0.001s, almost synchronous, and can be used directly; the latest frame point cloud timestamp of Lidar_Front is T-0.05s; based on the IMU data (velocity and angular velocity) of the mine car from T-0.05s to T, the system performs motion compensation on the entire point cloud, "extending" the positions of all point clouds to time T; the latest data timestamp of Radar_Left is T-0.02s, and the system obtains its observation value at time T through linear interpolation; the system obtains a set of sensor datasets that are spatially aligned at the precise time point T=123456.789s, ready to be sent to the next step for morphological constraint-based combination.

[0048] Furthermore, among multiple sensor data, the corresponding data type is determined based on the identification of each sensor data, and the first data combination is determined according to the multiple sensor data and the overall shape of the unmanned mining vehicle. This takes into account the overall consideration of multiple sensor data and the overall shape of the unmanned mining vehicle, ensuring the accuracy of the first data combination.

[0049] At this point, the system identifies which sensor each data block comes from based on the header information of the data packet or the sensor ID of the source (marked in S131); for different sensors, the system calls the corresponding parser to interpret its proprietary data format and determine its data type.

[0050] LiDAR: Parses its data packets to extract a series of three-dimensional coordinate points (x, y, z) and reflection intensity i. This set of points is classified as three-dimensional point cloud data. Camera: Parses its image data (such as RAW, YUV, or JPEG format) and decodes it into a two-dimensional pixel matrix. Each pixel contains color information (such as RGB values), which is classified as image data. Millimeter-wave radar: Parses its data stream to obtain a target list. Each target in the list is a structure containing information such as distance, radial velocity, azimuth angle, and signal-to-noise ratio (SNR), which is classified as target trace data. Inertial Measurement Unit: Parses its serial or CAN bus data to obtain three-axis acceleration (ax, ay, az) and three-axis angular velocity (wx, wy, wz), which is classified as inertial measurement data. All parsed data is encapsulated into a unified data structure. Each structure not only contains the data itself but also its data type label, timestamp, source sensor ID, and other metadata for easy subsequent processing.

[0051] The system loads a precise 3D model of the overall shape of the unmanned mining truck from the database. This model is usually a simplified bounding box combination, such as a cuboid representing the main body of the vehicle, plus more refined geometry representing the cab and hopper. This model is defined in the same vehicle coordinate system as the multi-dimensional detection space.

[0052] The system iterates through every point in the LiDAR point cloud; for each point, it determines whether it is located inside the 3D model of the unmanned mining truck through coordinate transformation; any points that fall inside the vehicle model are echoes generated by the laser beam hitting the vehicle itself, and these points are called "self-points" and are completely discarded; for targets reported by millimeter-wave radar, the system checks their position in the vehicle coordinate system; if a target's position report falls inside the vehicle model, this is usually a false target caused by ground metal reflection or multipath effect, and the target point will be marked as invalid and discarded.

[0053] Simultaneously, the system can generate a self-occlusion mask using the overall morphology model. This mask is a binary image in which areas corresponding to the vehicle's own components (such as the hood and A-pillar) are marked as "occluded." In subsequent image processing, any detection results within these areas will be downweighted or ignored. After the above-mentioned rigorous screening based on the overall morphology, all sensor data are re-integrated, and this integrated dataset is the first data combination. It contains effective observation information about the external environment from different sensors, which has been freed from its own interference.

[0054] Specifically, as the unmanned mining truck passes through the mine's intersection, the S131 has already prepared time-aligned sensor data for it. The system begins data type identification: the system receives a data packet from Lidar_Front, parses it to obtain a point cloud set containing 50,000 points, and marks its data type as PointCloud; it receives data from Camera_Front, decodes it to obtain a 1920x1080 RGB image, and marks its data type as Image; it receives data from Radar_Left, parses it to obtain a list containing 3 targets, and marks its data type as RadarTargets; it receives data from IMU_Main, parses it to obtain a set of six-axis floating-point numbers, and marks its data type as IMUData.

[0055] In the scene at the mine intersection, the system loaded the overall shape model of the unmanned mining truck: a cuboid measuring 10m x 3.5m x 3.8m, with a raised cab model at the front. The system began to traverse the 50,000 points in the Lidar_Front; it found that the coordinates of approximately 800 points fell inside the vehicle model, mainly distributed on the hood and the top of the cab; the system immediately removed these 800 points from the point cloud; the system checked the three targets in the Radar_Left, two of which were valid, located 10 meters to the left front and 20 meters to the left rear; however, the coordinates of the third target were reported as (X=-1.0m, Y=0.5m), exactly inside the left side of the mining truck; the system determined this to be a false target and removed it from the target list; the system generated a self-occlusion mask based on the vehicle model and the intrinsic and extrinsic parameters of the camera.

[0056] The system packages the cleaned point cloud (49,200 points), the cleaned radar target list (2 valid targets), the original image (and its self-occlusion mask), and the IMU data together to form the first data combination. The data in this combination has eliminated the interference from the mining truck itself and truly reflects the environmental conditions around the intersection. It can be safely used for subsequent fusion analysis.

[0057] Therefore, a second data combination is determined based on the data types of multiple sensing data and the overall shape of the unmanned mining vehicle. A corresponding fused data combination is determined based on the fusion of the first data combination and the first data combination. In the fused data combination, the data priority of each sensing data is marked, and anomaly screening is performed on multiple sensing data. This approach is compatible with the overall consideration of the fusion of the first data combination and the first data combination, ensuring the accuracy of the corresponding fused data combination.

[0058] At this point, the system analyzes the natural complementary relationships between different data types; for example, LiDAR provides accurate geometry and distance, but lacks color and semantic information; cameras provide rich color and texture, but lack accurate depth information, and these two are natural complementary pairs; similarly, IMU can measure angular velocity and acceleration at high frequency, but integration will produce drift; wheeled odometry can measure speed, but it will fail when the wheels slip; the overall shape (size, dynamic characteristics) of the unmanned mining truck provides a physical basis for grouping; the large size of the mining truck requires it to accurately perceive the geometric boundaries of the surrounding environment (LiDAR), while recognizing semantic information in the environment (such as other vehicles, people, from the camera); based on the above analysis, the system divides the data in the first data combination into several logical "second data combinations".

[0059] Within each second data combination, the system uses a fusion algorithm suitable for that data type. For the environmental perception combination, a sensor fusion framework is employed, such as variants of the Kalman filter (EKF, UKF) or more advanced optimization methods. For example, by accurately calibrating extrinsic parameters, feature points from the camera are projected onto the LiDAR point cloud to achieve data association, and a joint state vector is used to simultaneously optimize the environmental map and vehicle pose. For the self-state estimation combination, a Kalman filter is typically used. IMU data is used as a prediction step to update the vehicle's motion state, and wheel odometer or radar data is used as an observation step to correct the IMU's integral drift.

[0060] The results of local fusion are used as input for higher-level fusion. For example, the high-frequency, low-drift vehicle pose output from the "self-state estimation combination" is used as the prior input for the localization algorithm in the "environmental perception combination". Conversely, the high-precision absolute position obtained by matching the map in the "environmental perception combination" can be used to correct the accumulated error of the "self-state estimation combination". After two levels of fusion, the system generates a unified and comprehensive state estimate, i.e., the fused data combination. This combination usually includes: a high-precision global pose (position + attitude), a high-precision local environment map (point cloud with semantic labels), and the vehicle's own motion state (velocity, angular velocity).

[0061] In the final fused data combination, each data source or local fusion result is assigned a dynamic data priority, which is determined by multiple factors, such as environmental adaptability (visual data has a higher priority in sunny weather, radar data has a higher priority in foggy weather), motion state (forward sensors have a higher priority at high speeds, lateral sensors have a higher priority when turning at low speeds), and data consistency.

[0062] The system continuously monitors the performance of each sensor to identify anomalies. In frameworks such as Kalman filtering, innovation or residual (i.e., the difference between observed and predicted values) is a key indicator. If the residual of a sensor remains excessively large, the system determines that the sensor is abnormal. The system also checks the quality of the data itself, such as the point cloud density of LiDAR, the image contrast of the camera, and the signal-to-noise ratio of the radar. Once an anomaly is detected, the system takes appropriate measures, such as reducing the weight of the sensor's data in the fusion calculation, or temporarily removing the sensor from the fusion system in the case of severe anomalies.

[0063] Specifically, the system analyzes the first data combination generated by S132: {cleaned point cloud, image, cleaned radar target, IMU data}. The system identifies that the point cloud and image are complementary in terms of environmental perception, while the IMU data and radar target (which can be used for velocity estimation) are complementary in terms of estimating its own motion. Considering the large overall shape of the mining truck, accurate environmental perception and its own motion status are both crucial. The system determines two second data combinations: Combination A (environmental perception): {cleaned point cloud, image and its self-occlusion mask}; Combination B (self-motion): {IMU data, cleaned radar target data}.

[0064] In the scene at the mine intersection: In combination A, the system uses projection transformation to associate the "C mine truck" pixel in the image with the corresponding point cloud cluster in the point cloud, generating a precise 3D target labeled "vehicle"; In combination B, the system uses Kalman filtering to fuse the angular velocity of the IMU and the forward speed of the mine truck measured by the radar, outputting a smooth, high-frequency vehicle speed and heading angle; The system uses the smooth heading angle output by combination B as a priori to help combination A perform map matching more accurately; At the same time, the precise lateral position obtained by combination A through map matching is used to correct possible heading drift in combination B; The system generates the fused data combination: {Mine truck position:(X,Y), heading:85°, speed:4.8km / h, environmental target:{Target 1:(Type: vehicle, position:(x1,y1), speed:0km / h)}}.

[0065] Since it is a sunny day and the intersection requires precise geometric information, the system assigns high priority to the point cloud and image; the IMU also receives high priority due to its high frequency; radar targets are used as an auxiliary and have a medium priority; assuming that the left-side Radar_Left suddenly starts reporting a large number of target points at random locations; the system finds that these radar points cannot correspond to any objects in the LiDAR point cloud and camera image, and their residual values ​​increase sharply; the anomaly filtering module immediately determines that Radar_Left is under strong interference, reduces its priority to the lowest level, and ignores its data in subsequent global fusion, which ensures that the mine truck's positioning decision is entirely based on reliable LiDAR, camera, and IMU data.

[0066] refer to Figure 5 In step S14, the specific steps are as follows: S141: Collect the spatial region, determine the predicted range of the unmanned mining vehicle relative to the spatial region based on the matching between the unmanned mining vehicle and the spatial region, and determine multiple sub-prediction regions based on the identification of the predicted range; at the same time, collect the current image of the unmanned mining vehicle, and determine the current position event of the unmanned mining vehicle based on the current image of the unmanned mining vehicle and the multiple sub-prediction regions. S142: Collect and combine fused data, determine the current form event of the unmanned mining truck based on the fused data combination and the current image of the unmanned mining truck, and determine the current position event of the unmanned mining truck based on the current position event of the unmanned mining truck and the corresponding current form event.

[0067] In the embodiments of this application, the spatial region is collected, and the predicted range of the unmanned mining vehicle relative to the spatial region is determined based on the matching between the unmanned mining vehicle and the spatial region. Multiple sub-prediction regions are determined based on the identification of the predicted range. At the same time, the current image of the unmanned mining vehicle is collected, and the current position event of the unmanned mining vehicle is determined based on the current image of the unmanned mining vehicle and the multiple sub-prediction regions. This approach takes into account both the current image of the unmanned mining vehicle and the multiple sub-prediction regions as a whole, ensuring the accuracy of the current position event of the unmanned mining vehicle.

[0068] At this point, the system acquires a precise vector map of the spatial region (e.g., a specific transport road or an intersection) determined in S121; simultaneously, the system acquires the precise pose (position and attitude) and motion state (velocity, angular velocity) of the unmanned mining vehicle at the previous moment (t-1) from the fused data combination in S133.

[0069] The system uses a kinematic model to predict the vehicle's position at the current time (t). For low-speed, non-slip steering vehicles such as unmanned mining trucks, the constant speed and angular velocity model (CVTR model) is a commonly used and effective choice. This model calculates a theoretical pose at the current time based on the state and kinematic equations of the previous time step.

[0070] The system propagates the uncertainty of the positioning result at the previous moment (usually represented by the covariance matrix) and the noise of the motion model (such as engine vibration and uneven road surface) to obtain the uncertainty of the predicted pose at the current moment. This uncertainty is usually modeled as a two-dimensional Gaussian distribution, and its probability line is an ellipse. This elliptical region is the prediction range, which intuitively represents that "the unmanned mining truck may be located within this ellipse".

[0071] The system performs gridding on the ellipse prediction range generated in the previous step. It divides the ellipse into an M-row, N-column grid based on the size of the ellipse and the required positioning accuracy. Each cell in the grid is defined as a sub-prediction region. Each sub-region represents a specific vehicle position hypothesis: "If the center point of the unmanned mining truck is located within this sub-region." This process essentially generates a series of position hypotheses. For example, a 3x3 grid generates 9 different position hypotheses. The system assigns a unique ID to each sub-region and records the coordinates of its center point on the spatial region map.

[0072] The system acquires a high-resolution current image through the main camera of the unmanned mining vehicle; for each sub-prediction region (i.e. each position assumption), the system performs the following operations: the system calculates the six-degree-of-freedom pose of the virtual camera at this assumed position based on the center coordinates of the sub-region, the known attitude of the unmanned mining vehicle, and the precise extrinsic parameters of the camera.

[0073] Using the calculated virtual camera pose, the system renders a vector map of the spatial region. This is essentially a computer graphics process: projecting 3D line segments (such as road boundaries) and semantic features (such as signs) from the map onto a 2D image plane to generate a predicted view of "what a vehicle should see if it were here." The system compares the similarity between the real current image and the generated predicted view, typically by calculating the number and consistency of matching points using feature matching algorithms. The system compares the similarity scores of all sub-predicted regions. The sub-region with the highest score is the most likely real location. The center coordinates of this optimal sub-region, along with its similarity score, timestamp, and key feature point pairs used for matching, together constitute a current location event.

[0074] Specifically, the unmanned mining truck is departing from the "intersection of the main transport channel and the No. 3 connecting roadway" at a speed of 5 km / h; the spatial area collected by the system is the "main transport road on the east side of the intersection"; at the previous moment (t-1), the position of the mining truck was (X=100, Y=50), the speed was 5 km / h, and the heading angle was 90° (due east); the system uses the CVTR model to predict that the theoretical position after 1 second (time t) should be (X=101.39, Y=50); considering the uncertainty of positioning and motion, the system generates an elliptical prediction range centered at (101.39, 50), with a major axis (along the direction of travel) of 1.5 meters and a minor axis (perpendicular to the direction of travel) of 0.8 meters.

[0075] In the scenario of a mine intersection, the system divides the ellipse generated in the previous step into a 3x3 grid, resulting in 9 sub-prediction regions. The coordinates of the central region are approximately (101.39, 50), and the coordinates of the region in front of it are approximately (102.14, 50). And so on. These 9 regions are the "positional assumptions" that the system needs to verify one by one.

[0076] The system acquires a current image of the mining truck. The image shows the vehicle is basically centered in the lane, with a straight transport road ahead. The "Emergency Stop" sign on the left wall has just been left behind. The system then begins to verify nine hypotheses: When verifying the central region (101.39, 50), the system-rendered predicted view shows the transport road ahead centered, with the left sign located in the lower left corner of the image, which highly matches the real image with a similarity score of 98%. When verifying the sub-region to the right, the predicted view shows the transport road ahead skewed to the left of the image, which does not match the real image. When verifying the sub-region further back, the predicted view shows the left sign still within the image's field of view, but it has disappeared in the real image. After comparison, the central sub-predicted region (101.39, 50) has the highest matching degree. Therefore, the system determines the current location event: the unmanned mining truck is located at coordinates (101.39, 50) with a confidence level of 98%. The evidence is that the current image and the map rendering view of this location are highly consistent in key features.

[0077] Furthermore, the system collects and combines data, and determines the current form event of the unmanned mining truck based on the combined data and the current image of the unmanned mining truck. It also determines the current position event of the unmanned mining truck based on the current position event and the corresponding current form event, thus taking into account the overall consideration of the current position event and the corresponding current form event of the unmanned mining truck and ensuring the accuracy of the current position event of the unmanned mining truck.

[0078] At this point, the system acquires the fused data combination generated in step S133, the core of which includes a spatiotemporally aligned LiDAR local point cloud map and a high-precision vehicle pose estimate; the system acquires the current image used in S141 again; the system uses the vehicle pose and the extrinsic parameters of the LiDAR sensor in the fused data combination to project all the point clouds in the local point cloud map onto the two-dimensional pixel plane of the current image. This projection process generates a "point cloud contour" on the image, which accurately depicts what the unmanned mining truck itself and its surrounding environment should look like in the image from the perspective of LiDAR.

[0079] The system analyzes the degree of match between the projected point cloud contour and the actual visible object edges, shadows, and textures in the image. If the projected vehicle contour of the point cloud highly coincides with the edge of the mining truck body in the image, and the projected nearby environmental point cloud also matches the ground texture in the image, it is judged as morphologically consistent. Conversely, if there is a significant deviation, it is judged as morphologically inconsistent, which means that the sensor calibration is incorrect or faulty. The result of this comparison, namely the conclusion of "consistent" or "inconsistent", together with the quantified degree of consistency score, constitutes the current morphological event, which is an assessment of the overall status of the current multi-sensor perception system.

[0080] The system now possesses two key pieces of evidence: the current location event (from S141): a location hypothesis derived from "map-visual" matching, such as (x_v, y_v), with a visual matching score S_v; and the current morphology event (from S142-1): an assessment of the consistency of multi-sensor data, such as "consistency," with a similarity score S_c.

[0081] The system employs a decision fusion model to process these two pieces of evidence. In a high-consistency scenario: if the morphological event is "consistent" and the score S_c is high, it indicates that the fused data of S133 is reliable. In this case, the system will fuse the position (x_v, y_v) of S141 and the position (x_f, y_f) in the S133 fused data combination with a higher weight. The final output position will be a highly weighted average of the two, significantly reducing uncertainty and greatly improving confidence. In a low-consistency scenario: if the morphological event is "inconsistent," it indicates a problem with the fused data of S133. In this case, the system will reduce the weight of the S133 data source, and the final positioning result will mainly rely on the current position event (x_v, y_v) of S141. Simultaneously, the system will assign a lower confidence level to the final result and trigger sensor fault diagnosis. After the above fusion and calibration, the system generates the final, structured current position event of the unmanned mining vehicle. This event not only contains a precise 3D coordinate and attitude but, more importantly, includes a dynamically calculated confidence rating reflecting the current environment and sensor status.

[0082] Specifically, in the scenario at the mine intersection, the system collects and fuses data combined with the current image; the system performs projection: projecting the LiDAR point cloud onto the image; the results show that the point cloud accurately outlines the contours of the mine truck's hood, the A-pillar of the cab, and even the reflective strips on the front bumper correspond perfectly to their positions in the image; at the same time, the point cloud shows a small rock 1.5 meters to the left front, which is also clearly visible in the same position in the image; the system calculates a consistency score of 98%, determining the current morphological event as "highly consistent," indicating that the results from the LiDAR and Camera sensors are completely consistent.

[0083] In the scenario at the mine intersection, the system obtained two pieces of evidence: the current position event of S141: position (101.39, 50), with a visual matching score S_v of 98%; and the current morphological event of S142-1: "highly consistent," with a matching score S_c of 98%. Due to the "highly consistent" morphological event, the system determined that the fused data of S133 was absolutely reliable. The system extracted the position (101.41, 50.02) from the fused data combination of S133 and performed a weighted fusion with the position (101.39, 50) of S141. Because both were very close and had high confidence, the final result was almost the average of the two. The system generated the final current position event of the unmanned mining truck: position (101.40, 50.01), attitude angle 90.1°, and confidence rating of "high." This event was sent to the control system of the mining truck, which could confidently continue to execute the instructions to accelerate and keep the vehicle centered in the lane based on this high-precision positioning result.

[0084] refer to Figure 6 In step S15, the specific steps are as follows: S151: Collect the predicted positioning events of the unmanned mining vehicle, determine multiple positioning items based on the detection of the predicted positioning events of the unmanned mining vehicle, determine multiple positioning information according to the item content, corresponding item priority and predicted range of the unmanned mining vehicle relative to the spatial area, and mark the corresponding positioning dimensions. S152: Collect the overall shape of the unmanned mining vehicle, and determine the first positioning content based on the overall shape of the unmanned mining vehicle and multiple positioning information; S153: Determine the second positioning content based on the overall shape of the unmanned mining vehicle and the positioning dimensions of multiple positioning information, and determine the current positioning location of the unmanned mining vehicle based on the first positioning content, the second positioning content and the mine's transport distribution map.

[0085] In the embodiments of this application, the predicted positioning events of the unmanned mining vehicle are collected, and multiple positioning items are determined based on the detection of the predicted positioning events of the unmanned mining vehicle. Multiple positioning information is determined according to the item content, corresponding item priority and predicted range of the unmanned mining vehicle relative to the spatial area, and the corresponding positioning dimensions are marked. This approach takes into account the overall consideration of the item content, corresponding item priority and predicted range of the unmanned mining vehicle relative to the spatial area, ensuring the accuracy of the multiple positioning information.

[0086] At this point, the system obtains the final positioning result of the unmanned mining vehicle in the previous calculation cycle (t-1) and uses a kinematic model (such as a constant velocity model) to predict its pose in the current cycle (t). This prediction result, together with its uncertainty (such as the covariance matrix), constitutes the predicted positioning event; it provides a priori starting point for the current positioning. The system is based on a predefined library of localization items. Depending on the current scene and sensor status, it activates a set of localization items. Each item represents an independent localization technology or algorithm, is modular, and can be run and evaluated independently. Typical items include: LiDAR_Scan_Matching: determines pose by registering the current LiDAR scan point cloud with a pre-built map point cloud (e.g., using the ICP algorithm); Visual_Feature_Matching: extracts visual features from the current image and matches them with features in the map database, solving for pose using the PnP algorithm; IMU_Odometry_Fusion: performs dead reckoning by fusing IMU data and wheel odometry data, providing a high-frequency relative pose change.

[0087] The project content defines the specific algorithm, input data, and output format required to execute the project; for example, the LiDAR_Scan_Matching project content specifies the use of the ICP algorithm, the point cloud downsampling resolution, and the maximum number of iterations. The project priority is a dynamic weight, calculated in real time by step S133 based on factors such as sensor health status and environmental adaptability; for example, the priority of Visual_Feature_Matching is 0.8 on a clear day, while it drops to 0.2 on a foggy day. The prediction range is a key search constraint extracted from the predicted localization event; it is a geometric space (usually a two-dimensional or three-dimensional ellipsoid) within which the system has a high degree of confidence that the true location lies. Limiting the search range to this range can significantly improve computational efficiency, enhance algorithm robustness, and reduce false matches. For each localization project, the system uses its project content, priority, and prediction range as input to generate specific, executable localization information, which can be viewed as a "task package" containing all the necessary information.

[0088] The system defines a set of standard positioning dimensions to describe the physical quantities provided by the positioning information, ensuring that information from different sources can be understood within the same semantic framework. Each generated positioning information is labeled with one or more dimensions. Common dimensions include: 2D_Pose: position (X,Y) and heading angle (Yaw) on a two-dimensional plane; 3D_Pose: position (X,Y,Z) and attitude (Roll,Pitch,Yaw) in three-dimensional space; Linear_Velocity: three-dimensional linear velocity (vx,vy,vz); Angular_Velocity: three-dimensional angular velocity (wx,wy,wz). After labeling, each positioning information becomes a structured data packet containing: {item ID, item content, priority, search range (prediction range), positioning dimension label}. These data packets are uniformly sent to subsequent steps S152 and S153 for processing.

[0089] Furthermore, the overall shape of the unmanned mining vehicle is collected, and the first positioning content is determined based on the overall shape of the unmanned mining vehicle and multiple positioning information. This takes into account the overall shape of the unmanned mining vehicle and multiple positioning information, ensuring the accuracy of the first positioning content.

[0090] At this point, the system loads a precise 3D digital model of the overall shape of the unmanned mining truck from its configuration database. This model is usually not a detailed CAD model, but a simplified bounding volume model optimized for computational efficiency. Commonly used models include: bounding box: a minimal cuboid that completely encloses the vehicle's maximum outline in all poses; convex hull: a convex polyhedron that more closely matches the actual shape of the vehicle; and composite body: composed of multiple basic geometric shapes (such as cuboids and cylinders) that represent different parts such as the cab, hopper, and driver's cab, and can more accurately describe complex shapes. This shape model is defined in the same vehicle coordinate system as the multi-dimensional detection space, and its origin usually coincides with the vehicle's center of mass or rear axle center, ensuring the consistency of coordinate transformation.

[0091] The system iterates through all the positioning information generated by S151; for each positioning information that provides absolute position information (e.g., information labeled as 2D_Pose or 3D_Pose), the system extracts its core content: a hypothetical vehicle pose (x,y,yaw) or (x,y,z,roll,pitch,yaw).

[0092] The system loads the overall shape model and, based on the assumed pose provided by the positioning information, "places" it into the mine's transport distribution map (a three-dimensional environment model) through coordinate transformation. The system executes an efficient geometric algorithm to determine whether the instantiated vehicle model penetrates or severely overlaps with static environmental elements (such as transport road retaining walls, slope reinforcement structures, and fixed equipment) in the transport distribution map.

[0093] If the minimum distance between the vehicle model and the environment is greater than a preset safety threshold (e.g., 10 cm), the positioning information is considered geometrically feasible. If the vehicle model penetrates a wall or is placed in a physically inaccessible location (e.g., suspended in mid-air or embedded in a large rock), the positioning information is determined to be geometrically infeasible. All positioning information determined to be "geometrically feasible" is collected to form a set, which is the first positioning content. It represents all positioning assumptions that have passed the most basic physical common sense test, greatly improving the convergence and accuracy of subsequent fusion algorithms. The first positioning content generated by the system contains only one element: {positioning information A}. Through step S152, the system successfully identifies and discards two seriously deviating positioning results caused by visual mismatch and odometer cumulative error, ensuring that only the positioning assumption that best conforms to physical reality is sent to the next step for final fusion, thereby avoiding positioning failure.

[0094] Therefore, the second positioning content is determined based on the overall shape of the unmanned mining vehicle and the positioning dimensions of multiple positioning information. The current positioning location of the unmanned mining vehicle is determined based on the first positioning content, the second positioning content and the transportation distribution map of the mine. This takes into account the overall consideration of the first positioning content, the second positioning content and the transportation distribution map of the mine, and ensures the accuracy of the current positioning location of the unmanned mining vehicle.

[0095] At this point, the system iterates through all the positioning information in the first positioning content (i.e., information that has passed the S152 physical constraint check). For each piece of information, the system reads its positioning dimension label (such as 2D_Pose, Linear_Velocity, etc.). Although the information has passed the first round of physical screening, the overall shape of the unmanned mining vehicle can provide more refined grouping constraints here. For example, for information in the 2D_Pose dimension, the vehicle's width and turning radius (morphological attributes) determine that some positions, although not prone to collision, are "extremely unlikely" or "dynamically uncomfortable." The system can assign a lower intra-group weight to such positions. Based on the positioning dimension, the system divides the information in the first positioning content into different logical sets. Each set focuses on a specific physical quantity. For example: Set A (pose group): contains all positioning information labeled 2D_Pose or 3D_Pose; Set B (velocity group): contains all positioning information labeled Linear_Velocity and Angular_Velocity. These grouped sets constitute the second positioning content, which organizes a mixed set of information into several structured data packets oriented towards specific fusion tasks.

[0096] First fusion (intra-group fusion): Performed within each set of second localization content; the system uses a fusion algorithm suitable for this data type; for pose groups, if multiple pieces of information exist, the system will fuse them using a weighted average or an independent Kalman filter according to their priority and morphological constraint weights to obtain a more accurate "local optimal pose".

[0097] Second fusion (global state fusion): The system takes the various "local optimal" results (such as optimal pose and optimal velocity) generated by the first fusion as input and feeds them into a master state estimator (usually an extended Kalman filter EKF or an unscented Kalman filter UKF). This master filter maintains an estimate of the complete state of the unmanned mining vehicle and uses these local optimal results to update its state vector to obtain a globally consistent state estimate.

[0098] The system uses the pose obtained from global state fusion as the initial value and performs a final, high-precision match with the delivery distribution map. This is usually achieved through techniques such as graph optimization or bundle adjustment. In this optimization process, the system constructs a factor graph and obtains the most consistent and optimal pose solution on the global map by minimizing the overall error of all observation constraints (such as the matching error between LiDAR point cloud and map, and the reprojection error of visual features). The final pose solution after graph optimization is the current location of the unmanned mining vehicle. This result is not only highly accurate, but also achieves the highest level of confidence and robustness because it integrates all available effective information and has undergone final map calibration.

[0099] Please see Figure 7 , Figure 7 This is a schematic diagram of the structural composition of the unmanned mining truck positioning system based on multi-source heterogeneous sensor fusion in an embodiment of the present invention; the unmanned mining truck positioning system based on multi-source heterogeneous sensor fusion includes: The current image module 21 is used to collect the movement route of the unmanned mining truck in the internal conveying channel of the mine, identify multiple moving nodes based on the identification of the movement route of the unmanned mining truck, determine multiple dynamic images of the unmanned mining truck based on the tracing of multiple moving nodes, and determine the current image of the unmanned mining truck based on multiple dynamic images and the corresponding shooting time. The detection module 22 is used to determine the spatial area of ​​the unmanned mining truck relative to the mine based on the recognition of the current image of the unmanned mining truck, and to determine multiple multi-source heterogeneous sensors based on the detection of the spatial area. The data fusion module 23 is used to distribute multiple multi-source heterogeneous sensors around the unmanned mining vehicle and output corresponding sensing data; the data fusion combination is determined based on multiple sensing data, corresponding data types and the overall shape of the unmanned mining vehicle. The predictive positioning event module 24 is used to collect the predicted range of the unmanned mining vehicle relative to the spatial area, and determine the predicted positioning event of the unmanned mining vehicle based on the fused data combination, the current image of the unmanned mining vehicle and the predicted range of the unmanned mining vehicle relative to the spatial area. The current positioning module 25 is used to determine multiple positioning information based on the detection of predicted positioning events of the unmanned mining vehicle, and to determine the current positioning position of the unmanned mining vehicle based on the multiple positioning information, the corresponding positioning dimensions and the overall shape of the unmanned mining vehicle.

[0100] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion, characterized in that, include: In the internal transport channels of the mine, the movement route of the unmanned mining truck is collected, multiple moving nodes are identified based on the identification of the unmanned mining truck's movement route, multiple dynamic images of the unmanned mining truck are determined based on the tracing of multiple moving nodes, and the current image of the unmanned mining truck is determined based on multiple dynamic images and the corresponding shooting time. The spatial region of the unmanned mining truck relative to the mine is determined by recognizing the current image of the unmanned mining truck, and multiple multi-source heterogeneous sensors are identified based on the detection of this spatial region. Multiple heterogeneous sensors from multiple sources are distributed around the unmanned mining truck and output corresponding sensing data; The fusion data combination is determined based on multiple sensor data, the corresponding data types, and the overall shape of the unmanned mining vehicle. Collect the predicted range of the unmanned mining vehicle relative to the spatial area, and determine the predicted positioning event of the unmanned mining vehicle based on the fused data combination, the current image of the unmanned mining vehicle, and the predicted range of the unmanned mining vehicle relative to the spatial area. Multiple location information is determined based on the detection of predictive location events of unmanned mining vehicles. The current location of the unmanned mining vehicle is determined based on the multiple location information, the corresponding location dimensions, and the overall shape of the unmanned mining vehicle.

2. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 1, characterized in that, The process involves collecting the movement routes of unmanned mining vehicles (UAVs) within the mine's internal transport channels, identifying multiple movement nodes based on these routes, tracing these movement nodes to determine multiple dynamic images of the UAVs, and determining the current image of the UAV based on these dynamic images and their corresponding capture times. This includes: The internal transport channels of the mine are marked based on the detection of the internal transport channels. The transport distribution map of the mine is determined based on the detection of the internal transport channels. The movement route of the unmanned mining vehicle is determined based on the transport distribution map of the mine, the database of unmanned mining vehicles and the corresponding transport tasks, so as to collect the movement route of the unmanned mining vehicle. Multiple sub-movement areas are determined based on the detection of the unmanned mining truck's movement route. Based on the location of the multiple sub-movement areas, the corresponding area shape, and the stopping position of the unmanned mining truck, the corresponding moving nodes are determined. Multiple dynamic images of the unmanned mining truck are determined based on the tracing of multiple moving nodes. The shooting events of each dynamic image are marked, and the current image of the unmanned mining truck is determined based on multiple dynamic images and their corresponding shooting times.

3. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 1, characterized in that, The process involves determining the spatial region of the unmanned mining vehicle relative to the mine based on the recognition of the current image of the unmanned mining vehicle, and determining multiple multi-source heterogeneous sensors based on the detection of this spatial region, including: The system acquires current images of unmanned mining trucks, determines multiple sub-environmental regions based on the detection of the current images of unmanned mining trucks, determines the corresponding environmental features based on the detection of each sub-environmental region, and determines the spatial region of unmanned mining trucks relative to the mine based on the overall shape of the unmanned mining truck, the feature shapes of multiple environmental features, and the transportation distribution map of the mine. Multiple electronic devices are identified based on the detection of the spatial area, and a sensing combination is determined based on the devices of the multiple electronic devices. Multiple multi-source heterogeneous sensors are determined based on the sensing combination, the spatial position of the unmanned mining vehicle relative to the spatial area, and the response data of the unmanned mining vehicle. At this time, the multiple multi-source heterogeneous sensors are located around the unmanned mining vehicle and simultaneously sense the unmanned mining vehicle.

4. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 1, characterized in that, The multiple multi-source heterogeneous sensors are distributed around the unmanned mining vehicle and output corresponding sensing data; The fusion data combination is determined based on multiple sensor data, corresponding data types, and the overall shape of the unmanned mining vehicle, including: Multiple heterogeneous sensors are labeled, and the multi-dimensional detection space of the unmanned mining truck is determined based on the multiple heterogeneous sensors and the unmanned mining truck. In the multi-dimensional detection space of the unmanned mining truck, multiple heterogeneous sensors detect the unmanned mining truck along different directions and output corresponding sensing data. The sensing data are matched in the time dimension.

5. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 4, characterized in that, The multiple multi-source heterogeneous sensors are distributed around the unmanned mining vehicle and output corresponding sensing data; The fusion data combination is determined based on multiple sensor data, corresponding data types, and the overall shape of the unmanned mining vehicle, and also includes: Among multiple sensor data, the corresponding data type is determined based on the identification of each sensor data, and the first data combination is determined based on the multiple sensor data and the overall shape of the unmanned mining vehicle; The second data combination is determined based on the data types of multiple sensing data and the overall shape of the unmanned mining vehicle. The corresponding fused data combination is determined based on the fusion of the first data combination and the first data combination. In the fused data combination, the data priority of each sensing data is marked, and anomaly screening is performed on multiple sensing data.

6. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 1, characterized in that, The acquisition of the predicted range of the unmanned mining vehicle relative to the spatial area, based on the fused data combination, the current image of the unmanned mining vehicle, and the predicted range of the unmanned mining vehicle relative to the spatial area, determines the predicted positioning event of the unmanned mining vehicle, including: The system collects data on the spatial region, determines the predicted range of the unmanned mining vehicle relative to the spatial region based on the matching between the unmanned mining vehicle and the spatial region, and determines multiple sub-prediction regions based on the identification of the predicted range. At the same time, it collects the current image of the unmanned mining vehicle and determines the current position event of the unmanned mining vehicle based on the current image of the unmanned mining vehicle and the multiple sub-prediction regions.

7. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 6, characterized in that, The method of collecting the predicted range of the unmanned mining vehicle relative to the spatial area, and determining the predicted positioning event of the unmanned mining vehicle based on the fused data combination, the current image of the unmanned mining vehicle, and the predicted range of the unmanned mining vehicle relative to the spatial area, also includes: Collect and fuse data combinations, determine the current form event of the unmanned mining truck based on the fused data combinations and the current image of the unmanned mining truck, and determine the current position event of the unmanned mining truck based on the current position event of the unmanned mining truck and the corresponding current form event.

8. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 1, characterized in that, The detection of predictive positioning events based on unmanned mining vehicles determines multiple positioning information, and the current positioning location of the unmanned mining vehicle is determined based on the multiple positioning information, the corresponding positioning dimensions, and the overall shape of the unmanned mining vehicle, including: The predicted positioning events of the unmanned mining vehicle are collected. Based on the detection of the predicted positioning events of the unmanned mining vehicle, multiple positioning items are determined. Based on the content of the multiple positioning items, the corresponding priority of the items and the predicted range of the unmanned mining vehicle relative to the spatial area, multiple positioning information is determined and the corresponding positioning dimensions are marked.

9. The method for locating unmanned mining vehicles in open-pit mines based on multi-source heterogeneous sensor fusion according to claim 8, characterized in that, The method of determining multiple positioning information based on the detection of predictive positioning events of unmanned mining vehicles, and determining the current positioning location of the unmanned mining vehicle based on the multiple positioning information, the corresponding positioning dimensions, and the overall shape of the unmanned mining vehicle, further includes: Collect the overall shape of the unmanned mining vehicle, and determine the first positioning content based on the overall shape of the unmanned mining vehicle and multiple positioning information; The second positioning content is determined based on the overall shape of the unmanned mining vehicle and the positioning dimensions of multiple positioning information. The current positioning location of the unmanned mining vehicle is determined based on the first positioning content, the second positioning content, and the transportation distribution map of the mine.

10. A positioning system for unmanned mining trucks in open-pit mines based on multi-source heterogeneous sensor fusion, characterized in that, The open-pit mine unmanned mining truck positioning system based on multi-source heterogeneous sensor fusion is applied to the open-pit mine unmanned mining truck positioning method based on multi-source heterogeneous sensor fusion as described in any one of claims 1-9. The open-pit mine unmanned mining truck positioning system based on multi-source heterogeneous sensor fusion includes: The current image module is used to collect the movement route of unmanned mining trucks in the internal conveying channels of the mine, identify multiple moving nodes based on the identification of the movement route of the unmanned mining trucks, determine multiple dynamic images of the unmanned mining trucks based on the tracing of multiple moving nodes, and determine the current image of the unmanned mining trucks based on multiple dynamic images and the corresponding shooting time. The detection module is used to determine the spatial area of ​​the unmanned mining truck relative to the mine based on the recognition of the current image of the unmanned mining truck, and to determine multiple multi-source heterogeneous sensors based on the detection of the spatial area. The data fusion module is used to distribute multiple multi-source heterogeneous sensors around the unmanned mining vehicle and output corresponding sensing data; the data fusion combination is determined based on multiple sensing data, corresponding data types and the overall shape of the unmanned mining vehicle. The predictive positioning event module is used to collect the predicted range of the unmanned mining vehicle relative to the spatial area, and determine the predicted positioning event of the unmanned mining vehicle based on the fused data combination, the current image of the unmanned mining vehicle, and the predicted range of the unmanned mining vehicle relative to the spatial area. The current location module is used to determine multiple location information based on the detection of predicted location events of the unmanned mining vehicle. Based on the multiple location information, the corresponding location dimensions, and the overall shape of the unmanned mining vehicle, the current location of the unmanned mining vehicle is determined.

Images