An obstacle detection method and system based on unmanned mining trucks

Abstract

This invention relates to the field of artificial intelligence technology and discloses an obstacle detection method and system based on unmanned mining trucks. One obstacle detection method for unmanned mining trucks includes: collecting and preprocessing vehicle data; constructing a dust and fog evolution prediction network to obtain a defogging multispectral image; extracting the distribution information of different materials in the scene to form an enhanced scene feature representation; performing adaptive fusion and uncertainty evaluation to obtain a gating feature representation; using an obstacle detection model to detect obstacles; and constructing a multidimensional occupancy grid to output multidimensional obstacle data. This invention overcomes the performance bottleneck of traditional defogging algorithms in heavy dust and fog environments in mining areas by decoupling from the dust and fog environment through physical modeling. By constructing a multi-vehicle collaborative perception network with sparse feature sharing, it overcomes the problems of blind spots and insufficient long-distance perception capabilities in single-vehicle perception, enhancing the system's detection reliability in dynamic and complex environments and ensuring the safety of autonomous driving operations in mining areas.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an obstacle detection method and system based on unmanned mining trucks. Background Technology

[0002] In the context of unmanned transportation in smart mines, unmanned mining trucks undertake long-distance ore transfer tasks, and their obstacle detection performance directly affects driving safety and transportation efficiency. Mine roads are generally unstructured, with complex sections including dense convoy intersections, loading area checkpoints, and ramp / curve exits. Environmental factors include strong winds and dust, post-rain wetland reflections, low light and glare, falling rocks, and rapid equipment movement. Perception systems are primarily vision-based, supplemented by IMU attitude sensors and radar / stereo vision, but are often unstable and prone to misjudgments due to dust obstruction, mirror contamination, timestamp drift, and equipment degradation.

[0003] Chinese Patent CN120431552B discloses an obstacle detection method and an unmanned vehicle. The method includes: in response to a work vehicle detecting first obstacle data, matching the first obstacle data with second obstacle data; in response to a successful match between the first and second obstacle data, determining fusion parameters based on the obstacle type in the first obstacle data; based on the fusion parameters, fusing the shape of a first obstacle in the first obstacle data with the shape of a second obstacle in the second obstacle data to obtain an initial obstacle shape; smoothing the initial obstacle shape to obtain a target obstacle shape; and updating the second obstacle data based on the target obstacle shape. This invention solves the technical problem of low stability and accuracy in obstacle detection by work vehicles in related technologies.

[0004] However, at intersections, T-junctions, bend exits, and ramp merging points, multiple vehicle convoys merging, overtaking, and making U-turns in these sections generate multi-view mutual occlusion and wake superposition. When affected by crosswinds, headwinds, and heavy uphill loads, the preceding vehicle creates a semi-transparent trailing dust, with a particle size distribution concentrated between 10-100 μm, and visibility fluctuating between 5-30m. Dust, as a participating medium, generates strong false edges, low-contrast areas, and non-uniform patch occlusion in the visual image, leading to significant distortion and inconsistency in the cross-frame contours of the same obstacle. At this point, single-view shape projection fusion and smoothing update mechanisms struggle to effectively distinguish between the time-varying boundaries of dust and fog and the true topological changes of obstacles. The dynamic morphology of dust clumps is misjudged as structural changes in obstacles, triggering false merging and splitting decisions and producing erroneous outputs during tracking. Summary of the Invention

[0005] The purpose of this invention is to provide an obstacle detection method and system based on unmanned mining trucks in order to solve the above-mentioned problems.

[0006] This invention provides an obstacle detection method based on unmanned mining trucks, comprising the following steps: Collect and preprocess vehicle data; Based on the vehicle data, a dust and fog evolution prediction network is constructed and a dust and fog density field is obtained. Based on the dust and fog density field, the vehicle data is defogging to obtain a defogging multi-spectral domain image. Based on the polarization image data, polarization material decoupling is performed to extract the distribution information of different materials in the scene. This information is then combined with the defogging multi-spectral domain image to form an enhanced scene feature representation. The enhanced scene feature representations of multiple vehicles are compressed and spatiotemporally aligned to obtain a multi-vehicle sparse feature set. Based on the multi-vehicle sparse feature set, adaptive fusion and uncertainty evaluation are performed to obtain a gated feature representation. The gated feature representation is input into the obstacle detection model to perform obstacle detection, and the obstacle detection result is obtained. Based on the obstacle detection results and vehicle data, a multi-dimensional occupancy grid is constructed, and the multi-dimensional occupancy grid outputs multi-dimensional obstacle data.

[0007] Furthermore, the collection and preprocessing of vehicle data includes: Vehicle data includes multispectral image data, depth information, and vehicle status data. The multispectral image data includes visible light images, near-infrared images, and polarized images. The depth information includes three-dimensional spatial information. The vehicle status data includes vehicle attitude and motion status data. Preprocessing includes rolling shutter correction and temporal image stabilization for multispectral image data, as well as temporal alignment and sensor health assessment for data from different sensors.

[0008] Furthermore, the dehazed multispectral domain image obtained includes: A dust and fog evolution prediction network is constructed based on multispectral image data and vehicle status data in vehicle data. The dust and fog evolution prediction network outputs a dust and fog density field, which represents the dust and fog density per unit volume at each spatial location. Calculate the transmittance, which is a negative exponential function of the natural constant, with the exponent being the product of the scattering coefficient, dust density, and optical path distance; Based on transmittance, multispectral domain image data is dehazed using an atmospheric scattering model to obtain a dehazed multispectral domain image. When the transmittance is less than the preset transmittance threshold, the dust and fog mask value is set to 1; otherwise, it is set to 0. Based on the dust and fog mask information, the areas in the image affected by dust and fog are marked to obtain environmental status annotation information.

[0009] Furthermore, constructing the dust and fog evolution prediction network includes: The input to the dust and fog evolution prediction network includes wind speed field information extracted from multispectral image data, dust and fog density at the previous moment, and dust and fog source terms generated by vehicle movement. Among them, the dust and fog source terms are the basic dust intensity multiplied by the preset road surface dryness coefficient and then multiplied by the preset particulate matter density coefficient. The basic dust intensity is the square of the current vehicle speed multiplied by the tire contact area and then multiplied by the preset basic dust coefficient. The dust and fog evolution prediction network is modeled based on the convection-diffusion equation, where the rate of change of dust and fog density over time is equal to the negative divergence of dust and fog density and the wind speed field vector gradient, the product of the preset diffusion coefficient and the dust and fog density Laplace operator, and the algebraic sum of the dust and fog source terms.

[0010] Furthermore, dehazing of multispectral domain image data using atmospheric scattering models includes: According to the atmospheric scattering model, the observed image intensity is the sum of the true radiation term and the light value term. The true radiation term is the product of the scene's true radiation intensity and transmittance, and the light value term is the product of the atmospheric light value and the complement of transmittance. The observed image intensity is the light intensity signal actually received by the camera sensor, and the scene's true radiation intensity is the true light radiation intensity of the target object's surface in the scene under conditions without dust or fog interference. Atmospheric light values ​​are estimated using a dark channel prior method. The atmospheric light value is equal to the maximum value of the observed image pixels in the brightest preset percentage pixel region of the image. After dehazing, each pixel value of the multispectral domain image is equal to the difference between the observed image pixel value and the light value, divided by the larger of the transmittance and the preset minimum transmittance threshold. The observed image pixel value corresponds to the value of the observed image intensity at the spatial location of the pixel.

[0011] Furthermore, the gating feature representation obtained by adaptive fusion and uncertainty evaluation based on the multi-vehicle sparse features includes: A sparse feature compression algorithm is used to compress the enhanced scene feature representation to obtain sparse features; The sparse features are transmitted to other vehicles through the inter-vehicle communication module to perform cross-vehicle spatiotemporal alignment, resulting in a spatiotemporally aligned set of sparse features from multiple vehicles; among them, the other vehicles are unmanned mining trucks that share information and work collaboratively with this vehicle. Based on the multi-vehicle sparse feature set and the environmental state annotation information, an enhanced feature representation is obtained by adaptive fusion using a multi-head attention mechanism. Uncertainty assessment is performed on the fused enhanced feature representation and gating weights are calculated. The gating weights are then applied to the enhanced feature representation to obtain a gated feature representation, which is the element-wise product of the enhanced feature representation and the gating weights.

[0012] Furthermore, cross-vehicle spatiotemporal alignment includes: Represent the spatial coordinates corresponding to sparse features as homogeneous coordinates; The spatial transformation matrix is ​​multiplied with the homogeneous coordinates to obtain the transformed homogeneous coordinates. The spatial transformation matrix consists of a relative translation vector and a relative rotation matrix. The relative translation vector represents the position offset of the origin of other vehicle coordinate systems relative to the origin of the reference vehicle coordinate system, and the relative rotation matrix represents the attitude transformation of other vehicle coordinate systems relative to the reference vehicle coordinate system. Divide the first three components of the transformed homogeneous coordinates by the fourth component to obtain the transformed three-dimensional coordinates. At the transformed 3D coordinate position, the feature values ​​of the original sparse features are retained to obtain the aligned features; The spatiotemporally aligned multi-vehicle sparse feature set includes multiple aligned features, which are enhanced scene features observed by multiple vehicles from different perspectives.

[0013] Furthermore, the obstacle detection results include: The obstacle detection model is built on an improved YOLO network architecture, including a multi-scale feature extraction module, an efficient channel attention enhancement module, a bidirectional feature pyramid network, and a detection head. The gated feature representation is input into the multi-scale feature extraction module, and the multi-scale feature extraction module outputs multi-scale fused features. The multi-scale fusion features are input into the efficient channel attention module. Global average pooling is performed on the multi-scale fusion features. After applying one-dimensional convolution to the global average pooling result, the channel attention weights are calculated using the Sigmoid activation function. The multi-scale features and the channel attention weights are multiplied element-wise to obtain the feature weighted output. The weighted feature output is input into a bidirectional feature pyramid network for weighted fusion to obtain a fused multi-level feature representation. Based on the fused multi-level feature representation, the detection head outputs obstacle detection results, including obstacle bounding boxes, category labels, and detection confidence.

[0014] Furthermore, the multidimensional occupancy grid output multidimensional obstacle data includes: Multidimensional obstacle data includes three-dimensional spatial coordinates, time series, and occupancy confidence; the three-dimensional spatial coordinates represent the index positions of the spatial units occupied by obstacles in the horizontal, vertical, and longitudinal directions, and the time series represents the occupancy status of the spatial units occupied by obstacles at different times; The obstacle detection results are projected from the two-dimensional image plane onto a three-dimensional spatial grid. The three-dimensional spatial grid takes the vehicle's position as the origin and is uniformly divided into multiple spatial units in the horizontal and vertical directions. Each spatial unit represents a cubic voxel. For each obstacle bounding box, starting from the camera optical center, a ray is emitted along the line of sight corresponding to each pixel within the bounding box. The effective propagation distance of each ray in three-dimensional space is determined using depth information. The spatial units that the ray passes through are judged, and the vote count for each spatial unit that the ray passes through is increased by one. Calculate the three-dimensional occupancy probability. The three-dimensional occupancy probability is equal to the number of ray votes received by the spatial unit divided by the total number of perspectives participating in the vote. The total number of perspectives includes the perspective of this vehicle and the perspectives of other vehicles received through the inter-vehicle communication module. The exponential moving average method is used for time series fusion. The time series occupancy probability at the current time is the product of the preset time series fusion weight and the current three-dimensional occupancy probability, plus the product of the preset time series fusion weight complement and the time series occupancy probability at the previous time. Ghost targets are identified and suppressed through motion consistency test. For each spatial unit that was determined to be occupied in the previous moment, the predicted velocity is calculated by Kalman filter estimator. The observed velocity is calculated by analyzing the changes in the occupancy distribution between the current moment and the previous moment. The Euclidean distance between the predicted velocity vector and the observed velocity vector is calculated. If the Euclidean distance is less than the preset motion consistency threshold, the ghost penalty factor is set to 1. Otherwise, the ghost penalty factor is calculated. The ghost penalty factor is 1 minus the bias term. The bias term is the product of the normalized bias and the preset penalty intensity coefficient. The normalized bias is the Euclidean distance between the predicted velocity and the observed velocity minus the preset motion consistency threshold and then divided by the preset motion consistency threshold. Calculate the final confidence level, which is equal to the product of the gating weight, the temporal occupancy probability, and the ghost penalty factor. For spatial cells with a final confidence level higher than the preset occupancy threshold, they are determined to be occupied by obstacles. The final confidence level of a spatial cell occupied by an obstacle is the occupancy confidence level.

[0015] This invention provides an obstacle detection system based on unmanned mining trucks, which stores computer-readable instructions and, when read, can execute the aforementioned obstacle detection method based on unmanned mining trucks; the system includes: The data processing module collects and preprocesses vehicle data; The dust and fog modeling module constructs a dust and fog evolution prediction network based on the vehicle data and obtains a dust and fog density field. Based on the dust and fog density field, it performs defogging processing on the vehicle data to obtain a defogging multi-spectral domain image. Based on the polarization image data, it performs polarization material decoupling, extracts the distribution information of different materials in the scene, and combines it with the defogging multi-spectral domain image to form an enhanced scene feature representation. The collaborative perception module compresses and spatiotemporally aligns the enhanced scene feature representations of multiple vehicles to obtain a multi-vehicle sparse feature set, and performs adaptive fusion and uncertainty evaluation based on the multi-vehicle sparse feature set to obtain a gated feature representation; The obstacle detection module inputs the gated feature representation into the obstacle detection model to perform obstacle detection and obtain the obstacle detection result; The occupancy grid module constructs a multi-dimensional occupancy grid based on the obstacle detection results and vehicle data, and outputs multi-dimensional obstacle data from the multi-dimensional occupancy grid.

[0016] The beneficial effects of this invention are as follows: By employing a multi-source data preprocessing and time-series alignment mechanism, this invention effectively solves the data quality problems caused by severe vehicle vibration and asynchronous sensor acquisition under complex road conditions in mining areas. Through rolling shutter correction transformation and a time-domain image stabilization filter, the distortion effects of camera rolling shutter effect and vehicle vibration on the image are eliminated. Combined with asynchronous time-series alignment based on cross-correlation functions, precise synchronization of multi-sensor data in the time dimension is achieved. Simultaneously, the introduction of a sensor health assessment mechanism can adaptively adjust the weights of each sensor according to ambient light, vibration, and noise levels, providing a high-quality and highly reliable data foundation for subsequent processing and significantly improving the system's adaptability under harsh operating conditions.

[0017] By employing a physically-based modeling-based decoupling technique for dust and fog environments, the performance bottleneck of traditional defogging algorithms in heavy dust and fog conditions in mining areas was overcome. By constructing a dust and fog evolution prediction network incorporating computational fluid dynamics priors, accurate estimation of the dust and fog density and transmittance fields in three-dimensional space was achieved. Dark channel priors and the Beer-Lambert law were used for participating medium defogging, and combined with polarization material decoupling technology, reflected and scattered light in the scene were effectively separated. This physically restored clear scene information and material characteristics obscured by dust and fog, significantly improving the recognition of obstacle features under low visibility conditions.

[0018] By constructing a multi-vehicle collaborative perception network based on sparse feature sharing, the problems of blind spots and insufficient long-range perception capabilities in single-vehicle perception are effectively overcome. Principal component analysis is used to sparsely compress enhanced scene features, preserving key semantic information while meeting low-bandwidth communication constraints. Adaptive weighted fusion of multi-vehicle perspective features is achieved by utilizing the cross-attention fusion mechanism of the Transformer architecture and combining it with environmental state annotation information. In particular, an uncertainty gating mechanism is introduced to comprehensively evaluate accidental and cognitive uncertainties, effectively suppressing feature responses in high-uncertainty regions and significantly enhancing the system's robustness and detection reliability in dynamic and complex environments.

[0019] This invention significantly improves the detection performance of multi-scale and multi-morphological obstacles in mining areas through an improved YOLO detection network. Multi-scale feature extraction is achieved using grouped convolution, combined with an efficient channel attention module and a bidirectional feature pyramid network, enhancing the network's ability to represent and transfer features of small targets and targets with blurred boundaries. The full intersection-union loss function is used to optimize bounding box prediction, comprehensively considering overlap, center distance, and aspect ratio consistency, achieving accurate regression of obstacle position and size, effectively solving the problem of missed detections and false detections caused by drastic changes in obstacle scale on unstructured roads in mining areas.

[0020] This invention proposes a five-dimensional occupancy grid and consistency verification method, achieving a unified representation of obstacles in spatiotemporal and confidence dimensions. Through multi-view ray voting and three-dimensional digital differential analysis, the two-dimensional detection results are mapped to three-dimensional space. Combined with an anti-ghost suppression mechanism based on Kalman filtering, motion consistency checks effectively eliminate false targets generated by multi-view projections. Temporal consistency fusion smooths the temporal evolution of obstacle states, ultimately outputting a five-dimensional occupancy grid containing high-confidence information. This provides an accurate, stable, and reliable environmental perception model for path planning and obstacle avoidance decisions of unmanned mining trucks, comprehensively ensuring the safety of autonomous driving operations in mining areas. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an obstacle detection method based on an unmanned mining truck according to the present invention. Figure 2 This is an example diagram illustrating the formation of enhanced scene feature representation for an obstacle detection method based on unmanned mining trucks according to the present invention; Figure 3 This is an example diagram illustrating the gating feature representation obtained from an obstacle detection method based on an unmanned mining truck according to the present invention. Figure 4 This is a module example diagram of an obstacle detection system based on an unmanned mining truck according to the present invention. Detailed Implementation

[0022] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0023] An obstacle detection method and system based on unmanned mining trucks includes the following embodiments: Example 1: An obstacle detection method based on unmanned mining trucks, such as Figure 1 As shown, it includes the following steps: Step 100: Collect and preprocess vehicle data. Vehicle data includes multispectral image data, depth information, and vehicle state data. The multispectral image data includes visible light images, near-infrared images, and polarized images, each with a corresponding frame timestamp. Depth information, including 3D spatial information, is acquired using a depth sensing device. Vehicle state data, including vehicle attitude and motion data, is acquired using an inertial measurement unit (IMU). The depth sensing device includes a stereo vision camera or lightweight radar to acquire 3D spatial information, while the IMU acquires vehicle attitude and motion data. This raw data forms the foundation dataset for subsequent processing.

[0024] Because the camera uses a rolling shutter mechanism, the exposure times of different rows differ, requiring geometric correction. The rolling shutter correction transform adjusts the position of each pixel in the original image according to the exposure time of its corresponding row. The new coordinates of each pixel in the corrected image are determined by a row-related transformation matrix calculated based on vehicle state data obtained from the inertial measurement unit. This transformation matrix considers the vehicle's motion state at the exposure time of that row. The reason for using rolling shutter correction is that mining trucks experience severe vibrations and rapid movement when traveling on rough roads. During the row-by-row exposure process, the imaging times of different rows differ, causing geometric distortion of moving objects in the image, manifesting as tilting, bending, or stretching deformation. This distortion severely affects the accuracy of subsequent feature extraction and target detection; therefore, correction must be performed in the preprocessing stage to restore the true geometric relationships of the scene. The reason for choosing inertial measurement unit (IMU) data as input is that IMU can measure the angular velocity and acceleration of the vehicle body in real time at a high frequency. Its sampling frequency is usually 100 to 1000 times per second, which is much higher than the frame rate of the camera. It can accurately capture the instantaneous motion state of each line of exposure and provide accurate motion parameters for line-by-line correction.

[0025] Next, a temporal image stabilization filter is applied to eliminate the impact of vehicle vibration. The reason for using temporal image stabilization is that the uneven road surface in the mining area causes high-frequency vibrations during vehicle movement, resulting in jitter and displacement between consecutive frames. This jitter degrades image quality and affects subsequent optical flow estimation and feature matching accuracy. Temporal image stabilization smooths out temporal changes in the image sequence, improving the stability and continuity of visual perception. The temporal image stabilization process uses an exponentially weighted moving average method, which weights and fuses the corrected image of the current frame with the stabilization result of the previous frame. The exponentially weighted moving average method is chosen because it is computationally simple and efficient; it only requires storing the stabilization result of the previous frame to implement recursive filtering, has low memory footprint and low computational complexity, making it suitable for real-time processing. Furthermore, this method assigns exponentially decaying weights to historical frames, enabling rapid response to scene changes while maintaining temporal smoothness and avoiding motion blur. The preset weight coefficient for the current frame ranges from 0.3 to 0.7, with a default value of 0.5. This default value was determined through experimental statistical analysis based on the vibration frequency characteristics of mining trucks under standard road conditions. When the vehicle speed is below 20 km / h, the preset weight coefficient is 0.3 to enhance stability; when the vehicle speed is between 20 km / h and 40 km / h, the preset weight coefficient is 0.5 to balance stability and response speed; and when the vehicle speed is above 40 km / h, the preset weight coefficient is 0.7 to improve response speed. The weight of the previous frame's image stabilization result is the complement of the current frame's preset weight coefficient. After image stabilization processing, a geometrically corrected and stable multispectral image sequence is obtained.

[0026] Data acquisition from different sensors involves time delays, necessitating time alignment to ensure data consistency. This alignment is crucial because multispectral image data, depth information, and vehicle status data originate from different sensor devices. Each sensor has varying sampling frequencies, data processing delays, and transmission delays, leading to timestamp discrepancies in the system for observations at the same physical moment. Without time alignment, subsequent multi-source data fusion will incorrectly correlate data from different times, resulting in feature misalignment and detection errors. Delay estimation employs a cross-correlation function calculation method. This method determines the time delay between sensors by finding the time offset that maximizes the cross-correlation function of the two sensor feature signals. The cross-correlation function method is chosen because it automatically searches for the optimal time alignment relationship without prior knowledge of sensor delay parameters, exhibiting strong robustness and handling sensor delay drift over time. Furthermore, the cross-correlation function is insensitive to signal amplitude variations, focusing only on temporal correlation, making it suitable for processing sensor data with different physical quantities. Cross-correlation calculation involves the time-domain convolution of the characteristic signals of one sensor with the conjugates of the characteristic signals of another sensor.

[0027] The time window interpolation alignment function employs cubic spline interpolation. For image, depth, and state data requiring alignment, four control points are selected near a reference time point, and the aligned data values ​​are calculated using cubic polynomial interpolation. The cubic spline interpolation method is chosen because it generates smooth interpolation curves, ensuring the continuity of the first and second derivatives of the interpolation function at the control points. This avoids the piecewise linear effect of linear interpolation and the Runge phenomenon of high-order polynomial interpolation. For time-series interpolation of sensor data, cubic spline interpolation more accurately reflects the continuous change of physical quantities over time, improving alignment accuracy. The coefficients of the interpolation polynomial are determined using the values ​​of the control points and derivative constraints. After time-series alignment, all sensor data achieves precise synchronization in the time dimension.

[0028] Sensor health assessment employs a fault detection and isolation algorithm. For each sensor, its health score is obtained by performing an inner product operation between the quality index vector and the weight vector, followed by applying the Sigmoid activation function. Sensor health assessment is chosen because the mining environment is harsh; sensors operate under high temperature, high humidity, and high dust conditions for extended periods, making them prone to performance degradation, lens contamination, or partial malfunctions. Without real-time monitoring of sensor status, low-quality data from faulty sensors can contaminate the entire sensing system, leading to detection errors and safety hazards. Health assessment can identify abnormal sensors and reduce their data weights, improving the system's fault tolerance. The inner product operation between the quality index vector and the weight vector is chosen because this method integrates information from multiple quality dimensions, obtaining a single health score through weighted summation, facilitating subsequent data fusion and decision-making. Furthermore, the weight vector can be adjusted according to the needs of different application scenarios, offering good flexibility. The Sigmoid activation function is applied because it maps the real-valued result of the inner product operation to probability values ​​between zero and one, facilitating interpretation and comparison. Additionally, the non-linear characteristics of the Sigmoid function amplify health differences, making the distinction between high-quality and low-quality sensor scores more pronounced. The quality index vector consists of four components: signal-to-noise ratio, ambiguity, noise level, and consistency index.

[0029] The signal-to-noise ratio (SNR) is calculated as ten times the logarithm of the signal power to noise power ratio. Ambiguity is measured using the Laplacian operator variance to determine image sharpness. Noise level is estimated by analyzing the energy of high-frequency components in the image. The consistency index measures the correlation between the sensor data and data from other sensors. The preset weight vector of the quality index vector includes four components: SNR weight, ambiguity weight, noise level weight, and consistency index weight. The default values ​​for this preset weight vector are 0.3, 0.3, 0.2, and 0.2, determined based on statistical analysis of the impact of each quality index on sensor health. When ambient lighting conditions are good, the SNR and ambiguity weights are each increased by 0.05; when there is strong vibration in the environment, the noise level weight is increased by 0.1; and when a multi-sensor system is operating, the consistency index weight is increased by 0.1. After adjustment, each weight is normalized to a sum of 1.

[0030] The adaptive weight allocation of the weight vector employs a normalization method, where the weight of each sensor is equal to its health score divided by the sum of the health scores of all sensors. Based on the health assessment results, quality labeling information is added to each sensor's data, including data reliability, noise level, and recommended weights.

[0031] After step 100, the output is time-aligned vehicle data, including a time-aligned multispectral image dataset, depth information, vehicle status data, and corresponding sensor health assessment results. This preprocessed high-quality data provides a reliable input foundation for subsequent environmental modeling and dust / fog analysis.

[0032] Step 200: Construct a dust and fog evolution prediction network based on the vehicle data and obtain a dust and fog density field. Perform defogging processing on the vehicle data based on the dust and fog density field to obtain a defogging multispectral domain image. Decouple polarization materials based on the polarization image data, extract the distribution information of different materials in the scene, and combine this with the defogging multispectral domain image to form an enhanced scene feature representation, specifically as follows: Figure 2 As shown.

[0033] Based on the multispectral image dataset and sensor health assessment results output in step 100, step 200 performs physical modeling and decoupling processing on the complex environment of the mining area. Using preprocessed visible light images, near-infrared images, and polarization images, combined with depth information and vehicle status data, a physical model of the dust and fog environment is constructed. When the mining truck convoy passes through intersections, T-junctions, curve exits, or ramp merging sections, the vehicle's field of vision in the dust and fog trailing area is compensated for, and clear scene information is separated from it.

[0034] A dust and fog evolution prediction network is constructed by receiving multispectral domain image data and combining it with motion information from vehicle state data. This network, based on prior knowledge of computational fluid dynamics, employs a three-dimensional convolutional neural network structure. Inputs include wind speed field information extracted from the multispectral domain image data, dust and fog density at the previous moment, and dust and fog source terms generated by vehicle motion. The reason for constructing this network is that dust and fog concentrations are high and unevenly distributed in mining environments, severely degrading image quality and visibility, and affecting obstacle detection performance. By predicting the spatial distribution and temporal evolution of dust and fog, accurate dust and fog density field information can be provided for defogging algorithms, enabling adaptive defogging processing based on a physical model. Compared to traditional image enhancement methods, defogging based on a physical model can more realistically restore scene radiation information, avoiding over-enhancement or loss of detail. The reason for choosing to base the network on prior knowledge of computational fluid dynamics is that the propagation and diffusion of dust and fog in the air follow fluid dynamics laws. By embedding the convection-diffusion equation into the neural network structure, physical constraints can be used to guide network learning, improving the physical rationality and generalization ability of the predictions, and avoiding the overfitting problem of purely data-driven methods when training data is insufficient. The reason for adopting a three-dimensional convolutional neural network structure is that the dust and fog density field is a scalar field in three-dimensional space, requiring feature extraction and propagation modeling in three-dimensional space. Three-dimensional convolution can simultaneously capture local correlations in the length, width, and height directions of space, making it more suitable for processing volumetric data than two-dimensional convolution, and enabling more accurate modeling of the three-dimensional spatial distribution characteristics of dust and fog. The reason for choosing wind speed field information, the dust and fog density at the previous moment, and the dust and fog source term as inputs is that these three physical quantities are key parameters of the convection-diffusion equation. The wind speed field determines the convective transport direction and velocity of dust and fog, the dust and fog density at the previous moment provides the initial conditions for evolution, and the dust and fog source term describes the amount of newly generated dust and fog. Together, these three determine the spatiotemporal evolution process of the dust and fog density field.

[0035] Wind speed field information is extracted from multiple consecutive frames of images using an optical flow estimation algorithm. Utilizing the temporal variations of visible light and near-infrared images, the Lucas-Kanade optical flow method is employed to calculate pixel-level motion vectors. Then, depth information is used to project the two-dimensional optical flow field into three-dimensional space to form the wind speed field. The reason for using an optical flow estimation algorithm to extract the wind speed field is that it is difficult to deploy dense wind speed sensor arrays in mining environments. The movement of dust particles in the air forms observable motion patterns in the image sequence. The optical flow algorithm can indirectly estimate airflow velocity from the temporal variations of the images, achieving wind speed field reconstruction without additional sensors. The Lucas-Kanade optical flow method is chosen because it is based on the assumptions of constant pixel brightness and smooth motion within a local window. It solves the optical flow equation using the least squares method, resulting in high computational efficiency and robustness to noise, making it suitable for real-time processing. Compared to global optical flow methods, local optical flow methods can better handle independent motion in different regions of the scene, making them suitable for capturing spatial variations in the wind speed field. Visible light and near-infrared images were chosen as inputs because visible light images provide rich texture and edge information, facilitating optical flow calculations, while near-infrared images have strong penetration capabilities for dust and fog, maintaining good contrast even in dusty environments. Combining the two improves the robustness of optical flow estimation in complex environments. The wind speed field represents the three-dimensional vector distribution of airflow in the environment, including horizontal and vertical wind speed components. The horizontal wind speed component reflects the horizontal transport of dust and fog by ground winds, while the vertical wind speed component reflects the vertical diffusion of dust and fog caused by thermal convection and mechanical disturbances. The unit of the wind speed field is meters per second, and the spatial resolution is consistent with the spatial grid.

[0036] The dust and fog source term generated by vehicle movement is calculated using vehicle status data and environmental parameters. The dust and fog source term represents the mass of newly generated dust and fog per unit volume per unit time, and its magnitude is related to vehicle speed, tire-ground contact pressure, road surface material characteristics, and ambient humidity. When a vehicle is moving, the mechanical disturbance generated by tire-ground friction will stir up ground particles, forming dust and fog. The specific calculation method for the dust and fog source term is to first calculate the baseline dust intensity, which is equal to the square of the vehicle's current speed multiplied by the tire contact area, and then multiplied by a preset baseline dust coefficient.

[0037] The tire contact area is obtained by dividing the vehicle load by the tire pressure. The preset basic dust emission coefficient is determined according to the vehicle type. The default value of this preset basic dust emission coefficient for mining trucks is 0.05. This default value was obtained by conducting dust emission tests on different models of mining trucks in a standard mining area environment and performing regression analysis on the test data. When the load of the mining truck exceeds 80% of the rated load, the preset basic dust emission coefficient increases to 0.06; when the load of the mining truck is less than 50% of the rated load, the preset basic dust emission coefficient decreases to 0.04. Then, a preset road surface dryness coefficient is introduced for correction. The preset road surface dryness coefficient is calculated based on the ambient humidity. The default value of this preset road surface dryness coefficient is 1.0, which corresponds to the standard humidity conditions between 30% and 70%. When the ambient humidity is less than 30%, the preset road surface dryness coefficient is adjusted to 1.5 to reflect the physical characteristics of enhanced dust emission in dry environments. When the ambient humidity is greater than 70%, the preset road surface dryness coefficient is adjusted to 0.5 to reflect the dust suppression effect of particulate matter hygroscopic sedimentation in humid environments. Next, a preset particulate density coefficient is introduced for further correction. This preset particulate density coefficient is determined based on the characteristics of the road surface material. The default value of this preset particulate density coefficient is 0.8, corresponding to the most common gravel road surface material in mining areas. This default value is obtained by sampling and analyzing the particulate matter content of different road surface materials and combining it with statistical data from actual dust measurements. When the road surface material is sandy soil, the preset particulate density coefficient is adjusted to 1.2 to reflect the high loose particulate matter content of sandy soil. When the road surface material is hardened, the preset particulate density coefficient is adjusted to 0.3 to reflect the low particulate matter content of hardened road surfaces. Finally, the dust and fog source term equals the base dust intensity multiplied by the preset road surface dryness coefficient, and then multiplied by the preset particulate density coefficient. The dust and fog source term reaches its maximum value in the vehicle tire contact area and decreases exponentially with increasing spatial distance from the vehicle. The preset attenuation coefficient is 0.1 per meter, and this default value is also 0.1 per meter. This default value is obtained by fitting fluid dynamics simulation and field measurement data based on the diffusion and propagation law of dust and fog particles in the air. When the ambient wind speed is greater than 5 meters per second, the preset attenuation coefficient decreases to 0.08 per meter to reflect the effect of strong wind accelerating dust and fog diffusion. When the ambient wind speed is less than 1 meter per second, the preset attenuation coefficient increases to 0.12 per meter to reflect the slow diffusion characteristic of dust and fog under calm wind conditions. The unit of the dust and fog source term is kilograms per cubic meter per second.

[0038] The dust and fog evolution prediction network is modeled based on the convection-diffusion equation. This equation describes the rate of change of dust and fog density over time as equal to the negative divergence of dust and fog density with the wind speed field vector gradient, the product of the preset diffusion coefficient and the Laplace operator of dust and fog density, and the algebraic sum of the dust and fog source terms. Here, the wind speed field vector represents the wind speed field, the preset diffusion coefficient describes the diffusion characteristics of dust and fog, and the dust and fog source terms represent dust and fog generated by vehicle movement, etc. The preset diffusion coefficient characterizes the diffusion ability of dust and fog particles in the air, and its value directly affects the propagation speed and distribution range of dust and fog in space. The unit of the preset diffusion coefficient is square meters per second, and its value is related to factors such as ambient temperature, atmospheric pressure, particle size, and air humidity. Under standard atmospheric conditions, the default value of the preset diffusion coefficient for fine particulate matter is 0.01 square meters per second. This default value corresponds to standard mining area environmental conditions with an ambient temperature of 20 degrees Celsius, a standard atmospheric pressure of 1013 hPa, a particle size of 10 micrometers, and a relative humidity of 50%. This default value is obtained by calculation using Brownian motion theory and correction based on measured data from the mining area. When the ambient temperature is above 25 degrees Celsius, the preset diffusion coefficient increases by 20% to reflect the physical effect of increased molecular thermal motion due to temperature rise. When the ambient temperature is below 5 degrees Celsius, the preset diffusion coefficient decreases by 30% to reflect the physical effect of decreased molecular thermal motion due to temperature drop. When the atmospheric pressure is below standard atmospheric pressure, the preset diffusion coefficient increases linearly with pressure to reflect the effect of decreased air density due to reduced air pressure. When the relative humidity exceeds 80%, the preset diffusion coefficient decreases by 40% due to the increased weight of particulate matter from moisture absorption to reflect the effect of decreased diffusion capacity caused by increased particulate matter mass.

[0039] The dust and fog evolution prediction network structure consists of three parts: an encoder, intermediate layers, and a decoder. The encoder consists of three 3D convolutional layers, each using a 3×3×3 convolutional kernel. The intermediate layers include two fully connected layers with dimensions of 512 and 256, respectively. The decoder consists of three 3D deconvolutional layers, ultimately outputting the dust and fog density field. The training process of the dust and fog evolution prediction network includes the following steps: First, a training dataset is constructed by collecting multiple sets of time-series image data and corresponding ground truth dust and fog density data at different locations in the mining area and under different weather conditions. The ground truth dust and fog density is obtained through inversion of lidar scattering signals or measurement by dust concentration sensors. Each set of data includes a continuous sequence of multiple frames of images, vehicle status data, and corresponding dust and fog density field annotations. The dataset size is no less than 5000 time-series samples, covering various weather conditions such as sunny days, cloudy days, and strong winds, as well as different vehicle speeds and road surface types in the operation scenarios. The training data was then preprocessed, including image normalization, scaling, and data augmentation. Data augmentation employed methods such as random rotation, random cropping, and brightness jitter to expand the training samples and improve the network's generalization ability. Next, the network parameters were initialized. The weights of the convolutional layers in the encoder and decoder were initialized using the He initialization method, while the weights of the fully connected layers in the intermediate layers were initialized using the Xavier initialization method. The bias term was initialized to zero. The Adam optimizer was used during training, with an initial learning rate of 0.001, a batch size of 8, and 200 training epochs. The loss function was a weighted combination of physical constraint loss and data fitting loss. The physical constraint loss measures whether the network output satisfies the convection-diffusion equation, obtained by calculating the mean square error of the equation residuals. The data fitting loss measures the difference between the predicted dust density field and the true dust density field, using the mean square error loss function. The weight ratio of the two losses was set to 0.3 to 0.7, ensuring that the network follows physical laws while fitting the observed data. During training, network performance is evaluated on the validation set every 10 epochs. If the validation loss does not decrease for 5 consecutive epochs, the learning rate is reduced to 0.5 times its original value. Training stops when the learning rate drops below 0.00001 or the training epochs reach 200. After training, the model with the minimum loss on the validation set is selected as the final dust and fog evolution prediction network. The dust and fog density field is a scalar field describing the concentration distribution of dust and fog particles in three-dimensional space. It represents the dust and fog density per unit volume at each spatial location, and is the mass density of dust and fog particles. The unit of the dust and fog density field is kilograms per cubic meter, and its value typically ranges from 0 to 10 kilograms per cubic meter. The dust and fog density field exhibits a non-uniform spatial distribution, with higher density near the vehicle's path and gradually decreasing with increasing distance. The dust and fog density field also exhibits dynamic evolution characteristics over time, constantly changing under the combined influence of wind speed, diffusion coefficient, and dust and fog source terms. The dust and fog density field can accurately describe the spatial distribution and temporal evolution of dust and fog in the mining environment, providing a physical basis for subsequent transmittance calculation and defogging treatment.

[0040] The transmittance field is calculated using the Beer-Lambert law. Transmittance is equal to the negative exponent of the natural constant, which is the product of the preset scattering coefficient, dust density, and optical path distance. The preset scattering coefficient characterizes the ability of dust particles to scatter light, and its value depends on factors such as particle size distribution, shape characteristics, refractive index, and wavelength. The unit of the preset scattering coefficient is square meters per kilogram. For fine particles commonly found in mining areas, the default preset scattering coefficient in the visible light band is 5 square meters per kilogram. This default value corresponds to standard mining dust particles with a median particle size of 10 micrometers, an approximately spherical shape, and a refractive index of 1.5. This default value is obtained through calculation using Mie scattering theory combined with statistical analysis of multi-point measured data from the mining area. When the median particle size is less than 5 micrometers, the preset scattering coefficient is increased to 8 square meters per kilogram to reflect the high scattering efficiency of fine particles. When the median particle size is greater than 20 micrometers, the preset scattering coefficient is decreased to 2 square meters per kilogram to reflect the low scattering efficiency of coarse particles. The preset scattering coefficient is calculated using Mie scattering theory, requiring the input of particle size distribution parameters and complex refractive index parameters. The particle size distribution parameters are obtained through laser particle size analyzer measurement, while the complex refractive index parameter is obtained based on particle composition through table lookup or experimental determination. Optical path distance refers to the straight-line distance between the observation point and the target point in the scene, representing the path length of light propagation in a dusty or foggy medium. The unit of optical path distance is meters, and its value is obtained directly through depth sensing equipment measurement. For stereo vision cameras, the optical path distance is calculated from parallax, which is inversely proportional to the optical path distance. For lightweight radar, the optical path distance is calculated from the round-trip time of the laser pulse; the optical path distance equals the speed of light multiplied by half the round-trip time. In transmittance calculations, the optical path distance determines the degree of light attenuation in dust and fog; the greater the distance, the more severe the attenuation.

[0041] Using the sensor health assessment results provided in step 100, an adaptive dehazing strategy is applied to image data of different qualities. Dehazing is performed based on an atmospheric scattering model. The reason for adopting an adaptive dehazing strategy is that the image quality varies among different sensors. Sensors with high health have high signal-to-noise ratios and rich details, allowing for more aggressive dehazing parameters to maximize scene information recovery. Conversely, sensors with low health have higher noise levels, and using the same dehazing parameters would amplify the noise and further degrade image quality. By adaptively adjusting the dehazing intensity according to the sensor health, optimal dehazing results can be obtained on images of different qualities. The atmospheric scattering model is chosen because it is based on the physical process of light propagation in dust and fog media, accurately describing the attenuation and scattering effects of dust and fog on images. Compared to empirical image enhancement methods, dehazing based on a physical model can more realistically restore the radiation characteristics of the scene, avoiding color distortion and loss of detail. Furthermore, the parameters of the physical model have clear physical meanings, facilitating quantitative calculations in conjunction with dust and fog density field information.

[0042] The atmospheric scattering model describes the observed image intensity as the sum of the true radiance term and the light value term. The true radiance term is the product of the scene's true radiance and transmittance, while the light value term is the product of the atmospheric light value and the complement of the transmittance. The observed image intensity refers to the actual light intensity signal received by the camera sensor, representing the light intensity reaching the camera after attenuation and scattering by dust and fog. The unit of observed image intensity is watts per square meter per steradian, and its value is directly obtained through the camera sensor's analog-to-digital converter, taking into account parameters such as the sensor's quantum efficiency, gain settings, and exposure time. The scene's true radiance refers to the true light radiance intensity of the target object's surface in the scene without dust or fog interference, representing the original light intensity signal reflected or emitted by the object's surface. The unit of scene's true radiance is also watts per square meter per steradian, and its value is the target quantity to be recovered by the dehazing algorithm, estimated from the observed image intensity through the inverse transformation of the atmospheric scattering model. The observed image refers to the raw image data directly acquired by the camera sensor, including all the effects of dust and fog on light propagation. The pixel value of the observed image corresponds to the numerical value of the observed image intensity at the pixel's spatial location. The dynamic range of the pixel value depends on the camera's bit depth setting and is typically represented as an 8-bit or 16-bit integer. The observed image is acquired through the camera's image signal processor, and the processing includes steps such as dark current correction, white balance adjustment, gamma correction, and noise suppression.

[0043] The dehazed multispectral image is obtained through inverse transformation. Each pixel value in the dehazed multispectral image is equal to the difference between the observed image pixel value and the light value, divided by the larger of the transmittance and a preset minimum transmittance threshold. The default value of the preset minimum transmittance threshold is 0.1. This default value is determined based on the theoretical lower limit of transmittance under dense fog conditions and considering the stability of numerical calculations. This preset minimum transmittance threshold is used to prevent division by zero errors and avoid excessively amplified noise in extremely low transmittance regions.

[0044] Atmospheric light values ​​are estimated using a dark channel prior method. The atmospheric light value is equal to the maximum value of the observed image pixels in the brightest preset percentage pixel region. The default value for this preset percentage is 0.1%, determined based on statistical analysis of the proportion of sky areas in natural images. This preset percentage ensures that the selected candidate regions accurately represent atmospheric light characteristics without interference from ground objects. Atmospheric light values ​​represent the intensity of atmospheric scattered light at infinity or in dense fog regions, reflecting the combined effects of ambient lighting conditions and dust / fog scattering characteristics. The unit of atmospheric light value is the same as the observed image pixel value, typically a dimensionless digital intensity value. The estimation process for atmospheric light values ​​first requires calculating the dark channel image of the observed image. Each pixel value in the dark channel image is equal to the minimum value of all color channels within a preset neighborhood window at that pixel location. The default value for the preset neighborhood window size is 15×15 pixels. This default value is determined based on a comprehensive consideration of the statistical characteristics of local regions in natural images and computational efficiency. When the image resolution is higher than 1920×1080 pixels, the preset neighborhood window size is increased to 21×21 pixels to maintain the effectiveness of local statistics. When the image resolution is lower than 640×480 pixels, the preset neighborhood window size is reduced to 9×9 pixels to avoid over-smoothing. Then, the pixel values ​​of the dark channel image are sorted, and the top 0.1% of pixels with the largest values ​​are selected as candidate regions. These candidate regions typically correspond to the sky or distant fog areas in the image, where the true radiance of the scene is close to 0, and the observed image intensity is mainly contributed by atmospheric scattered light. Finally, the location with the largest pixel value in the observed image is searched within the candidate regions; the pixel value at this location is the estimated atmospheric light value. Accurate estimation of atmospheric light values ​​has a significant impact on the dehazing effect; an estimate that is too large will result in a darker image after dehazing, while an estimate that is too small will result in insufficient dehazing.

[0045] Dust and fog mask generation employs a threshold segmentation method. When the transmittance is less than a preset transmittance threshold, the dust and fog mask value is set to 1; otherwise, it is set to 0. The default value for the preset transmittance threshold is 0.5. This default value was determined based on human visual perception experiments and obstacle detection performance tests. When the transmittance is below 0.5, image quality significantly deteriorates and obstacle detection accuracy decreases markedly. This preset transmittance threshold is used to distinguish between clear areas and areas affected by dust and fog. Based on the dust and fog mask information, areas in the image affected by dust and fog are marked, obtaining environmental state annotation information.

[0046] Using the polarization image data output in step 100, material decoupling and scene feature extraction are performed. The degree of polarization is calculated using light intensity measurements in four different polarization directions. The reason for polarization material decoupling is that different materials respond differently to the polarization characteristics of light. Metal surfaces produce strong specular reflection, resulting in high polarization, while diffuse reflective surfaces such as roads and soil have lower polarization. Transparent or semi-transparent materials such as glass and plastic have specific polarization angles. By analyzing the polarization image, the distribution of different materials in the scene can be distinguished. This material information is of great value for obstacle recognition and classification; for example, it can distinguish between metal vehicles and non-metallic obstacles, improving detection accuracy. The reason for choosing four different polarization directions is that, according to Stokes parameter theory, at least four independent measurements are needed to fully describe the polarization state of light. By measuring the polarization intensity in four directions—0°, 45°, 90°, and 135°—complete polarization characteristic parameters such as the degree of polarization and polarization angle can be calculated. Compared to using only two directions, four-direction measurements provide more accurate and robust polarization information. The degree of polarization is equal to the square root of the difference between the intensity of 0-degree polarized light and the intensity of 90-degree polarized light plus the square root of the difference between the intensity of 45-degree polarized light and the intensity of 135-degree polarized light, and then divided by the sum of the intensity of 0-degree polarized light and the intensity of 90-degree polarized light.

[0047] The material consistency constraint loss function is defined as the average of the squared Euclidean distances between the predicted polarization degree and the polarization degree based on the physical model. This loss function calculates the squared difference between the predicted value and the physical model value at all pixels and then averages the results. The reason for using the material consistency constraint loss function is that the polarization-material decoupling network needs to learn the mapping relationship from polarization images to material distributions. To ensure that the material classification results output by the network are physically reasonable, physical constraints need to be introduced. The material consistency constraint requires that the polarization degree predicted by the network for each material should be consistent with the polarization degree of the physical model for that material. By minimizing the difference between the predicted value and the physical model value, prior knowledge of material optics can be incorporated into the network training process, improving the accuracy and physical reasonableness of material classification. The training process of the polarization-material decoupling network includes the following steps: First, a training dataset is constructed, collecting polarization image data containing different material targets. Each sample includes an image with four polarization directions and a corresponding material annotation map. The material annotation maps are obtained through manual annotation or measurement by a 3D scanner. The annotation categories include common materials in mining areas such as metals, road surfaces, soil, vegetation, and glass. The dataset size is no less than 3000 samples. Then, a polarization material decoupling network was constructed, employing an encoder-decoder structure. The encoder used ResNet50 as the backbone to extract polarization image features, while the decoder generated a material classification probability map through upsampling and convolutional layers. Network training employed a weighted combination of cross-entropy loss and material consistency constraint loss. The cross-entropy loss measures the accuracy of material classification, while the material consistency constraint loss measures physical plausibility. The weight ratio of the two losses was set to 0.6 to 0.4. The SGD optimizer was used during training, with an initial learning rate of 0.01, momentum of 0.9, batch size of 16, and 100 training epochs. Every 20 epochs, the learning rate was decayed to 0.1 times its original value. After training, the model with the highest classification accuracy on the validation set was selected as the final polarization material decoupling network.

[0048] By decoupling polarization materials, the distribution information of different materials in the scene is extracted, including the material features of metal surfaces, road surfaces, and obstacles. These material features are combined with the dehazed multispectral image to form an enhanced scene feature representation.

[0049] After step 200, the output includes a dehazed multispectral image, dust and fog density field, transmittance field, material distribution map, and environmental status annotation information. This data, after physical modeling and environmental decoupling, provides a clear and reliable environmental representation and feature foundation for subsequent multi-vehicle collaborative perception.

[0050] Step 300: Compress and spatiotemporally align the enhanced scene feature representations of multiple vehicles to obtain a multi-vehicle sparse feature set. Based on this multi-vehicle sparse feature set, perform adaptive fusion and uncertainty evaluation to obtain a gated feature representation, specifically as follows: Figure 3 As shown.

[0051] Step 300 overcomes the limitations of single-vehicle field of view through a multi-vehicle collaborative perception mechanism, achieving effective fusion of multi-view information and quantitative assessment of uncertainties in complex mining environments. This step addresses issues such as limited field of view of single-vehicle sensors, blind spots caused by dust and fog, and detection uncertainties caused by dynamic environmental changes in mining operations. By constructing a collaborative perception network among vehicles, it enhances the robustness and accuracy of obstacle detection by utilizing complementary information from multiple vehicle perspectives.

[0052] The input data for step 300 includes the dehazed multispectral image, dust / fog density field, transmittance field, material distribution map, and environmental status annotation information output from step 200. The dehazed multispectral image provides clear scene visual information, the material distribution map provides the spatial distribution characteristics of different materials in the scene, the dust / fog density field and transmittance field provide a quantitative description of the environmental status, and the environmental status annotation information includes annotation data such as dust / fog masks to identify areas affected by the environment. Additionally, it receives vehicle body status data from step 100 to calculate the relative pose relationships between vehicles, and the sensor health assessment results from step 100 to evaluate the reliability of the sensor data from each vehicle.

[0053] Based on the above input data, step 300 achieves multi-vehicle collaborative perception and multi-view information fusion. Utilizing clear environmental representations and feature bases, combined with the inter-vehicle communication module, a collaborative perception network is constructed to improve obstacle detection capabilities in complex mining environments. The inter-vehicle communication module supports inter-vehicle communication with bandwidth limited to less than 1 megabit per second.

[0054] The system receives the enhanced scene feature representation output from step 200, including dehazed multispectral image features and material distribution features. The enhanced scene feature representation is a high-dimensional feature vector extracted from the dehazed multispectral image output from step 200 using a convolutional neural network encoder, while also incorporating material type and spatial distribution information from the material distribution map. The dimensionality of this enhanced scene feature representation is typically between 512 and 2048 dimensions, with a spatial resolution of one-eighth to one-sixteenth of the original image resolution. The enhanced scene feature representation encodes semantic information of the scene in the channel dimension, including obstacle category features, texture features, and geometric shape features, while preserving obstacle location and scale information in the spatial dimension. Since the bandwidth of the inter-vehicle communication module is limited to less than 1 megabit per second, directly transmitting the high-dimensional feature vector would exceed the bandwidth limit, resulting in excessive communication latency. Therefore, the enhanced scene feature representation needs to be compressed to meet real-time communication requirements.

[0055] To achieve feature sharing within limited bandwidth, a sparse feature compression algorithm is employed to reduce the dimensionality of the enhanced scene feature representation. The feature compression function is based on Principal Component Analysis (PCA), which linearly transforms the enhanced scene features using a PCA transformation matrix to obtain sparse features. PCA is used because high-dimensional feature vectors contain a large amount of redundant information and correlations, and different feature dimensions are often not completely independent. PCA identifies several orthogonal directions with the largest variance in the feature space through eigenvalue decomposition. These principal component directions contain the main information of the original features. By projecting the features onto the principal component space, the most important feature information can be retained while significantly reducing dimensionality. Compared to random dimensionality reduction or simple feature selection methods, PCA is an optimal linear dimensionality reduction method that minimizes reconstruction error under a given compression ratio. The training process of the PCA transformation matrix includes the following steps: First, a large number of enhanced scene feature samples are collected. These samples come from mining area images under different scenes, lighting conditions, and dust / fog conditions, with no fewer than 10,000 samples to ensure coverage of the main distribution patterns in the feature space. Then, the mean vector of all sample feature vectors is calculated, and the mean vector is subtracted from each sample feature to obtain the centered feature. Next, the covariance matrix of the centered features is calculated. Each element of the covariance matrix represents the covariance between the corresponding two feature dimensions. Eigenvalue decomposition is performed on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors, which constitute the principal component directions. The eigenvalues ​​are sorted from largest to smallest, and the eigenvectors corresponding to the top k eigenvalues ​​are selected to form the principal component analysis transformation matrix. The value of k is determined according to a preset energy retention ratio. After training, the principal component analysis transformation matrix is ​​saved as offline parameters and can be directly loaded and used during actual runtime without online training. The principal component analysis method analyzes the variance distribution of feature vectors across various dimensions, identifies the principal component directions with the largest variance, and projects the original high-dimensional features onto these principal component directions, thereby achieving feature dimensionality reduction while retaining the most important feature information. The dimensionality of sparse features is much smaller than that of the enhanced scene features, typically compressed to one-tenth to one-fifth of the original dimension, ensuring that the feature data volume meets communication bandwidth limitations. Sparse features retain the main information of the original features while removing redundant and noisy information, improving the feature expression efficiency and transmission efficiency.

[0056] Principal Component Analysis (PCA) transformation matrices are obtained by eigenvalue decomposition of the feature covariance matrix. The feature covariance matrix describes the correlation between the dimensions of the eigenvectors. It is calculated by subtracting the feature mean from the eigenvectors of all training samples, multiplying the eigenvector by its transpose, and then averaging the product over all samples. The feature covariance matrix is ​​a symmetric positive definite matrix, with its dimension equal to the square of the eigenvector dimension. Eigenvalue decomposition of the feature covariance matrix decomposes it into the product of the transformation matrix, the diagonal eigenvalue matrix, and the transpose of the transformation matrix. Each column of the transformation matrix corresponds to an eigenvector, representing a principal component direction. The diagonal elements of the diagonal eigenvalue matrix are eigenvalues, representing the variance along the corresponding principal component direction. Larger eigenvalues ​​indicate richer information along that principal component direction and a greater contribution to the original features. By selecting the k principal components with the largest eigenvalues, the information of the original features can be preserved to the greatest extent while reducing the feature dimensionality.

[0057] Compression is achieved by retaining the first k principal components. The ratio of the sum of the first k eigenvalues ​​to the sum of all eigenvalues ​​must be no less than a preset energy retention ratio. The default value for this preset energy retention ratio is 0.9, determined through a balance optimization between feature reconstruction quality and communication bandwidth limitations. When communication bandwidth is sufficient, the preset energy retention ratio is increased to 0.95 to improve feature representation accuracy; when communication bandwidth is limited, it is decreased to 0.85 to reduce data transmission volume. The ratio of the sum of the first k eigenvalues ​​to the sum of all eigenvalues ​​is called the cumulative variance contribution rate, representing the proportion of the total variance of the original features that the retained k principal components can explain. A higher cumulative variance contribution rate indicates a more complete representation of the original features by the compressed sparse features, resulting in smaller feature reconstruction errors. By setting a threshold for the preset energy retention ratio, the number of principal components k to be retained can be automatically determined, ensuring that the compressed sparse features meet both communication bandwidth limitations and sufficient feature representation capability. The sparse feature vector is calculated by subtracting the mean of the enhanced scene features and then performing matrix multiplication with the first k columns of the transformation matrix to obtain a k-dimensional sparse feature vector. Each element of the sparse feature vector represents the projection coefficient of the original feature onto the corresponding principal component direction.

[0058] After obtaining the sparse features, they are transmitted to other vehicles in the collaborative sensing network via the vehicle-to-vehicle communication module. These other vehicles refer to unmanned mining trucks that share information and collaborate with this vehicle in the collaborative sensing network. This includes vehicles observing the scene from different directions, remote vehicles that may be outside the dust / fog area or observing from a direction with thinner dust / fog, and nearby vehicles in similar observation environments. All these vehicles are equipped with the same or compatible multi-spectral domain sensing systems and communication modules, enabling them to participate in data transmission and fusion computation within the collaborative sensing network. The vehicle-to-vehicle communication module employs dedicated short-range communication technology or cellular vehicle-to-everything (V2X) technology to support real-time data exchange between vehicles. Before transmission, the sparse features undergo quantization encoding, converting the floating-point representation of the feature values ​​into integer representations to further reduce data transmission volume. The quantization encoding uses a uniform quantization method, dividing the range of feature values ​​into several quantization intervals, each represented by an integer code. The number of quantization bits is determined based on communication bandwidth and feature precision requirements, typically using 8-bit or 16-bit quantization. The quantized sparse features are transmitted to other vehicles via a wireless channel. The receiving vehicle dequantizes the quantized features to restore them to sparse features represented by floating-point numbers.

[0059] After receiving sparse features shared by other vehicles, cross-vehicle spatiotemporal alignment is required to unify the features observed by different vehicles at different times and spatial locations into a single coordinate system. The cross-vehicle spatiotemporal alignment function considers the relative positions and attitudes between vehicles. Using the vehicle state data provided in step 100, including the GPS coordinates, heading angle, pitch angle, and roll angle of each vehicle, the relative pose relationship of each vehicle relative to the reference vehicle is calculated. The relative pose relationship includes a relative translation vector and a relative rotation matrix. The relative translation vector represents the positional offset of the origin of the other vehicle's coordinate system relative to the origin of the reference vehicle's coordinate system, calculated using the difference between the GPS coordinates of the two vehicles. The relative rotation matrix represents the attitude transformation of the other vehicle's coordinate system relative to the reference vehicle's coordinate system, constructed using the difference between the heading angle, pitch angle, and roll angle of the two vehicles. The relative rotation matrix is ​​constructed using the Euler angle to rotation matrix conversion method. First, a rotation matrix about the vertical axis is constructed based on the difference in heading angles. Then, a rotation matrix about the horizontal axis is constructed based on the difference in pitch angles. Finally, a rotation matrix about the vertical axis is constructed based on the difference in roll angles. The three rotation matrices are multiplied sequentially to obtain the relative rotation matrix.

[0060] The aligned feature is equal to the eigenvalue at the new spatial position after homogeneous coordinate transformation of the spatial coordinates corresponding to the sparse feature and the spatial transformation matrix. The specific calculation process is as follows: First, the spatial coordinates corresponding to the sparse feature are expressed in homogeneous coordinate form, i.e., a fourth component of 1 is added to the three-dimensional coordinates. Then, the spatial transformation matrix and the homogeneous coordinates are multiplied to obtain the transformed homogeneous coordinates. Finally, the first three components of the transformed homogeneous coordinates are divided by the fourth component to obtain the transformed three-dimensional coordinates. At the transformed three-dimensional coordinate position, the eigenvalues ​​of the original sparse feature are retained, thus achieving accurate spatial alignment of the feature, resulting in the aligned feature. The spatial transformation matrix consists of relative translation vectors and relative rotation matrices, used to transform features from other vehicle coordinate systems to the reference vehicle coordinate system. The spatial transformation matrix is ​​a 4x4 homogeneous transformation matrix; the top left 3x3 matrix is ​​the relative rotation matrix, the top right 3x1 matrix is ​​the relative translation vector, and the last row is 0, 0, 0, 1.

[0061] After spatiotemporal alignment, sparse features from different vehicles are unified into the coordinate system of the reference vehicle, sharing the same spatial and temporal reference bases. The spatiotemporally aligned multi-vehicle sparse feature set includes multiple aligned features, representing enhanced scene features observed from different perspectives by multiple vehicles, providing multi-view input data for subsequent cross-attention fusion. The accuracy of spatiotemporal alignment directly affects the multi-view fusion effect. Large alignment errors can lead to spatial misalignment of features from different vehicles, resulting in artifacts or reduced feature quality after fusion. Therefore, the spatiotemporal alignment process relies on high-precision vehicle state data, including centimeter-level GPS positioning accuracy and 0.1-degree-level attitude measurement accuracy. The temporal alignment process in step 100 ensures the temporal synchronization of vehicle state data and image data, providing a reliable pose information foundation for spatiotemporal alignment.

[0062] Based on the spatiotemporally aligned multi-vehicle sparse feature set, and combined with the environmental state annotation information provided in step 200, including dust and fog masks and transmittance field information, the multi-vehicle features are adaptively fused. The multi-vehicle sparse feature set includes features of the vehicle itself and features shared with other cooperating vehicles. Each feature corresponds to a spatial location in the scene, carrying semantic information and observation confidence information for that location. The environmental state annotation information provides the spatial pattern of dust and fog distribution in the scene. The dust and fog mask identifies areas severely affected by dust and fog, and the transmittance field quantifies the visibility level at different spatial locations. In areas with high dust and fog concentration and low transmittance, the feature quality observed by the vehicle itself is poor, with significant uncertainty, requiring supplementation and correction using features observed by other vehicles from different perspectives. In areas with good visibility, the feature quality of the vehicle itself is high, and it can serve as the primary information source. Therefore, it is necessary to adaptively adjust the fusion weights of different vehicle features according to the environmental state information to achieve multi-view fusion of environmental perception.

[0063] The multi-head attention mechanism employs a Transformer architecture to achieve adaptive weighted fusion of features. The Transformer architecture learns the correlations between features through a self-attention mechanism, automatically assigning appropriate fusion weights to each feature. The reason for using the Transformer architecture is that multi-vehicle collaborative perception requires fusing feature information from different vehicles and perspectives. The correlations between these features are dynamically changing, depending on the relative positions of the vehicles, the observation angle, and the environmental state. Traditional fixed-weight fusion methods cannot adapt to this dynamic change, while the Transformer's self-attention mechanism can automatically calculate the correlations between features based on the feature content, achieving data-driven adaptive fusion. Compared to convolutional neural networks, the Transformer can model long-distance dependencies and is suitable for handling multi-vehicle features with wide spatial distribution. The reason for choosing the multi-head attention mechanism is that a single attention head can only learn one feature correlation pattern, while there are various types of correlations between features in multi-vehicle collaborative scenarios, such as spatial proximity, semantic similarity, and observational complementarity. The multi-head attention mechanism computes multiple attention heads in parallel, each learning a different correlation pattern, and then concatenates and fuses the outputs of multiple heads, enabling a more comprehensive capture of the complex relationships between features. The training process of the query weight matrix, key weight matrix, and value weight matrix in the Transformer attention mechanism includes the following steps. First, a multi-vehicle cooperative perception training dataset is constructed. This dataset contains image data collected from multiple vehicles in the same scene from different perspectives, along with corresponding feature representations and obstacle annotations. Each sample group includes observation data from 3 to 5 vehicles, and the dataset size is no less than 5000 multi-vehicle scene samples. Then, a multi-vehicle cooperative perception network is constructed, comprising a feature compression module, a spatiotemporal alignment module, a Transformer fusion module, and an uncertainty evaluation module. The Transformer fusion module includes a multi-layer Transformer encoder, each layer consisting of a multi-head self-attention sublayer and a feedforward neural network sublayer. The number of attention heads is set to 8, and the dimension of each head is set to 64. Network training employs an end-to-end supervised learning approach. The loss function includes obstacle detection loss and uncertainty estimation loss. The obstacle detection loss uses a combination of cross-entropy loss and bounding box regression loss, while the uncertainty estimation loss uses negative log-likelihood loss. The training process uses the AdamW optimizer, with an initial learning rate of 0.0001, a weight decay coefficient of 0.01, a batch size of 4, and 150 training epochs. During training, the query weight matrix, key weight matrix, and value weight matrix are used as network parameters and automatically learned and updated through backpropagation. The learning objective is to minimize the end-to-end detection loss, enabling the fused features to more accurately predict obstacle locations and categories. After training, the model with the highest average detection accuracy on the validation set is selected as the final multi-vehicle cooperative perception network.The core idea of ​​the attention mechanism is to calculate attention weights based on the similarity between query features and key features, and then use these attention weights to perform a weighted sum of value features to obtain the fused features. In multi-vehicle collaborative perception scenarios, query features correspond to the feature locations that need enhancement for the current vehicle, key features correspond to the feature locations observed by all vehicles, and value features correspond to the semantic content of features from all vehicles. By calculating the similarity between query features and each key feature, it is possible to determine which vehicles' observation information is most relevant to the locations that need enhancement for the current vehicle, thereby assigning higher fusion weights to the features of these vehicles.

[0064] The calculation of attention weights first requires constructing a query matrix, a key matrix, and a value matrix. The query matrix is ​​equal to the product of the feature matrix and the query weight matrix. The feature matrix is ​​a two-dimensional matrix formed by arranging the sparse features of all vehicles according to their spatial locations. Rows correspond to different spatial locations, and columns correspond to different dimensions of the features. The query weight matrix is ​​a learnable parameter matrix, optimized through neural network training; its dimension is the feature dimension multiplied by the key vector dimension. The key vector dimension is the intermediate representation dimension of the attention mechanism, typically ranging from 64 to 256. Each row of the query matrix represents the query vector corresponding to its spatial location, used to calculate similarity with key vectors at other locations. The query matrix is ​​calculated using matrix multiplication, multiplying each row of the feature matrix by the query weight matrix to obtain the corresponding query vector.

[0065] The key matrix is ​​equal to the product of the feature matrix and the key weight matrix. The key weight matrix is ​​also a learnable parametric matrix, with dimensions equal to the feature dimension multiplied by the key vector dimension, the same as the query weight matrix. Each row of the key matrix represents the key vector corresponding to a spatial location, used to calculate similarity with the query vector. The key vector encodes key information about the features of that spatial location, used to determine whether the features of that location are relevant to the query location. The key matrix is ​​calculated similarly to the query matrix, through matrix multiplication of the feature matrix and the key weight matrix.

[0066] The value matrix is ​​equal to the product of the feature matrix and the value weight matrix. The value weight matrix is ​​the third learnable parameter matrix, and its dimension is the feature dimension multiplied by the value vector dimension. The value vector dimension can be the same as or different from the key vector dimension, and it is usually between 64 and 256. Each row of the value matrix represents the value vector at the corresponding spatial location, including the semantic content information of the features at that location. The value vectors are the final objects to be weighted and fused; the fused feature is the weighted sum of all value vectors. The value matrix is ​​calculated by matrix multiplication of the feature matrix and the value weight matrix.

[0067] After obtaining the query matrix, key matrix, and value matrix, the attention weights are calculated. The attention weights are obtained by multiplying the product of the query matrix and the transpose of the key matrix by the square root of the key vector dimension, multiplying by the vehicle confidence weight, and finally normalizing using the softmax function. The product of the query matrix and the transpose of the key matrix calculates the inner product between each pair of query vectors and key vectors. A larger inner product value indicates greater similarity between the two vectors and a stronger correlation in their spatial location features. Dividing the inner product value by the square root of the key vector dimension is for scale normalization, preventing excessively large inner product values ​​from causing the gradient to vanish in the softmax function. The square root of the key vector dimension serves as a normalization factor, ensuring that the variance of the inner product value remains stable and does not increase with the increase of the key vector dimension.

[0068] The vehicle reliability weight is determined by combining the sensor health assessment results from step 100 and the environmental status annotation information from step 200. For each vehicle, its reliability weight reflects the quality and reliability of its observation data. The calculation of the vehicle reliability weight considers three factors: sensor health score, transmittance of the observation location, and the reasonableness of the observation angle. The sensor health score comes from the sensor health assessment results from step 100, with a value ranging from 0 to 1; a higher value indicates a better sensor working condition. The transmittance of the observation location comes from the transmittance field from step 200. For the spatial location that needs enhancement for this vehicle, the transmittance value of that location is queried from the perspective of various vehicles; a higher transmittance indicates a clearer observation of that location by the vehicle. The reasonableness of the observation angle is evaluated by calculating the angle between the observation direction and the normal vector of the obstacle surface; a smaller angle indicates a more direct observation angle and more reliable observed features. The vehicle reliability weight is equal to the product of the sensor health score, the transmittance of the observation location, and the reasonableness factor of the observation angle, and then normalized for all vehicles so that the sum of the reliability weights of all vehicles is 1.

[0069] The original attention score matrix is ​​obtained by dividing the product of the query matrix and the transpose of the key matrix by the square root of the key vector dimension. Each element of the original attention score matrix represents the relevance score between the corresponding query position and the key position. Then, each row of the original attention score matrix is ​​multiplied by the credibility weight of the vehicle to which the corresponding key position belongs, resulting in a weighted attention score matrix. This weighting operation ensures that features from high-credibility vehicles receive higher attention scores and thus carry greater weight in subsequent fusion. Finally, each row of the weighted attention score matrix is ​​normalized using the softmax function to obtain the final attention weight matrix. The softmax function transforms the attention scores into a probability distribution, ensuring that the sum of the attention weights in each row is 1. Simultaneously, it amplifies score differences through an exponential function, giving higher weights to high-score positions and lower weights to low-score positions.

[0070] Based on the environmental state information provided in step 200, the feature fusion weights under different environmental conditions are adaptively adjusted. In areas with high dust and fog concentrations, the vehicle's observation is severely obstructed, resulting in low transmittance. In such cases, it is necessary to enhance the weight of remote vehicle information. Remote vehicles refer to vehicles that are far away from the vehicle and observe the scene from different directions. These vehicles may be outside the dust and fog area or observe from a direction with thinner dust and fog, providing clear information that the vehicle cannot obtain. The specific implementation of the adaptive adjustment is as follows: the areas affected by dust and fog in the vehicle's field of view are identified based on the dust and fog mask. For query locations in these areas, the credibility weight of remote vehicles is increased when calculating the attention weight. The increase is proportional to the dust and fog density at that location; the higher the dust and fog density, the greater the increase in the credibility weight of remote vehicles. At the same time, the credibility weights of the vehicle and nearby vehicles are reduced because these vehicles are in similar observation environments and are also affected by dust and fog obstruction. Through this adaptive adjustment mechanism, the complementary information of remote vehicles is fully utilized in dust and fog environments to compensate for the deficiencies in the vehicle's observation.

[0071] The collaborative feature fusion output is a weighted sum of the products of the attention weights and corresponding value matrices of all vehicles. Specifically, the calculation involves matrix multiplication of the attention weight matrix and the value matrix. Each row of the attention weight matrix represents the distribution of attention weights for all key positions at the corresponding query position, and each row of the value matrix represents the value vector for the corresponding key position. The result of the matrix multiplication is a fused feature matrix, where each row represents the fused feature vector for the corresponding query position. The fused feature vector is a weighted sum of the value vectors for all key positions, with the weights determined by the attention weights. For each spatial position that needs enhancement for this vehicle, the fused feature vector integrates the feature information observed by all vehicles at that position or related positions, automatically assigning fusion weights based on the credibility and feature relevance of each vehicle. Compared to single-vehicle features, the fused features have a higher signal-to-noise ratio, more complete information coverage, and stronger robustness.

[0072] The fused feature matrix, after undergoing residual connections and layer normalization, yields the enhanced feature representation. Residual connections add the fused feature matrix to the original feature matrix, preserving information from the original features and preventing information loss during fusion. Layer normalization normalizes each feature vector, ensuring a mean of 0 and a variance of 1, improving the training stability of subsequent neural network layers. The enhanced feature representation incorporates comprehensive information from multi-vehicle collaborative observations and environment-adaptive fusion weights, providing high-quality feature input for subsequent uncertainty assessment.

[0073] Based on the enhanced feature representation, the transmittance field and dust / fog density field information output in step 200 are combined with the sensor health assessment results from step 100 to perform a comprehensive uncertainty assessment. Although the enhanced feature representation incorporates multi-vehicle collaborative information, uncertainty still exists. This uncertainty stems from two aspects: first, the random uncertainty caused by random noise and environmental interference in the data itself; and second, the cognitive uncertainty caused by the limited cognitive ability of the model and insufficient training data. Random uncertainty is inherent in the data and cannot be eliminated; increasing model complexity or the amount of training data cannot reduce it. Cognitive uncertainty is model-related and can be eliminated; it can be reduced by improving the model structure or increasing training data. Quantitatively assessing these two types of uncertainty can provide a confidence reference for subsequent obstacle detection, allowing for a reduction in detection sensitivity or an increase in validation constraints in high-uncertainty regions, and an increase in detection sensitivity in low-uncertainty regions, thereby achieving a balance between accuracy and recall.

[0074] Uncertainty assessment comprises two parts: random uncertainty and cognitive uncertainty. Random uncertainty is directly predicted through a dedicated neural network subnetwork. This subnetwork employs a multilayer perceptron structure, including three fully connected layers, with ReLU activation functions used between each layer. The reason for uncertainty assessment is that although the quality of features after collaborative perception fusion is improved, uncertainty still exists. This uncertainty stems from various factors such as sensor noise, environmental interference, occlusion, and model limitations. Without quantifying and assessing uncertainty, the subsequent obstacle detection module cannot distinguish between high-confidence and low-confidence detections, easily leading to false positives or false negatives. By explicitly estimating the uncertainty level at each spatial location, a confidence reference can be provided for the detection results. More conservative decision-making strategies can be adopted in high-uncertainty regions, while detection sensitivity can be improved in low-uncertainty regions, thereby enhancing the overall system reliability and security. The distinction between random uncertainty and cognitive uncertainty lies in their different sources and natures. Random uncertainty stems from the randomness and noise inherent in the data itself, making it an inherent uncertainty that cannot be eliminated. Cognitive uncertainty arises from the incompleteness of model knowledge and can be reduced by increasing training data or improving the model. Quantifying these two types of uncertainty separately can provide more refined guidance for system optimization. For example, high random uncertainty suggests the need to improve sensor quality or environmental conditions, while high cognitive uncertainty suggests the need to supplement training data or adjust the model structure. The reason for using a neural network subnetwork to predict random uncertainty is that it is closely related to feature content and environmental state. The level of random uncertainty varies at different spatial locations and under different environmental conditions. Neural networks can learn the complex nonlinear mapping relationship between features and uncertainty, achieving adaptive uncertainty estimation. Compared to a fixed noise model, the neural network method can more accurately capture the spatial variation and environmental dependence of uncertainty. The subnetwork takes enhanced feature representations as input while considering environmental state annotation information, including transmittance and dust / fog density fields. Environmental status annotation information is fused with enhanced feature representation through feature concatenation. Specifically, the transmittance and dust / fog density values ​​at each spatial location are used as additional feature channels and concatenated with the enhanced feature representation along the channel dimension to form an expanded feature vector. This expanded feature vector is input to the first fully connected layer of the sub-network. After linear transformation and ReLU activation, the first hidden representation is obtained. The dimension of the first hidden representation is typically between 256 and 512. The first hidden representation is then input to the second fully connected layer, undergoing similar transformations and activations to obtain the second hidden representation. The dimension of the second hidden representation is typically between 128 and 256. Finally, the second hidden representation is input to the third fully connected layer, which does not use an activation function and directly outputs the variance estimate of random uncertainty.

[0075] The variance estimate of random uncertainty is a non-negative scalar representing the noise level of the spatial location feature. A larger variance estimate indicates more severe noise interference in the observed data at that location, and lower feature reliability. Random uncertainty is closely related to environmental conditions. In areas with low transmittance and high dust / fog density, light attenuation is severe, the image signal-to-noise ratio decreases, and random uncertainty is greater. In vehicles with low sensor health scores, sensor performance degrades, measurement noise increases, and random uncertainty is also greater. The sub-network learns the relationship between features and noise levels in the training data, enabling it to automatically predict random uncertainty based on feature content and environmental conditions. The sub-network is trained using a negative log-likelihood loss function, assuming the features follow a Gaussian distribution. Its variance is predicted by the sub-network, and its mean is predicted by the main network. The sub-network parameters are optimized by maximizing the likelihood probability of the observed data.

[0076] Cognitive uncertainty is estimated using the Monte Carlo Dropout method. Monte Carlo Dropout is a Bayesian approximate inference technique that maintains the activation of the Dropout layer during the inference phase, performs multiple random forward propagations to obtain multiple predictions, and then calculates the variance of these predictions as an estimate of cognitive uncertainty. The reason for using Monte Carlo Dropout is that it can estimate cognitive uncertainty without changing the network structure or training process. It only requires maintaining the activation of the Dropout layer and performing multiple samplings during the inference phase, making it simple to implement and computationally cost-effective. Compared to complete Bayesian neural network methods, Monte Carlo Dropout does not require maintaining the posterior distribution of parameters, resulting in higher training efficiency. Furthermore, this method has been theoretically proven to be equivalent to variational Bayesian inference on the neural network weights, providing a solid theoretical foundation. The reason for choosing to estimate cognitive uncertainty through the variance of the predictions is that variance directly reflects the dispersion of the model's predictions. When the model is uncertain about its prediction of a certain input, different Dropout samplings will produce significantly different predictions, leading to increased variance. When the model has high confidence in the input, the predictions from different samplings tend to be consistent, resulting in smaller variance. Therefore, variance is a natural measure of cognitive uncertainty. Dropout layers are used during the training phase to prevent overfitting by randomly discarding a portion of the neuron outputs. In traditional inference, dropout layers are disabled, and all neurons participate in computation. However, the Monte Carlo Dropout method keeps the dropout layers active during inference, randomly discarding different neurons with each forward propagation. This is equivalent to sampling different model instances from the posterior distribution of the neural network's parameters. Different model instances produce different predictions for the same input, and the dispersion of these predictions reflects the cognitive uncertainty of the model.

[0077] The calculation process for cognitive uncertainty variance involves performing a forward propagation on the enhanced feature representation for a predetermined number of samples. Each forward propagation uses a different Dropout random seed, resulting in a predetermined number of prediction results. The predetermined number of samples ranges from 50 to 100, with a default value of 75. This default value is determined by optimizing the trade-off between uncertainty estimation accuracy and computation time. When computational resources are sufficient, the predetermined number of samples is increased to 100 to improve estimation accuracy; when real-time requirements are strict, the predetermined number of samples is reduced to 50 to reduce computational latency. For each forward propagation, the enhanced feature representation is input into the subsequent feature extraction network. After processing through multiple convolutional and fully connected layers, the predicted feature vector for that sample is obtained. The dimension of the predicted feature vector is the same as the dimension of the enhanced feature representation. The mean of each of the predetermined number of predicted feature vectors is calculated in each dimension to obtain the overall mean vector. The overall mean vector represents the average result of all sampled predictions, reflecting the expected value of the model's prediction. Then, the difference between the predicted feature vector and the overall mean vector for each sample is calculated. The square of each element in the difference vector is then calculated to obtain the squared difference vector. The squared difference vectors for a predetermined number of samples are averaged over the sampling dimension to obtain the cognitive uncertainty variance vector. Each element of the cognitive uncertainty variance vector represents the magnitude of cognitive uncertainty in the corresponding feature dimension. The summation or averaging of all elements in the cognitive uncertainty variance vector yields the overall cognitive uncertainty variance for that spatial location.

[0078] Cognitive uncertainty reflects the model's confidence in the characteristics of a spatial location. Greater cognitive uncertainty indicates greater uncertainty in the model's prediction of that location, leading to greater discrepancies in predictions among different model instances. Cognitive uncertainty is typically higher in areas with insufficient training data coverage, such as rare obstacle types, extreme environmental conditions, or targets with ambiguous boundaries. Quantifying cognitive uncertainty can identify weaknesses in the model's cognitive abilities, providing guidance for active learning and data augmentation.

[0079] After obtaining the variances of random uncertainty and cognitive uncertainty, gating weights are calculated for feature selection. The gating weights are obtained by multiplying the sum of the variances of random uncertainty and cognitive uncertainty by a negative preset uncertainty weight coefficient, and then applying the Sigmoid function. The sum of the variances of random uncertainty and cognitive uncertainty represents the overall uncertainty level at that spatial location, comprehensively considering both data noise and model cognition as sources of uncertainty. Multiplying the overall uncertainty by the negative preset uncertainty weight coefficient yields a negative weighted uncertainty. The negative sign converts uncertainty into certainty; the greater the uncertainty, the smaller the weighted value. The preset uncertainty weight coefficient controls the strength of the influence of uncertainty on the gating weights; the larger the coefficient, the more significant the impact of uncertainty. The default value of the preset uncertainty weight coefficient is 1.0, determined through grid search optimization on the validation dataset. This default value ensures that the gating mechanism achieves optimal detection performance under standard environmental conditions.

[0080] The Sigmoid function maps negative weighted uncertainty to gating weights between 0 and 1. The input to the Sigmoid function is the negative weighted uncertainty, and the output is the gating weights. Mathematically, the Sigmoid function is 1 divided by 1 plus the negative power of the natural constant. When the input is 0, the Sigmoid function outputs 0.5. When the input is positive, the Sigmoid function outputs greater than 0.5, with larger inputs resulting in an output closer to 1. When the input is negative, the Sigmoid function outputs less than 0.5, with smaller inputs resulting in an output closer to 0. For spatial locations with low uncertainty, the negative weighted uncertainty is a small negative value, the Sigmoid function output is close to 0.5 and 1, the gating weights are high, and the features at this location are preserved and participate in subsequent detection. For spatial locations with high uncertainty, the negative weighted uncertainty is a large negative value, the Sigmoid function output is close to 0, the gating weights are low, and the features at this location are suppressed, reducing their impact on subsequent detection. Through this gating mechanism, the feature responses of high-uncertainty regions are weakened, reducing false detections and improving the reliability of detection results.

[0081] The preset uncertainty weighting coefficient is dynamically adjusted based on environmental complexity and sensor quality. Environmental complexity is evaluated using the dust and fog density field and transmittance field output in step 200; higher dust and fog density and lower transmittance indicate greater environmental complexity. Sensor quality is evaluated using the sensor health assessment results from step 100; lower health scores indicate poorer sensor quality. In cases of high dust and fog concentration or poor sensor quality, the uncertainty of the observed data is inherently greater. Therefore, it is necessary to enhance the suppression effect of the gating mechanism to prevent false detections caused by low-quality data. Specifically, the preset uncertainty weighting coefficient is increased to 1.5, resulting in a lower gating weight for the same level of uncertainty, suppressing more high-uncertainty features. In cases of clear environment and good sensor quality, the uncertainty of the observed data is lower, allowing for a more relaxed gating constraint to improve detection sensitivity and avoid missed detections. Specifically, the preset uncertainty weighting coefficient is decreased to 0.7, resulting in a higher gating weight for the same level of uncertainty, retaining more features for detection.

[0082] Gating weights are applied to enhance the feature representation, resulting in a gated feature representation. The gated feature representation is equal to the element-wise product of the enhanced feature representation and the gate weights. For each spatial location, each dimension of its feature vector is multiplied by the corresponding gate weight, achieving weighted modulation of the features. At locations with high gate weights, feature values ​​are preserved or amplified; at locations with low gate weights, feature values ​​are attenuated or suppressed. Gated feature representation, while preserving features in high-confidence regions, suppresses feature responses in high-uncertainty regions, improving the overall quality and reliability of the features.

[0083] After step 300, the output includes a gated feature representation, uncertainty assessment results, and environmental adaptability weights. The gated feature representation is the final feature representation obtained after sparse feature compression and spatiotemporal alignment, cross-attention fusion, and uncertainty gating. This representation integrates scene information observed by multiple vehicles at different spatial locations. Principal component analysis is used to achieve efficient feature compression and transmission; a spatial transformation matrix is ​​used to achieve precise alignment of features in different vehicle coordinate systems; a Transformer attention mechanism is used to achieve adaptive weighted fusion of multi-view features; and a gating mechanism is used to effectively suppress features in high-uncertainty regions. Compared to single-vehicle features, the gated feature representation has stronger expressive power, higher information integrity, and better robustness, especially in areas where the vehicle's field of view is obscured by dust or fog, or where sensor performance is degraded. The fusion of complementary information from other vehicles significantly improves feature quality.

[0084] The uncertainty assessment results include two components: random uncertainty variance and cognitive uncertainty variance. Random uncertainty variance, predicted through a neural network sub-network, quantifies the data noise uncertainty in the collaborative perception process, reflecting the degree to which observed data is affected by environmental interference and sensor noise. Random uncertainty variance is larger in areas with low transmittance and high dust / fog density, and also larger for vehicles with low sensor health scores. Cognitive uncertainty variance, estimated using the Monte Carlo Dropout method, quantifies the model's cognitive uncertainty, reflecting the difference in confidence of the model's predictions of features at different spatial locations. Cognitive uncertainty variance is larger in areas with insufficient training data coverage, and also larger in rare obstacle types or extreme environmental conditions. The uncertainty assessment results provide a confidence reference for subsequent obstacle detection and risk assessment. In high-uncertainty areas, more conservative detection strategies or additional validation constraints can be adopted, while in low-uncertainty areas, more aggressive detection strategies can be adopted to improve recall.

[0085] The environmental adaptability weight is a fusion weight parameter that is dynamically adjusted based on the environmental state annotation information provided in step 200. When calculating the attention weight, the environmental adaptability weight is reflected through the vehicle credibility weight, adaptively adjusting the contribution ratio of each vehicle's information in areas with high dust / fog concentration or poor sensor quality. For areas where the vehicle's field of vision is severely obscured by dust / fog, the environmental adaptability weight increases the fusion weight of remote vehicles and decreases the fusion weight of the vehicle itself and nearby vehicles, fully utilizing the clear information observed by remote vehicles from different perspectives. For areas with good visibility, the environmental adaptability weight primarily relies on the vehicle's own observations, appropriately fusing information from other vehicles to enhance robustness. The environmental adaptability weight enables the collaborative perception system to flexibly adjust its information fusion strategy according to real-time environmental conditions, maintaining stable perception performance in complex and changing mining environments.

[0086] Gating weights are feature selection weights calculated based on uncertainty assessment, used to suppress feature responses in high-uncertainty regions. Gating weights are obtained by multiplying the sum of the variance of accidental uncertainty and the variance of cognitive uncertainty by a negative preset uncertainty weight coefficient, and then applying a sigmoid function to map it to a range of 0 to 1. In regions of low uncertainty, the gating weight is close to 1, and the features at the corresponding location are fully preserved; in regions of high uncertainty, it is close to 0, and the features at the corresponding location are strongly suppressed. Gating weights are applied to enhance feature representations by weighted modulation of features through element-wise multiplication, resulting in gated feature representations. The gating mechanism effectively reduces the interference of high-uncertainty regions on subsequent obstacle detection, reduces false detections and spurious detections, and improves the accuracy and reliability of detection results.

[0087] These collaborative perception results provide more robust and accurate feature inputs for the subsequent step 400, improving the YOLO network detection. Gated feature representations, as input features for the improved YOLO network, directly enhance obstacle detection performance due to their high quality and reliability. Uncertainty assessment results can be combined with detection confidence scores to provide a more comprehensive credibility assessment for each detection result. Environmental adaptability weights reflect the differences in observation quality across different spatial regions and can guide threshold adjustment and result selection in post-detection processing. Especially in situations with limited single-vehicle visibility or harsh environments, the collaborative perception mechanism significantly improves detection performance through multi-vehicle information fusion and uncertainty quantification, enabling unmanned mining trucks to achieve reliable obstacle perception and safe operation in complex mining environments.

[0088] Step 400: Input the gated feature representation into the obstacle detection model to perform obstacle detection and obtain the obstacle detection result.

[0089] Step 400 achieves accurate detection and localization of obstacles in complex mining environments using an obstacle detection model built upon an improved YOLO network architecture. This step addresses the challenges of obstacles in mining areas, such as large scale variations, diverse shapes, blurred boundaries, and difficulties in detecting small targets. It enhances the network's perception and feature representation capabilities for obstacles of different scales by introducing improved modules such as multi-scale feature extraction, an efficient channel attention mechanism, and a bidirectional feature pyramid network, thereby improving detection accuracy and recall. The YOLO network architecture was chosen because it is a single-stage object detection method that transforms the object detection task into a regression problem. Through a single forward propagation, it simultaneously predicts the position and category of multiple targets. Compared to two-stage detection methods such as Faster R-CNN, YOLO offers faster detection speeds, meeting the real-time perception requirements of unmanned mining trucks. Furthermore, YOLO uses global features for prediction, making fuller use of the target's contextual information and exhibiting better robustness in complex environments. The reason for improving YOLO is that the original YOLO network has a performance bottleneck when dealing with multi-scale and small target detection. In mining scenes, obstacles have a wide range of scales, from small devices tens of centimeters to large vehicles several meters in size. The original YOLO's single-scale feature extraction is difficult to take into account targets of different scales at the same time. At the same time, small targets such as roadblocks and tools in the mining environment are easily overlooked, but they are crucial to safety. Therefore, it is necessary to enhance the network's multi-scale perception capability through multi-scale feature extraction and feature pyramid fusion.

[0090] The input data for step 400 is the gated feature representation output from step 300. This gated feature representation is a high-quality feature representation that has undergone multi-vehicle view fusion and uncertainty gating, including rich scene semantic and spatial structural information. The dimensionality of the gated feature representation matches the input layer dimension of the improved YOLO network, and the spatial resolution of the feature map maintains sufficient detail to support small object detection. Additionally, the uncertainty evaluation results output from step 300 are received, including the variance of random uncertainty and the variance of cognitive uncertainty, which are used to guide confidence adjustment and result selection during the detection process.

[0091] Based on the above input data, step 400 inputs the gated feature representation into the improved YOLO network for obstacle detection. The improved YOLO network adopts an end-to-end detection architecture, directly predicting the category, location, and bounding box parameters of obstacles from the feature map. The network structure of the improved YOLO network includes a multi-scale feature extraction module, an efficient channel attention enhancement module, a bidirectional feature pyramid network, and a detection head, etc. The modules form a complete detection process through feature transfer and information fusion.

[0092] The gated feature representation output from step 300 is received as input. The multi-scale feature extraction module first performs channel grouping processing on the gated feature representation. Because the scale of obstacles in the mining area varies greatly, from small devices at close range to large vehicles at long distances, a single-scale convolutional kernel cannot simultaneously capture the feature information of obstacles at different scales. By employing a channel grouping convolution strategy, features at multiple scales can be extracted in parallel within the same network layer, improving the network's ability to perceive obstacles at different scales. The reason for adopting the channel grouping convolution strategy is that traditional multi-scale feature extraction requires building multiple parallel network branches, each using convolutional kernels of different scales. This significantly increases the number of network parameters and computational cost. Channel grouping convolution divides the feature channels into several groups, each using convolutional kernels of different scales. This achieves multi-scale feature extraction while keeping the number of parameters and computational cost controllable. Compared to serial multi-scale processing, parallel grouping convolution can capture information at multiple scales simultaneously, avoiding information loss during serial transmission. The reason for choosing four sets of convolution kernels at different scales is that the four scales can cover the main scale range of obstacles in the mining area. The 1x1 convolution kernel corresponds to pixel-level details, the 3x3 convolution kernel corresponds to local structure, the 5x5 convolution kernel corresponds to medium-scale targets, and the 7x7 convolution kernel corresponds to large-scale targets and contextual information. The combination of the four scales achieves a good balance between feature representation ability and computational efficiency.

[0093] The multi-scale feature extraction module divides the gated feature representations into four groups evenly distributed according to channel dimension, with each group accounting for one-quarter of the total number of channels. The first group uses a 1×1 convolutional kernel for convolution operations. This kernel has the smallest receptive field, primarily capturing pixel-level local details and color texture information, suitable for extracting fine features of small-scale obstacles. The second group uses a 3×3 convolutional kernel for convolution operations. This kernel has a moderate receptive field, capable of capturing spatial structure features and edge contour information of local regions, suitable for extracting shape features of medium-scale obstacles. The third group uses a 5×5 convolutional kernel for convolution operations. This kernel has a larger receptive field, capable of capturing a wider range of contextual information and overall target features, suitable for extracting global features of large-scale obstacles. The fourth group uses a 7×7 convolutional kernel for convolution operations. This kernel has the largest receptive field, capable of capturing wide-area scene semantic information and spatial relationships between targets, suitable for extracting features of large-scale obstacles and distant targets.

[0094] After four sets of convolutional operations are executed in parallel, four feature maps at different scales are obtained. These four feature maps maintain a consistent spatial resolution but encode feature information at different scales along the channel dimension. To comprehensively utilize the multi-scale feature information, a multi-scale feature concatenation operation connects the four feature maps at different scales along the channel dimension to obtain a multi-scale fused feature. The concatenation operation is achieved by stacking four feature maps along the channel dimension. The number of channels in the concatenated multi-scale fused feature is equal to the sum of the number of channels in the four feature maps. Since each group occupies one-quarter of the total number of channels, the dimension of the concatenated feature remains the same as the dimension of the input gated feature representation. The multi-scale fused feature simultaneously contains multi-level feature information from details to the global at each spatial location, providing a rich feature foundation for subsequent attention enhancement and feature pyramid fusion.

[0095] Based on multi-scale fusion features, the efficient channel attention module adaptively weights the multi-scale fusion features according to the channel dimensions, enhancing the response of important feature channels and suppressing the interference of redundant feature channels. Since multi-scale fusion features contain feature information at four different scales, the contribution of different channels to obstacle detection varies. Some channels carry key discriminative features, while others may contain noise or redundant information. By learning the dependencies between channels, important channels can be automatically identified and their weights enhanced, thereby improving the effectiveness of feature representation. The reason for adopting the channel attention mechanism is that different channels of a convolutional neural network encode different types of visual patterns, such as edges, textures, shapes, and semantics. For a specific detection task, the features of some channels are more discriminative, while some channels may contain task-irrelevant information or noise. The channel attention mechanism allows the network to automatically learn which channels are more important and dynamically adjust the channel weights. Compared to fixed feature fusion, channel attention can adaptively select and enhance useful features based on the input content, improving the efficiency of feature representation and discriminative ability. The reason for choosing the efficient channel attention module is that it uses one-dimensional convolution to capture the local dependencies between channels. Compared with the traditional channel attention module that uses fully connected layers, one-dimensional convolution has fewer parameters and higher computational efficiency. At the same time, one-dimensional convolution can capture local correlation patterns in the channel dimension, avoiding the over-parameterization problem of fully connected layers, thus reducing model complexity while maintaining performance.

[0096] The efficient channel attention module first performs global average pooling on the multi-scale fused features. Global average pooling calculates the average value of all pixels in the spatial dimension of each channel, compressing the two-dimensional feature map of each channel into a scalar value. This scalar value represents the average activation intensity of the channel's feature across the entire spatial range, reflecting the global statistical characteristics of the channel's feature. Global average pooling yields a one-dimensional vector, the length of which is equal to the number of channels in the multi-scale fused feature, with each element of the vector corresponding to the global average activation value of one channel.

[0097] To learn the dependencies between channels, further processing is needed on the one-dimensional vector after global average pooling. The efficient channel attention module uses one-dimensional convolution operations to capture local dependencies between channels. The size of the one-dimensional convolution kernel determines the range of channel dependencies that can be captured; a larger kernel captures dependencies over longer distances, but also increases computational complexity. To strike a balance between computational efficiency and feature representation power, the preset one-dimensional convolution kernel size is determined using an adaptive method. The preset kernel size is equal to the base-2 logarithm of the number of channels divided by 2, plus the absolute value of half of that, and then the nearest odd number is taken. This adaptive method ensures that the kernel size grows moderately with the number of feature channels to capture longer-distance channel dependencies. The mathematical form of this method is derived from the theoretical analysis of the relationship between channel capacity and receptive field of the convolution kernel in information theory. For features with fewer channels, a smaller kernel is used to capture dependencies mainly between adjacent channels; for features with more channels, the kernel is appropriately enlarged to capture dependencies between channels at greater distances.

[0098] Channel attention weights are calculated by applying a one-dimensional convolution to the global average pooling result, followed by a sigmoid activation function. The one-dimensional convolution operation slides along the channel dimension, performing a weighted summation of the local channel neighborhood at each location to learn the correlation patterns between channels. The sigmoid activation function maps the convolution output to a value between 0 and 1, resulting in the attention weight for each channel. An attention weight closer to 1 indicates a more important feature in that channel and should be enhanced; a weight closer to 0 indicates a less important feature and should be suppressed. The dimension of the channel attention weights is the same as the number of channels in the multi-scale fused feature, with each channel corresponding to one attention weight value.

[0099] The feature-weighted output is obtained by element-wise multiplying the multi-scale fused features with the channel attention weights. For each channel of the multi-scale fused feature, the feature values ​​at all spatial locations are multiplied by the corresponding channel attention weight, achieving feature recalibration along the channel dimension. The recalibrated features amplify the responses in important channels and weaken the responses in unimportant channels, improving the discriminative power and representational efficiency of the features. The feature-weighted output maintains the same spatial resolution and number of channels as the multi-scale fused features, but the feature quality is significantly improved, providing more discriminative feature inputs for subsequent bidirectional feature pyramid network fusion.

[0100] Based on feature-weighted output, the bidirectional feature pyramid network performs bidirectional fusion of features at different levels, enabling information flow from bottom to top and from top to bottom, enhancing information interaction between features of different scales. Since obstacle detection requires processing targets of different scales simultaneously, bottom-level features, with their high resolution and rich detail, are suitable for detecting small targets, while top-level features, with their large receptive field and rich semantic information, are suitable for detecting large targets. The bidirectional feature pyramid network can transmit detailed information from the bottom to the top and semantic information from the top to the bottom, ensuring that each layer of features possesses both detailed and semantic information, thus improving the expressive power of each layer. The reason for adopting a bidirectional feature pyramid network is that traditional feature pyramid networks only have a top-down unidirectional information flow. While they can transmit high-level semantic information to lower levels, they cannot effectively transmit detailed information from lower levels to higher levels, resulting in insufficient localization accuracy of high-level features. The bidirectional feature pyramid network adds a bottom-up information flow path, allowing bottom and top-level features to mutually enhance each other. Bottom-level features, after acquiring semantic information, can more accurately identify the category of small targets, while top-level features, after acquiring detailed information, can more accurately locate the boundaries of large targets. The reason for choosing learnable weights for feature fusion is that the contribution of features at different levels to the fusion output of the current layer is dynamic and depends on the content of the input image and the scale distribution of the target. With learnable weights, the network can automatically learn the optimal fusion strategy. Compared with fixed weight fusion, learnable weights have stronger adaptability and expressive power.

[0101] The bidirectional feature pyramid network first constructs a multi-level feature pyramid by weighting the output features and then using multiple convolutional layers and downsampling operations. The feature pyramid consists of multiple feature layers; from the bottom to the top, the spatial resolution of the feature layers gradually decreases, the receptive field gradually increases, and the semantic abstraction level gradually increases. Each feature layer corresponds to a specific scale range for detecting obstacles within that scale. During the construction of the feature pyramid, each layer's features are extracted through convolutional operations, and then the spatial resolution is reduced through downsampling operations to obtain the next layer's features.

[0102] The bidirectional feature pyramid network employs learnable weights for feature fusion, achieving adaptive multi-level feature integration. For each feature layer *i* in the feature pyramid, its fusion output depends not only on the features of that layer itself but also on the features of its neighboring layers. The neighboring layers include features from the previous layer and features from the next layer. Features from the previous layer come from a bottom-up feature propagation path, and features from the next layer come from a top-down feature propagation path. Since the spatial resolution of features from different neighboring layers differs from that of the current layer, a resizing operation is required to adjust the spatial resolution of the neighboring layer features to match that of the current layer. For features from the previous layer, since their spatial resolution is higher than that of the current layer, a downsampling operation is used for resizing; for features from the next layer, since their spatial resolution is lower than that of the current layer, an upsampling operation is used for resizing.

[0103] The fused output of feature layer i is equal to the weighted sum of the products of all neighboring layer features (after size adjustment) and their corresponding weights. Each neighboring layer feature corresponds to a learnable weight parameter, which is automatically learned during network training via backpropagation and reflects the contribution of that neighboring layer feature to the fused output of the current layer. To ensure the rationality and numerical stability of the fused weights, a weight normalization method is used to normalize the learnable weight parameters. The weight normalization method equals the normalized weight of each connection equal to the value of the learnable parameter after ReLU activation, divided by the sum of the values ​​of all learnable parameters connected to that layer after ReLU activation, plus a preset small constant. The ReLU activation function ensures that the learnable parameters are non-negative, avoiding feature cancellation caused by negative weights. The default value of the preset small constant is 0.0001, which is determined based on the numerical computation stability requirements. This preset small constant is used to prevent division by zero errors and ensure the numerical stability of the normalization calculation. The sum of the normalized weights is 1, ensuring that the numerical range of the fused output is consistent with the input features.

[0104] Through weighted fusion using a bidirectional feature pyramid network, each feature layer integrates feature information from different levels. Lower-level features are fused with semantic information from the upper levels, improving semantic understanding of small targets, while upper-level features are fused with detailed information from the lower levels, improving localization accuracy for large targets. This fused multi-level feature representation provides rich multi-scale feature inputs for subsequent detection heads, enabling the detection network to accurately detect obstacles at different scales.

[0105] Based on the fused multi-level feature representation, the detection head module performs obstacle detection and prediction on each feature layer, outputting the obstacle's category, location, and bounding box parameters. The detection head employs multiple parallel convolutional layers to predict the obstacle's category probability, bounding box center coordinate offset, bounding box width, and height, respectively. The category prediction branch outputs the probability distribution of each spatial location belonging to each obstacle category, while the bounding box prediction branch outputs the bounding box parameters corresponding to each spatial location.

[0106] To improve the accuracy of bounding box prediction, the Full Cross-Union Loss (MCR) is used for optimization during network training. The MCR comprehensively considers three aspects: bounding box overlap, center distance, and aspect ratio consistency. Compared to the traditional cross-union loss function, it can more comprehensively evaluate the matching degree between the predicted and ground truth bounding boxes. The reason for using the MCR is that the traditional cross-union loss function only considers the overlap area of ​​the bounding boxes. When two bounding boxes do not overlap, the gradient of the loss function is zero, failing to provide an optimization direction. Furthermore, the cross-union loss is not sensitive enough to the positional offset and shape changes of the bounding boxes, which may result in bounding boxes with high cross-union but with offset center positions or mismatched aspect ratios. The MCR, by introducing center distance and aspect ratio consistency penalties, overcomes the limitations of the traditional cross-union loss, providing effective gradient signals even when the bounding boxes do not overlap, while imposing finer constraints on the position and shape of the bounding boxes, thus improving the accuracy of bounding box prediction. The improved training process of the YOLO network includes the following steps. First, a training dataset for obstacle detection in mining areas is constructed. This dataset contains images collected under different mining areas, weather conditions, and lighting conditions, along with corresponding obstacle annotations. The annotations include the bounding box coordinates and category labels of obstacles, such as mining trucks, excavators, loaders, pedestrians, roadblocks, and stockpiles—common obstacles in mining areas. The dataset contains at least 20,000 images. Next, the training data undergoes preprocessing and data augmentation. Preprocessing includes image scaling to a uniform size and normalization. Data augmentation methods include random cropping, random flipping, color dithering, and random occlusion to enhance the network's robustness to different scales, viewpoints, and lighting conditions. Then, the improved YOLO network parameters are initialized. The backbone network uses weights pre-trained on the ImageNet dataset, while the detection head weights are randomly initialized. The training process uses the SGD optimizer with an initial learning rate of 0.01, momentum of 0.9, weight decay coefficient of 0.0005, batch size of 32, and 300 training epochs. The loss function consists of three parts: classification loss, bounding box regression loss, and confidence loss. The classification loss uses cross-entropy loss, the bounding box regression loss uses full intersection-union (MIU) loss, and the confidence loss uses binary cross-entropy loss. The weights of the three losses are set to 1:5:1. The training process uses a cosine annealing learning rate scheduling strategy, with the learning rate decaying cosinely with each training epoch. Every 10 epochs during training, the network performance is evaluated on the validation set, and the average accuracy is calculated. After training, the model with the highest average accuracy on the validation set is selected as the final obstacle detection model. The MUI loss function is equal to 1 minus the MUI value. A larger MUI value indicates that the predicted bounding box is closer to the ground truth bounding box, and the smaller the loss function value.

[0107] Calculating the perfect intersection-union ratio (CIU) first requires calculating the standard intersection-union ratio (SUI). The SUI equals the area of ​​the intersection between the predicted and ground truth bounding boxes divided by the area of ​​their union, reflecting the degree of overlap between the two bounding boxes. The intersection area is the area of ​​the overlapping region between the two bounding boxes, while the union area is the total area covered by both bounding boxes. The SUI ranges from 0 to 1; it is 0 when the two bounding boxes do not overlap at all, and 1 when they are completely coincident.

[0108] Building upon the standard Intersection over Union (IoU), the Complete Intersection over Union (CIO) introduces a center point distance penalty and an aspect ratio consistency penalty. The center point distance penalty is equal to the square of the Euclidean distance between the centers of the predicted and ground truth bounding boxes, divided by the square of the diagonal length of the smallest bounding rectangle enclosing both boxes. This penalty measures the deviation between the centers of the predicted and ground truth bounding boxes; a larger center point distance results in a larger penalty and a smaller CIO value. The smallest bounding rectangle is the smallest rectangle that can simultaneously enclose both the predicted and ground truth bounding boxes. Its diagonal length serves as a normalization factor, ensuring that the center point distance penalty ranges from 0 to 1, unaffected by the absolute size of the bounding boxes.

[0109] The aspect ratio consistency penalty measures the shape similarity between the predicted and ground truth bounding boxes. The aspect ratio consistency term is equal to 4 divided by the square of pi, multiplied by the square of the difference between the arctangent of the aspect ratio of the ground truth bounding box and the arctangent of the aspect ratio of the predicted bounding box. The arctangent maps the aspect ratio to between 0 and half of pi, allowing aspect ratio differences to be measured as angular differences. The aspect ratio consistency term ranges from 0 to 1; it is 0 when the two bounding boxes have the same aspect ratio, and close to 1 when the difference in aspect ratio between the two bounding boxes is maximum.

[0110] To balance the impact of the aspect ratio consistency penalty, a tradeoff factor is introduced to weight the aspect ratio consistency term. The tradeoff factor equals the aspect ratio consistency term divided by 1 minus the sum of the standard intersection-union ratio (CUI) and the aspect ratio consistency term. The design of the tradeoff factor ensures that when the CUI is low, the weight of the aspect ratio consistency penalty term is small, and the optimization focus is on improving the overlap of the bounding boxes; conversely, when the CUI is high, the weight of the aspect ratio consistency penalty term is large, and the optimization focus is on improving the shape matching of the bounding boxes.

[0111] The perfect intersection-union ratio (MIRR) equals the standard MIRR minus the center point distance penalty term, and then minus the product of the tradeoff factor and the aspect ratio consistency term. By minimizing the MIRR loss function, the network training process not only optimizes the overlap of bounding boxes, but also simultaneously optimizes the center point position and aspect ratio of the bounding boxes, so that the predicted bounding boxes and the ground truth bounding boxes achieve a high degree of consistency in position, size, and shape.

[0112] After step 400, the obstacle detection results are output, including the detected obstacle bounding boxes, category labels, and detection confidence scores. The obstacle bounding boxes are represented by rectangles and include four parameters: center coordinates, width, and height. The center coordinates indicate the obstacle's position on the image plane, while the width and height represent the obstacle's spatial dimensions. Category labels identify the obstacle type, including common mine obstacle categories such as vehicles, pedestrians, large equipment, and stockpiles. Detection confidence scores represent the network's degree of certainty about the detection result, ranging from 0 to 1; a higher value indicates a more reliable detection result. The calculation of detection confidence scores comprehensively considers the category prediction probability and bounding box prediction quality, while also adjusting for the uncertainty assessment results output in step 300. Detection confidence scores are reduced in high-uncertainty regions and maintained or increased in low-uncertainty regions, ensuring that the detection confidence scores more accurately reflect the reliability of the detection results.

[0113] Step 500: Construct a multi-dimensional occupancy grid based on the obstacle detection results and vehicle data, and output multi-dimensional obstacle data from the multi-dimensional occupancy grid.

[0114] By constructing a multi-dimensional occupancy grid, a unified representation of obstacles in three-dimensional space plus time and confidence dimensions is achieved. False detections are eliminated through multi-view consistency verification and temporal consistency constraints, improving the accuracy and stability of obstacle localization and yielding multi-dimensional obstacle data. This step addresses the problems of lack of depth information in two-dimensional image detection results, the potential for ghost targets in multi-view detection, and temporal jumps in obstacle states in dynamic scenes. A multi-view ray voting mechanism projects the two-dimensional detection results into three-dimensional space. Motion consistency verification suppresses ghost targets that do not conform to the laws of physical motion, and temporal fusion smooths the temporal evolution of obstacle states, ultimately obtaining reliable obstacle occupancy confidence. The reason for constructing a multi-dimensional occupancy grid is that two-dimensional image detection can only provide the positional information of obstacles on the image plane, lacking spatial information in the depth dimension, making it unsuitable for direct use in path planning and obstacle avoidance decisions. By combining the two-dimensional detection results with depth information and projecting them into a three-dimensional spatial grid, a three-dimensional spatial occupancy representation of obstacles can be established, providing an accurate environmental model for subsequent three-dimensional path planning. Simultaneously, the multi-dimensional occupancy grid expands the time and confidence dimensions, enabling the tracking of the temporal evolution of obstacles and quantifying the reliability of occupancy, providing richer information for dynamic obstacle avoidance and risk assessment. The reason for adopting a multi-view ray voting mechanism is that detection from a single viewpoint may be affected by occlusion, lighting, and viewpoint limitations, resulting in false detections or missed detections. In multi-vehicle collaborative scenarios, different vehicles observe the same scene from different viewpoints. The ray voting mechanism can fuse the detection results from multiple viewpoints. Real obstacles will receive a high number of votes at the intersection of rays from multiple viewpoints, while false detections from a single viewpoint will have a low number of votes due to the lack of support from other viewpoints. By statistically analyzing the votes, we can effectively distinguish between real obstacles and false detections, thereby improving the accuracy and robustness of detection.

[0115] The input data for step 500 includes the obstacle detection results, depth information, and vehicle state data output from step 400. The obstacle detection results provide the position and bounding box information of the obstacles on the 2D image plane; the depth information provides the 3D spatial structure of the scene; and the vehicle state data provides the vehicle's pose and motion state for coordinate system transformation. Additionally, the gating weights output from step 300 are received as the initial confidence level of the detection results. These gating weights reflect the assessment of detection uncertainty during the collaborative perception stage. For multi-vehicle collaborative scenarios, detection results and pose information shared by other vehicles through the inter-vehicle communication module are also received for multi-view occupancy voting.

[0116] Based on the above input data, step 500 constructs a multi-dimensional occupancy grid for consistency verification. The multi-dimensional occupancy grid is a data structure formed by extending the time dimension and confidence dimension on the basis of a three-dimensional spatial grid, and is used to uniformly represent the state information of obstacles in three aspects: space, time and confidence.

[0117] First, the obstacle detection results on the 2D image plane output from step 400 need to be projected into 3D space to establish a 3D spatial representation of the obstacles. The 3D spatial mesh is constructed with the vehicle's position as the origin, and the spatial range of the mesh is determined according to the effective detection range of the depth sensing device. The mesh is uniformly divided into several spatial units in the horizontal and vertical directions. Each spatial unit represents a cubic voxel. The side length of the voxel is determined according to the accuracy requirements of obstacle detection, and is usually between 0.2 meters and 0.5 meters. Smaller voxel side lengths can provide finer spatial resolution, but increase computational and storage overhead. Larger voxel side lengths can reduce computational complexity, but may lose spatial detail information. The default value for the voxel side length is 0.3 meters. This default value is determined comprehensively based on the typical size of obstacles in the mining area and the limitations of computing resources, which can maintain real-time requirements while ensuring detection accuracy.

[0118] For each obstacle bounding box detected in step 400, it needs to be projected from the 2D image plane onto a 3D spatial mesh. The projection process uses a ray-casting mechanism. Starting from the camera's optical center, rays are emitted along the viewing direction corresponding to each pixel within the bounding box. These rays propagate in 3D space, intersecting with spatial cells along their path. Using the depth information provided in step 100, the effective propagation distance of each ray in 3D space is determined. The depth information provides the 3D spatial depth value corresponding to each pixel in the scene, representing the distance from the scene surface corresponding to that pixel to the camera's optical center. For each pixel within the bounding box, the ray propagation distance is determined by the pixel's depth value. The ray originates from the camera's optical center, propagates along the viewing direction, and terminates at the spatial position corresponding to the depth value.

[0119] The intersection determination of rays and spatial cells employs a three-dimensional digital differential analysis (DDA) algorithm. This algorithm iteratively advances along the ray direction to determine which spatial cells the ray passes through. The reason for using DDA is that it is an efficient voxel traversal method, capable of quickly determining the intersection relationship between rays and the 3D mesh. Compared to the brute-force method of checking each voxel for intersection with the ray, DDA only accesses the voxels actually traversed by the ray, significantly reducing computational complexity. The algorithm's time complexity is proportional to the number of voxels traversed by the ray, and independent of the total number of voxels in the mesh, making it suitable for handling ray projections of large-scale 3D meshes. A ray can be represented by a parametric equation in 3D space. The coordinates of any point on the ray are equal to the coordinates of the ray's starting point plus the product of the parameter and the ray's direction vector. The parameter represents the distance along the ray direction from the starting point. Starting from the ray's starting point, the DDA algorithm determines the ray's step size and direction of advancement within the 3D mesh based on the ray's direction vector. The algorithm maintains the index of the spatial cell where the current ray is located, calculates the next intersecting spatial cell based on the intersection point of the ray and the mesh boundary, and gradually advances until the ray reaches its endpoint or exceeds the mesh range. For each spatial cell that the ray passes through, the vote count for that spatial cell is incremented by one, indicating that one ray believes that the spatial cell may be occupied by an obstacle.

[0120] In multi-vehicle cooperative scenarios, different vehicles observe the same scene from different perspectives. Each vehicle generates its own obstacle detection results and projects them into a 3D spatial grid through a ray-casting mechanism. Due to the different observation perspectives of different vehicles, their ray-casting results will form different voting distribution patterns in 3D space. For real obstacles, rays from different perspectives will converge at the obstacle's location, resulting in a high vote count at that location. For false detections or ghost targets, due to the lack of consistency support across multiple perspectives, the corresponding spatial location will have a lower vote count. By counting the number of votes received by each spatial unit, the 3D occupancy probability of that spatial unit can be calculated. The 3D occupancy probability equals the number of ray votes received by that spatial unit divided by the total number of perspectives participating in the voting. The total number of perspectives includes the vehicle's own perspective and perspectives from other vehicles received through the inter-vehicle communication module. The numerical range of the 3D occupancy probability is between zero and one; a higher value indicates a higher probability that the spatial unit is occupied by an obstacle, while a lower value indicates a higher probability that the spatial unit is vacant.

[0121] After obtaining the 3D occupancy probability at the current moment, it needs to be temporally fused with the occupancy probabilities at historical moments to improve the temporal stability and robustness of the occupancy estimation. Temporal fusion employs the exponential moving average method, which weights the observation results at the current moment and the fused results at historical moments. The reason for temporal fusion is that single-frame detection results may be affected by instantaneous noise, occlusion, or detection errors, causing the occupancy probability to jump and become unstable over time. This instability can affect subsequent path planning and decision control. Temporal fusion can utilize historical information to smooth the current observation, reduce the impact of random errors, and improve the temporal continuity and reliability of the occupancy estimation. The exponential moving average method is chosen because it is computationally simple and efficient, requiring only the fused results from the previous moment for recursive updates. It has a small memory footprint and is suitable for real-time processing. Furthermore, the exponential moving average assigns exponentially decaying weights to historical observations, with more recent historical information having a larger weight and more distant historical information having a smaller weight. This time-weighted strategy can smooth short-term fluctuations and quickly respond to real-world changes in the scene, avoiding over-reliance on outdated historical information. The temporal occupancy probability at the current moment is equal to the product of the preset temporal fusion weight and the current 3D occupancy probability, plus the product of the preset temporal fusion weight complement and the temporal occupancy probability at the previous moment. The preset temporal fusion weight controls the relative contribution of the current observation and historical information; a larger weight indicates greater trust in the current observation, while a smaller weight indicates greater reliance on historical information. The preset temporal fusion weight ranges from 0.1 to 0.3, with a default value of 0.2, determined through optimization based on a balance between temporal smoothness and response speed. When obstacles move quickly, the scene changes rapidly, requiring a faster response to the current observation; in this case, the preset temporal fusion weight is increased to 0.3 to improve the response speed to dynamic changes. When the environment is relatively static, scene changes slowly, allowing for greater reliance on historical information to enhance stability; in this case, the preset temporal fusion weight is reduced to 0.1 to enhance temporal stability. Through temporal fusion processing, the temporal occupancy probability maintains continuity and smoothness in the time dimension, avoiding the impact of random fluctuations in single-frame detection results on occupancy estimation.

[0122] Based on temporal occupancy probability, a motion consistency test is used to identify and suppress ghost targets that do not conform to the laws of physical motion. Ghost targets refer to false occupancy generated during multi-view voting due to the geometric relationship of the viewpoints or detection errors. These false occupancy may gain some voting support in space, but their motion state does not conform to the physical motion laws of real obstacles. By introducing motion model constraints, real obstacles and ghost targets can be effectively distinguished, improving the reliability of occupancy estimation. The reason for conducting the motion consistency test is that although multi-view voting can improve the accuracy of detection, under certain special geometric configurations, false detection rays from multiple views may unexpectedly converge in space, forming false occupancy with some voting support. If these ghost targets are not identified and suppressed, they will lead to unnecessary detours or incorrect obstacle avoidance decisions in path planning. Through the motion consistency test, using the prior knowledge that obstacle motion should follow physical laws, false occupancy with abnormal motion states can be effectively identified. The motion trajectory of real obstacles should be continuous and smooth, conforming to dynamic constraints, while the motion state of ghost targets, generated by detection errors, often exhibits abrupt, discontinuous, or physically incompatible characteristics. The reason for choosing the Kalman filter estimator is that it is an optimal linear state estimation method that can integrate motion model predictions and sensor observations to provide the optimal estimate in the sense of minimum mean square error in the presence of noise. Kalman filtering updates the state estimate recursively, which is computationally efficient and suitable for real-time processing. At the same time, Kalman filtering can provide uncertainty information of the state estimate, providing a confidence reference for motion consistency judgment.

[0123] The core idea of ​​motion consistency testing is to compare the consistency between the predicted motion state and the observed motion state of a spatial unit. For each spatial unit determined to be occupied in the previous moment, its position and velocity at the current moment are predicted based on its historical trajectory and motion model. The predicted velocity is calculated using a Kalman filter estimator. Kalman filtering is a recursive state estimation method that estimates the target's motion state by fusing motion model predictions and sensor observations. Kalman filtering maintains the state vector for each occupied spatial unit, which includes the spatial unit's three-dimensional position coordinates and three-dimensional velocity components. The state transition matrix describes the evolution of the state vector over time without external control input. For a uniform motion model, the current position equals the previous position plus the product of the previous velocity and the time interval, and the current velocity equals the previous velocity. The control matrix describes the influence of external control input on the state vector. For obstacle motion estimation, the control input is usually zero or set based on prior knowledge of the scene. The predicted velocity at the current moment equals the product of the state transition matrix and the velocity component in the previous state vector, plus the product of the control matrix and the previous control input.

[0124] After obtaining the predicted velocity, the observed velocity needs to be extracted from the observation data at the current moment for comparison. The observed velocity is calculated by analyzing the changes in the occupancy distribution between the current and previous moments. For a spatial cell occupied in the previous moment, the spatial cell with the highest occupancy probability within its neighborhood is searched in the occupancy distribution at the current moment. The difference in the spatial cell positions at the two moments is divided by the time interval to obtain the observed velocity. The observed velocity reflects the actual movement of the occupied area in space and is a motion estimate directly calculated from sensor observation data.

[0125] Motion consistency testing assesses the reliability of an occupation by comparing the difference between predicted and observed velocities. The Euclidean distance between the predicted and observed velocity vectors is calculated; this distance represents the straight-line distance between the two velocity vectors in three-dimensional space, reflecting the degree of deviation between the predicted and observed motion states. If the Euclidean distance is less than a preset motion consistency threshold, the motion state of the spatial unit is considered to meet the consistency condition, and the occupation is likely a genuine obstacle. If the Euclidean distance is greater than or equal to the preset motion consistency threshold, the motion state of the spatial unit is considered not to meet the consistency condition, and the occupation may be a ghost target or a detection error. The preset motion consistency threshold ranges from 2 m / s to 5 m / s, with a default value of 3.5 m / s. This default value was determined based on statistical analysis of typical movement patterns of mining vehicles and sensor measurement errors. The speed of mining vehicles typically ranges from 0 km / h to 60 km / h, with a corresponding speed change rate generally not exceeding 3 m / s. Considering sensor measurement errors and motion model errors, setting the preset motion consistency threshold to 3.5 m / s allows for effective identification of abnormal movements while tolerating reasonable errors. When sensor measurement accuracy is high and measurement error is small, a stricter consistency threshold can be used. In this case, the preset motion consistency threshold is reduced to 2 meters per second to improve the strictness of ghost suppression and reduce the retention of false occupancy. When sensor noise is high, measurement error increases, and the consistency threshold needs to be relaxed to avoid misclassifying real obstacles as ghost targets. In this case, the preset motion consistency threshold is increased to 5 meters per second to avoid misclassifying real obstacles.

[0126] For spatial cells that do not meet the motion consistency condition, a ghost penalty factor is calculated to reduce their occupancy confidence. The ghost penalty factor is determined based on the degree of deviation between the predicted velocity and the observed velocity; the larger the deviation, the heavier the penalty. Specifically, the calculation method involves subtracting a preset motion consistency threshold from the Euclidean distance between the predicted and observed velocities to obtain the deviation exceeding the threshold. This deviation is then normalized by dividing the deviation by the preset motion consistency threshold to obtain the normalized deviation. The ghost penalty factor equals one minus the deviation term, which is the product of the normalized deviation and the preset penalty intensity coefficient, and is limited to between zero and one. The preset penalty intensity coefficient controls the intensity of the penalty; a larger coefficient results in a heavier penalty. The default value of the preset penalty intensity coefficient is 0.5. This default value significantly reduces the occupancy confidence of spatial cells with high motion inconsistency, while spatial cells with low motion inconsistency receive a relatively mild penalty. For spatial cells that meet the motion consistency condition, the ghost penalty factor is set to one, indicating no penalty. Multiplying the temporal occupancy probability of each spatial cell by the corresponding ghost penalty factor yields the occupancy probability after ghost suppression processing. By employing motion consistency testing and ghost target suppression, false occupancy generated during multi-view voting is effectively eliminated, thus improving the accuracy of occupancy estimation.

[0127] Based on the temporal occupancy probability and the ghost penalty factor, confidence information from multiple sources is comprehensively fused to obtain the final occupancy confidence. The final occupancy confidence needs to comprehensively consider the uncertainty assessment of the collaborative perception stage, the consistency of multi-view voting, the continuity of temporal states, and the physical rationality of the motion model, quantitatively evaluating the reliability of occupancy from multiple dimensions. The reason for adopting multi-source confidence fusion is that single-dimensional confidence assessment is often insufficient and may miss some important reliability factors. By integrating confidence information from multiple independent dimensions, the authenticity of occupancy can be verified from different angles, improving the accuracy and robustness of the confidence assessment. Collaborative perception confidence reflects the quality of observational data, multi-view consistency confidence reflects the degree of support of spatial geometry, and motion consistency confidence reflects the physical rationality of temporal evolution. These three complement and constrain each other, jointly constituting a comprehensive confidence assessment system. The reason for choosing the product fusion method is that product fusion requires high confidence levels in all dimensions to obtain a high final confidence level, which reflects the rigor of multiple verification. Low confidence levels in any dimension will lead to a decrease in the final confidence level. This fusion strategy can effectively filter out occupants with problems in a certain dimension and reduce the false detection rate. Compared with fusion methods such as weighted summation, product fusion is more sensitive to outliers and is more suitable for safety-critical obstacle detection applications.

[0128] The final confidence score is calculated using a multi-weighted fusion method. First, the gating weights output from step 300 are introduced as the collaborative perception confidence score. The gating weights reflect the assessment of feature uncertainty during the collaborative perception stage; lower gating weights are used in high-uncertainty regions, and higher gating weights are used in low-uncertainty regions. For each spatial cell in the 3D grid, the gating weight value at its corresponding location is queried based on its projection position on the image plane. If a spatial cell corresponds to multiple image pixel locations, the average of these gating weights is taken as the collaborative perception confidence score for that spatial cell. The collaborative perception confidence score transfers the uncertainty assessment based on feature quality and environmental state from step 300 to the 3D spatial occupancy estimation, enabling the occupancy confidence score to reflect the quality differences in the observed data.

[0129] Then, the temporal occupancy probability is introduced as a multi-view consistency confidence level. The temporal occupancy probability integrates the ray voting results from multiple perspectives and temporal fusion information, reflecting the occupancy consistency of the spatial unit in multi-view observation and temporal evolution. The higher the temporal occupancy probability, the stronger the multi-view support obtained by the spatial unit, the more stable the temporal state, and the higher the reliability of occupancy.

[0130] Next, a ghost penalty factor is introduced as a confidence level for motion consistency. The ghost penalty factor reflects whether the motion state of the spatial unit conforms to the laws of physical motion. Spatial units that meet the motion consistency condition have a ghost penalty factor of one, while spatial units that do not meet the motion consistency condition have a ghost penalty factor of less than one. The motion consistency confidence level introduces motion model constraints into occupancy estimation, effectively suppressing spurious occupancy that does not conform to physical laws.

[0131] The final confidence score is obtained by productizing and fusing the collaborative perception confidence score, multi-view consistency confidence score, and motion consistency confidence score. Specifically, the final confidence score is equal to the product of the gating weight, the temporal occupancy probability, and the ghost penalty factor. This product-fusion method ensures that only spatial units with high confidence scores in all three aspects achieve a high final confidence score; a low confidence score in any one aspect will lead to a decrease in the final confidence score. This fusion strategy reflects the complementarity and constraint of multi-source information, improving the reliability of occupancy determination through confidence assessment across multiple independent dimensions.

[0132] After obtaining the final confidence score for each spatial unit, occupancy determination needs to be performed based on a preset occupancy threshold. Spatial units with a final confidence score higher than the preset occupancy threshold are determined to be occupied by obstacles, and the final confidence score of these occupied units is the occupancy confidence score. Spatial units with a final confidence score lower than or equal to the preset occupancy threshold are determined to be free space. The preset occupancy threshold needs to strike a balance between detection accuracy and recall. The default value for the preset occupancy threshold is 0.6, which was determined through optimization based on the balance between obstacle detection accuracy and recall. At this threshold, the system can maintain a high recall rate while ensuring high detection accuracy, meeting the safety and availability requirements of unmanned driving in mining areas. When the application scenario requires higher detection accuracy and needs to minimize false detections, the preset occupancy threshold can be increased to 0.75. In this case, only spatial units with very high confidence scores will be determined as occupied, reducing the false detection rate but potentially increasing the missed detection rate. When the application scenario has higher requirements for detection recall and needs to minimize false negatives, the preset occupancy threshold can be lowered to 0.45. At this point, more spatial units will be judged as occupied, which improves the recall rate but may increase the false positive rate.

[0133] All identified occupied spatial units and their final confidence scores are stored in multidimensional obstacle data, which includes three-dimensional spatial coordinates, time series, and occupancy confidence scores. The multidimensional obstacle data is a multidimensional array structure. The first three dimensions correspond to three-dimensional spatial coordinates, representing the index positions of the occupied spatial units in the horizontal, vertical, and longitudinal directions, respectively. The fourth dimension corresponds to the time series, storing the occupancy status of the occupied spatial units at different times, supporting the tracking and analysis of the temporal evolution of obstacles. The fifth dimension stores the occupancy confidence score, recording the final confidence score of each spatial unit at each time point. Multidimensional obstacle data unifies the spatial location, temporal evolution, and detection confidence scores of obstacles in a single data structure, providing complete environmental occupancy information for subsequent path planning, risk assessment, and decision control. By querying the multidimensional obstacle data, the occupancy status and confidence score of any spatial location at any time can be quickly obtained, supporting real-time obstacle avoidance and safety decisions.

[0134] Example 2 See Figure 4 As shown, an obstacle detection system based on an unmanned mining truck is provided. This system stores computer-readable instructions, which, when read, can execute the aforementioned obstacle detection method based on an unmanned mining truck. The system includes: Data processing module 101 collects and preprocesses vehicle data; The dust and fog modeling module 102 constructs a dust and fog evolution prediction network based on the vehicle data and obtains a dust and fog density field. Based on the dust and fog density field, it performs defogging processing on the vehicle data to obtain a defogging multi-spectral domain image. Based on the polarization image data, it performs polarization material decoupling, extracts the distribution information of different materials in the scene, and combines it with the defogging multi-spectral domain image to form an enhanced scene feature representation. The collaborative perception module 103 compresses and spatiotemporally aligns the enhanced scene feature representations of multiple vehicles to obtain a multi-vehicle sparse feature set, and performs adaptive fusion and uncertainty evaluation based on the multi-vehicle sparse feature set to obtain a gated feature representation; The obstacle detection module 104 inputs the gated feature representation into the obstacle detection model to perform obstacle detection and obtain the obstacle detection result; The occupancy grid module 105 constructs a multi-dimensional occupancy grid based on the obstacle detection results and vehicle data, and outputs multi-dimensional obstacle data from the multi-dimensional occupancy grid.

[0135] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. An obstacle detection method based on unmanned mining trucks, characterized in that, Includes the following steps: Collect and preprocess vehicle data; Based on the vehicle data, a dust and fog evolution prediction network is constructed and a dust and fog density field is obtained. Based on the dust and fog density field, the vehicle data is defogging to obtain a defogging multi-spectral domain image. Based on the polarization image data, polarization material decoupling is performed to extract the distribution information of different materials in the scene. This information is then combined with the defogging multi-spectral domain image to form an enhanced scene feature representation. The enhanced scene feature representations of multiple vehicles are compressed and spatiotemporally aligned to obtain a multi-vehicle sparse feature set. Based on the multi-vehicle sparse feature set, adaptive fusion and uncertainty evaluation are performed to obtain a gated feature representation. The gated feature representation is input into the obstacle detection model to perform obstacle detection, and the obstacle detection result is obtained. Based on the obstacle detection results and vehicle data, a multi-dimensional occupancy grid is constructed, and the multi-dimensional occupancy grid outputs multi-dimensional obstacle data.

2. The obstacle detection method based on unmanned mining trucks according to claim 1, characterized in that, The collection and preprocessing of vehicle data includes: Vehicle data includes multispectral image data, depth information, and vehicle status data. The multispectral image data includes visible light images, near-infrared images, and polarized images. The depth information includes three-dimensional spatial information. The vehicle status data includes vehicle attitude and motion status data. Preprocessing includes rolling shutter correction and temporal image stabilization for multispectral image data, as well as temporal alignment and sensor health assessment for data from different sensors.

3. The obstacle detection method based on unmanned mining trucks according to claim 1, characterized in that, The obtained dehazed multispectral domain image includes: A dust and fog evolution prediction network is constructed based on multispectral image data and vehicle status data in vehicle data. The dust and fog evolution prediction network outputs a dust and fog density field, which represents the dust and fog density per unit volume at each spatial location. Transmittance is calculated as a negative exponential function of the natural constant, with the exponent being the product of the scattering coefficient, dust density, and optical path distance. Based on transmittance, multispectral domain image data is dehazed using an atmospheric scattering model to obtain a dehazed multispectral domain image. When the transmittance is less than the preset transmittance threshold, the dust and fog mask value is set to 1; otherwise, it is set to 0. Based on the dust and fog mask information, the areas in the image affected by dust and fog are marked to obtain environmental status annotation information.

4. The obstacle detection method based on unmanned mining trucks according to claim 3, characterized in that, Constructing a dust and fog evolution prediction network includes: The input to the dust and fog evolution prediction network includes wind speed field information extracted from multispectral image data, dust and fog density at the previous moment, and dust and fog source terms generated by vehicle movement. Among them, the dust and fog source terms are the basic dust intensity multiplied by the preset road surface dryness coefficient and then multiplied by the preset particulate matter density coefficient. The basic dust intensity is the square of the current vehicle speed multiplied by the tire contact area and then multiplied by the preset basic dust coefficient. The dust and fog evolution prediction network is modeled based on the convection-diffusion equation, where the rate of change of dust and fog density over time is equal to the negative divergence of dust and fog density and the wind speed field vector gradient, the product of the preset diffusion coefficient and the dust and fog density Laplace operator, and the algebraic sum of the dust and fog source terms.

5. The obstacle detection method based on unmanned mining trucks according to claim 3, characterized in that, Dehazing of multispectral domain image data using atmospheric scattering models includes: According to the atmospheric scattering model, the observed image intensity is the sum of the true radiation term and the light value term. The true radiation term is the product of the scene's true radiation intensity and transmittance, and the light value term is the product of the atmospheric light value and the complement of transmittance. The observed image intensity is the light intensity signal actually received by the camera sensor, and the scene's true radiation intensity is the true light radiation intensity of the target object's surface in the scene under conditions without dust or fog interference. Atmospheric light values ​​are estimated using a dark channel prior method. The atmospheric light value is equal to the maximum value of the observed image pixels in the brightest preset percentage pixel region of the image. After dehazing, each pixel value of the multispectral domain image is equal to the difference between the observed image pixel value and the light value, divided by the larger of the transmittance and the preset minimum transmittance threshold. The observed image pixel value corresponds to the value of the observed image intensity at the spatial location of the pixel.

6. The obstacle detection method based on unmanned mining trucks according to claim 1, characterized in that, Based on the aforementioned multi-vehicle sparse features, adaptive fusion and uncertainty evaluation are performed to obtain the gating feature representation, which includes: A sparse feature compression algorithm is used to compress the enhanced scene feature representation to obtain sparse features; The sparse features are transmitted to other vehicles through the inter-vehicle communication module to perform cross-vehicle spatiotemporal alignment, resulting in a spatiotemporally aligned set of sparse features from multiple vehicles; among them, the other vehicles are unmanned mining trucks that share information and work collaboratively with this vehicle. Based on the multi-vehicle sparse feature set and the environmental state annotation information, an enhanced feature representation is obtained by adaptive fusion using a multi-head attention mechanism. Uncertainty assessment is performed on the fused enhanced feature representation and gating weights are calculated. The gating weights are then applied to the enhanced feature representation to obtain a gated feature representation, which is the element-wise product of the enhanced feature representation and the gating weights.

7. The obstacle detection method based on unmanned mining trucks according to claim 6, characterized in that, Cross-vehicle spatiotemporal alignment includes: Represent the spatial coordinates corresponding to sparse features as homogeneous coordinates. The spatial transformation matrix is ​​multiplied with the homogeneous coordinates to obtain the transformed homogeneous coordinates. The spatial transformation matrix consists of a relative translation vector and a relative rotation matrix. The relative translation vector represents the position offset of the origin of other vehicle coordinate systems relative to the origin of the reference vehicle coordinate system, and the relative rotation matrix represents the attitude transformation of other vehicle coordinate systems relative to the reference vehicle coordinate system. Divide the first three components of the transformed homogeneous coordinates by the fourth component to obtain the transformed three-dimensional coordinates. At the transformed 3D coordinate position, the feature values ​​of the original sparse features are retained to obtain the aligned features; The spatiotemporally aligned multi-vehicle sparse feature set includes multiple aligned features, which are enhanced scene features observed by multiple vehicles from different perspectives.

8. The obstacle detection method based on unmanned mining trucks according to claim 1, characterized in that, Obstacle detection results include: The obstacle detection model is built on an improved YOLO network architecture, including a multi-scale feature extraction module, an efficient channel attention enhancement module, a bidirectional feature pyramid network, and a detection head. The gated feature representation is input into the multi-scale feature extraction module, and the multi-scale feature extraction module outputs multi-scale fused features. The multi-scale fusion features are input into the efficient channel attention module. Global average pooling is performed on the multi-scale fusion features. After applying one-dimensional convolution to the global average pooling result, the channel attention weights are calculated using the Sigmoid activation function. The multi-scale features and the channel attention weights are multiplied element-wise to obtain the feature weighted output. The weighted feature output is input into a bidirectional feature pyramid network for weighted fusion to obtain a fused multi-level feature representation. Based on the fused multi-level feature representation, the detection head outputs obstacle detection results, including obstacle bounding boxes, category labels, and detection confidence.

9. The obstacle detection method based on unmanned mining trucks according to claim 1, characterized in that, Multidimensional occupancy grid output multidimensional obstacle data includes: Multidimensional obstacle data includes three-dimensional spatial coordinates, time series, and occupancy confidence; the three-dimensional spatial coordinates represent the index positions of the spatial units occupied by obstacles in the horizontal, vertical, and longitudinal directions, and the time series represents the occupancy status of the spatial units occupied by obstacles at different times; The obstacle detection results are projected from the two-dimensional image plane onto a three-dimensional spatial grid. The three-dimensional spatial grid takes the vehicle's position as the origin and is uniformly divided into multiple spatial units in the horizontal and vertical directions. Each spatial unit represents a cubic voxel. For each obstacle bounding box, starting from the camera optical center, a ray is emitted along the line of sight corresponding to each pixel within the bounding box. The effective propagation distance of each ray in three-dimensional space is determined using depth information. The spatial units that the ray passes through are judged, and the vote count for each spatial unit that the ray passes through is increased by one. Calculate the three-dimensional occupancy probability. The three-dimensional occupancy probability is equal to the number of ray votes received by the spatial unit divided by the total number of perspectives participating in the vote. The total number of perspectives includes the perspective of this vehicle and the perspectives of other vehicles received through the inter-vehicle communication module. The exponential moving average method is used for time series fusion. The time series occupancy probability at the current time is the product of the preset time series fusion weight and the current three-dimensional occupancy probability, plus the product of the preset time series fusion weight complement and the time series occupancy probability at the previous time. Ghost targets are identified and suppressed through motion consistency test. For each spatial unit that was determined to be occupied in the previous moment, the predicted velocity is calculated by Kalman filter estimator. The observed velocity is calculated by analyzing the changes in the occupancy distribution between the current moment and the previous moment. The Euclidean distance between the predicted velocity vector and the observed velocity vector is calculated. If the Euclidean distance is less than the preset motion consistency threshold, the ghost penalty factor is set to 1. Otherwise, the ghost penalty factor is calculated. The ghost penalty factor is 1 minus the bias term. The bias term is the product of the normalized bias and the preset penalty intensity coefficient. The normalized bias is the Euclidean distance between the predicted velocity and the observed velocity minus the preset motion consistency threshold and then divided by the preset motion consistency threshold. Calculate the final confidence level, which is equal to the product of the gating weight, the temporal occupancy probability, and the ghost penalty factor. For spatial cells with a final confidence level higher than the preset occupancy threshold, they are determined to be occupied by obstacles. The final confidence level of a spatial cell occupied by an obstacle is the occupancy confidence level.

10. An obstacle detection system based on unmanned mining trucks, characterized in that, It is used to store computer-readable instructions, which, when read, can execute the obstacle detection method based on an unmanned mining truck as described in any one of claims 1-9; the system includes: The data processing module collects and preprocesses vehicle data; The dust and fog modeling module constructs a dust and fog evolution prediction network based on the vehicle data and obtains a dust and fog density field. Based on the dust and fog density field, it performs defogging processing on the vehicle data to obtain a defogging multi-spectral domain image. Based on the polarization image data, it performs polarization material decoupling, extracts the distribution information of different materials in the scene, and combines it with the defogging multi-spectral domain image to form an enhanced scene feature representation. The collaborative perception module compresses and spatiotemporally aligns the enhanced scene feature representations of multiple vehicles to obtain a multi-vehicle sparse feature set, and performs adaptive fusion and uncertainty evaluation based on the multi-vehicle sparse feature set to obtain a gated feature representation; The obstacle detection module inputs the gated feature representation into the obstacle detection model to perform obstacle detection and obtain the obstacle detection result; The occupancy grid module constructs a multi-dimensional occupancy grid based on the obstacle detection results and vehicle data, and outputs multi-dimensional obstacle data from the multi-dimensional occupancy grid.

Images