Mine unmanned vehicle cooperative perception method based on internet of vehicles

Abstract

The application discloses a mine unmanned vehicle cooperative perception method based on Internet of Vehicles, first, through a degradation feature compensation module based on a space quality map guide, a sparse Transform network is used for structural reconstruction of a point cloud degradation area caused by dust and water mist; secondly, aiming at pose drift caused by mine GPS rejection, a cross-vehicle registration module based on feature correlation is designed, and feature adaptive registration is realized by means of a deformable convolution implicit regression space offset; finally, a dynamic feature fusion mechanism based on a blind area mask guide is constructed, a double gate is used for extracting a front vehicle semantic increment, and complete fusion in a perception blind area is realized. The mine unmanned vehicle cooperative perception method based on Internet of Vehicles can effectively improve the robustness and three-dimensional detection precision of unmanned vehicle perception under extreme working conditions for complex roadway scenes such as long straight, curved and cross intersection in a coal mine.

Description

Technical Field

[0001] This invention relates to a perception method for unmanned vehicles in mines, specifically a collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) technology, belonging to the field of perception technology for unmanned driving in mines. Background Technology

[0002] Unmanned transportation in mines is a crucial aspect of smart mine construction. However, the narrow, tubular topology of the enclosed space in mines poses significant challenges to environmental perception. Since single-vehicle autonomous driving systems heavily rely on onboard LiDAR and cameras, when vehicles are traveling in platoons, the preceding vehicle often severely obstructs the view of the following vehicles, creating blind spots that fail to meet safe transportation requirements. Simultaneously, high concentrations of dust and water mist within the tunnels cause scattering and energy attenuation of laser pulses, resulting in sparse cloud features at distant points, sharp drops in echo intensity, and even missing target structures. Existing single-vehicle denoising algorithms struggle to handle this severe and non-uniform localized perception degradation, leading to low-quality data at the source of perception.

[0003] In terms of positioning, due to the complete rejection of GPS signals in deep space, vehicles rely heavily on relative positioning methods such as inertial measurement units (IMUs) and laser SLAM. The inherent cumulative errors of sensors will produce significant pose drift, making it impossible for the feature maps transmitted during multi-vehicle collaborative perception to be accurately aligned in space, thus inducing severe "ghosting" or semantic mismatch phenomena. In addition, existing collaborative perception methods are mostly designed for multi-view scenarios on urban roads and do not consider the characteristics of highly overlapping views between vehicles in a single-lane longitudinal following mode in mines. If a global feature fusion strategy is directly adopted, it will not only cause communication bandwidth redundancy due to the transmission of a large number of repetitive features, but also lead to low collaborative efficiency because it cannot effectively extract the "semantic increment" of occluded areas.

[0004] Existing collaborative sensing frameworks have significant limitations when facing the degradation of non-uniform mine environments, pose errors under weak positioning conditions, and the need for fusion of asymmetric vision. There is an urgent need to develop a mine-specific collaborative sensing method that can take into account environmental compensation, pose correction, and on-demand fusion. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) technology. This method can effectively improve the robustness of unmanned vehicle perception and the accuracy of 3D detection under extreme working conditions, especially in complex roadway scenarios such as long straight sections, curves, and intersections in coal mines.

[0006] To achieve the above objectives, this collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) specifically includes the following steps:

[0007] Step 1: Mine map scene construction and point cloud data extraction for autonomous and cooperative vehicles: First, a simulated mine map that conforms to the actual physical characteristics of underground coal mines is constructed using 3D scene modeling tools; then, the simulated mine map is imported into the autonomous driving simulation platform, and the dynamic parameters of multiple unmanned mining vehicles are configured. Multi-sensor fusion configuration is adopted, and the acquisition frequency is set to simulate a V2X cooperative scenario; through the clock synchronization mechanism of the simulation platform, multiple unmanned mining vehicles achieve inter-frame time synchronization when acquiring data in each frame, acquiring multiple independent driving scenarios, setting the duration of each scenario, and generating high-fidelity point cloud data;

[0008] Step 2: Feature compensation is performed using a single-vehicle degradation compensation module guided by spatial quality map: physical prior is used to estimate the local quality of point cloud and construct spatial quality map, which guides sparse Transformer network to dynamically identify and reconstruct the structure of local point cloud degradation areas caused by dust and water mist, thereby achieving reliable enhancement of the perception source.

[0009] Step 3: Feature space registration is performed using a cross-vehicle alignment module based on feature correlation: By calculating the global correlation of dense features across viewpoints and utilizing deformable convolutional implicit regression spatial offset, adaptive registration of heterogeneous features is achieved under weak localization conditions.

[0010] Step 4: Feature aggregation is performed using a dynamic feature fusion mechanism guided by blind spot masking: the semantic increment of the preceding vehicle is extracted using adaptive dual gating, and the system is guided to perform complete fusion only within the blind spot of the following vehicle.

[0011] Step 5: Use a 3D detection head to perform target attribute regression and output results on the fused features: First, input the complete fused feature map generated in Step 4 into the 3D detection head, and use the decoupled classification branch and regression branch to regress the 3D bounding box attributes of the target from the enhanced features; then use the 3D non-maximum suppression algorithm to filter the output candidate boxes, and eliminate redundant predictions according to the intersection-over-union ratio threshold, and finally output the decoded high-precision target list;

[0012] Step 6: Use an end-to-end joint loss function to train and optimize the overall perception framework: Use an end-to-end joint training paradigm to achieve closed-loop optimization of perception accuracy.

[0013] Furthermore, Step 2 is detailed below:

[0014] Step 2-1: First, the raw point cloud data is voxelized, and the average echo intensity of each voxel is extracted. Local point cloud density and the radial distance from the voxel center to the radar Physical attributes are concatenated along the channel dimension to construct a 3D initial feature vector. Then, a lightweight multilayer perceptron (MLP) is constructed to learn the joint distribution of degradation features. The MLP employs a two-layer fully connected architecture. First, the 3D input is mapped to a 16-dimensional hidden layer. The boundary features of the degradation region are extracted using the ReLU activation function. The output layer compresses the boundary features into a single scalar and normalizes them using the Sigmoid function, ultimately outputting the spatial quality weights. , means as follows:

[0015]

[0016] In the formula: Spatial quality weight; Use the Sigmoid activation function; It is a multilayer perceptron; The average echo intensity of the voxel unit; The local point cloud density of a voxel unit; The radial distance from the voxel center to the lidar;

[0017] Step 2-2 introduces a mask reconstruction mechanism based on sparse Transformer. During the forward propagation in the training phase, the original point cloud feature space is randomly sampled and a mask is applied to force some feature tensors to be set to zero, as follows:

[0018] set up Let the feature tensor of the region to be filled be . for Contextual features of spatial neighborhood will affect the spatial quality map. As a spatial prior, it incorporates attention mechanisms by aggregating high-confidence neighborhood context features. Implement the feature to be completed The structural reconstruction is represented as follows:

[0019]

[0020] In the formula: Features to be completed The query vector of a linear mapping; and Neighborhood context features The key vector and value vector obtained from the mapping; This is the mass weight matrix for the corresponding neighborhood unit; This represents the element-wise multiplication operation;

[0021] Step2-3, use Perform quality-perception soft fusion operations and output compensation features. , means as follows:

[0022]

[0023] In the formula: These are the original features.

[0024] Furthermore, Step 3 is detailed below:

[0025] Step 3-1, set and These are the characteristics of the cooperative vehicle after processing by the degradation compensation module and the characteristics of the autonomous vehicle after processing by the degradation compensation module, respectively. , ,in Indicates the number of channels is Space dimensions are The three-dimensional feature tensor, , , Let represent the number of channels, height, and width of the feature map, respectively. First, the heterogeneous features are concatenated along the channel dimension to obtain a joint feature tensor; then, this tensor is input into _____. Convolution and A lightweight decoder using convolutions evaluates the geometric bias between features pixel by pixel, regressing a dense two-dimensional offset field for the transformation of the preceding vehicle's features. and modulation weight matrix , , ,in This indicates that each spatial location corresponds to A two-dimensional offset set of sampling points This indicates that each spatial location corresponds to The set of modulation weights for each sampling point , , These represent the number of convolution sampling points, the height of the feature map, and the width of the feature map, respectively.

[0026] Step 3-2: In the feature resampling stage, deformable convolution is introduced, using fixed local grid coordinates. Based on this, the non-integer spatial offsets predicted by the network are superimposed. The feature tensor of the collaborative vehicle after degradation compensation module is obtained through bilinear interpolation. Up-adaptive sampling, multiplied by the corresponding modulation weights. The final output is the accurately registered collaborative features. , means as follows:

[0027]

[0028] Where: Where: The center coordinate vector of the vehicle; For the first The learnable convolutional kernel weight vector corresponding to each sampling position; The coordinates of the predefined fixed local sampling grid; The first one predicted by the network Two-dimensional spatial offset of each sampling location; For the first Modulation weights at each sampling position; , This represents the total number of local sampling points;

[0029] Step 3-3: Design a soft gating mechanism to evaluate the overall matching quality from the correlation matrix, extract the maximum matching response along the vehicle spatial dimension, and output the cross-vehicle registration confidence score through global average pooling and a multilayer perceptron. , means as follows:

[0030]

[0031] In the formula: Use the Sigmoid activation function; It is a multilayer perceptron; This indicates a global average pooling operation; This represents the maximization operation along the spatial dimension of the vehicle feature map; and These are the spatial coordinates in the feature maps of the cooperating vehicle and the autonomous vehicle, respectively. For the collaborative vehicle feature location With the location of the vehicle's features Elements of the correlation matrix between them; Registration confidence levels across vehicles;

[0032] Steps 3-4: The final registration and coordination features of the gated modulation are represented as follows:

[0033]

[0034] In the formula: This indicates the registration coordination characteristics of the final gated modulation.

[0035] Furthermore, Step 4 is detailed below:

[0036] Step 4-1, first set the vehicle features Collaborative features after registration By stitching along the channel dimension, intermediate fusion features are generated. , ,in Indicates the number of channels is Space dimensions are The three-dimensional feature tensor, , , These represent the number of channels, height, and width of the feature map, respectively; a stair-step dilated convolution is introduced to construct a spatial gating generator, which outputs a single-channel spatial gating weight matrix. Representation space fusion weights, ,in Indicates the space size as A two-dimensional matrix, and Here, the height and width of the feature map are represented as follows:

[0037]

[0038] In the formula: This is the spatial gating weight matrix; This represents a stepped dilated convolution operation; This is a feature of intermediate fusion;

[0039] Step 4-2: Introduce channel attention branches in parallel, first processing the intermediate fused features. Perform global average pooling to compress its spatial dimensions and obtain channel features. , Subsequently, cross-channel interaction weights are extracted via a two-layer multilayer perceptron, and channel gating vectors are output. , , means as follows:

[0040]

[0041] In the formula: This is the channel gating vector; and These are the learnable first-layer weight matrix and the learnable second-layer weight matrix of the fully connected layer, respectively. It is the ReLU activation function;

[0042] Step 4-3: Feature aggregation is performed using the residual soft fusion paradigm, focusing on vehicle features. Using this as a basis, we will incorporate collaborative features. through and After element-wise weighted modulation, it is used as incremental information to supplement the vehicle features. In the process, the final fused features are output. , means as follows:

[0043]

[0044] In the formula: For the final fusion feature; This indicates an element-wise multiplication operation.

[0045] Furthermore, Step 6 is detailed below:

[0046] The overall total loss function of the end-to-end joint training paradigm Including the loss of the main task of 3D object detection Compensation for degradation features and auxiliary loss Two parts;

[0047] 3D object detection main task loss It directly drives the output of 3D detection results and is composed of a combination of category classification loss, bounding box regression loss and orientation angle classification loss.

[0048] For the degradation feature compensation module, a self-supervised reconstruction mechanism is adopted: Let For the original features without a mask, To reconstruct features, Given a binary random sampling mask matrix, the self-supervised reconstruction loss... Defined as the mean square error within the masked area, it is expressed as follows:

[0049]

[0050] In the formula: Losses due to self-monitoring reconstruction;

[0051] Introducing a smoothing regularization constraint, as follows:

[0052]

[0053] In the formula: This represents a smoothing regularization constraint. Indicates the location of the masked feature; Indicates position The neighborhood feature location; Indicates position Reconstruction features at the location; For position The original features of the location;

[0054] Degradation compensation auxiliary loss Losses due to self-monitoring reconstruction With smoothing regularization term constraint A linear combination of is represented as follows:

[0055]

[0056] In the formula: and These are the weight hyperparameters of the self-supervised reconstruction loss term and the weight hyperparameters of the smoothing regularization term, respectively.

[0057] Then the overall total loss function It is expressed as follows:

[0058]

[0059] In the formula: This serves as the auxiliary supervisory signal for jointly training hyperparameters and balancing the compensation mechanism.

[0060] Compared with existing technologies, this collaborative perception method for unmanned mining vehicles based on vehicle-to-everything (V2X) firstly uses a degradation feature compensation module guided by spatial quality maps to dynamically identify and reconstruct the structure of point cloud degradation regions under dust and water mist interference using physical priors and sparse Transformer networks. Secondly, it uses a cross-vehicle registration module based on feature correlation to calculate the global correlation of dense features across viewpoints, and achieves adaptive resampling alignment of heterogeneous features without the need for high-precision absolute pose by implicitly regressing spatial offsets through deformable convolution. Finally, it constructs a dynamic feature fusion mechanism guided by blind zone masks, uses adaptive dual gating to extract semantic increments from the preceding vehicle, guides the system to perform complete fusion only within the blind zone of the following vehicle, and outputs the final target perception result from the 3D detection head. On the one hand, the complementary fusion mechanism proposed in this invention can effectively suppress information redundancy in the shared field of vision of the front and rear vehicles, and reduce communication and computing overhead. On the other hand, the algorithm of this invention has high robustness and can significantly improve the reliability of collaborative perception. It can be applied to key areas such as unmanned driving assisted transportation in mines, safe operation in coal mines, and multi-machine collaborative obstacle avoidance, and can effectively solve the problem of limited perception in the narrow and enclosed space of mines. Attached Figure Description

[0061] Figure 1 This is an overall framework diagram of the present invention;

[0062] Figure 2 This is a mine map scene construction diagram according to an embodiment of the present invention;

[0063] Figure 3 This is a structural diagram of the degradation feature compensation module based on spatial quality map guided by the present invention;

[0064] Figure 4 This is a structural diagram of the cross-vehicle feature dynamic registration module based on deformable convolution of the present invention;

[0065] Figure 5 This is a structural diagram of the dynamic feature fusion mechanism guided by blind zone mask in this invention;

[0066] Figure 6 This is a schematic diagram of a degradation scenario sample according to an embodiment of the present invention;

[0067] Figure 7 This is a schematic diagram of a pose error scenario according to an embodiment of the present invention;

[0068] Figure 8 This is a visual comparison chart of the detection results in an embodiment of the present invention. Detailed Implementation

[0069] This vehicle-to-everything (V2X)-based collaborative perception method for unmanned vehicles in mines aims to address the limited perception capabilities of individual vehicles due to the harsh environment and pose errors within the enclosed space of a mine. For example... Figure 1 As shown, this vehicle-to-everything (V2X) collaborative perception method for unmanned vehicles in mines firstly uses a degradation feature compensation module guided by a spatial quality map. This module estimates the local quality of the point cloud using physical priors and constructs a spatial quality map, guiding a sparse Transformer network to dynamically identify and reconstruct the structure of local point cloud degradation areas caused by dust and water mist, thus enhancing the credibility of the perception source. Secondly, addressing the vehicle pose drift and feature mismatch issues caused by GPS signal rejection in mines, a cross-vehicle registration module based on feature correlation is designed. By calculating the global correlation of dense features across different viewpoints and utilizing deformable convolution to implicitly regress spatial offsets, adaptive registration of heterogeneous features is achieved under weak localization conditions. Finally, a dynamic feature fusion mechanism guided by blind zone masks is constructed. This mechanism uses stepped dilated convolution and adaptive dual gating to accurately extract the semantic increment of the preceding vehicle, guiding the following vehicle to perform complete fusion only within the perception blind zone, effectively suppressing feature redundancy and misalignment in the shared field of view.

[0070] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0071] This collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) technology specifically includes the following steps:

[0072] Step 1: Mine map scene construction and point cloud data extraction for autonomous and collaborative vehicles:

[0073] Step 1-1, Parametric Modeling of High-Precision Digital Map of the Mine: Utilizing 3D scene modeling tools (such as MATLAB RoadRunner), construct a highly realistic mine map that conforms to the physical characteristics of a real underground coal mine. For example... Figure 2 As shown, this map covers three typical driving environments in mine auxiliary transportation:

[0074] ① Long straight tunnel scenario: Simulates long-distance longitudinal following travel to test feature redundancy suppression capability.

[0075] ② Curved bend tunnel scenario: Simulates the working condition of obstructed line of sight due to large curvature, used to test the nonlinear feature registration capability.

[0076] ③ Crossroads and alleyway scenario: Simulates multi-directional vehicle convergence to verify the robustness of collaborative perception under complex topology.

[0077] The physical dimensions of the tunnel, as per the embodiment, are set as follows: total length 2.92km, width 3.50m, and height 5.90m, accurately simulating the topological constraints of the confined space underground.

[0078] Step 1-2, Building and Dynamically Configuring the Collaborative Perception Simulation Environment:

[0079] Import the mine map constructed in Step 1-1 into an autonomous driving simulation platform (such as CARLA), and configure the dynamic parameters of the unmanned mining vehicles. In this example, the average vehicle speed is set to 20 km / h, and a multi-sensor fusion configuration is adopted, with a focus on 32-line or 64-line LiDAR, and the acquisition frequency is set to 10 Hz. To simulate a V2X collaborative scenario, at least two unmanned mining vehicles with communication capabilities (the autonomous vehicle and the collaborative vehicle) are deployed simultaneously in the simulation environment, and the relative extrinsic parameters and installation height between the sensors are fixed.

[0080] Steps 1-3: Spatiotemporally synchronized data acquisition and large-scale storage to obtain raw point cloud data:

[0081] The simulation platform's clock synchronization mechanism ensures strict inter-frame time synchronization among multiple vehicles during each data acquisition frame. The example collected 368 independent driving scenarios, generating a total of 33,111 frames of high-fidelity point cloud data; each scenario lasted approximately 10 seconds, guaranteeing data continuity in the temporal dimension and comprehensive scene coverage.

[0082] Step 2: Feature compensation is performed using a single-vehicle degradation compensation module guided by spatial quality maps.

[0083] Step 2-1, as follows Figure 3 As shown, the original point cloud data is first voxelized, and the average echo intensity of each voxel unit is extracted. Local point cloud density and the radial distance from the voxel center to the radar The physical properties are then concatenated along the channel dimension to construct a 3D initial feature vector. A lightweight multilayer perceptron (MLP) is then constructed to learn the joint distribution of degradation features. This MLP employs a two-layer fully connected architecture. First, the 3D input is mapped to a 16-dimensional hidden layer. The boundary features of the degradation region are extracted using the ReLU activation function. The output layer further compresses the features into a single scalar and normalizes it using the Sigmoid function, ultimately outputting spatial quality weights. , means as follows:

[0084]

[0085] In the formula: The spatial quality weight is used to characterize the degree of degradation and feature confidence of the current voxel unit. This is the Sigmoid activation function, used to normalize the network output to confidence levels; It is a lightweight multilayer perceptron used to learn the joint distribution characteristics of average echo intensity, local point cloud density, and radial distance; The average echo intensity of the voxel unit; The local point cloud density of a voxel unit; The radial distance from the voxel center to the lidar is denoted as .

[0086] Step 2-2: In order to compensate for the structural loss caused by degradation at the feature level, a mask reconstruction mechanism based on sparse Transformer is introduced. Drawing on the idea of ​​mask autoencoder, the original point cloud feature space is randomly sampled and a mask is applied during the forward propagation in the training phase, forcing some feature tensors to be set to zero.

[0087] set up Let the feature tensor of the region to be filled be . for Contextual features of spatial neighborhood. Unlike traditional uniform attention mechanisms, this invention utilizes spatial quality maps... As a spatial prior, it incorporates attention mechanisms by aggregating high-confidence neighborhood context features. To achieve the completion of features The structural reconstruction is represented as follows:

[0088]

[0089] In the formula: Features to be completed The query vector of a linear mapping; and Neighborhood context features The key vector and value vector obtained from the mapping; This is the mass weight matrix for the corresponding neighborhood unit; This indicates an element-wise multiplication operation.

[0090] By performing element-wise multiplication, this mechanism enables the network to adaptively block noise propagation in low-confidence neighborhoods under the constraint of mask occlusion, and guides the network to aggregate key geometric information from high-fidelity regions, significantly enhancing the model's ability to reason about contextual missing local structures.

[0091] Steps 2-3, considering the direct use of reconstruction features Replacing the original features will irreversibly destroy the distribution characteristics of the observations. Perform quality-perception soft fusion operations and output compensation features. , means as follows:

[0092]

[0093] In the formula: These are the original features.

[0094] Unlike hard truncation strategies with fixed thresholds, spatial quality weights have continuity. It can preserve the proportion of original information at different scales and suppress secondary artifacts caused by feature mutations.

[0095] Step 3: Feature space registration is performed using a cross-vehicle alignment module based on feature correlation.

[0096] Multi-vehicle collaborative perception systems in mine environments heavily rely on inertial measurement units (IMUs) to provide initial relative pose priors. However, limited by sensor accuracy, coarse registration based solely on sensor extrinsic parameters often leaves significant nonlinear spatial misalignment, introducing relative pose errors. To address this, this invention designs a cross-vehicle feature dynamic registration module based on deformable convolution to suppress registration features, as shown in the following structure... Figure 4 As shown.

[0097] Step 3-1, set and These are the characteristics of the cooperative vehicle after processing by the degradation compensation module and the characteristics of the autonomous vehicle after processing by the degradation compensation module, respectively. , ,in Indicates the number of channels is Space dimensions are The three-dimensional feature tensor, , , These represent the number of channels, height, and width of the feature map, respectively. To capture the spatial mapping relationship between the two vehicle perspectives, the heterogeneous features are first concatenated along the channel dimension to obtain a joint feature tensor; then, this tensor is input into... Convolution and A lightweight decoder composed of convolutions evaluates the geometric bias between features pixel by pixel, and then regresses a dense two-dimensional offset field for the transformation of the preceding vehicle features. and modulation weight matrix , , ,in This indicates that each spatial location corresponds to A two-dimensional offset set of sampling points This indicates that each spatial location corresponds to The set of modulation weights for each sampling point , , These represent the number of convolution sampling points, the height of the feature map, and the width, respectively.

[0098] Step 3-2: In the feature resampling stage, deformable convolution is introduced, using fixed local grid coordinates. Based on this, the non-integer spatial offsets predicted by the network are superimposed. The feature tensor of the collaborative vehicle after degradation compensation module is obtained through bilinear interpolation. Up-adaptive sampling, multiplied by the corresponding modulation weights. The final output is the accurately registered collaborative features. , means as follows:

[0099]

[0100] In the formula: The center coordinate vector of the vehicle; For the first The learnable convolutional kernel weight vector corresponding to each sampling position; The coordinates of the predefined fixed local sampling grid; The first one predicted by the network Two-dimensional spatial offset of each sampling location; For the first Modulation weights at each sampling position; , This represents the total number of local sampling points.

[0101] This mechanism enables nonlinear deformation registration of collaborative features to the vehicle's perspective with minimal computational overhead.

[0102] Step 3-3: In scenarios with heavy dust obscuring the view in mines or on steep curves, the physical common area between vehicles may be completely lost, causing the correlation matrix to degenerate into uniformly distributed noise. To address this, the module further designs a soft gating mechanism to avoid the "negative migration" phenomenon when matching fails by sensing the strength of cross-vehicle feature space correlation. Specifically, the overall matching quality is evaluated from the correlation matrix, the maximum matching response is extracted along the vehicle spatial dimension, and the cross-vehicle registration confidence is output through global average pooling and a multilayer perceptron. , means as follows:

[0103]

[0104] In the formula: Use the Sigmoid activation function; It is a multilayer perceptron used to generate registration confidence based on the overall matching response; This indicates a global average pooling operation, used to perform global statistics on the maximum matching response; This represents the maximization operation along the spatial dimension of the vehicle feature map, used to extract the most salient matching response overall; and These are the spatial coordinates in the feature maps of the cooperating vehicle and the autonomous vehicle, respectively. For the collaborative vehicle feature location With the location of the vehicle's features Correlation matrix elements between them; Cross-vehicle registration confidence is used to measure the reliability of the current cross-vehicle feature alignment results.

[0105] Steps 3-4: Final registration and coordination features of gated modulation This provides robustness to the model under extreme and harsh conditions.

[0106] Step 4: Feature aggregation is performed using a dynamic feature fusion mechanism guided by blind zone masking.

[0107] In unstructured mining scenarios, the perspectives of vehicles at different points in the field of view differ significantly. Directly fusing features from multiple vehicles globally not only easily leads to feature redundancy and spatial ghosting in shared viewing areas but also makes it difficult to effectively extract incremental information based on occlusion conditions. Therefore, this invention proposes a dynamic feature fusion mechanism guided by blind zone masks to achieve on-demand aggregation of heterogeneous features. Its structure is as follows: Figure 5 As shown.

[0108] Step 4-1: To adaptively evaluate the feature complementarity requirements of different spatial regions, firstly, the vehicle features... Collaborative features after registration By stitching along the channel dimension, intermediate fusion features are generated. , ,in Indicates the number of channels is Space dimensions are The three-dimensional feature tensor, , , These represent the number of channels, height, and width of the feature map, respectively. Considering that the determination of occlusion and co-view relationships highly depends on the wide-area context, a stepped dilated convolution (dilation rate) is introduced. A spatial gating generator (=1, 2, 5) is constructed to significantly expand the receptive field with zero additional parameters, fully capturing long-range geometric distributions. The joint features are then compressed by this convolutional group and activated by a sigmoid function to output a single-channel spatial gating weight matrix. Representation space fusion weights, ,in Indicates the space size as A two-dimensional matrix, and Here, the height and width of the feature map are represented as follows:

[0109]

[0110] In the formula: This is the spatial gating weight matrix; This represents a stepped dilated convolution operation; This is an intermediate fusion feature.

[0111] Step 4-2: In addition to the spatial dimension, to filter out background noise at the semantic level, a channel attention branch is introduced in parallel. First, the intermediate fused features are processed... Perform global average pooling to compress its spatial dimensions and obtain channel features. , Subsequently, cross-channel interaction weights are extracted via a two-layer multilayer perceptron, and channel gating vectors are output. , , means as follows:

[0112]

[0113] In the formula: This is the channel gating vector; and These are the learnable first-layer weight matrix and the learnable second-layer weight matrix of the fully connected layer, respectively. This is the ReLU activation function.

[0114] Step 4-3: After obtaining the dual modulation weights, feature aggregation is performed using the residual soft fusion paradigm, with the vehicle features as the basis. Using this as a basis, we will incorporate collaborative features. through and After element-wise weighted modulation, it is used as incremental information to supplement the vehicle features. In the process, the final fused features are output. , means as follows:

[0115]

[0116] In the formula: For the final fusion feature; This indicates an element-wise multiplication operation.

[0117] Step 5: Use a 3D detection head to perform target attribute regression on the fused features and output the results:

[0118] The complete fusion feature map generated in Step 4 is input into the 3D detection head. The decoupled classification and regression branches are used to regress the target's category, center coordinates, geometric size, and orientation angle, etc., 3D bounding box attributes from the enhanced features. Then, the 3D non-maximum suppression (NMS) algorithm is used to filter the large number of candidate boxes output. Redundant predictions are eliminated according to the intersection-union ratio threshold to suppress the ghosting phenomenon caused by feature overlap or misalignment in multi-vehicle collaborative perception. Finally, the system outputs the decoded high-precision target list to the decision planning module of the unmanned vehicle, realizing a complete closed loop from feature enhancement, adaptive registration, dynamic fusion to target perception in the harsh environment of the mine.

[0119] Step 6: Use an end-to-end joint loss function to train and optimize the overall perception framework:

[0120] Employing an end-to-end joint training paradigm, the overall total loss function... Loss due to 3D object detection main task Compensation for degradation features and auxiliary loss It consists of two weighted components.

[0121] 3D object detection main task loss It directly drives the output of 3D detection results and is composed of a combination of category classification loss (FocalLoss), bounding box regression loss (Smooth L1 Loss), and orientation angle classification loss (Cross Entropy Loss).

[0122] For the degradation feature compensation module, a self-supervised reconstruction mechanism is adopted: Let For the original features without a mask, To reconstruct features, Given a binary random sampling mask matrix, the self-supervised reconstruction loss... Defined as the mean square error within the masked area, it is expressed as follows:

[0123]

[0124] In the formula: Losses due to self-monitoring and reconstruction.

[0125] To ensure the geometric continuity between the reconstructed features and their neighborhood context, a smoothing regularization term is further introduced as a constraint, as follows:

[0126]

[0127] In the formula: This represents a smoothing regularization constraint, used to constrain the geometric continuity between reconstructed features and their neighborhood context; Indicates the location of the masked feature; Indicates position The neighborhood feature location; Indicates position Reconstruction features at the location; For position The original features of the location.

[0128] Degradation compensation auxiliary loss Losses due to self-monitoring reconstruction With smoothing regularization term constraint A linear combination of is represented as follows:

[0129]

[0130] In the formula: and These are the weight hyperparameters for the self-supervised reconstruction loss term and the weight hyperparameters for the smoothing regularization term, respectively.

[0131] Then the overall total loss function It is expressed as follows:

[0132]

[0133] In the formula: This serves as the auxiliary supervisory signal for jointly training hyperparameters and balancing the compensation mechanism.

[0134] Using joint training hyperparameters By balancing the monitoring signals from different modules, the network is driven to automatically learn the underlying physical laws under extreme working conditions in the mine through the backpropagation algorithm, thereby achieving closed-loop optimization of perception accuracy.

[0135] The collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) was applied to the scenario of the implementation example for experimentation. Precision (P), recall (R), and average precision (AP) were used as evaluation metrics. Positive and negative samples were determined based on the 3D intersection-union ratio (IoU) between the predicted bounding box and the actual 3D detection box, and two thresholds, IoU=0.5 and IoU=0.7, were taken and denoted as AP@0.5 and AP@0.7, respectively. Model training and testing were performed on a deep learning workstation running Ubuntu 22.04, with an Intel Core i9-10900X CPU and an NVIDIA RTX 6000 Blackwell GPU. The algorithm was built using the PyTorch 2.9.1 framework. The training epochs were 120, the batch size was 2, the optimizer was Adam, the initial learning rate was 0.0002, and it decayed by 0.1 times every 10 epochs. The vehicle perception range was x∈[-140.8,140.8] m, y∈[-12.8,12.8] m, z∈[-2,4.4] m, the voxel size was 0.4×0.4×0.4 m, the maximum communication distance for vehicle-to-everything (V2X) communication was 70 m, the deformable convolution sampling point number K was 9, and the feature compensation loss weights were [missing information]. .

[0136] To verify the applicability of the method of this invention in complex mining conditions, conventional scenarios, degraded scenarios, and pose error scenarios were constructed respectively. Among them, as shown... Figure 6 As shown, the degradation scenario enhances the physical laws of laser propagation by introducing dust and water mist media, simulating point cloud sparsity, local missing data, and noise enhancement phenomena in the harsh environment of a mine; for example... Figure 7 As shown, the pose error scenario is constructed by injecting translational noise and heading angle rotation noise into the relative poses between vehicles to simulate the spatial registration deviation between front and rear vehicles under weak localization conditions. Based on this, a mid-term feature fusion scheme based on PointPillars in the OpenV2V framework is selected as the baseline model and compared with Single (single-vehicle perception), Early Fusion, Late Fusion, and representative mid-term fusion methods such as F-Cooper, V2VNet, Where2comm, CoBEVT, and V2X-ViT. To ensure fairness in the comparison, all methods use PointPillars as the backbone network for single-vehicle feature extraction, and the communication compression rate and transmission strategy are kept consistent. The experimental results comparing the detection performance of different methods in the mine simulation scenario are shown in Table 1 below.

[0137]

[0138] As shown in Table 1, the AP@0.5 of the method of the present invention in the normal scenario, the degradation scenario and the pose error scenario are 0.902, 0.813 and 0.884, respectively, and the corresponding AP@0.7 are 0.791, 0.652 and 0.765, respectively. In conventional scenarios, the AP@0.7 of the method of this invention is higher than that of Single (0.556), the baseline (0.682), V2X-ViT (0.745), and EarlyFusion (0.785). In degraded scenarios, the AP@0.7 of the method of this invention is 0.652, higher than that of the baseline (0.336), CoBEVT (0.386), and V2X-ViT (0.412). In pose error scenarios, the AP@0.7 of the method of this invention is 0.765, higher than that of the baseline (0.504), CoBEVT (0.615), and the suboptimal method V2X-ViT (0.642). Therefore, this invention can maintain high 3D target detection accuracy in conventional mining scenarios, perception degradation scenarios, and positioning error scenarios.

[0139] To verify the independent contribution of each module, ablation experiments were conducted in three test scenarios. The comparison results of ablation experiments using different methods in the mine simulation scenario are shown in Table 2 below. Here, w / Comp indicates that only the degradation feature compensation module is introduced on the baseline model, w / Align indicates that only the feature registration module is introduced, and w / Comp+Align indicates that both the degradation feature compensation module and the feature registration module are introduced simultaneously, while the global self-attention mechanism is still used in the fusion stage.

[0140]

[0141] As shown in Table 2, the ablation experiment results indicate that in the degradation scenario, the AP@0.5 and AP@0.7 of w / Comp are improved from 0.521 and 0.336 at the baseline to 0.795 and 0.628, respectively; in the pose error scenario, the AP@0.7 of w / Align is improved from 0.504 at the baseline to 0.735; after simultaneously introducing the degradation compensation module and the feature registration module, the AP@0.7 of w / Comp+Align in the normal, degradation, and pose error scenarios reaches 0.768, 0.635, and 0.740, respectively; based on this, after further introducing the dynamic feature fusion mechanism guided by the blind zone mask, the AP@0.7 of the method of this invention is further improved by 2.3%, 2.7%, and 2.7% in the three scenarios, respectively, finally reaching 0.791, 0.652, and 0.765. The results show that the degradation feature compensation module mainly improves the feature quality under degradation environment, the cross-vehicle registration module mainly improves the registration robustness under pose error scenario, and the blind spot mask-guided dynamic feature fusion mechanism further improves the blind spot completion capability and the final detection accuracy.

[0142] To intuitively verify the detection effect of the method of the present invention under complex mining conditions, the three-dimensional detection visualization results of different methods in the embodiments are given as follows: Figure 8 As shown, the green 3D bounding box represents the ground truth bounding box, the red 3D bounding box represents the predicted bounding box, and the red arrows are used to mark typical false positives or false negatives. Figure 8 It can be seen that the CoBEVT model exhibits severe false negatives when fusing features with significant pose deviations. The underlying mechanism lies in the severe feature smoothing and blurring effects caused by local convolution operations when processing multi-vehicle feature maps with inadequate spatial registration. This leads to the loss of high-frequency spatial details of the target, and the network fails to output effective bounding box predictions due to a sharp decline in confidence. While V2X-ViTV2 maintains target recall within the main field of view, it generates a series of false positives that are severely deviated from the true position. This reveals the limitations of the global attention mechanism in handling pose interference. In the absence of explicit geometric correction constraints, the network misclassifies misaligned feature flows as independent physical entities, thus inducing high-frequency rotation and translation false positives. ERMVP reduces some distortion through a multi-scale alignment mechanism, but in complex areas such as curves and intersection edges, box offsets and false negatives due to incomplete alignment still remain. In contrast, the method of this invention suppresses the propagation of low-quality perceptual noise through a degradation feature compensation module, achieves adaptive alignment of heterogeneous features using a cross-vehicle registration module, and extracts semantic increments of the preceding vehicle on demand through a dynamic feature fusion mechanism guided by blind zone masks. Therefore, in... Figure 8 It can obtain more complete and accurate target detection results, with higher consistency between predicted boxes and ground truth boxes, and significantly reduced false positives and false negatives. Figure 8The qualitative results in Table 2 are consistent with the quantitative results in Table 2, further demonstrating that the method of the present invention has good sensing robustness and detection accuracy under the conditions of mine degradation and pose error.

Claims

1. A collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) communication, characterized in that, Specifically, the following steps are included: Step 1: Mine map scene construction and point cloud data extraction for autonomous and cooperative vehicles: First, a simulated mine map that conforms to the actual physical characteristics of underground coal mines is constructed using 3D scene modeling tools; then, the simulated mine map is imported into the autonomous driving simulation platform, and the dynamic parameters of multiple unmanned mining vehicles are configured. Multi-sensor fusion configuration is adopted, and the acquisition frequency is set to simulate a V2X cooperative scenario; through the clock synchronization mechanism of the simulation platform, multiple unmanned mining vehicles achieve inter-frame time synchronization when acquiring data in each frame, acquiring multiple independent driving scenarios, setting the duration of each scenario, and generating high-fidelity point cloud data; Step 2: Feature compensation is performed using a single-vehicle degradation compensation module guided by spatial quality map: physical prior is used to estimate the local quality of point cloud and construct spatial quality map, which guides sparse Transformer network to dynamically identify and reconstruct the structure of local point cloud degradation areas caused by dust and water mist, thereby achieving reliable enhancement of the perception source. Step 3: Feature space registration is performed using a cross-vehicle alignment module based on feature correlation: By calculating the global correlation of dense features across viewpoints and utilizing deformable convolutional implicit regression spatial offset, adaptive registration of heterogeneous features is achieved under weak localization conditions. Step 4: Feature aggregation is performed using a dynamic feature fusion mechanism guided by blind spot masking: the semantic increment of the preceding vehicle is extracted using adaptive dual gating, and the system is guided to perform complete fusion only within the blind spot of the following vehicle. Step 5: Use a 3D detection head to perform target attribute regression and output results on the fused features: First, input the complete fused feature map generated in Step 4 into the 3D detection head, and use the decoupled classification branch and regression branch to regress the 3D bounding box attributes of the target from the enhanced features; then use the 3D non-maximum suppression algorithm to filter the output candidate boxes, and eliminate redundant predictions according to the intersection-over-union ratio threshold, and finally output the decoded high-precision target list; Step 6: Use an end-to-end joint loss function to train and optimize the overall perception framework: Use an end-to-end joint training paradigm to achieve closed-loop optimization of perception accuracy.

2. The collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) as described in claim 1, characterized in that, Step 2 is as follows: Step 2-1: First, the raw point cloud data is voxelized, and the average echo intensity of each voxel is extracted. Local point cloud density and the radial distance from the voxel center to the radar Physical attributes are concatenated along the channel dimension to construct a 3D initial feature vector. Then, a lightweight multilayer perceptron (MLP) is constructed to learn the joint distribution of degradation features. The MLP employs a two-layer fully connected architecture. First, the 3D input is mapped to a 16-dimensional hidden layer. The boundary features of the degradation region are extracted using the ReLU activation function. The output layer compresses the boundary features into a single scalar and normalizes them using the Sigmoid function, ultimately outputting the spatial quality weights. , means as follows: In the formula: Spatial quality weight; Use the Sigmoid activation function; It is a multilayer perceptron; The average echo intensity of the voxel unit; The local point cloud density of a voxel unit; The radial distance from the voxel center to the lidar; Step 2-2 introduces a mask reconstruction mechanism based on sparse Transformer. During the forward propagation in the training phase, the original point cloud feature space is randomly sampled and a mask is applied to force some feature tensors to be set to zero, as follows: set up Let the feature tensor of the region to be filled be . for Contextual features of spatial neighborhood will affect the spatial quality map. As a spatial prior incorporation attention mechanism, it aggregates high-confidence neighborhood context features. Implement the feature to be completed The structural reconstruction is represented as follows: In the formula: Features to be completed The query vector of a linear mapping; and Neighborhood context features The key vector and value vector obtained from the mapping; This is the mass weight matrix for the corresponding neighborhood unit; This represents the element-wise multiplication operation; Step2-3, use Implement quality-perception soft fusion operations and output compensation features. , means as follows: In the formula: These are the original features.

3. The collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) as described in claim 2, characterized in that, Step 3 is as follows: Step 3-1, set and These are the characteristics of the cooperative vehicle after processing by the degradation compensation module and the characteristics of the autonomous vehicle after processing by the degradation compensation module, respectively. , ,in Indicates the number of channels is Space dimensions are The three-dimensional feature tensor, , , Let represent the number of channels, height, and width of the feature map, respectively. First, the heterogeneous features are concatenated along the channel dimension to obtain a joint feature tensor; then, this tensor is input into _____. Convolution and A lightweight decoder using convolutions evaluates the geometric bias between features pixel by pixel, regressing a dense two-dimensional offset field for the transformation of the preceding vehicle's features. and modulation weight matrix , , ,in This indicates that each spatial location corresponds to A two-dimensional offset set of sampling points This indicates that each spatial location corresponds to The set of modulation weights for each sampling point , , These represent the number of convolution sampling points, the height of the feature map, and the width of the feature map, respectively. Step 3-2: In the feature resampling stage, deformable convolution is introduced, using fixed local grid coordinates. Based on this, the non-integer spatial offsets predicted by the network are superimposed. The feature tensor of the collaborative vehicle after degradation compensation module is obtained through bilinear interpolation. Up-adaptive sampling, multiplied by the corresponding modulation weights. The final output is the accurately registered collaborative features. , means as follows: In the formula: The center coordinate vector of the vehicle; For the first The learnable convolutional kernel weight vector corresponding to each sampling position; The coordinates of the predefined fixed local sampling grid; The first one predicted by the network Two-dimensional spatial offset of each sampling location; For the first Modulation weights at each sampling position; , This represents the total number of local sampling points; Step 3-3: Design a soft gating mechanism to evaluate the overall matching quality from the correlation matrix, extract the maximum matching response along the vehicle spatial dimension, and output the cross-vehicle registration confidence score through global average pooling and a multilayer perceptron. , means as follows: In the formula: Use the Sigmoid activation function; It is a multilayer perceptron; This indicates a global average pooling operation; This represents the maximization operation along the spatial dimension of the vehicle feature map; and These are the spatial coordinates in the feature maps of the cooperating vehicle and the autonomous vehicle, respectively. For the collaborative vehicle feature location With the location of the vehicle's features Correlation matrix elements between them; Registration confidence levels across vehicles; Steps 3-4: The final registration and coordination features of the gated modulation are represented as follows: In the formula: This indicates the registration coordination characteristics of the final gated modulation.

4. The collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) as described in claim 3, characterized in that, Step 4 is as follows: Step 4-1, first set the vehicle features Collaborative features after registration By stitching along the channel dimension, intermediate fusion features are generated. , ,in Indicates the number of channels is Space dimensions are The three-dimensional feature tensor, , , These represent the number of channels, height, and width of the feature map, respectively; a stair-step dilated convolution is introduced to construct a spatial gating generator, which outputs a single-channel spatial gating weight matrix. Representation space fusion weights, ,in Indicates the space size as A two-dimensional matrix, and Here, the height and width of the feature map are represented as follows: In the formula: This is the spatial gating weight matrix; This represents a stepped dilated convolution operation; This is an intermediate fusion feature; St In ep4-2, a parallel channel attention branch is introduced, first fusing intermediate features. Perform global average pooling to compress its spatial dimensions and obtain channel features. , Subsequently, cross-channel interaction weights are extracted via a two-layer multilayer perceptron, and channel gating vectors are output. , , means as follows: In the formula: This is the channel gating vector; and These are the learnable first-layer weight matrix and the learnable second-layer weight matrix of the fully connected layer, respectively. It is the ReLU activation function; Step 4-3: Feature aggregation is performed using the residual soft fusion paradigm, focusing on vehicle features. Using this as a basis, we will incorporate collaborative features. through and After element-wise weighted modulation, it is used as incremental information to supplement the vehicle features. In the process, the final fused features are output. , means as follows: In the formula: For the final fusion feature; This indicates an element-wise multiplication operation.

5. The collaborative perception method for unmanned vehicles in mines based on vehicle-to-everything (V2X) as described in claim 4, characterized in that, Step 6 is as follows: The overall total loss function of the end-to-end joint training paradigm Including the loss of the main task of 3D object detection Compensation for degradation features and auxiliary loss Two parts; 3D object detection main task loss It directly drives the output of 3D detection results and is composed of a combination of category classification loss, bounding box regression loss and orientation angle classification loss. For the degradation feature compensation module, a self-supervised reconstruction mechanism is adopted: Let For the original features without a mask, To reconstruct features, Given a binary random sampling mask matrix, the self-supervised reconstruction loss... Defined as the mean square error within the masked area, it is expressed as follows: In the formula: Losses due to self-monitoring reconstruction; Introducing a smoothing regularization constraint, as follows: In the formula: This represents a smoothing regularization constraint. Indicates the location of the masked feature; Indicates position The neighborhood feature location; Indicates position Reconstruction features at the location; For position The original features of the location; Degradation compensation auxiliary loss Losses due to self-monitoring reconstruction With smoothing regularization term constraint A linear combination of is represented as follows: In the formula: and These are the weight hyperparameters of the self-supervised reconstruction loss term and the weight hyperparameters of the smoothing regularization term, respectively. Then the overall total loss function It is expressed as follows: In the formula: This serves as the auxiliary supervisory signal for jointly training hyperparameters and balancing the compensation mechanism.

Images