Mine unmanned vehicle environment perception system and method based on deep learning

Abstract

This invention relates to the field of deep learning technology and discloses a deep learning-based environmental perception system and method for unmanned mining vehicles. The deep learning-based environmental perception system for unmanned mining vehicles includes: a transition anomaly extraction module for calculating a normalized transition anomaly rate; a polar feature construction module for constructing a polar feature tensor; a polar feature extraction module for performing convolutional encoding on the polar feature tensor to obtain a hidden representation; a fusion distance calculation module for calculating the fusion distance; a final distance calculation module for calculating the final distance; and a feasible region generation module for generating a feasible region. This invention generates a highly reliable feasible region by dividing the single-frame point cloud into scanning rings and extracting the transition anomaly rate, combined with dynamic convolutional encoding using polar feature tensor construction and normalized transition anomaly rate modulation, and further filtered by dynamic tolerance ground occupancy judgment and passage score selection. This provides a reliable basis for path planning of unmanned mining vehicles.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically, to a deep learning-based environmental perception system and method for unmanned mining vehicles. Background Technology

[0002] The safe operation of unmanned mining vehicles relies heavily on environmental perception systems. LiDAR, as a core sensor, generates feasible regions by collecting point cloud data, providing crucial information for path planning. Mining environments are complex, with water accumulation areas often formed in mine tunnels due to dust suppression spraying and groundwater seepage. Some rock surfaces also exhibit strong specular reflectivity when wet.

[0003] When the laser beam emitted by the lidar shines on these mirror-reflecting surfaces, the beam is reflected in a specific direction rather than being transmitted back to the sensor, resulting in a continuous void sector without echo within the corresponding azimuth angle range. This void is not a real ground defect, but a ranging defect caused by signal reflection loss. However, traditional environmental perception methods have difficulty distinguishing between mirror voids and real obstacles or ground depressions.

[0004] Existing technologies either directly classify empty areas as impassable or fill in missing distance measurements with fixed values ​​or random numbers. These methods fail to accurately identify the geometric features of the mirrored voids or to provide distance data that conforms to the physical laws of the ground. Consequently, the generated feasible region either omits actually passable mirrored areas or contains false obstacles, severely impacting the accuracy of path planning and posing safety hazards to unmanned mining vehicles. Summary of the Invention

[0005] This invention provides an environmental perception system and method for unmanned mining vehicles based on deep learning, which solves the technical problems mentioned in the background.

[0006] This invention provides a deep learning-based environmental perception system for unmanned mining vehicles, comprising:

[0007] The transition anomaly extraction module is used to divide a single frame point cloud into scanning rings according to the elevation angle, obtain its azimuth sequence, ranging sequence and effective mask, and determine the void sector, range transition, angle width and transition anomaly rate accordingly. In the non-void sector, the ground polar change rate set is obtained by the finite difference between adjacent azimuth and adjacent ranging, and the upper bound of the polar change rate and the normalized transition anomaly rate are calculated.

[0008] The polar feature construction module is used to combine normalized ranging, effective mask, normalized transition anomaly rate and polar gradient into a polar feature tensor, and to perform boundary value preservation on the virtual sector and set the polar gradient in the virtual sector to zero.

[0009] The polar feature extraction module is used to map the normalized transition anomaly rate to obtain the dynamic expansion rate and conditional kernel weights, and to perform convolutional encoding on the polar feature tensor in one-dimensional polar direction to output the hidden representation.

[0010] The fusion distance calculation module is used to calculate the void probability and ground-compensated distance based on the hidden representation, and fuse the original ranging and ground-compensated distance with the void probability as the weight to obtain the fusion distance.

[0011] The final distance calculation module is used to apply forward and backward constraints to the fusion distance based on the upper bound of the polar change rate to obtain the final distance;

[0012] The feasible region generation module is used to restore the final distance, azimuth, elevation and sensor installation height into three-dimensional coordinates and project them onto the top-view grid. Within each grid, it determines the ground occupation based on the reference ground height and tolerance, and accumulates a score for each grid with the void probability as the weight. The score is then compared with the lowest passability score to generate a feasible region.

[0013] This invention provides a deep learning-based environmental perception method for unmanned mining vehicles, comprising the following steps:

[0014] Step S201: Divide the single-frame point cloud into scanning rings according to the elevation angle, obtain its azimuth sequence, ranging sequence and effective mask, and determine the void sector, range transition, angle width and transition anomaly rate accordingly. In the non-void sector, obtain the set of ground polar change rate through the finite difference between adjacent azimuth and adjacent ranging, and calculate the upper bound of the polar change rate and the normalized transition anomaly rate.

[0015] Step S202: The normalized ranging, effective mask, normalized transition anomaly rate and polar gradient are combined to form a polar feature tensor, and boundary value preservation is performed on the virtual sector and the polar gradient in the virtual sector is set to zero.

[0016] Step S203: Map the normalized transition anomaly rate to obtain the dynamic expansion rate and conditional kernel weights, perform convolutional encoding on the polar feature tensor in one-dimensional polar orientation, and output the hidden representation.

[0017] Step S204: Calculate the void probability and ground compensation distance based on the hidden representation, and fuse the original ranging and ground compensation distance with the void probability as the weight to obtain the fused distance;

[0018] Step S205: Apply forward and backward constraints to the fusion distance based on the upper bound of the polar change rate to obtain the final distance;

[0019] Step S206: The final distance, azimuth, elevation and sensor installation height are restored to three-dimensional coordinates and projected onto the top-view grid. In each grid, the ground occupancy is determined according to the reference ground height and tolerance. The grid is weighted and accumulated with the void probability as the weight to obtain a score, and compared with the lowest passability score to generate a feasible region.

[0020] The beneficial effects of this invention are as follows: Addressing the problem of missing LiDAR measurements due to specular reflection from wet surfaces and other surfaces in mining environments, this invention achieves accurate calculation of the void probability and ground compensation distance by dividing the single-frame point cloud into scanning loops and extracting the transition anomaly rate, combined with dynamic convolutional coding using pichiometric feature tensor construction and normalized transition anomaly rate modulation. Furthermore, by using bidirectional geometric constraints and probability-weighted fusion, ground ranging data is supplemented, and a highly reliable feasible region is generated through dynamic tolerance ground occupancy judgment and passage score filtering. This avoids the shortcomings of traditional methods that misjudge voids as obstacles or fill them with false data, while also adapting to the dynamic changes in complex mining environments, significantly improving the accuracy and robustness of environmental perception, providing a reliable basis for path planning of unmanned mining vehicles, and effectively reducing driving safety hazards. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the environmental perception system for unmanned mining vehicles based on deep learning, as described in this invention.

[0022] Figure 2 This is a flowchart of the deep learning-based environmental perception method for unmanned mining vehicles according to the present invention.

[0023] In the figure: Transition anomaly extraction module 101, polar feature construction module 102, polar feature extraction module 103, fusion distance calculation module 104, final distance calculation module 105, feasible region generation module 106. Detailed Implementation

[0024] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0025] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the term encompasses the elements or objects listed following the term and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] like Figures 1-2 As shown, the deep learning-based environmental perception system for unmanned mining vehicles includes:

[0027] The transition anomaly extraction module 101 is used to divide a single frame point cloud into scanning rings according to the elevation angle, obtain its azimuth sequence, ranging sequence and effective mask, and determine the void sector, range transition, angle width and transition anomaly rate accordingly. In the non-void sector, the set of ground polar change rate is obtained by the finite difference between adjacent azimuth and adjacent ranging, and the upper bound of polar change rate and normalized transition anomaly rate are calculated.

[0028] The polar feature construction module 102 is used to combine normalized ranging, effective mask, normalized transition anomaly rate and polar gradient into a polar feature tensor, and to perform boundary value preservation on the virtual sector and set the polar gradient in the virtual sector to zero.

[0029] The polar feature extraction module 103 is used to map the normalized transition anomaly rate to obtain the dynamic expansion rate and conditional kernel weight, and to perform convolutional encoding on the polar feature tensor in one-dimensional polar direction to output the hidden representation.

[0030] The fusion distance calculation module 104 is used to calculate the void probability and ground compensation distance based on the hidden representation, and fuse the original ranging and ground compensation distance with the void probability as the weight to obtain the fusion distance.

[0031] The final distance calculation module 105 is used to apply forward and backward constraints to the fusion distance based on the upper bound of the polar change rate to obtain the final distance;

[0032] The feasible region generation module 106 is used to restore the final distance, azimuth, elevation and sensor installation height into three-dimensional coordinates and project them onto the top-view grid. Within each grid, it determines the ground occupation based on the reference ground height and tolerance, accumulates a score for each grid with the void probability as the weight, and compares it with the lowest pass score to generate a feasible region.

[0033] In one embodiment of the present invention, a single frame point cloud is divided into several scanning rings according to the elevation angle, and the azimuth sequence, ranging sequence and effective mask of each scanning ring are obtained, wherein each value of the effective mask is used to identify whether there is an echo at the corresponding azimuth angle.

[0034] Find the largest continuous corner region marked as echo-free in the effective mask, treat the corner region as a void sector, and record the left and right azimuth angles of the void sector.

[0035] Extract the corresponding range value from the nearest marker with echo on the left side of the left azimuth angle, and extract the corresponding range value from the nearest marker with echo on the right side of the right azimuth angle. Calculate the difference between the two range values ​​as the range jump, calculate the difference between the left and right azimuth angles as the angle width, and then calculate the ratio of the range jump to the angle width to obtain the jump anomaly rate.

[0036] It should be noted that a single-frame point cloud represents the collection of all three-dimensional point data acquired by the LiDAR (Light Detection and Ranging) system mounted on the mining unmanned vehicle within a single complete scan cycle, with each point containing spatial location information. The elevation angle represents the angle between the laser beam emitted by the LiDAR and the horizontal plane. LiDAR systems typically divide into different laser channels according to a fixed elevation angle, and the laser beams of the same elevation angle channel form a scanning loop. A scanning loop represents a sequence of measurement points acquired by the laser beam of the same fixed elevation angle channel as the horizontal azimuth angle changes during a single 360° scan cycle; each scanning loop corresponds to a fixed elevation angle. The azimuth angle sequence represents the numerical sequence of horizontal azimuth angles corresponding to all measurement points within a single scanning loop, arranged in scanning order, reflecting the angular position of the measurement points on the horizontal plane. The distance measurement sequence represents the numerical sequence of distance values ​​from the LiDAR to the target object corresponding to all measurement points within a single scanning loop, arranged in scanning order, reflecting the spatial distance from the measurement point to the sensor.

[0037] It should be noted that the effective mask represents a binary numerical sequence with the same length as the azimuth sequence and ranging sequence. It is used to identify whether the lidar at the corresponding azimuth angle has received an echo signal from a target object. 0 indicates no echo, and 1 indicates an echo. An echo represents the signal reflected back to the sensor from the laser beam emitted by the lidar after encountering a target object. An echo indicates the presence of a detectable target object at that azimuth angle, while no echo indicates that no target object was detected at that azimuth angle (or signal loss due to specular reflection, etc.). The azimuth range with no echo represents the range of azimuth angles corresponding to multiple consecutive values ​​of 0 in the effective mask (i.e., multiple consecutive azimuth angles with no echo). The maximum consecutive azimuth range with no echo represents the azimuth range with the largest azimuth span (the difference between the right and left azimuth angles) among all consecutive azimuth ranges corresponding to the effective mask. The void sector represents the largest continuous angular region without echo in the effective mask. This region is an abnormal area that needs to be addressed in mining scenarios because of the lack of near-field target echo due to specular reflection (such as water accumulation or wet surfaces) or signal absorption.

[0038] It should be noted that the left and right azimuth angles represent the minimum and maximum azimuth angles corresponding to the void sector (the largest continuous unecho angular domain), respectively, i.e., the starting and ending azimuth angles of this angular domain. The nearest echoing position to the left of the left azimuth angle indicates an azimuth value less than the left azimuth angle, a valid mask indicating an echo, and the position corresponding to the azimuth angle with the smallest angle difference from the left azimuth angle. The nearest echoing position to the right of the right azimuth angle indicates an azimuth value greater than the right azimuth angle, a valid mask indicating an echo, and the position corresponding to the azimuth angle with the smallest angle difference from the right azimuth angle. The range transition represents the difference between the nearest valid range values ​​at both ends of the void sector, used to quantify the degree of jump in target distance before and after the void sector. The angular width represents the azimuth span of the void sector, used to quantify the horizontal coverage area of ​​the void sector. The transition anomaly rate is the ratio of distance transition to angle width. It is a core indicator representing the characteristics of large distance jumps and small angle leaps in void sectors, and is used to identify mirror voids.

[0039] In one embodiment of the present invention, in a non-void sector, positions where the effective masks of three consecutive azimuth angles are all identified as having echoes are selected. Based on the difference between adjacent azimuth angles and the difference between adjacent range values ​​of these positions, the ground polar change rate corresponding to each selected position is calculated by the finite difference method to form a set of ground polar change rates.

[0040] Statistical analysis was performed on the set of ground polar variability rates, and the 95th percentile value was taken as the upper bound of the polar variability rate. The normalized polar variability rate was obtained by calculating the ratio of the transition anomaly rate to the upper bound of the polar variability rate.

[0041] Specifically, the ground polar variation rate at the i-th location The calculation formula is as follows:

[0042]

[0043] in and These represent the distance measurements at positions i-1 and i+1, respectively. and These represent the azimuth angles at the (i-1)th and (i+1)th positions, respectively.

[0044] It should be noted that non-empty sectors refer to the areas in the effective mask other than empty sectors, including echo areas and non-maximum continuous non-echo areas. The difference between adjacent azimuth angles represents the difference between the azimuth angle corresponding to the selected location and its previous adjacent azimuth angle; the difference between adjacent range values ​​represents the absolute value of the difference between the range value corresponding to the selected location and its previous adjacent azimuth angle. The finite difference method approximates the derivative (rate of change) at a point within an interval by using the ratio of the difference in function values ​​between two adjacent discrete points to the difference in the independent variable; that is, it is used to approximate the ground polar variability rate. The ground polar variability rate represents the rate at which the ground distance changes with the azimuth angle at the selected location, reflecting the geometric characteristics of normal ground (non-empty areas). The 95th percentile value represents the value located at the 95th percentile after sorting the values ​​in the ground polar variability rate set from smallest to largest. This value excludes extreme outliers and represents the statistical upper limit of the normal ground polar variability rate. The normalized transition anomaly rate is the ratio of the transition anomaly rate to the upper bound of the polarity change rate. It is a dimensionless index used for subsequent modulation network parameters (such as convolution dilation rate and kernel weights).

[0045] It should be noted that the specific method for finding the maximum continuous echo-free angle domain includes the following steps:

[0046] Step S301: Initialize the starting azimuth of the current continuous non-echo angle domain to an invalid value, the span of the current continuous non-echo angle domain to 0, the maximum span to 0, and the azimuth of the left end of the void sector and the azimuth of the right end of the void sector to invalid values;

[0047] Step S302: Traverse the valid masks in ascending order of the azimuth sequence. When the first valid mask with a value of 0 is encountered, record the corresponding azimuth as the starting azimuth of the current continuous no-echo angular domain.

[0048] Step S303: Continue to traverse the subsequent valid masks. If the result is 0, then continuously record the current azimuth as the end azimuth of the current continuous no-echo angle domain and calculate the current continuous no-echo angle domain span, which is the difference between the current end azimuth and the current start azimuth.

[0049] Step S304: When a valid mask with a value of 1 is encountered, stop recording the current continuous no-echo angle domain, compare the current span with the maximum span, if the current span is larger, update the maximum span to the current span, and set the left end azimuth of the void sector as the current starting azimuth and the right end azimuth of the void sector as the current ending azimuth.

[0050] Step S305: Repeat steps S302 to S304 until all valid masks have been traversed. The angular domains corresponding to the left and right azimuth angles of the final determined void sector are the largest continuous echo-free angular domains (void sectors).

[0051] It should be noted that in mining scenarios, lidar is susceptible to interference from dust and mineral debris, which may result in isolated false echoes or no-echo points. Calculating the rate of change using only two adjacent points is easily affected by these interfering points. However, three consecutive echo points ensure that the selected location is in a stable ground area, and the calculated ground polar variability accurately reflects the geometric characteristics of a normal ground surface. Furthermore, small-scale no-echo areas are often due to random noise (such as obstruction by a single dust particle), while large-scale, continuous no-echo areas are often effective voids caused by specular reflection from water accumulation, slippery mine surfaces, etc. This definition accurately distinguishes between noise-induced no-echo areas and specular voids, avoiding wasted processing resources in ineffective noise regions and improving the efficiency and accuracy of environmental perception.

[0052] In one embodiment of the present invention, the ranging values ​​corresponding to the positions with echoes identified by the effective mask in the non-empty sector are filtered out, and the ranging mean and ranging standard deviation are calculated based on these ranging values. Then, the ranging values ​​of all positions are normalized according to the ranging mean and ranging standard deviation to obtain normalized ranging.

[0053] The effective masks of three consecutive azimuth angles in the non-void sector are all marked as positions with echoes and where these three azimuth angles are not within the void sector. Based on the difference between adjacent azimuth angles and the difference between adjacent range values ​​at these positions, the pole gradient corresponding to each selected position is calculated, and the pole gradient of other positions is set to zero.

[0054] For each location within a void sector, the normalized transition anomaly rate remains unchanged; for each location outside a void sector, the normalized transition anomaly rate is set to zero.

[0055] For positions within the void sector that are marked as echo-free by the effective mask, the normalized ranging is set to the normalized ranging of the nearest effective position on the left side of the void sector, and the poloidal gradient is set to zero.

[0056] For each location, the normalized ranging, effective mask, normalized transition anomaly rate, and poloidal gradient of that location are combined to obtain the poloidal feature tensor corresponding to each location.

[0057] It should be noted that normalized ranging represents the result of standardizing the original ranging values ​​at each location according to the ranging mean and standard deviation, eliminating the influence of dimensions on the ranging values ​​and making the characteristics of different distance ranges comparable. The polar gradient represents the rate of change of the ranging value with the azimuth angle at a specific selected location, reflecting the trend of ranging changes around that location and providing geometric change information for the feature tensor. The calculation range of the ranging mean and standard deviation is limited to the locations where the effective mask identifies echoes within the non-empty sectors of a single scan ring. That is, the ranging mean and standard deviation are calculated independently for each scan ring, ensuring that the normalization process adapts to the distance distribution characteristics of scan rings with different elevation angles and avoiding standardization distortion caused by cross-ring distance differences.

[0058] It should be noted that the epitropic gradient = (the difference between the distance measured at the next adjacent azimuth angle after the selection position and the distance measured at the previous adjacent azimuth angle before the selection position) ÷ (2 × the difference between the next adjacent azimuth angle after the selection position and the previous adjacent azimuth angle before the selection position). That is, the difference between adjacent distance values ​​does not need to be taken as an absolute value, and the information of the direction of change is preserved. Normalized ranging provides standardized distance information, effectively masks the validity of data, normalized transition anomaly rate accurately points to the void region, epitropic gradient reflects geometric changes, and the four-dimensional combination of epitropic feature tensor covers numerical magnitude, data reliability, anomaly identification, and geometric trend, thereby matching the subsequent conditional convolution's requirement for perceiving the void region, enabling the network to accurately identify and process mirror voids based on multi-dimensional features simultaneously. Furthermore, the echo-free area caused by the mirror void is not a real ground loss, but a signal reflection loss. The distance measurement at the effective position on its left can reflect the ground base distance characteristics of the area. Using the left edge boundary value can avoid feature breakage caused by filling zero or mean values, and at the same time, it does not introduce false distance changes, so that the feature tensor still maintains physical rationality in the void area. This will not be elaborated here.

[0059] In one embodiment of the present invention, the normalized transition anomaly rate at each position is obtained, and it is mapped by the hyperbolic tangent function. The mapping result is multiplied by a fixed constant and rounded down. Then, the integer part is added by one to obtain the integer dynamic expansion rate corresponding to each position.

[0060] Based on the normalized transition anomaly rate at each position, the conditional kernel weights at each corresponding position within the receptive field radius are obtained by performing a linear operation between the first fixed parameter vector and the normalized transition anomaly rate, where the receptive field radius is an integer and the two first fixed parameter vectors have the same dimension as the polar feature tensor.

[0061] The polar feature tensor is sampled according to the dynamic expansion rate corresponding to each position to obtain the sampled polar feature tensor sequence. If the sampling exceeds both ends of the polar feature tensor sequence, the polar feature tensor corresponding to the nearest valid position in the sequence is taken.

[0062] The sampled poloidal feature tensor is multiplied channel by channel with the corresponding conditional kernel weights, and all multiplication results are summed to obtain the summation result. The summation result is then processed by channel-wise modified linear unit activation to obtain the hidden representation corresponding to each position.

[0063] Specifically, the dynamic expansion rate at position i. The calculation formula is as follows:

[0064]

[0065] in This represents the normalized transition anomaly rate at position i. and Let represent the first and second fixed constants, respectively, and tanh represent the hyperbolic tangent function. This indicates rounding down to the nearest integer.

[0066] Specifically, the conditional weighting corresponding to the t-th offset position The calculation formula is as follows:

[0067]

[0068] in This represents the normalized transition anomaly rate at position i. and Let represent the first weight parameter and the first bias parameter corresponding to the t-th offset position, respectively, and T represent the receptive field radius.

[0069] Specifically, the summation result at the i-th position The calculation formula is as follows:

[0070]

[0071] in This represents the polar feature tensor after sampling according to the dynamic expansion rate. This indicates multiplication by channel.

[0072] It should be noted that the dynamic dilation rate represents the convolution sampling interval dynamically adjusted according to the normalized transition anomaly rate, and is an integer used to control the sampling range of the epitropic feature tensor; both fixed constants are positive constants determined through network training. The conditional kernel weights represent the convolution kernel weights modulated by the normalized transition anomaly rate, dynamically adjusted with position, used to ensure the convolution focuses on features adapted to the virtual scene. The first fixed parameter vector represents two fixed vectors with the same dimension as the epitropic feature tensor, obtained through network training, used to construct the linear relationship between the conditional kernel weights and the normalized transition anomaly rate. The receptive field radius represents the radius of the position range of the epitropic feature tensor that the convolution operation can cover, and is a non-negative integer, determining the size of the convolution's receptive range. Channel-wise multiplication means performing element-wise multiplication of the sampled epitropic feature tensor and the corresponding conditional kernel weights on the same feature channel, achieving channel-wise modulation of the features by the weights. Modified linear unit activation processing means processing using the ReLU activation function, used here to enhance the network's non-linear expressive ability and filter invalid features. The sampled poloidal feature tensor represents the feature vectors selected from the original poloidal feature tensor according to the dynamic dilation rate. When the feature vectors exceed the range of the original sequence, the feature vectors at the nearest valid position are selected to ensure sampling integrity. The hidden representation contains key information from the poloidal feature tensor after normalized transition anomaly rate modulation convolutional encoding, which is used for subsequent calculation of the void probability and ground compensation distance.

[0073] It should be noted that the training dataset is LiDAR point cloud data from mining scenarios (including mirror-like void scenes such as water accumulation and wet surfaces). The training objective function uses the root mean square loss of the void region distance compensation error. During training, a fixed constant is iteratively optimized using a gradient descent algorithm to ensure that the dynamic expansion rate optimally adapts to void scenes with different transition anomaly rates (e.g., the expansion rate increases when the transition anomaly rate is high, and the expansion rate approaches 1 when the transition anomaly rate is low). Finally, the fixed constant after training convergence is used as the fixed constant during deployment. Since the dimension of the polar feature tensor is 4, the dimensions of the first weight parameter and the first bias parameter in the first fixed parameter vector are both 4. The receptive field radius is determined based on the point cloud density of the scanning ring. The azimuth interval of the scanning ring in mining LiDAR is usually 0.1° to 0.5°, and the number of point clouds in a single ring is 720 to 3600. Therefore, the receptive field radius is usually in the range of 3 to 7. Specifically, when the number of point clouds in a single ring exceeds 2000, the receptive field radius is 3 to 5; when the number of point clouds in a single ring is less than 2000, the receptive field radius is 5 to 7. This ensures that the convolution can cover sufficient contextual information while avoiding a surge in computation due to an excessively large sensing range.

[0074] In one embodiment of the present invention, after performing a linear operation between the hidden representation and the second fixed parameter vector, the result of the operation is processed by a logical function to obtain the void probability of each position.

[0075] The hidden representation is linearly operated on with the third fixed parameter vector to obtain the ground compensation distance at each location;

[0076] Calculate the complement of the void probability, multiply the original range by the complement, and simultaneously multiply the ground compensation distance by the void probability. Sum the results of the two multiplications to obtain the fused distance.

[0077] It should be noted that the void probability represents the confidence that each location belongs to the mirror void region, with a value ranging from 0 to 1; the logistic function is the Sigmoid activation function. The second fixed parameter vector represents a fixed vector obtained through network training, with the same dimension as the hidden representation. It is used to extract void-related features from the hidden representation using weights, providing feature weights for void probability calculation. The third fixed parameter vector also represents a fixed vector obtained through network training, with the same dimension as the hidden representation. It is used to extract ground geometric features from the hidden representation using weights, providing feature weights for ground compensation distance calculation. The ground compensation distance represents the ground ranging value that should exist in the void region estimated based on the hidden representation. It is used to fill in the missing ranging measurements in the void region, and its value range is consistent with the original ranging. The complement of the void probability is obtained by subtracting the void probability from 1, with a value ranging from 0 to 1, used to quantify the confidence of non-void regions. The fused distance represents the result of fusing the original ranging and the ground compensation distance by weighting them according to the void probability and its complement, balancing the accuracy of the original data in non-void regions and the rationality of the supplementary data in void regions.

[0078] It should be noted that the dimension of the hidden representation is consistent with the dimension of the epitropic feature tensor. Therefore, the dimension of the second weight parameter in the second fixed parameter vector and the dimension of the third weight parameter in the third fixed parameter vector are both 4, and the dimension of the dimension of the second bias parameter in the second fixed parameter vector and the dimension of the third bias parameter in the third fixed parameter vector are both 1. Similarly, a LiDAR point cloud dataset from a mining autonomous driving scenario is used, including different environments such as sunny days, rainy days, and dusty days. The dataset is constructed by labeling real ground distance measurements (ground values ​​for void region compensation) and void region labels (ground values ​​for void probability). A combined loss of binary cross-entropy loss (for void probability) and mean squared error loss (for ground compensation distance) is used, with a weight ratio of 1:5 (because the accuracy of distance compensation has a greater impact on the subsequent feasible region generation). In addition, if the ground compensation distance is less than 0, it should be adjusted to 0 (LiDAR cannot measure negative distances); if the ground compensation distance is greater than the maximum ranging range of the LiDAR (the maximum ranging range of mining LiDAR is usually 200 meters, which needs to be set according to the actual hardware parameters), it should be adjusted to the maximum ranging range, which will not be elaborated here.

[0079] Specifically, the transpose of the second weight parameter is first multiplied by the hidden representation, and then added to the second bias parameter. The result of the addition is processed by the Sigmoid activation function to obtain the void probability.

[0080] Specifically, the transpose of the third weight parameter is multiplied by the hidden representation, and then added to the third bias parameter to output the ground compensation distance.

[0081] In one embodiment of the present invention, for each position, the difference between the azimuth of that position and the previous azimuth is calculated as the azimuth increment, wherein the azimuth increment of the first position is set to zero, and the azimuth increment of the last position is not calculated separately.

[0082] Using the fusion distance of the first position as the initial value, for the second to the last position, the forward constraint distance of each position is calculated in turn. The forward constraint distance is the smaller of the fusion distance of the current position, the forward constraint distance of the previous position, the product of the upper bound of the polar change rate and the current azimuth increment.

[0083] Using the forward constraint distance of the last position as the initial value, for each position from the second to last to the first position, calculate the backward constraint distance of each position in turn. The backward constraint distance is the smaller of the forward constraint distance of the current position, the backward constraint distance of the next position, the product of the upper bound of the polar change rate and the azimuth increment of the next position, where the azimuth increment of the next position represents the difference between the azimuth of the next position and the azimuth of the current position. Use the backward constraint distance of each position as the final distance.

[0084] Specifically, an initial value is set, where the forward limiting distance of the first position is equal to the fusion distance of that position; then, for positions numbered from 2 to N (the total number of positions within a single scan ring) (from the second to the last position), the forward limiting distance is calculated sequentially: first, the forward limiting distance of the previous position is calculated, and then the product of the upper bound of the polar change rate and the azimuth increment of the current position is added to obtain the maximum allowable distance value; then, the fusion distance of the current position is compared with this maximum allowable distance value, and the smaller of the two is taken as the forward limiting distance of the current position.

[0085] Specifically, an initial value is set, where the backward limiting distance of the last position (number N) is equal to the forward limiting distance of that position; then, for positions numbered from N-1 to 1 (from the second to the first position), the following calculations are performed sequentially: first, the backward limiting distance of the next position is calculated, and then the product of the upper bound of the polar change rate and the backward azimuth increment of the current position is added to obtain the maximum allowable distance value; then, the forward limiting distance of the current position is compared with this maximum allowable distance value, and the smaller of the two is taken as the backward limiting distance of the current position.

[0086] It should be noted that the azimuth increment represents the azimuth difference between two adjacent positions, used to quantify the angular span in the horizontal direction, providing a scale basis for the physical constraints of distance changes. The azimuth increment of the first position is fixed at 0. The backward azimuth increment represents the azimuth difference between the current position and the next position, used to quantify the angular span of adjacent positions in the opposite direction when constrained backward, and is calculated only for positions other than the last position. The forward constrained distance represents the distance value after constraining the fused distance in the azimuth increment direction (from the first to the last position) combined with the upper bound of the polar rate of change. The backward constrained distance represents the distance value after constraining the forward constrained distance again in the azimuth decrement direction (from the last to the first position) combined with the upper bound of the polar rate of change. The total number of positions in a single scan loop represents the total number of positions of all measurement points in a single scan loop, determined by the scanning resolution (azimuth interval) of the lidar. The final distance represents the distance value after two rounds of physical constraints in the forward and backward directions, accurately reflecting the ranging situation on the real ground.

[0087] It should be noted that in mining scenarios, the ground may have local protrusions (such as rock piles) or depressions (such as small pits). The forward constraint may fail due to the abnormal influence of the previous position (such as the distance of the previous position being too large, and the subsequent constraint also being too large). The backward constraint is verified from the end (usually far away from the vehicle body and less affected by interference), which can correct the cumulative error of the forward constraint, thereby reducing the error of the final distance and adapting to the complex terrain of mining ground.

[0088] In one embodiment of the present invention, the final distance, azimuth, and elevation angle of each location are processed by trigonometric functions, and combined with the sensor installation height, the three-dimensional coordinates of each location are obtained.

[0089] Set the origin coordinates and grid side length of the top-view grid. Calculate the difference between the x-coordinate and the x-coordinate of the grid origin, and the difference between the y-coordinate and the y-coordinate of the grid origin in the three-dimensional coordinates of each position. Calculate the ratio of the two differences to the grid side length and round down to obtain the top-view grid cell coordinates corresponding to each position.

[0090] Specifically, the horizontal axis of the three-dimensional coordinate system is the cosine of the final distance × elevation angle × azimuth angle, the vertical axis is the sine of the final distance × elevation angle × azimuth angle, and the vertical axis is the sensor installation height + the sine of the final distance × elevation angle.

[0091] It should be noted that the three-dimensional coordinates represent Cartesian coordinates after the polar coordinates (final distance, azimuth, elevation) have been converted; the sensor installation height represents the vertical installation height of the lidar sensor relative to the ground; the origin coordinates of the top-view grid represent the coordinates of the starting reference point of the top-view grid, with the x-coordinate being the minimum x-coordinate value in the three-dimensional coordinates of a single frame point cloud, and the y-coordinate being the minimum y-coordinate value in the three-dimensional coordinates of a single frame point cloud; the grid side length represents the side length of each square cell in the top-view grid, and the top-view grid cell coordinates represent the index of the cell to which each point belongs in the top-view grid, with the x-coordinate representing the horizontal index and the y-coordinate representing the vertical index, used to locate the position of the point in the top-view grid.

[0092] Specifically, first subtract the x-coordinate of the origin of the top-view grid from the x-coordinate of the three-dimensional coordinates, then divide by the grid side length, and finally round down to obtain the x-coordinate of the top-view grid; the y-coordinate of the top-view grid is calculated in the same way, which will not be elaborated here.

[0093] In one embodiment of the present invention, the mean value of the vertical coordinates in the three-dimensional coordinates of all positions within each top-view grid cell is calculated as the reference ground height of the top-view grid cell, the mean value of the final distances of all positions within each top-view grid cell is calculated as the average final distance of the top-view grid cell, the elevation angle interval between adjacent scanning rings is obtained, and the product of the elevation angle interval and the average final distance is calculated to obtain the tolerance of the top-view grid cell.

[0094] If the difference between the vertical coordinate of a location within each top-view grid cell and the reference ground height of that top-view grid cell does not exceed the tolerance, then that top-view grid cell is determined to be ground-occupied.

[0095] For each top-view grid cell determined to be occupied by the ground, the void probabilities of all positions within the cell are summed to obtain a score, and the score is compared with a predefined minimum passage score. Top-view grid cells with scores not lower than the minimum passage score are considered as feasible regions.

[0096] It should be noted that: Reference ground height represents the arithmetic mean of the vertical coordinates of all points within the top-view grid cell, used to represent the average vertical height of the ground within that cell; average final distance represents the arithmetic mean of the final distances of all points within the top-view grid cell, used to represent the average horizontal distance from points within that cell to the sensor; the elevation angle interval between adjacent scanning rings represents the absolute value of the elevation angle difference between two adjacent scanning rings of the LiDAR; tolerance represents the allowable range of ground height fluctuations within the top-view grid cell, dynamically adjusted with the average final distance to avoid ground judgment errors caused by distance differences. A ground-occupied cell represents a top-view grid cell where the absolute value of the difference between the vertical coordinate of at least one point and the reference ground height does not exceed the tolerance, indicating the presence of real ground within that cell. The passage score represents the sum of the virtual probabilities of all points within a ground-occupied cell, used to quantify the reliability of ground information replenishment within that cell; a higher score indicates stronger reliability. The feasible region represents the set of all ground-occupied cells with a passage score not lower than the minimum passage score, used for path planning of unmanned mining vehicles to identify areas where safe travel is possible.

[0097] It should be noted that the grid side length can be set according to the minimum turning radius of the mining unmanned vehicle. For example, multiply the minimum turning radius of the vehicle by 0.1. The minimum turning radius of mining vehicles is usually 5 to 10 meters, and the corresponding grid side length is 0.5 to 1 meter. This ensures that the feasible domain resolution meets the path planning accuracy while avoiding a surge in computation due to an excessive number of grids. The adjacent scanning ring elevation angle interval can be read from the LiDAR hardware manual and is usually 2° or 4°. The minimum passage score is determined statistically through mining scene datasets. For example, after collecting 1000 frames of mining scene point cloud data, the passage score of all ground-occupied units is calculated. Then, all scores are sorted from smallest to largest, and the 30th percentile value is taken as the minimum passage score. This percentile value ensures that low-confidence replenishment units are excluded while retaining more than 90% of high-confidence ground units. It is usually set to 5 to 10 (adjusted according to the point cloud density of a single frame). The determination of ground occupation units indicates that there must be at least one point within the unit. Since some ground units in mining scenarios may only contain a few points (such as distant ground), requiring multiple points would result in missing effective surfaces. Therefore, setting at least one point within the unit avoids misjudging empty units as ground occupation.

[0098] In one embodiment of the present invention, a deep learning-based environmental perception method for unmanned mining vehicles includes the following steps:

[0099] Step S201: Divide the single-frame point cloud into scanning rings according to the elevation angle, obtain its azimuth sequence, ranging sequence and effective mask, and determine the void sector, range transition, angle width and transition anomaly rate accordingly. In the non-void sector, obtain the set of ground polar change rate through the finite difference between adjacent azimuth and adjacent ranging, and calculate the upper bound of the polar change rate and the normalized transition anomaly rate.

[0100] Step S202: The normalized ranging, effective mask, normalized transition anomaly rate and polar gradient are combined to form a polar feature tensor, and boundary value preservation is performed on the virtual sector and the polar gradient in the virtual sector is set to zero.

[0101] Step S203: Map the normalized transition anomaly rate to obtain the dynamic expansion rate and conditional kernel weights, perform convolutional encoding on the polar feature tensor in one-dimensional polar orientation, and output the hidden representation.

[0102] Step S204: Calculate the void probability and ground compensation distance based on the hidden representation, and fuse the original ranging and ground compensation distance with the void probability as the weight to obtain the fused distance;

[0103] Step S205: Apply forward and backward constraints to the fusion distance based on the upper bound of the polar change rate to obtain the final distance;

[0104] Step S206: The final distance, azimuth, elevation and sensor installation height are restored to three-dimensional coordinates and projected onto the top-view grid. In each grid, the ground occupancy is determined according to the reference ground height and tolerance. The grid is weighted and accumulated with the void probability as the weight to obtain a score, and compared with the lowest passability score to generate a feasible region.

[0105] It should be noted that the interval and threshold sizes are set for ease of comparison. The size of the threshold depends on the amount of sample data and the base number set by those skilled in the art for each set of sample data, as long as it does not affect the proportional relationship between the parameter and the quantized value. Furthermore, the above formulas are all dimensionless calculations, and the formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0106] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A deep learning-based environmental perception system for unmanned mining vehicles, characterized in that, include: The transition anomaly extraction module is used to divide a single frame point cloud into scanning rings according to the elevation angle, obtain its azimuth sequence, ranging sequence and effective mask, and determine the void sector, range transition, angle width and transition anomaly rate accordingly. In the non-void sector, the ground polar change rate set is obtained by the finite difference between adjacent azimuth and adjacent ranging, and the upper bound of the polar change rate and the normalized transition anomaly rate are calculated. The polar feature construction module is used to combine normalized ranging, effective mask, normalized transition anomaly rate and polar gradient into a polar feature tensor, and to perform boundary value preservation on the virtual sector and set the polar gradient in the virtual sector to zero. The polar feature extraction module is used to map the normalized transition anomaly rate to obtain the dynamic expansion rate and conditional kernel weights, and to perform convolutional encoding on the polar feature tensor in one-dimensional polar direction to output the hidden representation. The fusion distance calculation module is used to calculate the void probability and ground-compensated distance based on the hidden representation, and fuse the original ranging and ground-compensated distance with the void probability as the weight to obtain the fusion distance. The final distance calculation module is used to apply forward and backward constraints to the fusion distance based on the upper bound of the polar change rate to obtain the final distance; The feasible region generation module is used to restore the final distance, azimuth, elevation and sensor installation height into three-dimensional coordinates and project them onto the top-view grid. Within each grid, it determines the ground occupation based on the reference ground height and tolerance, accumulates a score for each grid with the void probability as the weight, and compares it with the lowest pass score to generate a feasible region. The single-frame point cloud is divided into several scanning rings according to the elevation angle. The azimuth sequence, ranging sequence and effective mask of each scanning ring are obtained. Each value of the effective mask is used to identify whether there is an echo at the corresponding azimuth angle. Find the largest continuous corner region marked as echo-free in the effective mask, treat the corner region as a void sector, and record the left and right azimuth angles of the void sector. Extract the corresponding range value from the nearest location marked with an echo on the left side of the left azimuth angle, and extract the corresponding range value from the nearest location marked with an echo on the right side of the right azimuth angle. Calculate the difference between the two range values ​​as the range jump, calculate the difference between the left and right azimuth angles as the angle width, and then calculate the ratio of the range jump to the angle width to obtain the jump anomaly rate. In the non-void sector, the effective masks of three consecutive azimuth angles are selected and marked as locations with echoes. Based on the difference between adjacent azimuth angles and the difference between adjacent range values ​​at these locations, the ground polar change rate corresponding to each selected location is calculated by the finite difference method, forming a set of ground polar change rates. Statistical analysis was performed on the set of ground polar variability rates, and the 95th percentile value was taken as the upper bound of the polar variability rate. The normalized transition anomaly rate is obtained by calculating the ratio of the transition anomaly rate to the upper bound of the polar change rate. The normalized transition anomaly rate at each position is obtained, mapped using the hyperbolic tangent function, the mapping result is multiplied by a fixed constant and rounded down, and then one is added to the integer part to obtain the integer dynamic expansion rate corresponding to each position. Based on the normalized transition anomaly rate at each position, the conditional kernel weights at each corresponding position within the receptive field radius are obtained by performing a linear operation between the first fixed parameter vector and the normalized transition anomaly rate, where the receptive field radius is an integer and the two first fixed parameter vectors have the same dimension as the polar feature tensor. The polar feature tensor is sampled according to the dynamic expansion rate corresponding to each position to obtain the sampled polar feature tensor sequence. If the sampling exceeds both ends of the polar feature tensor sequence, the polar feature tensor corresponding to the nearest valid position in the sequence is taken. The sampled poloidal feature tensor is multiplied channel by channel with the corresponding conditional kernel weights, and all multiplication results are summed to obtain the summation result. The summation result is then processed by channel-wise modified linear unit activation to obtain the hidden representation corresponding to each position.

2. The deep learning-based environmental perception system for unmanned mining vehicles according to claim 1, characterized in that, Filter the effective mask identifiers in the non-empty sectors to identify the locations with echoes. Calculate the mean and standard deviation of the distance measurement based on these values. Then, normalize the distance measurement values ​​of all locations according to the mean and standard deviation to obtain normalized distance measurement. The effective masks of three consecutive azimuth angles in the non-void sector are all marked as positions with echoes and where these three azimuth angles are not within the void sector. Based on the difference between adjacent azimuth angles and the difference between adjacent range values ​​at these positions, the pole gradient corresponding to each selected position is calculated, and the pole gradient of other positions is set to zero. For each location within a void sector, the normalized transition anomaly rate remains unchanged; for each location outside a void sector, the normalized transition anomaly rate is set to zero. For positions within the void sector that are marked as echo-free by the effective mask, the normalized ranging is set to the normalized ranging of the nearest effective position on the left side of the void sector, and the poloidal gradient is set to zero. For each location, the normalized ranging, effective mask, normalized transition anomaly rate, and poloidal gradient of that location are combined to obtain the poloidal feature tensor corresponding to each location.

3. The deep learning-based environmental perception system for unmanned mining vehicles according to claim 1, characterized in that, After performing a linear operation between the hidden representation and the second fixed parameter vector, the result is processed by a logical function to obtain the void probability at each position. The hidden representation is linearly operated on with the third fixed parameter vector to obtain the ground compensation distance at each location; Calculate the complement of the void probability, multiply the original range by the complement, and simultaneously multiply the ground compensation distance by the void probability. Sum the results of the two multiplications to obtain the fused distance.

4. The deep learning-based environmental perception system for unmanned mining vehicles according to claim 1, characterized in that, For each position, the difference between the azimuth of that position and the previous azimuth is calculated as the azimuth increment, where the azimuth increment of the first position is set to zero, and the azimuth increment of the last position is not calculated separately. Using the fusion distance of the first position as the initial value, for the second to the last position, the forward constraint distance of each position is calculated in turn. The forward constraint distance is the smaller of the fusion distance of the current position, the forward constraint distance of the previous position, the product of the upper bound of the polar change rate and the current azimuth increment. Using the forward constraint distance of the last position as the initial value, for each position from the second to last to the first position, calculate the backward constraint distance of each position in turn. The backward constraint distance is the smaller of the forward constraint distance of the current position, the backward constraint distance of the next position, the product of the upper bound of the polar change rate and the azimuth increment of the next position, where the azimuth increment of the next position represents the difference between the azimuth of the next position and the azimuth of the current position. Use the backward constraint distance of each position as the final distance.

5. The deep learning-based environmental perception system for unmanned mining vehicles according to claim 1, characterized in that, The final distance, azimuth, and elevation angles of each location are processed using trigonometric functions, and combined with the sensor installation height, to obtain the three-dimensional coordinates of each location. Set the origin coordinates and grid side length of the top-view grid. Calculate the difference between the x-coordinate and the x-coordinate of the grid origin, and the difference between the y-coordinate and the y-coordinate of the grid origin in the three-dimensional coordinates of each position. Calculate the ratio of the two differences to the grid side length and round down to obtain the top-view grid cell coordinates corresponding to each position.

6. The deep learning-based environmental perception system for unmanned mining vehicles according to claim 5, characterized in that, The mean value of the vertical coordinates of all positions in the three-dimensional coordinates of each top-view grid cell is calculated as the reference ground height of the top-view grid cell. The mean value of the final distance of all positions in each top-view grid cell is calculated as the average final distance of the top-view grid cell. The elevation angle interval of adjacent scan rings is obtained. The product of the elevation angle interval and the average final distance is calculated to obtain the tolerance of the top-view grid cell. If the difference between the vertical coordinate of a location within each top-view grid cell and the reference ground height of that top-view grid cell does not exceed the tolerance, then that top-view grid cell is determined to be ground-occupied. For each top-view grid cell determined to be occupied by the ground, the void probabilities of all positions within the cell are summed to obtain a score, and the score is compared with a predefined minimum passage score. Top-view grid cells with scores not lower than the minimum passage score are considered as feasible regions.

7. A deep learning-based environmental perception method for unmanned mining vehicles, characterized in that, Implementing the deep learning-based environmental perception system for unmanned mining vehicles as described in any one of claims 1 to 6 includes the following steps: Step S201: Divide the single-frame point cloud into scanning rings according to the elevation angle, obtain its azimuth sequence, ranging sequence and effective mask, and determine the void sector, range transition, angle width and transition anomaly rate accordingly. In the non-void sector, obtain the set of ground polar change rate through the finite difference between adjacent azimuth and adjacent ranging, and calculate the upper bound of the polar change rate and the normalized transition anomaly rate. Step S202: The normalized ranging, effective mask, normalized transition anomaly rate and polar gradient are combined to form a polar feature tensor, and boundary value preservation is performed on the virtual sector and the polar gradient in the virtual sector is set to zero. Step S203: Map the normalized transition anomaly rate to obtain the dynamic expansion rate and conditional kernel weights, perform convolutional encoding on the polar feature tensor in one-dimensional polar orientation, and output the hidden representation. Step S204: Calculate the void probability and ground compensation distance based on the hidden representation, and fuse the original ranging and ground compensation distance with the void probability as the weight to obtain the fused distance; Step S205: Apply forward and backward constraints to the fusion distance based on the upper bound of the polar change rate to obtain the final distance; Step S206: The final distance, azimuth, elevation and sensor installation height are restored to three-dimensional coordinates and projected onto the top-view grid. In each grid, the ground occupancy is determined according to the reference ground height and tolerance. The grid is weighted and accumulated with the void probability as the weight to obtain a score, and compared with the lowest passability score to generate a feasible region.

Images