Mine automatic driving visual perception method and system based on YOLO real-time inference

Abstract

This invention discloses a visual perception method and system for autonomous driving in mines based on YOLO real-time inference, relating to the field of environmental perception technology for autonomous driving in mines. The method employs an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation to calculate the image plane displacement caused by low-frequency, high-amplitude vibrations from continuous image frames acquired by an onboard camera. Deconvolution operations are then performed on directional motion blur and edge texture degradation. The processed image frames are input into a YOLO real-time inference network with reconfigured anchor frame parameters. These anchor frame parameters are obtained by clustering the labeled dimensions of fallen rocks and eroded rock blocks from the slope in the mine. By performing optical flow deconvolution restoration operations on the blurred frames from the mine's bumpy terrain and reconfiguring the anchor frame parameters based on the clustering results of mine obstacle sizes, this invention enables the YOLO real-time inference network to maintain stable bounding box outputs and continuous trajectory markings for fallen rocks and eroded rock blocks under vibration conditions on unpaved roads.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology for autonomous driving in mines, specifically to a visual perception method and system for autonomous driving in mines based on YOLO real-time reasoning. Background Technology

[0002] In mining autonomous vehicles operating on unpaved roads, the onboard vision perception system is responsible for the real-time detection and localization of dangerous obstacles within the work area. The YOLO real-time inference architecture, with its end-to-end detection framework and high computational efficiency, has been applied in general road object detection scenarios. This architecture unifies the object detection task into a single forward propagation regression problem, predicting the position and size of bounding boxes using pre-set anchor boxes, and outputting the object category and spatial coordinates.

[0003] In the mining environment, vehicles travel on unpaved transport roads, subjecting their bodies to low-frequency, high-amplitude vibrations. This vibration is transmitted through the vehicle structure to the onboard camera mounting base, causing image plane displacement within a single frame exposure cycle. This results in motion blur extending along the direction of motion and attenuation of edge texture contrast in consecutive image frames. Simultaneously, dangerous obstacles commonly found in mining operations, such as fallen rocks and eroded rock fragments from slopes, exhibit different geometric shapes and scales compared to conventional targets like vehicles and pedestrians in general road scenarios, typically displaying smaller scale and irregular outlines.

[0004] The anchor frame scale distribution and aspect ratio prior parameters fixed in the initialization phase of the conventional YOLO real-time inference architecture originate from statistical fitting of labeled data of general road scenes. This statistical fitting process is based on the distribution law of target size under general scenes, which differs from the actual size distribution range of rolled-down gravel and eroded rock blocks on the slope in the mining working face. Under the above-mentioned mining driving conditions, the continuous image frames acquired by the vehicle-mounted camera are subject to motion blur interference induced by vibration, and also face the correspondence deviation between the preset anchor frame parameterized expression and the morphological characteristics of irregular small-scale dangerous obstacles in the mine. These factors together constitute the working conditions that need to be addressed in the visual perception task of autonomous driving in mining.

[0005] The limitations of existing technologies include at least the following problems: When the conventional YOLO real-time inference architecture is directly applied to the visual perception scenario of autonomous driving in mines, the road smoothness assumption on which this architecture relies is incompatible with the low-frequency, high-amplitude vibration conditions generated by vehicles traveling on unpaved roads in mines. The vehicle's bumps cause the on-board camera to produce image plane displacement within a single frame exposure period, which introduces directional motion blur and edge texture degradation into continuous image frames. The scale distribution and aspect ratio prior parameters of the anchor frames preset by the conventional YOLO model are derived from statistical fitting of general road scenes. There is a structural representation gap between this parameterized expression and the arbitrary morphological features of irregular small-scale dangerous obstacles such as rolling gravel and peeling rock blocks on the slope in the mine working face. The frame-level feature degradation caused by motion blur and the morphological representation deviation caused by anchor frame mismatch cause the detection response of the perception front end to intermittently interrupt the continuous temporal sequence of such targets. Single-frame missed detections evolve into frequent switching and loss of target trajectory markers after tracking and association. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a visual perception method and system for autonomous driving in mines based on YOLO real-time inference, which solves the problem of target continuous detection and trajectory tracking failure caused by bumpy blur and anchor frame mismatch in existing YOLO autonomous driving in mines.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a visual perception method for autonomous driving in mines based on YOLO real-time inference, comprising the following steps: For continuous image frames acquired by the onboard camera of an autonomous driving vehicle in a mine, an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation is used to calculate the image plane displacement caused by low-frequency, high-amplitude vibrations generated by vehicles traveling on unpaved roads in the mine; based on the calculation results, deconvolution operations are performed on directional motion trails and edge texture degradation existing in the continuous image frames; the continuous image frames after deconvolution are input into the prior parameter re-anchor frame. The configured YOLO real-time inference network reconfigures the anchor frame prior parameters by replacing the preset anchor frame scale distribution and aspect ratio prior parameters obtained through statistical fitting of general road scenes in the conventional YOLO real-time inference architecture with parameters obtained by clustering the labeled sizes of irregular small-scale dangerous obstacles such as rolling gravel and eroded rock blocks on the slope within the mining face. The YOLO real-time inference network performs feature extraction and bounding box regression on the input continuous image frames, outputting the bounding box coordinates of rolling gravel and eroded rock blocks on the slope within the mining face. Target trajectory identification is associated with the bounding box coordinates output in continuous time series.

[0008] Furthermore, the specific steps for performing deconvolution operations on directional motion blur and edge texture degradation in consecutive image frames are as follows: feature point extraction and pyramid optical flow tracking are performed on two adjacent frames of consecutive images to obtain pixel-level motion vector fields; the image plane displacement component caused by vibration and the scene change displacement component in the vector field are separated, and the vibration displacement component is extracted as the initial estimate of the motion blur kernel; adaptive deconvolution iteration is performed with the initial estimate of the blur kernel, and the blur kernel parameters and image estimation are updated synchronously. After the iteration converges, the processed consecutive image frames are output.

[0009] Furthermore, the specific steps for generating the initial estimate of the motion blur kernel are as follows: perform frequency domain transformation on the pixel-level motion vector field to obtain the motion vector spectrum distribution; perform bandpass filtering on the spectrum based on the vibration prior frequency band to filter out high-frequency noise and DC components; perform inverse transformation on the filtered spectrum to obtain the image plane displacement components caused by vibration; map the displacement components to the point spread function space to generate the initial estimate of the motion blur kernel.

[0010] Furthermore, the specific steps for replacing clustering parameters are as follows: obtain the width and height dimensions of the labeled bounding boxes of the mine's rolled-off gravel and the slope's eroded rock blocks; perform clustering operations on the width and height data, using the cluster centers as the reconfigured anchor frame scale and aspect ratio parameters; and replace the preset anchor frame parameters of the general road scene item by item with the scale and aspect ratio values ​​corresponding to the cluster centers.

[0011] Furthermore, the initial cluster center selection steps for clustering operations are as follows: randomly select the first cluster center from the width and height dataset; calculate the shortest distance between each point in the dataset and the existing centers; select the next cluster center with a probability based on the squared distance ratio; repeat the selection until a preset number of initial cluster centers are obtained.

[0012] Furthermore, the specific steps for trajectory identification association are as follows: obtain the bounding box coordinates of the mine obstacles in the current frame and the previous frame; calculate the intersection-union ratio (IU / U) of the bounding box coordinates of adjacent frames, and generate the IU / U cost matrix.

[0013] Further, after generating the cost matrix, the following steps are performed: apply prior position weights to the obstacle coordinates in the matrix based on the vehicle's travel and rockfall directions; input the weighted matrix into the Hungarian algorithm matcher to perform optimal matching between bounding boxes and trajectory labels; assign the original trajectory labels to successfully matched boxes and assign new trajectory labels to unmatched boxes.

[0014] Further, the specific steps of feature extraction are as follows: input the processed image into the YOLO backbone network and extract multi-scale feature maps layer by layer; input the multi-scale feature maps into the feature pyramid network and perform bidirectional feature fusion to obtain multi-scale fused feature maps; perform channel attention weighted suppression on the channel responses of the degraded residual frequency bands in the fused feature maps; input the processed fused feature maps into the YOLO detection head.

[0015] Furthermore, the specific steps of bounding box regression are as follows: obtain the coordinates of the predicted bounding boxes output by the YOLO detector head; calculate the loss function values ​​of the predicted boxes and the ground truth boxes, the loss function including position and aspect ratio loss terms; adaptively adjust the penalty weight of the aspect ratio loss term based on the obstacle annotation size variance; perform backpropagation and parameter update based on the loss function until the loss function converges.

[0016] A vision perception system for autonomous driving in mining based on YOLO real-time inference includes: an image acquisition module for acquiring continuous image frames captured by the onboard camera of an autonomous driving vehicle in the mine; a deblurring module for applying an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation to the continuous image frames, calculating the image plane displacement caused by low-frequency, high-amplitude vibrations generated by vehicles traveling on unpaved roads in the mine, and performing deconvolution operations on directional motion trails and edge texture degradation in the continuous image frames based on the calculation results; and an anchor frame parameter reconfiguration module for reconfiguring preset anchor frames obtained by statistical fitting of general road scenes in a conventional YOLO real-time inference architecture. The scale distribution and aspect ratio prior parameters are replaced with parameters obtained by clustering irregular small-scale hazardous obstacles labeled with dimensions from rolling boulders and detached rock blocks on the mine face. The YOLO real-time inference module receives continuous image frames after deconvolution from the deblurring module and anchor frame scale distribution and aspect ratio prior parameters provided by the anchor frame parameter reconfiguration module. It performs feature extraction and bounding box regression on the input continuous image frames and outputs the bounding box coordinates of rolling boulders and detached rock blocks on the mine face. The trajectory association module performs target trajectory identification association on the bounding box coordinates output by the YOLO real-time inference module over continuous time.

[0017] The present invention has the following beneficial effects: (1) The visual perception method for autonomous driving in mines based on YOLO real-time inference uses an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation to calculate the image plane displacement caused by low-frequency high-amplitude vibration of the unpaved road surface in the mine and performs deconvolution operation on the directional motion trail and edge texture degradation in the continuous image frames. After extracting the pixel-level motion vector field between adjacent frames, the image plane displacement component caused by vibration is separated from the scene change displacement component. The vibration displacement component is used as the initial estimate of the motion blur kernel. The blur kernel parameters and image estimation are updated synchronously in the adaptive deconvolution iteration until convergence. The above process infers the vibration disturbance from the optical flow change of the image sequence itself and performs the restoration operation. It does not rely on additional sensor calibration or fixed blur kernel prior model, so that the texture edges degraded by bump blur in the continuous image frames can be restored.

[0018] (2) In the process of reconfiguring the anchor frame prior parameters, the preset anchor frame scale distribution and aspect ratio prior parameters derived from the statistical fitting of general road scenarios in the conventional YOLO real-time inference architecture are replaced with parameters obtained by clustering the irregular small-scale dangerous obstacles labeled with the dimensions of rolling gravel and peeling rock blocks on the slope in the mining working face. The improved initial center selection operation of K-means clustering is performed on the width and height data of the obstacle label bounding box. The coordinates of the cluster center are directly used as the values ​​of the anchor frame scale and aspect ratio after reconfiguration. Since the clustering operation is based entirely on the statistical law of the size of real obstacles in the mining scene, it is no longer constrained by the target scale distribution of urban roads. A more direct correspondence is established between the parameterized expression of the anchor frame and the arbitrary shape characteristics of irregular small-scale obstacles in the mine. The YOLO real-time inference network adjusts the initial anchor frame that is closer to the real shape of the target in the bounding box regression stage, which alleviates the regression convergence delay problem caused by the mismatch between the preset anchor frame and the shape of the mine obstacle.

[0019] (3) The visual perception method for autonomous driving in mines based on YOLO real-time inference generates an intersection-union cost matrix for adjacent frames and applies a priori weights based on the vehicle driving direction and the rolling direction of the rock to the coordinate positions of the rolling rubble and the detached rock on the slope in the continuous temporal bounding box coordinate trajectory label association process. After the weighted matrix is ​​input into the Hungarian algorithm matcher, the bounding box is matched with the existing trajectory label. If the match is successful, the original label is used; if the match is not successful, a new label is assigned. This process introduces the physical prior constraint of the obstacle movement trend in the mining environment on the basis of spatial intersection-union, so that the trajectory label of the same rock or detached rock in the continuous frames remains continuous. At the same time, in the feature extraction stage, attention weighted suppression is performed on the channel response of the directional motion trail and the residual frequency band of edge texture degradation in the multi-scale feature map after feature pyramid fusion, so as to reduce the influence of residual degradation components on feature expression after deblurring. The penalty weight of the aspect ratio loss term in the bounding box regression loss function is adaptively adjusted according to the variance of the obstacle label size, so that the network maintains balanced regression fitting for irregular small targets with significant morphological differences.

[0020] (4) The mine autonomous driving vision perception system based on YOLO real-time inference acquires continuous image frames through the image acquisition module. The deblurring module performs adaptive deconvolution operation based on inter-frame optical flow estimation to process directional motion trails and edge texture degradation. The anchor frame parameter reconfiguration module independently completes the clustering operation of the annotation size of irregular small-scale obstacles in the mine and provides the anchor frame scale and aspect ratio parameters to the YOLO real-time inference module. The YOLO real-time inference module receives the deblurred image frames and reconfigured anchor frame parameters, performs feature extraction and bounding box regression to output bounding box coordinates. The trajectory association module applies position prior weights to the bounding box coordinates in continuous time sequence and completes Hungarian matching to maintain the continuity of the trajectory labels of falling rocks and detached rock blocks.

[0021] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0022] Figure 1 This is a flowchart of a visual perception method for autonomous driving in mines based on YOLO real-time inference, according to the present invention.

[0023] Figure 2 This is a block diagram of a mine autonomous driving visual perception system based on YOLO real-time inference, according to the present invention. Detailed Implementation

[0024] Please see Figure 1 This invention provides a technical solution: a visual perception method for autonomous driving in mines based on YOLO real-time inference, comprising the following steps: For continuous image frames acquired by the onboard camera of an autonomous driving vehicle in a mine, an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation is used to calculate the image plane displacement caused by low-frequency, high-amplitude vibrations generated by vehicles traveling on unpaved roads in the mine; based on the calculation results, deconvolution operations are performed on directional motion trails and edge texture degradation existing in the continuous image frames; the continuous image frames after deconvolution are input into the YOLO system after anchor frame prior parameter reconfiguration. The OLO real-time inference network reconfigures the anchor frame prior parameters by replacing the preset anchor frame scale distribution and aspect ratio prior parameters obtained through statistical fitting of general road scenes in the conventional YOLO real-time inference architecture with parameters obtained by clustering the labeled dimensions of irregular small-scale dangerous obstacles such as rolling rocks and eroded rock blocks on the slope within the mining face. The YOLO real-time inference network performs feature extraction and bounding box regression on the input continuous image frames, outputting the bounding box coordinates of rolling rocks and eroded rock blocks on the slope within the mining face. Target trajectory identification is then performed on the bounding box coordinates output in continuous time series.

[0025] Among them, the iterative convergence condition of the adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation is that the number of iterations reaches a preset threshold or the image pixel error between two adjacent iterations is less than a preset error threshold. The number of clusters for the parameters obtained from clustering is set according to the actual size distribution of irregular small-scale dangerous obstacles in the mining face, to ensure that the anchor frame parameters are adapted to different sizes of rolling gravel and slope spalling rock blocks.

[0026] Specifically, the steps for performing deconvolution on directional motion blur and edge texture degradation in consecutive image frames are as follows: Feature point extraction and pyramid optical flow tracking are performed on two adjacent frames of a continuous image to obtain a pixel-level motion vector field, specifically as follows: The SIFT algorithm is used to extract feature points from two adjacent frames (denoted as the current frame image and the next frame image). A Gaussian difference pyramid is constructed from the extracted feature points. The number of pyramid layers is set to an appropriate level, and the image size of each layer is half that of the previous layer. The motion vector of each feature point between two frames is calculated using the pyramid optical flow tracking algorithm. The motion vector is represented by the displacement of the pixel in two vertical directions. Among them, the two vertical coordinates correspond to the pixel positions of the feature points in the current frame image. The displacement in the two directions together constitute the motion vector of each feature point. The motion vectors of all feature points are combined to form a pixel-level motion vector field, thereby capturing the image pixel displacement caused by the vibration of mining vehicles. The image plane displacement component induced by vibration in the vector field is separated from the scene change displacement component. The vibration displacement component is extracted as the initial estimate of the motion blur kernel. Specifically: A Gaussian mixture model is used to model the pixel-level motion vector field. Leveraging the model's ability to effectively classify mixed distribution data, the motion vectors are divided into two categories: One type is the image plane displacement component caused by vibration (characterized by concentrated distribution and small variance), and the other type is the image plane displacement component caused by changes in scene content (characterized by dispersed distribution and large variance). The two types of components satisfy the vector superposition relationship. By setting a variance threshold for motion vectors, motion vectors with a variance less than the threshold are identified as displacement components caused by vibration, while motion vectors with a variance greater than or equal to the threshold are identified as displacement components caused by scene changes. The selected vibration displacement components are statistically averaged to obtain the average vibration displacement vector, which is used as the initial estimate of the motion fuzzy kernel. For example, when the low-frequency vibration amplitude of mining vehicles is small, the corresponding average vibration displacement will have reasonable values ​​in the two vertical directions. An adaptive deconvolution iteration is performed using an initial blur kernel estimate, synchronously updating the blur kernel parameters and image estimation. After the iteration converges, a series of processed image frames are output, specifically: Set an initial estimation kernel for motion blur; the initial sharp image is estimated as the degraded image of the current frame. During the iteration process, the blur kernel and the sharp image estimate are updated synchronously in each iteration. The update logic is as follows: Minimize image error and regularization constraints, where two regularization parameters are set and given reasonable value ranges to avoid overfitting during the iteration process; Each iteration calculates the pixel error estimated between two adjacent clear images. When the error is less than a preset error threshold or the number of iterations reaches a preset number, the iteration stops, and a clear image is output. All consecutive image frames are processed sequentially according to the above process to obtain the processed consecutive image frames.

[0027] The specific steps for generating the initial estimate of the motion fuzzy kernel are as follows: Perform a frequency domain transformation on the pixel-level motion vector field to obtain the spectral distribution of the motion vector, specifically as follows: For the vibration displacement components in the pixel-level motion vector field, perform two-dimensional Fourier transforms on the displacement components in the two perpendicular directions respectively. Utilize the characteristic of Fourier transform to convert time-domain signals into frequency-domain signals, convert the vibration displacement components in the time domain into the frequency-domain spectral distribution, and obtain the spectrum corresponding to the displacement components in the two directions. The horizontal axis of the spectral distribution is the frequency, and the vertical axis is the spectral amplitude. The low-frequency, high-amplitude vibration in the mine corresponds to a specific frequency range, and the amplitude of this frequency band is significantly higher than that of other frequency bands. Based on the prior frequency range of low-frequency, high-amplitude vibrations generated by vehicles traveling on unpaved roads in mines, bandpass filtering is applied to the spectral distribution to filter out high-frequency random noise components and DC components. Specifically: A priori frequency band range for low-frequency, high-amplitude vibrations in mines is defined. Based on the actual characteristics of mine vehicle vibrations, a Butterworth bandpass filter is designed. This filter has the advantages of stable amplitude-frequency characteristics and good attenuation characteristics. The spectrum is multiplied by the filter transfer function to filter out DC components with frequencies below the range and high-frequency random noise components with frequencies above the range, while retaining the vibration-related spectrum components in the range. The filter is set with reasonable low cutoff frequency, high cutoff frequency, and order to ensure the filtering effect. An inverse transform is performed on the filtered spectral distribution to obtain the image plane displacement components induced by low-frequency, high-amplitude vibrations. Specifically: Two-dimensional inverse Fourier transforms are performed on the filtered spectrum to convert the frequency domain spectral components back to the time domain, resulting in the filtered vibration displacement component. This component is the pure image plane displacement component caused by low-frequency high-amplitude vibration, eliminating the interference of scene changes and noise. For example, the variance of the filtered displacement components can be reduced to below a reasonable range; The displacement components are mapped to the point spread function parameter space to generate an initial estimate of the motion fuzzy kernel, specifically as follows: The motion fuzz kernel is essentially a representation of the point spread function. The parameters of the point spread function are directly related to the vibration displacement components. The average displacement and vibration direction angle of the vibration displacement components are set, where the vibration direction angle is the angle between the displacement vector and one of the perpendicular directions. The point spread function is constructed based on the average displacement and direction angle. This point spread function is the initial estimate of the motion fuzz kernel. The size of the fuzz kernel is set to a reasonable size to ensure that it can cover the maximum displacement range caused by vibration.

[0028] In this implementation scheme, a pixel-level motion vector field is constructed from adjacent image frames through SIFT feature point extraction and pyramid optical flow tracking. The image plane displacement component caused by vibration is separated from the scene change displacement component by using a Gaussian mixture model. This allows for the initial estimation of the motion blur kernel caused by turbulence without the need for an additional inertial measurement unit. Frequency domain transformation and bandpass filtering are performed on the separated vibration displacement component. High-frequency noise and DC interference are filtered out based on the prior frequency band of low-frequency, high-amplitude vibration in the mine. After inverse transformation, the pure image plane displacement is restored and mapped to the point spread function parameter space to generate a motion blur kernel that matches the actual vibration state. Subsequently, an adaptive deconvolution iteration is performed using this blur kernel as the initial value. Regularization constraints are introduced during the process of synchronously updating the blur kernel parameters and estimating the clear image, so that directional motion trails and edge texture degradation in continuous image frames can be specifically restored.

[0029] Specifically, the steps for replacing clustering parameters are as follows: Obtain the width and height dimensions of the labeled bounding boxes for fallen rock and eroded rock blocks from the mine and slopes. Specifically: Collect relevant images of different areas within the mining face under different weather conditions, with a quantity sufficient for annotation, covering various irregular small-scale dangerous obstacles of different sizes; Manual annotation is used to label targets in each image with axis-aligned rectangular boxes to ensure annotation accuracy; Record the width and height of each labeled bounding box to form a dataset with width and height dimensions, where the total number of datasets meets the clustering requirements; For example, if the actual size of the rolled-off gravel corresponding to a certain bounding box is appropriate, the width and height of the bounding box will also have corresponding reasonable values ​​based on the image pixel resolution. Clustering operations are performed on the width and height data, and the cluster centers are used as the reconfigured anchor frame size and aspect ratio parameters. Specifically: The width and height data are normalized, with the normalization benchmark being the width and height of the input image to the YOLO network. The purpose is to eliminate the influence of image size on the clustering results. Clustering algorithms are used to cluster the normalized width and height data. The number of clusters is set within a reasonable range. Euclidean distance is used as the similarity measure to measure the differences between different width and height data. The clustering objective is to minimize the sum of the distances from all data points to their respective cluster centers, ensuring that the cluster centers can accurately represent the size distribution of mine obstacles. After clustering, several cluster centers are obtained. The cluster centers are restored to their actual pixel size to obtain the anchor frame size. The aspect ratio of the cluster centers is the aspect ratio of the anchor frame. The preset anchor frame parameters for the general road scene are replaced item by item with the scale and aspect ratio values ​​corresponding to the cluster centers, specifically as follows: The common anchor box parameters of the conventional YOLO architecture are replaced item by item with the anchor box parameters of the mining scenario obtained by clustering. After the replacement, the network anchor box configuration file is updated to ensure that the reconfigured parameters are called during network inference. During the replacement process, the number and hierarchical distribution of anchor boxes are kept consistent with the conventional YOLO architecture, and only the scale and aspect ratio parameters of the anchor boxes are updated.

[0030] The steps for selecting initial centers in clustering operations are as follows: The first cluster center is randomly selected from the width and height dataset, specifically as follows: From the normalized width and height dataset, a data point is randomly selected as the first cluster center using uniform random sampling. During the sampling process, it is ensured that each data point has an equal probability of being selected, and data points located in the central region of the dataset can be selected preferentially. The shortest distance between each point in the dataset and the existing center is calculated as follows: Given the currently selected set of cluster centers, for each data point in the dataset, calculate its Euclidean distance to each existing cluster center, and take the minimum value among all distances as the shortest distance for that data point; For example, if a data point is located at a distance that is large or small from two existing cluster centers, the smaller value is taken as the shortest distance to the data point. The next cluster center is selected based on the probability of the ratio of the square of the shortest distance to the sum of the squares of the shortest distances of all data points. Specifically: Calculate the sum of squares of the shortest distances of all data points, and then calculate the selection probability of each data point. This probability is the ratio of the square of the shortest distance to the sum of squares. Use the roulette wheel method to select the next cluster center according to the probability. This method can ensure that data points that are far away from existing cluster centers have a higher probability of being selected, and can effectively avoid the initial cluster centers being too concentrated. Repeatedly calculate the shortest distance and select it according to probability until a preset number of initial cluster centers are obtained, specifically: Repeat the above two steps, updating the cluster center set after each selection, until the number of cluster centers reaches the preset value; After selection, ensure that the shortest distance between the initial cluster centers is not less than the preset distance. If it is less, reselect to ensure uniform distribution.

[0031] In this implementation scheme, the width and height data of the labeled bounding boxes of rolled-down gravel and eroded rock blocks on the slope in different regions and weather conditions are collected within the mining working face. The width and height values ​​are normalized and clustered. The cluster centers are used as the reconfigured anchor frame scale and aspect ratio parameters to replace the preset anchor frame parameters derived from statistical fitting of general road scenarios in the conventional YOLO real-time inference architecture. The selection of the initial cluster centers adopts a roulette wheel method based on the probability of squared distance, making the initial center distribution more uniform and reducing the sensitivity of the clustering results to random starting points. The replaced anchor frame parameters are closer to the actual shape characteristics of irregular small-scale dangerous obstacles in the mine in terms of scale distribution and aspect ratio values. During the bounding box regression stage, the YOLO real-time inference network can adjust the position and size based on the initial anchor frame that is closer to the actual contour of the target, thereby alleviating the regression convergence lag problem caused by the mismatch between the preset anchor frame and the shape of the mine obstacle.

[0032] Specifically, the steps for associating trajectory identifiers are as follows: To obtain the bounding box coordinates of the mine obstacles in the current frame and the previous frame, the specific steps are as follows: Set the current frame and the previous frame, and output the bounding box coordinate sets of the two frames through the YOLO network. Each set contains several bounding box coordinates. Each bounding box coordinate is represented by an axis-aligned rectangle, which is determined by the pixel coordinates of the top-left and bottom-right corners of the rectangle. For example, a bounding box in the current frame corresponds to a rockfall target within a certain area of ​​the image; Calculate the intersection-union ratio (CIU) of the bounding box coordinates of adjacent frames to generate the CIU cost matrix, which is as follows: The intersection-union ratio (IUU) measures the degree of overlap between two bounding boxes. It is an indicator of bounding box matching in target tracking. It is calculated as the ratio of the intersection area to the union area of ​​two bounding boxes. The value ranges from zero to one. The closer it is to one, the higher the degree of overlap, that is, the greater the probability that the two bounding boxes correspond to the same target. Calculate the intersection-union ratio (IUR) of each bounding box in the current frame with that of each bounding box in the previous frame, and generate a cost matrix of the corresponding dimension. The matrix element is one minus the IUR. That is, the larger the IUR, the smaller the cost. In this way, the bounding box matching problem is transformed into a cost minimization problem. For example, if the intersection and union of two sets of bounding boxes is relatively high, then the corresponding cost matrix elements are smaller, indicating that the two sets have a high degree of matching.

[0033] Execute after generating the cost matrix: Prior positional weights for vehicle travel and rockfall direction are applied to the obstacle coordinates in the matrix, specifically as follows: Based on the characteristics of the mining scenario, the vehicle's driving direction is set as one of the positive perpendicular directions, and the direction of falling rocks is set as the other positive perpendicular direction. A prior position weight is applied to each element in the cost matrix. This weight is obtained by weighting the vehicle driving direction weight and the rock rolling direction weight. The weight coefficients of the two directions are set to reasonable values, and the weights of the two directions decrease as the distance between the center of the bounding box increases. Reasonable distance thresholds are set for the two directions respectively. The cost after applying weights is the product of the original cost and the weights. This method can improve the matching priority of bounding boxes at reasonable locations and avoid mismatches caused by the complex environment of the mine. The weighted matrix is ​​then input into the Hungarian algorithm matcher to perform optimal matching between the bounding box and the trajectory label, specifically as follows: The Hungarian algorithm can find the minimum matching scheme in a finite time. The weighted cost matrix is ​​input into the Hungarian algorithm. The goal of the algorithm is to find the optimal matching scheme that minimizes the sum of matching costs, and each bounding box can be matched at most once. Set a matching threshold. When the weighted cost is less than the threshold, the two bounding boxes are considered to be matched; otherwise, they are not matched. After the current frame bounding box is matched with the previous frame bounding box, there may be some current frame bounding boxes that are difficult to match. These bounding boxes correspond to newly appearing targets. The original trajectory identifier is assigned to the successfully matched bounding boxes, and a new trajectory identifier is assigned to the unmatched bounding boxes, specifically as follows: Each bounding box in the previous frame corresponds to a unique trajectory identifier, which is incremented by a positive integer. The bounding box in the current frame that is successfully matched uses the corresponding identifier from the previous frame. A new identifier is assigned to the unmatched current frame bounding box, which is the current maximum identifier plus one. The flags of unmatched frames from the previous frame are retained for a certain number of frames. If there are still no matches, they are deleted, and resources are released.

[0034] In this implementation, the coordinates of the bounding boxes of obstacles in the mine output by the YOLO network in the current frame and the previous frame are obtained, and the intersection-union ratio (CUI) between the bounding boxes of adjacent frames is calculated to generate an CUI cost matrix. The cost matrix incorporates prior position weights set according to the vehicle driving direction and the rock rolling direction, so that the bounding box matching process no longer depends solely on the degree of spatial overlap, but reasonably adjusts the matching priority based on the physical laws of obstacle movement in the mine scene. The weighted cost matrix is ​​input into the Hungarian algorithm matcher to perform global optimal matching. The existing trajectory labels are used for successfully matched bounding boxes, and new trajectory labels are assigned to unmatched bounding boxes. In this way, the frequent switching and loss of trajectory labels caused by single-frame missed detection or deblurring residual degradation are effectively suppressed in the continuous temporal bounding box coordinate association.

[0035] Specifically, the steps for feature extraction are as follows: The processed image is input into the YOLO backbone network, and multi-scale feature maps are extracted layer by layer, specifically as follows: The processed image consists of consecutive image frames after the deconvolution operation. The image size is adjusted to the standard input size of the YOLO real-time inference network and normalized to normalize the pixel values ​​to a reasonable range. The normalized image is input into the YOLO backbone network, which contains several convolutional layers and residual blocks. Each convolutional layer uses a convolutional kernel of appropriate size and a stride of reasonable value. The residual blocks are connected by shortcuts to avoid the gradient vanishing problem that occurs during the training of deep networks. Multi-scale feature maps of the image are extracted layer by layer by the backbone network, resulting in three feature maps of different scales, corresponding to shallow features, medium features and deep features respectively. The shallow feature map is larger in size and contains detailed information such as the edge and texture of the image. The medium feature map is of moderate size and contains the contour information of the target. The deep feature map is the smallest in size and contains the semantic information of the target. For example, shallow feature maps can clearly show the edge texture of rolled-up gravel, while deep feature maps can accurately represent the overall outline of rolled-up gravel. The multi-scale feature map is input into the feature pyramid network, and bidirectional feature fusion is performed to obtain the multi-scale fused feature map, which is as follows: The three feature maps of different scales output by the backbone network are input into the feature pyramid network, which contains two fusion paths, one from top to bottom and one from bottom to top, with the aim of fusing features from different levels to improve the detection performance of small-scale targets. Top-down path: The deep feature map is upsampled (using bilinear interpolation with an upsampling factor of two) to obtain a feature map with the same size as the middle feature map. This feature map is then fused with the middle feature map element by element to obtain the fused middle feature map. Then, the mid-layer fusion feature map is upsampled to obtain a feature map with the same size as the shallow feature map. This feature map is then added to the shallow feature map element by element to obtain a shallow fusion feature map, thus realizing the transfer of deep semantic features to the shallow layer. Bottom-up path: The shallow fusion feature map is downsampled through a convolutional layer (with an appropriate kernel size and stride) to obtain a feature map with the same size as the middle fusion feature map. This feature map is then added to the middle fusion feature map element by element to obtain the updated middle fusion feature map. The updated mid-layer fusion feature map is then downsampled to obtain a feature map with the same size as the deep feature map. This feature map is then added to the deep feature map element by element to obtain the deep fusion feature map, thus realizing the transfer of shallow detailed features to the deep layer. The final result is three multi-scale fused feature maps, each with a size equal to or smaller than the original three feature maps. Figure 1 In summary, the fused feature map contains both shallow detail features and deep semantic features, which can effectively improve the feature representation capability of small-scale targets. Channel attention-weighted suppression is applied to the channel responses of degraded residual frequency bands in the fused feature map, specifically as follows: A channel attention mechanism is adopted, which can adaptively adjust the weights of each channel of the feature map, highlighting effective features and suppressing ineffective features, and processing each multi-scale fused feature map separately. Taking a certain fused feature map as an example, global average pooling is first performed on it to obtain channel feature vectors, and then channel attention weights are obtained through fully connected layers and activation functions to evaluate the importance of each channel; The channel weights corresponding to the degraded residual frequency bands are multiplied by a suppression coefficient, while the weights of the other channels remain unchanged. This is done to suppress the residual vibration degradation features after deblurring and prevent them from interfering with target detection. The adjusted weights are multiplied channel by channel with the fused feature map to suppress degradation residues and improve the purity of the feature map; The multi-scale fused feature map, after undergoing channel attention-weighted suppression, is output to the detection head of the YOLO real-time inference network, specifically as follows: The three processed multi-scale fused feature maps are then input into the three corresponding YOLO detection heads. Each detection head contains two convolutional layers, one for feature extraction and the other for target parameter output. The number of output channels of the detection head meets the target detection requirements, corresponding to several anchor frames, bounding box parameters, confidence levels, and categories (dangerous obstacles). The outputs of the three detection heads are concatenated to obtain the final feature extraction result, which is then used for subsequent bounding box regression.

[0036] In this implementation, the continuous image frames after deconvolution are adjusted to the standard input size of the YOLO real-time inference network and normalized. The backbone network extracts feature maps at three scales: shallow, medium, and deep, layer by layer. This ensures that the edge texture, contour shape, and semantic information of the rolled-off rubble and the eroded rock on the slope are preserved at different levels. The feature pyramid network performs bidirectional fusion of the multi-scale feature maps from top to bottom and from bottom to top, transferring deep semantic features to the shallow layer and feeding back shallow detail features to the deep layer. The fused feature map has both the ability to express details that are crucial for small-scale obstacles and the ability to make overall discrimination. A channel attention mechanism is introduced to apply weighted suppression to the channel responses corresponding to the directional motion trails and the residual frequency bands of edge texture degradation in the fused feature map, reducing the interference of residual degradation components after deblurring on target detection.

[0037] Specifically, the steps for bounding box regression are as follows: Obtain the coordinates of the predicted bounding box output by the YOLO detector head, specifically as follows: The YOLO detector outputs the offset relative to the anchor box. Combined with the reconfigured anchor box parameters and the center coordinates of the anchor box on the feature map, it is converted into the predicted bounding box coordinates in the image pixel coordinate system. During the conversion process, the center coordinate offset is normalized by the activation function to ensure that the predicted center is within the anchor box grid. Convert the center coordinates and width and height into the coordinates of the top left and bottom right corners of the axis-aligned rectangle; these are the coordinates of the predicted bounding box. Calculate the loss function values ​​for the predicted bounding box and the ground truth bounding box. The loss function includes a location loss term and an aspect ratio loss term, specifically: The true bounding box is manually labeled with coordinates. After being converted to center coordinates and width and height, the loss is calculated with the predicted box to ensure that the deviation between the predicted box and the true target is minimized. The location loss adopts an appropriate loss calculation method, which adds a correction term based on the measurement of the degree of overlap of the bounding boxes, which can effectively solve the problem that the loss is zero when the bounding boxes do not overlap and that gradient updates are difficult. The aspect ratio loss uses logarithmic loss to accurately measure the deviation between the predicted and the actual aspect ratio, avoiding aspect ratio prediction deviations caused by irregularities in mine obstacles; The total loss is the weighted sum of the two, with the initial weighting coefficients set to reasonable values ​​to balance the importance of location prediction and aspect ratio prediction. Based on the statistical variance of the dimensions of fallen rocks and detached rock fragments on the slope within the mining face, the penalty weight for the aspect ratio loss term is adaptively adjusted, specifically as follows: Calculate the statistical variance of the aspect ratio of the labeled bounding box. This variance can reflect the degree of difference in the aspect ratio of mine obstacles. The larger the variance, the more irregular the shape of the obstacle, and the more difficult it is to predict the aspect ratio. The penalty weight for aspect ratio loss is adaptively adjusted based on the variance. The adjustment logic is that the larger the variance, the larger the penalty weight, thereby increasing the penalty for aspect ratio prediction error and improving the aspect ratio prediction accuracy of irregular targets. The smaller the variance, the smaller the penalty weight, thereby reducing the penalty and avoiding overfitting caused by excessive penalty. Set a reasonable adjustment coefficient; For example, when the variance is at a moderate level, the penalty weight can be adjusted to an appropriate value while updating the total loss function. This adjustment logic fits the actual distribution characteristics of mine obstacles. Backpropagation and network parameter updates are performed based on the loss function value until the loss function converges, specifically as follows: The stochastic gradient descent optimization algorithm is adopted. This algorithm is a commonly used parameter optimization method in deep learning. It can gradually reduce the value of the loss function through gradient descent. Backpropagation is performed based on the total loss function to calculate the gradient of all trainable parameters in the YOLO real-time inference network and determine the parameter update direction. The network parameters are updated according to the gradient descent strategy, the learning rate is set to a reasonable value, and an appropriate strategy is used to dynamically decay the learning rate to balance the network convergence speed and convergence accuracy. The average loss function value is calculated once for each iteration. When the change in the average loss function value is less than the preset value for several consecutive training rounds, it indicates that the network parameters have stabilized, and backpropagation and parameter updates are stopped. At this point, the bounding box regression has reached the convergence state.

[0038] In this implementation, the offset output by the YOLO detector head is combined with the reconfigured anchor box parameters to convert the predicted bounding box coordinates in the image pixel coordinate system. A loss function containing position loss and aspect ratio loss is constructed to constrain the prediction results. The position loss term introduces a correction mechanism based on the degree of boundary box overlap to avoid gradient stagnation caused by the lack of overlap between the predicted and ground truth boxes. The aspect ratio loss term adopts a logarithmic loss form to accurately measure the deviation between the predicted and ground truth aspect ratios. Based on this, the penalty weight of the aspect ratio loss term is adaptively adjusted according to the statistical variance of the aspect ratio of the bounding boxes labeled with fallen boulders and detached rock blocks on the mine face. When the obstacle shape varies greatly, the aspect ratio penalty is automatically increased, and when the shape is relatively regular, the penalty is reduced accordingly to avoid overfitting. The loss function value is backpropagated through the stochastic gradient descent algorithm to update the network parameters until the loss change in multiple consecutive training rounds tends to stabilize. The above process enhances the adaptability of the bounding box regression to the shape of irregular small-scale obstacles in the mine, and the network convergence process is more stable.

[0039] Please see Figure 2This invention provides a technical solution: a visual perception system for autonomous driving in mines based on YOLO real-time inference, comprising: an image acquisition module for acquiring continuous image frames captured by the onboard camera of an autonomous driving vehicle in a mine; a deblurring module for applying an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation to the continuous image frames, calculating the image plane displacement caused by low-frequency, high-amplitude vibrations generated by vehicles traveling on unpaved roads in the mine, and performing deconvolution operations on directional motion trails and edge texture degradation in the continuous image frames based on the calculation results; and an anchor frame parameter reconfiguration module for reconfiguring the parameters obtained by statistical simulation of general road scenes in a conventional YOLO real-time inference architecture. The pre-defined anchor frame scale distribution and aspect ratio prior parameters are replaced with parameters obtained by clustering irregular small-scale hazardous obstacles labeled with dimensions of rolled-down gravel and eroded rock blocks on the slope within the mining face; the YOLO real-time inference module receives the continuous image frames after deconvolution operation output by the deblurring module and the anchor frame scale distribution and aspect ratio prior parameters provided by the anchor frame parameter reconfiguration module, performs feature extraction and bounding box regression on the input continuous image frames, and outputs the bounding box coordinates of rolled-down gravel and eroded rock blocks on the slope within the mining face; the trajectory association module performs target trajectory identification association on the bounding box coordinates output by the YOLO real-time inference module over continuous time.

[0040] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0041] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A visual perception method for automated mining operations based on YOLO real-time inference, characterized in that, Includes the following steps: For continuous image frames captured by the onboard camera of the autonomous driving vehicle in the mine, an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation is used to calculate the image plane displacement caused by low-frequency high-amplitude vibration of the vehicle traveling on the unpaved road in the mine. Based on the calculation results, deconvolution operation is performed on the directional motion trails and edge texture degradation in the continuous image frames. The continuous image frames after deconvolution are input into the YOLO real-time inference network that has completed anchor frame prior parameter reconfiguration. The anchor frame prior parameter reconfiguration process replaces the preset anchor frame scale distribution and aspect ratio prior parameters obtained by statistical fitting of general road scenes in the conventional YOLO real-time inference architecture with parameters obtained by clustering the labeled size of irregular small-scale dangerous obstacles such as rolling gravel and detached rock blocks on the slope in the mining face. The YOLO real-time inference network performs feature extraction and bounding box regression on the input continuous image frames, and outputs the bounding box coordinates of the rolling gravel and the detached rock blocks on the slope within the mining face. Perform target trajectory identification association on the bounding box coordinates output in continuous time series.

2. The visual perception method for automated mining operations based on YOLO real-time inference as described in claim 1, characterized in that, The specific steps for performing deconvolution on directional motion blur and edge texture degradation in consecutive image frames are as follows: Feature point extraction and pyramid optical flow tracking are performed on two adjacent frames of a continuous image to obtain pixel-level motion vector fields. The image plane displacement component induced by vibration in the vector field is separated from the scene change displacement component, and the vibration displacement component is extracted as the initial estimate of the motion blur kernel. An adaptive deconvolution iteration is performed with an initial estimate of the blur kernel, simultaneously updating the blur kernel parameters and image estimation. After the iteration converges, the processed continuous image frames are output.

3. The visual perception method for automated mining operations based on YOLO real-time reasoning according to claim 2, characterized in that, The specific steps for generating the initial estimate of the motion fuzzy kernel are as follows: Perform frequency domain transformation on the pixel-level motion vector field to obtain the motion vector spectrum distribution; Bandpass filtering is performed on the spectrum based on the vibration prior frequency band to filter out high-frequency noise and DC components; Perform an inverse transform on the filtered spectrum to obtain the image plane displacement components induced by vibration; The displacement components are mapped to the point spread function space to generate an initial estimate of the motion fuzzy kernel.

4. The visual perception method for automated mining operations based on YOLO real-time inference as described in claim 1, characterized in that, The specific steps for replacing clustering parameters are as follows: Obtain the width and height dimensions of the annotation bounding box for fallen rocks and eroded rock blocks on the slope; Perform clustering operations on the width and height data, and use the cluster centers as the reconfigured anchor frame size and aspect ratio parameters; Replace each of the preset anchor frame parameters for the general road scene with the scale and aspect ratio values ​​corresponding to the cluster centers.

5. The visual perception method for automated mining operations based on YOLO real-time inference as described in claim 4, characterized in that, The steps for selecting initial centers in clustering operations are as follows: The first cluster center is randomly selected from the width and height dataset; Calculate the shortest distance between each point in the dataset and the existing center; The next cluster center is selected based on the probability of the squared distance ratio. Repeat the selection process until a preset number of initial cluster centers are obtained.

6. The visual perception method for automated mining operations based on YOLO real-time inference according to claim 1, characterized in that, The specific steps for associating trajectory identifiers are as follows: Get the bounding box coordinates of the mine obstacles in the current frame and the previous frame; Calculate the intersection-union ratio (IUR) of the bounding box coordinates of adjacent frames and generate the IUR cost matrix.

7. The visual perception method for automated mining operations based on YOLO real-time inference as described in claim 6, characterized in that, Execute after generating the cost matrix: Apply prior positional weights based on the vehicle's travel direction and the direction of rockfall to the obstacle coordinates in the matrix; The weighted matrix is ​​then input into the Hungarian algorithm matcher to perform optimal matching between the bounding box and the trajectory label. The original trajectory identifier is assigned to the successfully matched boxes, and a new trajectory identifier is assigned to the unmatched boxes.

8. The visual perception method for automated mining operations based on YOLO real-time inference as described in claim 1, characterized in that, The specific steps of feature extraction are as follows: The processed image is input into the YOLO backbone network, and multi-scale feature maps are extracted layer by layer. The multi-scale feature map is input into the feature pyramid network, and bidirectional feature fusion is performed to obtain the multi-scale fused feature map. Channel attention-weighted suppression is applied to the channel responses of degraded residual frequency bands in the fused feature map; The processed fused feature map is then input into the YOLO detection head.

9. The visual perception method for automated mining operations based on YOLO real-time reasoning according to claim 1, characterized in that, The specific steps of bounding box regression are as follows: Obtain the coordinates of the predicted bounding box output by the YOLO detector head; Calculate the loss function values ​​for the predicted bounding box and the ground truth bounding box. The loss function includes loss terms for position and aspect ratio. Based on the variance of obstacle labeled dimensions, the penalty weight of the aspect ratio loss term is adaptively adjusted; Backpropagation and parameter updates are performed based on the loss function until the loss function converges.

10. A mine autonomous driving visual perception system based on YOLO real-time reasoning, employing the mine autonomous driving visual perception method based on YOLO real-time reasoning as described in any one of claims 1-9, characterized in that, include: The image acquisition module is used to acquire continuous image frames captured by the onboard camera of the autonomous driving vehicle in the mine; The deblurring module is used to apply an adaptive deconvolution deblurring algorithm based on inter-frame optical flow estimation to continuous image frames. It calculates the image plane displacement caused by low-frequency, high-amplitude vibrations of vehicles traveling on unpaved roads in mines, and performs deconvolution operations on directional motion trails and edge texture degradation in continuous image frames based on the calculation results. The anchor frame parameter reconfiguration module is used to replace the preset anchor frame scale distribution and aspect ratio prior parameters obtained by statistical fitting of general road scenarios in the conventional YOLO real-time inference architecture with parameters obtained by clustering the labeled size of irregular small-scale dangerous obstacles such as rolling gravel and detached rock blocks on the slope in the mining operation face. The YOLO real-time inference module receives the continuous image frames after deconvolution operation output by the deblurring module and the prior parameters of anchor frame scale distribution and aspect ratio provided by the anchor frame parameter reconfiguration module. It performs feature extraction and bounding box regression on the input continuous image frames and outputs the bounding box coordinates of the rolling gravel and the peeling rock on the slope in the mining face. The trajectory association module is used to perform target trajectory identification association on the bounding box coordinates output by the YOLO real-time inference module over continuous time.

Images