An unmanned driving method for underground mines based on light field camera and deep learning

Abstract

The present application discloses an unmanned driving method for underground mining based on light field camera and deep learning, which involves unmanned driving, including: collecting light field data and performing spatiotemporal synchronization operation; performing denoising, color correction and geometric correction on the light field data after spatiotemporal synchronization operation in sequence; generating corresponding disparity map and depth map according to the corrected light field data; converting the depth map into a 3D point cloud model; inputting the 3D point cloud model into a pre-trained HyperVoxelNet obstacle detection model, and outputting the category and position information of the detected obstacle; performing path planning through the A star algorithm according to the category and position information of the detected obstacle, and obtaining the driving route of the underground mining unmanned vehicle; generating corresponding control instructions according to the driving route, and controlling the underground mining unmanned vehicle to drive according to the driving route. In view of the weak dynamic perception ability of unmanned driving in underground mining environment in the prior art, the present application improves the ability of underground unmanned driving to perceive the dynamic environment of underground mining.

Description

Technical Field

[0001] The present application relates to unmanned driving, and in particular to an unmanned driving method for underground mines based on light field cameras and deep learning. Background Art

[0002] In the unmanned driving system of underground mining, perceiving and understanding the underground environment is the basis for autonomous navigation and obstacle avoidance. At present, unmanned vehicles in underground mining mainly use sensors such as visible light cameras, ultrasonic sensors and lidar to perceive the environment.

[0003] Visible light cameras install multiple cameras on unmanned vehicles to capture real-time images of the environment around the vehicle, and detect and identify obstacles in combination with image processing algorithms. However, due to the complex and changeable lighting conditions in underground mines, traditional visible light cameras have difficulty obtaining high-quality images when the light is insufficient or changes dramatically, resulting in unsatisfactory obstacle detection results.

[0004] Ultrasonic sensors estimate the distance between the vehicle and surrounding obstacles by measuring the flight time of sound waves. By arranging multiple ultrasonic sensors around the vehicle body, accurate detection of close-range obstacles can be achieved. However, ultrasonic waves are easily interfered by environmental noise, and their detection accuracy will be significantly reduced in underground environments with heavy dust and smoke.

[0005] LiDAR generates a high-resolution three-dimensional point cloud by scanning the environment, and then uses point cloud processing algorithms to identify and classify obstacles. LiDAR can provide relatively accurate distance information, but it is also easily affected by environmental factors such as dust. In addition, LiDAR is expensive and has certain limitations in practical applications.

[0006] In addition to being greatly affected by external interference, existing perception methods also generally have the problem of insufficient detection capabilities for dynamic obstacles. There are often a large number of moving workers and vehicles in underground mining environments. Traditional methods are difficult to detect these dynamic objects in real time and sensitively, which can easily lead to collision accidents. At the same time, due to the low data processing efficiency, existing methods are difficult to meet the real-time requirements of unmanned driving systems when faced with massive sensor data. Summary of the invention

[0007] In view of the problem of weak dynamic perception ability of unmanned driving in underground mining environment in the prior art, this application provides an unmanned driving method for underground mining based on light field camera and deep learning, which collects light field data through light field camera array, performs spatiotemporal synchronization, denoising and correction, and generates disparity map and depth map based on the corrected data. On this basis, through HyperVoxelNet obstacle detection, according to the detected obstacle information, the A star algorithm is used for path planning, which improves the ability of underground unmanned driving to perceive the dynamic environment of underground mining.

[0008] The purpose of this application is achieved through the following technical solutions.

[0009] The present application provides an unmanned driving method for underground mining based on light field cameras and deep learning, including: collecting light field data through a light field camera array arranged in an underground mining environment; performing spatiotemporal synchronization operations on the collected light field data; performing denoising, color correction and geometric correction on the light field data after the spatiotemporal synchronization operations in sequence to obtain corrected light field data; generating corresponding disparity maps and depth maps according to the corrected light field data; converting the depth map into a 3D point cloud model for providing three-dimensional reconstruction information of the environment; inputting the 3D point cloud model into a pre-trained HyperVoxelNet obstacle detection model to perform obstacle detection and recognition, and outputting the category and location information of the detected obstacles; performing path planning through the A star algorithm according to the category and location information of the detected obstacles to obtain the driving route of the underground mining unmanned vehicle; generating corresponding control instructions according to the driving route to control the underground mining unmanned vehicle to drive according to the driving route. Among them, the A star algorithm is an algorithm based on graph search, which is used to find the optimal path from the starting point to the target point in the state space. It combines the advantages of best-first search and Dijkstra algorithm, and guides the search in the most promising direction by estimating the cost from each node to the target. In this application, the underground mining environment is represented as a grid map through a 3D point cloud model, and each grid represents a state in the environment. According to the obstacle position information detected by HyperVoxelNet, the grids occupied by the obstacles are marked in the grid map. According to the Euclidean distance or Manhattan distance between the current position of the unmanned vehicle and the target position, the heuristic function h(n) is designed to estimate the cost from each grid to the target position. Taking the current position of the unmanned vehicle as the starting point and the target position as the end point, the A star algorithm is applied to search for the optimal path from the starting point to the target position on the grid map. The searched path is post-processed, such as smoothing, local optimization, etc., to generate a driving route that is more suitable for the unmanned vehicle to execute. According to the planned driving route, the corresponding control instructions are generated to control the unmanned vehicle to drive along the route, while considering the vehicle dynamics constraints and safety requirements.

[0010] Wherein, light field data is collected by a light field camera array arranged in an underground mining environment, including: the light field camera array includes multiple light field camera units, and the light field camera units are connected to a central control unit via a wireless communication network; wherein the step of collecting light field data includes: sending an acquisition instruction to each light field camera unit via the central control unit, the acquisition instruction includes parameters such as acquisition frame rate, exposure time, white balance mode, etc.; each light field camera unit sets corresponding acquisition parameters according to the acquisition instruction, and uses the set acquisition parameters to image the underground mining environment to obtain an original light field image sequence; each light field camera unit sends the collected original light field image sequence to the central control unit in real time via the wireless communication network; the central control unit receives the original light field image sequence sent by each light field camera unit, and summarizes them to obtain light field data of the underground mining environment.

[0011] Furthermore, a spatiotemporal synchronization operation is performed on the collected light field data, including: obtaining timestamp information of the collected original light field image sequence; selecting an original light field image sequence as a reference sequence according to a preset time synchronization strategy; calculating the time offset between the original light field image sequence and the reference sequence; performing time offset correction on the original light field image sequence according to the calculated time offset to obtain a time-aligned light field image sequence; establishing a spatial mapping relationship between each light field camera according to internal and external parameters of the light field camera; and using the spatial mapping relationship, performing geometric transformation and projection operations on the time-aligned light field image sequence to unify the light field image sequences collected by each light field camera into the same coordinate system to obtain spatiotemporal synchronized light field data.

[0012] Furthermore, the light field data after the spatiotemporal synchronization operation is subjected to denoising, including: dividing the light field data into blocks to obtain a plurality of light field data sub-blocks; for each light field data sub-block, denoising is performed using a sparse representation model based on K-SVD dictionary learning to obtain a denoising result of the corresponding light field data sub-block; wherein the objective function of K-SVD dictionary learning is: , where D is the dictionary matrix and X is the sparse coefficient matrix. is the extraction operator for extracting the i-th sub-block from the original light field data y, F is the synthesis operator, For sub-block The sparse representation coefficient in the dictionary D, N is the total number of sub-blocks, is a sparsity control parameter; the denoising results of each light field data sub-block are spliced ​​to obtain the denoised light field data; represents the L0 norm; represents the L2 norm.

[0013] The block denoising method based on K-SVD dictionary learning can effectively remove noise from light field data while retaining the detailed information of the light field data. Dictionary learning can adaptively learn the optimal sparse representation basis from the data. Compared with the traditional fixed basis representation, it has stronger denoising ability and better data adaptability. The block processing strategy can reduce the computational complexity and improve the denoising efficiency. At the same time, it can reduce the visual artifacts caused by the block effect through sub-block overlap and splicing smoothing.

[0014] Further, color correction is performed on the light field data after the spatiotemporal synchronization operation, including: calculating the color vector value of each pixel in the light field data according to the denoised light field data , where the color vector value Contains three components: R, G, and B; constructs a feature matrix M based on the color vector value of each pixel; constructs a color transformation function ,in is a preset color transformation function used to transform the feature matrix M into a color space; the color transformation function In the form of a polynomial function, The coefficients of are solved by the least squares regression method; using the obtained Transform the feature matrix M to obtain the feature matrix after color correction ,Will The pixel color vector value in is filled into the position of the corresponding pixel to obtain the color-corrected light field data.

[0015] Color correction can effectively eliminate color deviation and inconsistency in light field data, and improve the color authenticity and visual quality of light field data. The construction of the feature matrix M can flexibly select features according to the needs of color correction, such as color values, spatial coordinates, texture features, etc., to make full use of the rich information of light field data. Color transformation function In the form of a polynomial function, the coefficients are estimated by least squares regression, which can adaptively fit the color transformation law of light field data, with good robustness and generalization ability. The color-corrected light field data can provide high-quality color information for subsequent applications, such as 3D reconstruction and virtual viewpoint rendering.

[0016] Furthermore, geometric correction is performed on the light field data after the spatiotemporal synchronization operation, including: calibrating the internal and external parameters of the light field camera through a calibration plate, wherein the internal parameters include the principal point coordinates of the light field camera , focal length f and pixel size ; The external parameters include the rotation matrix R and translation vector t of the light field camera relative to the world coordinate system; the geometric model of light field camera imaging is established by using the internal and external parameters obtained by calibration, which is used to describe the mapping relationship between points in the three-dimensional world and corresponding points in the light field camera image; the feature points including the tunnel edge, borehole, anchor and support device are extracted from the color-corrected light field data, and the Harris-Stephens corner detection algorithm is used to calculate the pixel coordinates of the feature points by performing quadratic surface fitting interpolation on the pixel grayscale values ​​within the preset range of the feature points;

[0017] The calculated pixel coordinates of the feature points are input into the established geometric model of light field camera imaging. The pixel coordinates of the feature points are calibrated with the calibrated intrinsic and extrinsic parameters of the light field camera, and the perspective projection mapping relationship between the calibrated feature points in the image coordinate system and the world coordinate system is established to eliminate the aberration distortion in the imaging process of the light field camera. According to the calibrated feature point coordinates, the Fourier interpolation algorithm is used to interpolate and reconstruct the light field data in the frequency domain to generate geometrically corrected light field data.

[0018] Through the geometric correction method, the aberration distortion in the imaging process of the light field camera can be effectively eliminated, and the geometric accuracy and consistency of the light field data can be improved. The internal and external parameters obtained by the light field camera calibration provide the necessary camera model and transformation relationship for geometric correction. Feature point extraction and coordinate calibration further refine the position information of key features, and by mapping them with real-world coordinates, the light field data is accurately aligned with the physical space. Fourier interpolation reconstruction uses the calibrated feature point coordinates to perform smooth interpolation in the frequency domain to generate high-quality light field data after geometric correction.

[0019] Further, generating corresponding disparity maps and depth maps according to the corrected light field data includes: rearranging the geometrically corrected light field data in angle dimension and spatial dimension to obtain a set of sub-aperture images, selecting and The sub-aperture image at the position is taken as the central view sub-image; where, and They are the central view angle coordinates of the light field camera in the U and V directions respectively;

[0020] Taking the central view sub-image as the reference image, the Lucas-Kanade optical flow method is used to calculate the pixel displacement vectors between the central view sub-image and other sub-aperture images to obtain the initial disparity map. ; The pixel displacement vector represents the offset of the corresponding point of the pixel point on the central view sub-image in other sub-aperture images; construct the energy function of optimizing the disparity map , energy function Contains data items and smoothing term ; Among them, the data items Measuring the disparity map With the initial disparity map The degree of difference, smoothing term Measuring the disparity map The local smoothness of Solving the optimal disparity map , get the optimized disparity map ; Use the checkerboard calibration plate to jointly calibrate the imaging parameters and disparity values ​​of the light field camera and establish a disparity-depth mapping model ; Where Z is the depth value, D is the disparity value, is the mapping function from disparity to depth, including the scale factor and offset between disparity value and depth value; the optimized disparity map Substitute into the parallax-depth mapping model to get the corresponding depth map .

[0021] By generating disparity maps and depth maps, the three-dimensional structural information of the scene can be obtained from the geometrically corrected light field data. Light field data rearrangement and central view selection provide multi-perspective sub-aperture images, laying the foundation for disparity estimation. The Lucas-Kanade optical flow method obtains the initial disparity map by calculating the pixel displacement vector, which describes the disparity distribution of the scene. Energy function optimization further improves the smoothness and continuity of the disparity map, and obtains more accurate and stable disparity estimation results. Through the disparity-depth mapping model, the disparity value is converted into the real depth value, and the corresponding depth map is generated, which provides effective depth information for subsequent three-dimensional reconstruction, target detection and other tasks.

[0022] Furthermore, the 3D point cloud model is input into the pre-trained HyperVoxelNet obstacle detection model to perform obstacle detection and recognition, and output the category and location information of the detected obstacles, including: calculating the corresponding 3D point cloud model according to the depth map Z (u, v) , The coordinates of each point in With Depth Map The corresponding pixel Coordinates and depth values The following relationship is satisfied: , , ,in, is the principal point coordinate of the light field camera, and are the focal lengths of the light field camera in the x and y directions respectively; the 3D point cloud model Divide into voxel grids of equal size, each voxel has a size of d×d×d; generate a sparse feature representation of the voxel grid by calculating the point cloud density and normal vector information within each voxel, and obtain the normalized voxel feature ; Voxel features Input the pre-trained HyperVoxelNet obstacle detection model to perform obstacle detection; the HyperVoxelNet obstacle detection model outputs the category information C(n) and bounding box information B(n) of the detected obstacle, where n represents the number of the detected obstacle, C(n) represents the category label of the nth obstacle, and B(n) represents the bounding box coordinates of the nth obstacle; according to the bounding box information B(n) of the detected obstacle, determine the voxel area occupied by each obstacle instance.

[0023] The depth map is converted into 3D point cloud data, and geometric features at the voxel level are extracted. The conversion from depth map to point cloud is based on the pinhole camera model. Points in the 2D image space are projected into 3D space through the mapping relationship between pixel coordinates and depth values. Voxel partitioning divides the point cloud data into a regular cubic grid, and each voxel represents a local spatial area. By calculating the point cloud density and normal vector features inside the voxel, the occupancy and geometric structure information of the voxel can be characterized.

[0024] Furthermore, the HyperVoxelNet obstacle detection model includes: an input module that receives voxel feature descriptors; a hierarchical encoding module that divides the input voxel feature descriptors into voxel representations of different scales; hierarchical encoding of voxel features at different scales, and adjusting the encoding weights of each scale through the attention mechanism; a four-dimensional voxel spatiotemporal convolution module 4D-VSTC that uses a 4D convolution kernel to convolve in space. Convolution operation is performed on the layered encoded voxel features in the time T dimension to output the feature map ; Among them, the convolution kernel in the 4D-VSTC module is a five-dimensional tensor ; Indicates the size of the channel dimension, which is used to represent the feature information at each position. Indicates the size of the time dimension, used to capture dynamic changes in time; The convolution kernel size representing the spatial dimension is used to capture the details of the target in space; the context interaction module GLM uses the self-attention mechanism and graph convolution network to perform feature map Perform global context modeling to generate feature maps that fuse global and local context information .

[0025] The spatiotemporal feature association graph module DSTFG uses the graph structure to map the feature graph. Modeling is performed on the time dimension to obtain the updated feature map ; According to the updated feature map Calculate the weights of feature maps of different scales in different scenes, and obtain the final feature map through weighted aggregation according to the weights. ; Feature compression module, which uses quantum state simulation method to compress the feature map output by DSTFG module Compress in the channel dimension, and restore the compressed feature map through feature decoding to obtain the feature map that is the same as the input feature map in the spatial dimension. Compressed feature maps of the same size; the detection head module uses the region candidate network RPN to detect obstacle candidate regions on the feature map output by the feature compression module, and then classifies and regresses the candidate regions through the regional convolutional neural network R-CNN to obtain the final obstacle detection result, including the obstacle category and 3D bounding box position coordinates.

[0026] Furthermore, a feature map is generated that integrates global and local context information, including: using an attention mechanism to calculate the feature map The correlation between different positions in the feature map is used to generate the global feature G. As a graph structure, each voxel represents a node, and the connection between voxels represents an edge. The graph convolutional network GCN performs convolution operations on the graph structure to obtain the local feature L; the global feature G and the local feature L are weightedly fused to obtain a feature map that integrates global and local context information.

[0027] Among them, the spatiotemporal feature association graph module DSTFG uses the graph structure to map the feature graph Modeling is performed on the time dimension to obtain the updated feature map :The feature map As input, build a dynamic graph structure ; Among them, nodes represent feature points, and edges represent spatiotemporal associations; feature propagation is performed on the dynamic graph structure G through graph convolution to obtain the updated feature graph .

[0028] Among them, according to the updated feature map Calculate the weights of feature maps of different scales in different scenes, and obtain the final feature map through weighted aggregation according to the weights. , including: using convolution kernels of multiple scales to update the feature map Perform convolution operation to obtain multi-scale feature maps , where i represents different scales; according to the updated feature map The time series information is obtained by using the attention mechanism to calculate the importance weight of each time step. , according to the weight Feature maps of each scale in the time dimension Perform weighted summation to obtain the aggregate feature map ; Aggregate feature map As input, through the pre-trained weight network, the aggregate feature map of each scale is , assign a weight , Reflects the importance of features of different scales in the current scene; according to the weight Aggregate feature map Aggregate to get the final feature map .

[0029] Compared with the prior art, the advantages of this application are:

[0030] The light field data of the underground mining environment is collected through the light field camera array, and pre-processed by time and space synchronization, denoising, color correction and geometric correction. By recording the direction and intensity of light, the light field camera can more comprehensively perceive the three-dimensional environmental characteristics of the underground mining environment, making up for the shortcomings of the traditional monocular / binocular camera imaging method.

[0031] The disparity map and depth map are generated based on the corrected light field data. By combining the disparity information with the depth information, the three-dimensional structure of the underground mining environment can be accurately restored. The disparity map reflects the relative distance relationship of objects in the scene, and the depth map provides the absolute depth information of the objects. The depth map is converted into a point cloud model, which provides a unified three-dimensional representation for obstacle detection.

[0032] The HyperVoxelNet obstacle detection model can fully mine and fuse multi-dimensional feature information in light field data. It constructs voxel feature descriptors by extracting disparity features and geometric features, uses 4D convolution to extract features in the spatiotemporal dimension, uses attention mechanisms and graph convolution to achieve global context modeling of features, and compresses redundant information through quantum state simulation and feature decoding. Multi-dimensional and multi-scale feature fusion enables the obstacle detection model to fully understand the semantic information of underground mining scenes, greatly improving the detection accuracy and robustness in complex environments.

[0033] The spatiotemporal feature association graph module DSTFG uses graph structure to model the association and evolution of temporal features, introduces time dimension information, and enhances the model's ability to perceive dynamic environments. Graph convolution is used to achieve information propagation and updating between different time steps, so that the model can capture the dynamic changes of obstacles in underground mining environments and improve the accuracy of dynamic obstacle detection. The multi-scale feature weight adaptive mechanism further enhances the model's ability to detect obstacles of different scales.

[0034] Quantum state simulation technology was introduced to compress features, removing redundancy while retaining key information, reducing model calculations and storage overhead, and significantly improving the real-time performance of obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0036] Figure 1 is an exemplary flow chart of an unmanned driving method for underground mines based on a light field camera and deep learning according to some embodiments of the present application;

[0037] Figure 2 is an exemplary flow chart of the data input and fusion stage according to some embodiments of the present application;

[0038] Figure 3 is an exemplary flow chart of the feature encoding stage according to some embodiments of the present application;

[0039] Figure 4 is an exemplary flow chart of the advanced feature extraction stage according to some embodiments of the present application;

[0040] Figure 5 It is an exemplary flow chart of the detection and post-processing stages shown in some embodiments of the present application. DETAILED DESCRIPTION

[0041] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0042] Figure 1 This is an exemplary flow chart of an underground mining unmanned driving method based on light field cameras and deep learning according to some embodiments of the present application, including: collecting light field data through a light field camera array arranged in an underground mining environment; performing spatiotemporal synchronization operations on the collected light field data; performing denoising, color correction and geometric correction on the light field data after the spatiotemporal synchronization operations in sequence to obtain corrected light field data; generating corresponding disparity maps and depth maps based on the corrected light field data; converting the depth map into a 3D point cloud model for providing three-dimensional reconstruction information of the environment; inputting the 3D point cloud model into a pre-trained HyperVoxelNet obstacle detection model to perform obstacle detection and recognition, and outputting the category and position information of the detected obstacles; performing path planning through the A star algorithm based on the category and position information of the detected obstacles to obtain a driving route for the underground mining unmanned vehicle; generating corresponding control instructions based on the driving route to control the underground mining unmanned vehicle to drive according to the driving route.

[0043] Figure 2 It is an exemplary flow chart of the data input and fusion stage shown in some embodiments of the present application, in which light field data is collected by a light field camera array arranged in an underground mining environment; arrangement and installation of the light field camera array: according to the characteristics of the underground mining environment and the structure of the unmanned mine car, light field cameras are installed in key positions such as the front, left and right sides and rear of the mine car to form an all-round perception field of view. Each light field camera is installed with an independent anti-vibration fixed bracket. The bracket design needs to consider factors such as impact resistance and corrosion resistance to ensure the stability of the camera in harsh environments. The camera bracket is equipped with a multi-dimensional adjustment mechanism, which can flexibly adjust the pitch and roll angles of the camera according to the on-site conditions to obtain the best imaging viewing angle. There must be a certain overlap in the fields of view of adjacent cameras in the camera array, and the robustness of perception is improved by the fusion of multi-view data.

[0044] Power supply and data transmission of light field camera array: Each light field camera is equipped with an independent power supply module, which adopts a low-power, highly reliable power supply solution, such as lithium battery power supply, to ensure the camera's continuous working ability during long-term operation. The camera power supply module is designed with waterproof, dustproof and other protective measures to improve the reliability of the power supply system. The light field camera is connected to the data processing unit through a high-speed data interface (such as USB, Gigabit Ethernet, etc.) to ensure real-time transmission of light field data. The data interface of each camera is connected to the data processing unit through a bus topology. The bus adopts industrial standard protocols (such as CAN, EtherCAT, etc.) and has the characteristics of high bandwidth, low latency and high synchronization. Electromagnetic shielding measures need to be taken when laying the data bus to reduce the interference of strong electrical equipment on the transmission signal.

[0045] The collected light field data is subjected to a spatiotemporal synchronization operation, including: extracting timestamp information of the original light field image sequence collected by each light field camera unit, and performing time alignment on the original light field image sequence according to the timestamp information to obtain the time-aligned light field image sequence; wherein the time alignment step includes: selecting an original light field image sequence collected by a light field camera unit as a reference sequence according to a preset time synchronization strategy, and recording the collection time T0 of the reference sequence; wherein the preset time synchronization strategy includes: selecting the original light field image sequence with the earliest collection time, selecting the original light field image sequence with the most central spatial position of the light field camera unit; calculating the collection time of the original light field image sequence collected by other light field camera units Acquisition time with reference sequence The time offset between , where i is the number of the light field camera unit; the time offset The calculation formula is: ; For the i-th light field camera unit, according to its time offset , in its original light field image sequence, the acquisition time is selected as The light field image of the i-th light field camera unit at the reference time Corresponding light field image; each light field camera unit reference time The corresponding light field images are sequentially combined to obtain a time-aligned light field image sequence, which is recorded as ,in, represents the light field image of the i-th light field camera unit, and n is the total number of light field camera units.

[0046] According to the spatial position relationship of each light field camera unit, the time-aligned light field image sequence is Perform spatial registration to obtain time-space synchronized light field data; wherein the spatial registration step includes: calibrating the intrinsic parameter matrix of each light field camera unit and the extrinsic parameter matrix ,in, is the intrinsic parameter matrix of the i-th light field camera unit, and are the rotation matrix and translation vector of the i-th light field camera unit relative to the global world coordinate system, according to , and Establish the homography matrix between the i-th light field camera unit and the world coordinate system , the homography matrix The calculation formula is: ; Light field image sequence after time alignment In the example, a frame of light field image is selected as the reference image, denoted as ;use and The epipolar constraints between arrive The basic matrix ,Will Decompose into rotation matrix and translation vectors , and combined with the homography matrix of the i-th light field camera unit , through projection transformation, The light field image in is projected onto On the imaging plane, we get Perspective-consistent light field images , where the calculation formula for the projection transformation is: ;right Each frame of the light field image in the above projection transformation is performed to obtain a light field image sequence with the same viewing angle. , a sequence of light field images with consistent viewing angles This is the light field data after time and space synchronization. Time synchronization is performed using the Network Time Protocol (NTP) and hardware trigger signals.

[0047] Spatial synchronization obtains internal and external parameters through camera calibration, and converts the data into the vehicle coordinate system. Camera intrinsic parameter calibration: Use a standard checkerboard calibration plate to take multiple calibration plate images at different positions and angles to ensure that the projection of the calibration plate in the image has a sufficient range of variation. Extract the pixel coordinates of the checkerboard corner points in the calibration plate image, and achieve high-precision corner point positioning through sub-pixel corner point detection algorithms (such as Harris corner point detection). According to the physical size of the checkerboard and the pixel coordinates of the corner points, construct the geometric constraint equations for camera imaging, and use the Zhang calibration method to solve the camera's intrinsic parameter matrix and distortion coefficients. Fine-tune the intrinsic parameter matrix and distortion coefficients through optimization algorithms (such as the least squares method) to minimize the reprojection error and improve the accuracy of intrinsic parameter estimation.

[0048] Camera extrinsic calibration: Set up a global coordinate system (such as the vehicle body coordinate system) on the unmanned mining vehicle as the reference system for extrinsic calibration. Use high-precision measurement equipment (such as total stations, laser trackers, etc.) to measure the position and posture of each light field camera relative to the vehicle body coordinate system to obtain the initial extrinsic parameters of the camera. Optimize the camera extrinsic parameters through the hand-eye calibration method, and establish a rigid body transformation relationship between the camera coordinate system and the vehicle body coordinate system. Use an extrinsic optimization algorithm based on corresponding points (such as the PnP algorithm) and use feature points in the scene (such as landmarks, natural features, etc.) to further optimize the camera extrinsic parameters and reduce measurement errors and calibration errors.

[0049] Spatial transformation of light field data: Perform distortion correction on the light field data collected by each light field camera, and use the distortion coefficient obtained by intrinsic calibration to dedistort the image to eliminate the influence of optical distortion. According to the intrinsic parameter matrix of the light field camera and the dedistorted light field data, perform ray tracing on each pixel to calculate the light direction and starting position corresponding to the pixel. Using the extrinsic parameter matrix of the light field camera, convert the light direction and starting position in the light field data from the camera coordinate system to the vehicle coordinate system to achieve a unified representation of the light field data in the global coordinate system. Through the spatial transformation of the light field data, the local light field data collected by different cameras are integrated into a unified global coordinate framework to facilitate subsequent data processing and analysis.

[0050] Stitching and optimization of multi-camera light field data: In the vehicle body coordinate system, the overlapping area of ​​light field data from different cameras is determined based on the external parameters and imaging field of view of the light field camera. In the overlapping area, the correspondence between the light field data is calculated through ray tracing and triangulation principles, and matching constraints for light field data from different perspectives are established. Using the matching constraints of the light field data, the external parameters and light field data of all cameras are jointly optimized through global optimization methods (such as Bundle Adjustment) to minimize the geometric error and photometric error between the light field data. The optimized light field data is seamlessly stitched in the vehicle body coordinate system to generate a complete and globally consistent light field representation, providing a unified data foundation for subsequent environmental perception tasks.

[0051] The light field data after the spatiotemporal synchronization operation is subjected to denoising, including: block processing of light field data: the light field data after spatiotemporal synchronization is divided into sub-blocks of equal size. The sub-block size is selected according to the resolution and noise characteristics of the light field data, and a sub-block size of 8×8 or 16×16 is usually selected. The overlapping block strategy is adopted, and there is a certain overlap area between adjacent sub-blocks. The pixels in the overlapping area can be used to reduce the block effect at the sub-block boundary. The light field data sub-block after block division is represented as , i=1,2,...,N, where Represents the i-th sub-block, and N is the total number of sub-blocks. Sub-block denoising based on K-SVD dictionary learning: For each light field data sub-block , using the K-SVD dictionary learning algorithm to construct an overcomplete dictionary D and sparse coefficients , dictionary D and sparse coefficients It is obtained by minimizing the following objective function: ,in, represents an overcomplete dictionary, n is the number of sub-block pixels, K is the number of dictionary atoms (K>n), and K is usually several times of n, where Represents a sparse coefficient matrix, where each column corresponds to the sparse coefficient of a sub-block . represents the extraction operator for extracting the i-th sub-block from the original light field data y, and M is the total number of pixels in the light field data. F: Represents dictionary D and sparse coefficients The synthesis operator is used to reconstruct the denoised sub-block. represents the i-th light field data sub-block, λ represents the sparsity control parameter, which is used to balance the data fidelity term and the sparse regularization term and control the sparsity of the sparse coefficients. represents the L2 norm, represents the L0 norm, that is, the number of non-zero elements.

[0052] The K-SVD algorithm is used to alternately optimize the dictionary D and the sparse coefficient X. The dictionary D is fixed and the orthogonal matching pursuit (OMP) algorithm is used to solve the sparse coefficient. , . Fix the sparse coefficient X and update the dictionary atoms using singular value decomposition (SVD). , where Y is the matrix composed of all sub-blocks, representing the Frobenius norm. Iteratively update the dictionary D and sparse coefficients X until the objective function converges or the maximum number of iterations is reached. Using the learned dictionary D and sparse coefficients , reconstruct the denoised sub-block , and get the denoising result of the i-th sub-block. , i=1, 2, ..., N are spliced ​​according to the original position to obtain the complete denoised light field data y ̂. For the overlapping areas of sub-blocks, weighted average is used for fusion to reduce splicing traces.

[0053] Perform color correction on the light field data after spatiotemporal synchronization, including: Calculate the color vector value of the pixel: For the denoised light field data, extract the RGB color value of each pixel and represent it as a three-dimensional color vector . The color vector values ​​of all pixels in the light field data are combined into a matrix C. Each row of the matrix corresponds to the color vector value of a pixel, and the number of rows of the matrix is ​​the total number of pixels N. Construct the feature matrix M: Construct the feature matrix M based on the color vector value matrix C. Each row of the feature matrix M corresponds to the feature vector of a pixel. The selection of feature vectors can be designed according to the requirements of color correction and prior knowledge. Common feature vectors include: Color vector value: ; High order of color vector values: ;The spatial coordinates of the pixel point: ; Texture features of pixel points: Arrange the selected eigenvectors in order to form a feature matrix M, where the number of rows is the total number of pixels N, and the number of columns is the dimension of the eigenvector. Construct a color transformation function : Color transformation function Used to map the feature matrix M to the corrected color space, It can take the form of a polynomial function, such as: ,in, is the coefficient of the polynomial function, and k is the order of the polynomial. Color transformation function The coefficients of can be estimated by the least squares regression method, the goal is to minimize the error between the transformed color value and the reference color value, that is: ,in, is the feature vector of the i-th pixel, is the reference color value of the i-th pixel. The above optimization problem is solved by least squares regression to obtain the color transformation function The optimal coefficient estimates of .

[0054] Color correction and data filling: Using the estimated color transformation function , transform the feature matrix M to obtain the feature matrix after color correction , , from the color-corrected feature matrix Extract the color vector value of each pixel , is a three-dimensional vector representing the corrected RGB color value. Fill the position of the pixel corresponding to the light field data to obtain the color-corrected light field data.

[0055] Perform geometric correction on the light field data after the spatiotemporal synchronization operation, including: Light field camera calibration: Use a standard calibration plate (such as a chessboard calibration plate) to calibrate the light field camera to obtain the internal and external parameters of the light field camera. The internal parameters include the coordinates of the principal point of the light field camera. , focal length f and pixel size , the intrinsic parameters can be estimated by Zhang Zhengyou calibration method or multi-plane calibration method. The external parameters include the rotation matrix R and translation vector t of the light field camera relative to the world coordinate system, which can be estimated by PnP (Perspective-n-Point) algorithm or Bundle Adjustment method. Establish the geometric model of light field camera imaging: Use the intrinsic and extrinsic parameters obtained by calibration to establish the geometric model of light field camera imaging. The model describes the mapping relationship between points (X, Y, Z) in the three-dimensional world and the corresponding points (u, v) in the light field camera image. The geometric model of light field camera imaging can be expressed as: , where K is the intrinsic parameter matrix of the light field camera, and R and t are the extrinsic parameter matrices of the light field camera.

[0056] Feature point extraction and pixel coordinate calculation: Extract feature points including key features such as tunnel edges, boreholes, anchors, and support devices from the color-corrected light field data. The Harris-Stephens corner detection algorithm is used to extract feature points. This algorithm determines whether a pixel is a corner point by calculating the eigenvalue of the second-order moment matrix of the grayscale value in the neighborhood of the pixel point. For the extracted feature points, perform quadratic surface fitting interpolation in the pixel neighborhood within the preset range, and obtain the sub-pixel coordinates of the feature points by solving the extreme points of the interpolation surface. .

[0057] Feature point coordinate calibration: The pixel coordinates of the extracted feature points Input into the established light field camera imaging geometric model. Combined with the calibrated light field camera internal parameter K and external parameter , calibrate the pixel coordinates of the feature points, and establish the perspective projection mapping relationship between the calibrated feature points in the image coordinate system (u, v) and the world coordinate system (X, Y, Z): ,Through coordinate calibration, the aberration distortion in the imaging process of the light field camera is eliminated, so that the coordinates of the calibrated feature points are consistent with the real-world coordinates.

[0058] Light field data interpolation and reconstruction: According to the calibrated feature point coordinates, the Fourier interpolation algorithm is used to interpolate and reconstruct the light field data. The light field data is transformed from the spatial domain to the frequency domain, interpolated in the frequency domain according to the calibrated feature point coordinates, and then the interpolation result is transformed back to the spatial domain through the inverse Fourier transform. The Fourier interpolation algorithm has the advantages of high efficiency, smoothness, and no ringing effect, and can generate high-quality light field data after geometric correction.

[0059] Generate the corresponding disparity map and depth map based on the corrected light field data, including: light field data rearrangement and center view selection: rearrange the geometrically corrected light field data according to the angle dimension and spatial dimension to obtain a set of sub-aperture images. The light field data can be represented as a 4D function , where (u, v) represents the angle dimension and (x, y) represents the spatial dimension. The rearranged subaperture image can be expressed as ,in is the angular coordinate of the (i, j)th sub-aperture. In the rearranged sub-aperture image array, select The sub-aperture image of the position is used as the central view sub-image ,in It is the central view angle coordinate of the light field camera in the U and V directions.

[0060] Initial disparity map estimation: center view sub-image The Lucas-Kanade optical flow method is used to estimate the pixel displacement vector between the central view sub-image and other sub-aperture images. For each pixel point (x, y) on the central view sub-image, the pixel displacement vector between the central view sub-image and other sub-aperture images is estimated. Search for the corresponding pixel in , and calculate the displacement vector . Combine the displacement vectors of all pixels into an initial disparity map , represents the offset of each pixel on the central view sub-image in other sub-aperture images.

[0061] Disparity map optimization: Constructing an energy function to optimize the disparity map , the energy function is composed of the data term and smoothing term composition: , where λ is the weight coefficient of the smoothing term. Data term Measuring the disparity map With the initial disparity map The degree of difference can be measured using L1 norm or L2 norm: , where p = 1 or 2, indicating the L1 norm or L2 norm. Smooth term Measuring the disparity map The local smoothness of , the total variation regularization term can be used: ,in, represents the gradient operator. By minimizing the energy function Solving the optimal disparity map , we can use optimization methods such as variational method and graph cut algorithm. Optimized disparity map While maintaining the initial parallax Figure 1 While maintaining consistency, it also has better smoothness and continuity.

[0062] Disparity-depth mapping and depth map generation: Use a checkerboard calibration plate to jointly calibrate the light field camera imaging parameters and disparity values ​​to establish a disparity-depth mapping model Place the checkerboard at different depths and capture the light field image and the corresponding disparity map at the same time. Obtain the disparity values ​​and true depth values ​​at different depths by manually marking or automatically detecting the checkerboard corners. Use the obtained disparity-depth correspondence data to estimate the mapping function from disparity to depth through methods such as least squares fitting or polynomial fitting. The parameters of the optimized disparity map include the scale factor and offset. Substitute the parallax-depth mapping model In the calculation, the depth value of each pixel is calculated to obtain the corresponding depth map : .

[0063] Calculate the 3D point cloud model based on the depth map and generate voxel features. Calculate the 3D point cloud model based on the coordinates (u, v) and corresponding depth values ​​of each pixel in the depth map Z (u, v) , calculate the 3D point cloud model The three-dimensional coordinates of the corresponding points in . 3D point cloud model The coordinates of each point in With Depth Map The corresponding pixel Coordinates and depth values The following relationship is satisfied: , , ,in, is the principal point coordinate of the light field camera, and are the focal lengths of the light field camera in the x and y directions, respectively. This relationship is based on the pinhole camera model, which converts pixel coordinates and depth values ​​into point coordinates in three-dimensional space. By traversing all the pixels in the depth map and applying the above coordinate transformation relationship, a complete 3D point cloud model is generated. .

[0064] Figure 3 is an exemplary flow chart of the feature encoding stage shown in some embodiments of the present application, voxel division and feature calculation: converting the 3D point cloud model Divide into voxel grids of equal size, each voxel is d×d×d in size. For each voxel, calculate the density and normal vector information of the point cloud data contained in it to generate the feature representation of the voxel. Point cloud density feature: Count the number of point clouds inside each voxel to obtain the density feature of the voxel The density feature reflects the occupancy and distribution density of the point cloud inside the voxel. Normal vector feature: For each point cloud inside the voxel, calculate its normal vector information to obtain the normal vector feature of the voxel. The normal vector feature reflects the direction and geometric structure information of the point cloud inside the voxel. The normal vector of the point cloud data inside the voxel can be estimated using methods such as principal component analysis (PCA). The estimated normal vectors are averaged or weighted averaged to obtain the normal vector feature of the voxel. . The point cloud density feature D_voxel and the normal vector feature The feature vector of the voxel : . For voxel feature vector Normalization is performed so that the feature values ​​of each dimension are in the range of [0, 1] to obtain a standardized voxel feature representation .

[0065] HyperVoxelNet obstacle detection model, this application proposes a new 3D target detection network structure HyperVoxelNet, which is designed for processing 3D data in dynamic and complex environments. The network integrates multiple advanced modules and technologies to improve the accuracy and efficiency of target detection. The following are the main innovations of the HyperVoxelNet network structure.

[0066] Input module, receiving voxel feature descriptors: HyperVoxelNet first proposed a fusion method for 3D information generated by multiple light field cameras. This module receives input data from multiple light field cameras and integrates light field data from different perspectives into a unified 4D voxel grid (including X, Y, Z spatial dimensions and T time dimension) through light field feature fusion technology. Receives input data from multiple light field cameras, each of which generates a 4D light field data , where (u, v) represents the angle dimension and (x, y) represents the spatial dimension. Through the light field feature fusion technology, the light field data from different perspectives are integrated into a unified four-dimensional voxel grid , where (x, y, z) represents the spatial dimension and t represents the temporal dimension. Light field feature fusion technology can use weighted averaging, maximum pooling and other methods to fuse light field data from different perspectives at corresponding spatial positions to obtain a unified voxel representation.

[0067] Hierarchical coding module, multi-scale voxelization and adaptive hierarchical coding: Multi-scale voxelization is introduced to divide the input data into voxel representations of different scales to capture the details and global features of the target. Perform multi-scale voxelization to divide it into voxel representations of different scales , where s represents the scale level and S is the total number of scales. At each scale level s, the voxels are encoded using adaptive hierarchical coding to obtain the encoded voxel representation Adaptive layered coding preserves important information at low resolution and captures details at high resolution by learning scale-specific encoders. The encoder can be implemented using methods such as convolutional neural networks.

[0068] Figure 4 This is an exemplary flow chart of the advanced feature extraction stage shown in some embodiments of the present application. 4D Voxel Spatio-Temporal Convolution (4D-VSTC): HyperVoxelNet introduces a 4D voxel spatio-temporal convolution module, which has never appeared in the previous network structure. This module uses a 4D convolution kernel to perform convolution operations in both spatial (X, Y, Z) and temporal (T) dimensions to capture subtle changes in the target in space and time. Compared with traditional 3D convolution, this 4D convolution method can more effectively handle complex targets in dynamic scenes. The convolution kernel in the 4D-VSTC module is a five-dimensional tensor, represented as ,in Indicates the size of the channel dimension and determines the amount of feature information at each position. Indicates the size of the time dimension, which is used to capture dynamic changes in time. They represent the convolution kernel size of the spatial dimension, which is used to capture the details of the target in space.

[0069] Global-Local Interaction Module (GLIM): In HyperVoxelNet, a new GLIM module is introduced to realize the interaction between global and local contexts between different layers of the network. This module combines the self-attention mechanism with the graph convolutional network (GCN) to enhance the expression of local features while retaining global context information. The GLIM module can effectively capture the spatial position and posture of the target and improve the detection accuracy of the target in complex scenes. Specific implementation steps: If the output dimension of the previous layer of the network is ,in is the number of channels, usually used to indicate the amount of information in the feature map. It is the time dimension, which is used to capture the changes of the target at different time points. Represents the width, height and depth of the spatial dimension, respectively, and is used to describe the position of the target in three-dimensional space. Global feature extraction: Use the self-attention mechanism to extract the input feature map Perform global context modeling and generate global features , where the attention mechanism captures global context information and the output dimension is Local feature enhancement: The local information of the input feature map is enhanced through the graph convolutional network (GCN) to obtain local features. , , the output dimension is Feature fusion: weighted fusion of global features and local features to obtain the fused output feature map , , the dimension of the output feature remains .

[0070] Dynamic Spatio-Temporal Feature Graph (DSTFG): HyperVoxelNet introduces a dynamic spatio-temporal feature graph to express temporal and spatial information through a graph structure. This module based on a graph neural network can flexibly model the spatial changes of targets in a time series and adapt to the rapid changes of targets through a dynamic update mechanism. Specific implementation steps: Graph structure construction: Construct a graph based on the input feature graph. Dynamic graph structure , where nodes represent feature points and edges represent spatiotemporal associations. Feature propagation and update: Feature propagation is performed on the dynamic graph through graph convolution to obtain the feature graph after spatiotemporal update. , the output feature dimension is Dynamic association update: Through the flow of information in the time dimension, the graph structure is dynamically updated to adapt to the changes in time and space. The dynamically updated graph structure can capture the changes of the target in time and space.

[0071] Dynamic feature aggregation and selection mechanism: In order to cope with the complexity of targets and backgrounds in 3D scenes, HyperVoxelNet has designed a dynamic feature aggregation and selection mechanism. This mechanism uses a trainable dynamic weight network and the time series information provided by DSTFG to automatically select and aggregate important information in multi-scale feature maps, thereby reducing computational overhead while enhancing the ability to capture targets. This dynamic feature selection mechanism enables the network to adaptively adjust feature aggregation strategies in different scenarios to improve detection performance. Specific implementation steps: Multi-scale feature extraction: When processing 3D data, features are usually extracted at multiple scales to capture different levels of details and global information of the target. The specific steps are as follows: Input: Assume that the dimension of the input feature map is Multi-scale convolution: Use convolution kernels of multiple scales to perform convolution operations on input features to obtain feature maps of multiple scales. For example: , where each Represents feature maps at different scales, with dimensions of Extraction of time dimension information: Use the time series information provided by the dynamic spatiotemporal feature graph (DSTFG) to process the multi-scale feature graph in the time dimension. Input: Multi-scale feature graph . Attention mechanism in the time dimension: Use the attention mechanism to calculate the importance weight of each time step , This operation is performed independently on the feature maps of each scale. Temporal information aggregation: In the temporal dimension, a weighted sum is performed on the feature maps of each scale to aggregate the temporal information: , the resulting aggregate feature map The dimension is Dynamic Weight Network: Through a trainable dynamic weight network, the most important features are adaptively selected and aggregated according to the changes in the scene. Input: Aggregated feature map Dynamic weight calculation: A weight is assigned to each scale feature map through a trainable dynamic weight network. , These weights reflect the importance of features at different scales in the current scenario. Multi-scale feature selection and aggregation: Based on dynamic weights Select and aggregate multi-scale feature maps: The output feature map The dimension is .

[0072] Figure 5 This is an exemplary flow chart of the detection and post-processing stage shown in some embodiments of the present application. Feature compression module, feature compression based on quantum state simulation: through feature compression technology based on quantum state simulation, further compress feature representation, reduce redundant information, reduce computational complexity, and retain important features. Assume that the output feature map dimension of the previous layer network is , where H and W are the height and width of the feature map respectively, is the number of input channels. Quantum state simulation: Input feature map is a two-dimensional matrix , the shape is . Randomly initialize a quantum circuit Q, the number of quantum bits in the circuit is ,in is the target output channel number. Each row of is mapped to the quantum state space through the encoding circuit E to obtain the encoded quantum state , i=1, 2, ..., HW. For each quantum state , through the quantum circuit Q, the output quantum state is obtained . For the output quantum state Measure and get the encoded feature vector . All The equal lengths are concatenated into a two-dimensional matrix S with a shape of (H*W, C_out). Reshape S into a three-dimensional tensor , the shape is , and get the compressed feature map.

[0073] Feature decoding and compression: The compressed feature map Through a 1x1 convolution layer, the number of channels is increased from Expand back , get the decoded feature map .calculate With the original input feature map The mean square error , which is used as the reconstruction loss. The gradient descent method is used to optimize the parameters of the quantum circuit Q and the encoding circuit E, minimize the reconstruction loss, and make the compressed feature map retain the information of the original feature map as much as possible. The trained quantum circuit Q and encoding circuit E are used in the inference stage. Decoded as , and get the spatial dimension with the input features Figure 1 The compressed feature map.

[0074] The detection head module uses the region candidate network RPN to detect the obstacle candidate area on the feature map output by the feature compression module, and then classifies and regresses the candidate area through the regional convolutional neural network R-CNN to obtain the final obstacle detection result, including the category of the obstacle and the position coordinates of the three-dimensional bounding box. The voxel feature F_voxel is input into the pre-trained HyperVoxelNet obstacle detection model for obstacle detection. The HyperVoxelNet obstacle detection model outputs the category information C(n) and bounding box information B(n) of the detected obstacle, where n represents the number of the detected obstacle, C(n) represents the category label of the nth obstacle, and B(n) represents the bounding box coordinates of the nth obstacle.

[0075] Data enhancement and generation technology, voxel-level data enhancement: Design voxel-level data enhancement technology, such as perturbing points inside voxels, adding virtual points, etc., to improve the robustness of the model. Point cloud generation based on generative adversarial network (GAN): Generate high-quality point cloud data using generative adversarial network to help HyperVoxelNet obtain more diverse data during training and improve the generalization ability of the model. Model training and deployment, use the data set in the underground mining environment to train the model, including the labeled data of static and dynamic obstacles. Model training includes steps such as data enhancement, feature extraction, and network optimization. The trained model is deployed to the obstacle detection module to detect and identify the newly collected data in real time.

[0076] Path planning: Building an environmental map: Based on the actual layout of the mine, a two-dimensional or three-dimensional grid map is built, where each grid represents a location in the environment. The category and location information of the detected obstacles are mapped to the grid map, and the corresponding locations are marked as obstacles. The map is expanded to add a certain safety distance around the obstacles to prevent the unmanned vehicle from colliding with the obstacles. A star algorithm path search: Define the coordinates of the starting point and target point of the unmanned vehicle on the grid map. Design heuristic function , estimate the cost from the current node n to the target node. Commonly used heuristic functions include Manhattan distance or Euclidean distance. Initialize the open list and closed list, and add the starting point to the open list. Iterative search: take the estimated total cost from the open list The smallest node n, where is the actual cost from the starting point to the current node n. Move node n from the open list to the closed list. If node n is the target node, the optimal path is found and the algorithm ends. Expand the adjacent nodes of node n and calculate the and , and update its parent node. Add the adjacent nodes that are not in the closed list to the open list. Obtain the optimal path from the starting point to the target point, and convert the node coordinates on the path into the driving route of the unmanned vehicle.

[0077] Obstacle avoidance strategy and path tracking control: Dynamic obstacle avoidance: During the driving process of the unmanned vehicle, dynamic obstacles in the environment are detected in real time. According to the current speed, acceleration and other motion states of the unmanned vehicle, the motion trajectory of the unmanned vehicle in the future is predicted. The motion of the detected dynamic obstacles is predicted to determine whether it will conflict with the predicted trajectory of the unmanned vehicle. If there is a conflict, local adjustments are made on the original path to generate a new safe obstacle avoidance path. The obstacle avoidance path and the original path are smoothly transitioned to ensure the motion continuity of the unmanned vehicle. Path tracking control: The planned driving route is discretized into a series of target points. According to the current position and posture of the unmanned vehicle, the lateral error and heading angle error with the target point are calculated. A PID controller or other control algorithm is designed to generate the steering and speed control instructions of the vehicle according to the error. The control instructions are converted into underlying control signals such as the speed and angle of the motor to drive the unmanned vehicle along the planned route. During the driving process, positioning and error detection are performed in real time, and the movement of the unmanned vehicle is continuously corrected through feedback control to ensure that it accurately tracks the planned route.

[0078] Communication module, data transmission, the communication module realizes real-time data transmission between the unmanned mine car and the ground control center and other vehicles. Use wireless communication technology (such as Wi-Fi, 4G / 5G) to ensure the stability and low latency of data transmission. Command reception, receive commands from the ground control center to ensure the coordination and remote control capabilities of the system. Feedback the operation status and detection results of the mine car for real-time monitoring by the control center. User interface module, system status display, provide system status display for operators, including mine car location, path planning information, obstacle detection results, etc. Real-time display of system operation status, convenient for operators to monitor and manage. Fault diagnosis, provide fault diagnosis function, automatically detect system faults and prompt operators. Including detection and diagnosis of various fault types such as hardware failure and data anomalies. Data monitoring, real-time monitoring of the operation data and environmental data of the mine car. Provide a graphical user interface (GUI) for easy viewing and operation by operators.

Claims

1. An unmanned driving method for underground mines based on light field cameras and deep learning, characterized in that: Light field data is collected by using a light field camera array arranged in an underground mining environment; Performing spatiotemporal synchronization operations on the collected light field data; The light field data after the spatiotemporal synchronization operation is subjected to denoising, color correction and geometric correction in sequence to obtain corrected light field data; Generate corresponding disparity map and depth map based on the corrected light field data; convert the depth map into a 3D point cloud model to provide three-dimensional reconstruction information of the environment; The 3D point cloud model is input into the pre-trained HyperVoxelNet obstacle detection model to perform obstacle detection and recognition, and output the category and location information of the detected obstacles; According to the type and location information of the detected obstacles, the path planning is carried out through the A star algorithm to obtain the driving route of the underground mining unmanned vehicle; Generate corresponding control instructions according to the driving route to control the underground mining unmanned vehicle to drive according to the driving route; Generate the corresponding disparity map and depth map based on the corrected light field data, including: The geometrically corrected light field data is rearranged in the angular dimension and the spatial dimension to obtain a set of sub-aperture images. and The sub-aperture image at the position is taken as the central view sub-image; where, and They are the central view angle coordinates of the light field camera in the U and V directions respectively; Taking the central view sub-image as the reference image, the Lucas-Kanade optical flow method is used to calculate the pixel displacement vectors between the central view sub-image and other sub-aperture images to obtain the initial disparity map D init (u, v); where the pixel displacement vector represents the offset of the corresponding point of the pixel point on the central view sub-image in other sub-aperture images; Construct the energy function E(D) of the optimized disparity map. The energy function E(D) contains the data item E data (D) and the smoothing term E smooth (D); where data item E data (D) Weigh the disparity map D(u, v) and the initial disparity map D init The difference between (u, v), the smoothing term E smooth (D) measures the local smoothness of the disparity map D(u, v); solves the optimal disparity map D by minimizing the energy function E(D) opt (u, v), get the optimized disparity map D init ′(u, v); The imaging parameters and disparity values ​​of the light field camera are jointly calibrated using a checkerboard calibration plate to establish a disparity-depth mapping model Z = g(D); where Z is the depth value, D is the disparity value, and g(D) is the mapping function from disparity to depth, including the scale factor and offset between the disparity value and the depth value; the optimized disparity map D opt Substitute (u, v) into the disparity-depth mapping model to obtain the corresponding depth map Z(u, v); Input the 3D point cloud model into the pre-trained HyperVoxelNet obstacle detection model to detect and identify obstacles, and output the category and location information of the detected obstacles, including: According to the depth map Z(u, v), the corresponding 3D point cloud model P(x, y, z) is calculated. The coordinates (x, y, z) of each point in P(x, y, z) and the coordinates and depth value Z(u, v) of the corresponding pixel point (u, v) in the depth map Z(u, v) satisfy the following relationship: z=Z(u,v) Among them, (u0, v0) is the principal point coordinate of the light field camera, f x and f y are the focal lengths of the light field camera in the x and y directions, respectively; The 3D point cloud model P(x, y, z) is divided into voxel grids of equal size, and the size of each voxel is d×d×d; by calculating the point cloud density and normal vector information in each voxel, a sparse feature representation of the voxel grid is generated to obtain the normalized voxel feature F voxel ; The voxel feature F voxel Input the pre-trained HyperVoxelNet obstacle detection model to perform obstacle detection; The HyperVoxelNet obstacle detection model outputs the category information C(n) and bounding box information B(n) of the detected obstacles, where n represents the number of the detected obstacles, C(n) represents the category label of the nth obstacle, and B(n) represents the bounding box coordinates of the nth obstacle; According to the bounding box information B(n) of the detected obstacles, the voxel area occupied by each obstacle instance is determined.

2. The unmanned driving method for underground mines based on light field camera and deep learning according to claim 1 is characterized in that: Perform spatiotemporal synchronization operations on the collected light field data, including: Obtaining timestamp information of the collected original light field image sequence; According to a preset time synchronization strategy, an original light field image sequence is selected as a reference sequence; Calculate the time offset between the original light field image sequence and the reference sequence; According to the calculated time offset, the original light field image sequence is time-shifted and corrected to obtain a time-aligned light field image sequence; According to the internal and external parameters of the light field cameras, a spatial mapping relationship between the light field cameras is established; By utilizing the spatial mapping relationship, geometric transformation and projection operations are performed on the time-aligned light field image sequences, and the light field image sequences collected by each light field camera are unified into the same coordinate system to obtain the spatiotemporally synchronized light field data.

3. The unmanned driving method for underground mines based on light field cameras and deep learning according to claim 2 is characterized in that: De-noising the light field data after the spatiotemporal synchronization operation, including: Processing the light field data in blocks to obtain a plurality of light field data sub-blocks; For each light field data sub-block, denoising is performed using a sparse representation model based on K-SVD dictionary learning to obtain the denoising result of the corresponding light field data sub-block; wherein the objective function of K-SVD dictionary learning is: Among them, D is the dictionary matrix, X is the sparse coefficient matrix, R i is the extraction operator for extracting the i-th sub-block from the original light field data y, F is the synthesis operator, and a i For sub-block y i The sparse representation coefficient in the dictionary D, N is the total number of sub-blocks, and λ is the sparsity control parameter; The denoising results of each light field data sub-block are spliced ​​to obtain denoised light field data.

4. The unmanned driving method for underground mines based on light field cameras and deep learning according to claim 3 is characterized in that: Color correction is performed on the light field data after spatiotemporal synchronization operations, including: According to the denoised light field data, the color vector value C of each pixel in the light field data is calculated. i , where the color vector value C i Contains three components: R, G, and B; Construct a feature matrix M based on the color vector value of each pixel; Constructing a color transformation function f(M), wherein f(M) is a preset color transformation function for performing color space transformation on the feature matrix M; the color transformation function f(M) is in the form of a polynomial function, and the coefficients of f(M) are solved by a least squares regression method; The obtained f(M) is used to transform the feature matrix M to obtain the color-corrected feature matrix M′, and the pixel color vector values ​​in M′ are filled into the positions of the corresponding pixels to obtain the color-corrected light field data.

5. The unmanned driving method for underground mines based on light field cameras and deep learning according to claim 4 is characterized in that: Perform geometric correction on the light field data after the spatiotemporal synchronization operation, including: The intrinsic and extrinsic parameters of the light field camera are calibrated by the calibration plate, where the intrinsic parameters include the principal point coordinates (u0, v0), focal length f and pixel size (d u , d v ); the external parameters include the rotation matrix R of the light field camera relative to the world coordinate system and the translation vector t; Using the intrinsic and extrinsic parameters obtained through calibration, a geometric model of light field camera imaging is established to describe the mapping relationship between points in the three-dimensional world and corresponding points in the light field camera image. Extract the feature points of the tunnel edge, borehole, anchor and support device from the color-corrected light field data, and use the Harris-Stephens corner detection algorithm to calculate the pixel coordinates of the feature points by performing quadratic surface fitting interpolation on the pixel grayscale values ​​within the preset range of the feature points. The pixel coordinates of the feature points calculated are input into the geometric model of light field camera imaging. The pixel coordinates of the feature points are calibrated by combining the calibrated intrinsic and extrinsic parameters of the light field camera. The perspective projection mapping relationship between the calibrated feature points in the image coordinate system and the world coordinate system is established to eliminate the aberration distortion in the imaging process of the light field camera. According to the calibrated feature point coordinates, the Fourier interpolation algorithm is used to interpolate and reconstruct the light field data in the frequency domain to generate geometrically corrected light field data.

6. The unmanned driving method for underground mines based on light field camera and deep learning according to claim 1, characterized in that: HyperVoxelNet obstacle detection model, including: An input module, receiving a voxel feature descriptor; The hierarchical encoding module divides the input voxel feature descriptor into voxel representations of different scales; hierarchical encoding is performed on the voxel features at different scales, and the encoding weights of each scale are adjusted through the attention mechanism; The four-dimensional voxel spatiotemporal convolution module 4D-VSTC performs convolution operations on the hierarchically encoded voxel features in the spatial (X, Y, Z) and temporal T dimensions through a 4D convolution kernel, and outputs a feature map O VSTC (C, T, X, Y, Z); the convolution kernel in the 4D-VSTC module is a five-dimensional tensor K C ×K T ×K X ×K Y ×K Z ; K C Indicates the size of the channel dimension, which is used to represent the feature information at each position, K T Indicates the size of the time dimension, used to capture dynamic changes in time; K X , K Y , K Z The convolution kernel size represents the spatial dimension, which is used to capture the details of the target in space; The context interaction module GLM uses the self-attention mechanism and graph convolutional network to perform the feature map O VSTC (C, T, X, Y, Z) performs global context modeling and generates a feature map O that integrates global and local context information GLIM ; The spatiotemporal feature association graph module DSTFG uses the graph structure to map the feature graph O GLIM Modeling is performed in the time dimension to obtain the updated feature map O DSTFG ; According to the updated feature map O DSTFG Calculate the weights of feature maps of different scales in different scenes, and obtain the final feature map F through weighted aggregation according to the weights. final ; The feature compression module uses the quantum state simulation method to compress the feature map F output by the DSTFG module. final Compress in the channel dimension, and restore the compressed feature map through feature decoding to obtain the feature map F that is the same as the input feature map in the spatial dimension. final Compressed feature maps of consistent size; The detection head module uses the region candidate network RPN to detect obstacle candidate regions on the feature map output by the feature compression module, and then uses the regional convolutional neural network R-CNN to classify and regress the candidate regions to obtain the final obstacle detection result, including the obstacle category and the three-dimensional bounding box position coordinates.

7. The unmanned driving method for underground mines based on light field camera and deep learning according to claim 6 is characterized in that: Generate a feature map that fuses global and local context information, including: Use the attention mechanism to calculate the feature map O VSTC The correlation between different positions in (C, T, X, Y, Z) generates the global feature G; The feature map O VSTC (C, T, X, Y, Z) is a graph structure, each voxel represents a node, and the connection between voxels represents an edge. The graph convolution network GCN performs convolution operations on the graph structure to obtain the local feature L; The global feature G and the local feature L are weightedly fused to obtain a feature map that integrates global and local context information.

Images