Mineral dynamic identification and precise sampling method and system based on deep learning

Abstract

The application provides a kind of mineral dynamic identification and accurate sampling method and system based on deep learning, comprising: obtaining mineral image data and laser radar ranging data on mine car in underground environment, and obtaining mineral feature data by preprocessing; design a deep convolutional neural network architecture with multi-scale feature fusion and conduct multi-stage training to determine the mineral identification model; use the mineral identification model for target detection, and realize multi-target tracking through Kalman filter and Hungarian algorithm; use FAST corner detection, ORB descriptor and optical flow algorithm to calculate the motion parameters of the mine car; use multi-sensor fusion and extended Kalman filter to predict the future position of the target mineral; based on the predicted position, accurate sampling operation is performed through quintic polynomial interpolation and dynamic compensation control. The application can realize high-precision mineral identification and dynamic accurate sampling in complex underground environment, significantly improving the representativeness and production efficiency of mineral sampling.

Description

A Deep Learning-Based Method and System for Dynamic Mineral Identification and Precise Sampling Technical Field

[0001] This invention relates to the field of mineral identification and intelligent sampling, and in particular to a method and system for dynamic mineral identification and precise sampling based on deep learning, which is applicable to the identification, tracking and precise sampling of moving minerals on mining trucks in underground environments. Background Technology

[0002] Mineral identification and sampling technology is a crucial link in the mining production process, and it is of great significance for ore quality control, production decision-making, and resource optimization. Traditional mineral identification and sampling mainly rely on manual experience or simple mechanical equipment, which presents many technical challenges in practical applications.

[0003] Currently, common technologies in the mining industry include fixed-point mechanical sampling devices and static identification systems based on spectral analysis. The former performs periodic sampling at preset time or location intervals, while the latter analyzes mineral composition using near-infrared or X-ray fluorescence techniques when the ore is in a static state. These technologies have certain application value in controlled environments, but they are difficult to adapt to the complex and ever-changing underground environment.

[0004] The closest prior art to this invention uses computer vision technology to identify and classify minerals on an ore conveyor belt, and performs sampling operations using a fixed robotic arm. This technology utilizes traditional image processing algorithms to extract features such as color and texture of minerals, and can achieve basic mineral classification under good lighting conditions and low ore movement speed.

[0005] However, the existing technology has obvious shortcomings: First, the recognition accuracy is low, especially in environments with high dust concentration and poor lighting conditions underground, the false recognition rate is as high as 15% or more; second, the sampling mechanism lacks flexibility and cannot achieve dynamic and accurate sampling of high-speed moving mining trucks. Sampling operations can often only be performed after the truck has stopped, which seriously affects production efficiency; third, the lack of motion compensation capability leads to large deviations in sampling positions, poor sample representativeness, and affects the reliability of subsequent analysis results. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for dynamic mineral identification and precise sampling based on deep learning. This method can achieve high-precision identification and dynamic precise sampling of minerals on mobile mining trucks in complex underground environments, greatly improving the representativeness and efficiency of mineral sampling.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A deep learning-based method for dynamic mineral identification and precise sampling, characterized by the following steps:

[0009] Mineral image data and lidar ranging data on a mining car in the underground environment are acquired. The mineral image data and lidar ranging data are preprocessed to obtain mineral feature data.

[0010] A deep convolutional neural network architecture for multi-scale feature fusion was designed and trained in multiple stages to determine the mineral recognition model.

[0011] The mineral identification model is used to detect the mineral feature data, a unique ID is assigned to the detected target mineral and depth and appearance features are extracted, a target motion model is established using a Kalman filter and prediction is performed, and multi-target association matching is performed using the Hungarian algorithm to obtain the motion trajectory data of the target mineral.

[0012] Based on the motion trajectory data, feature points of continuous image frames are extracted using the improved FAST corner detection algorithm, feature matching is performed using ORB descriptors and RANSAC algorithm, and pixel-level displacement vector field is calculated using the pyramid Lucas-Kanade optical flow algorithm to obtain the speed and displacement data of the mine car.

[0013] Based on the velocity and displacement data, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned and the three-dimensional spatial position of the target mineral is reconstructed. A nonlinear dynamic model is established, and the multi-sensor data is fused using an extended Kalman filter to predict the target mineral position at future moments, thereby obtaining the target spatial coordinate data.

[0014] Based on the target spatial coordinate data, the joint spatial trajectory is generated using the fifth-order polynomial interpolation method and optimized for time-optimized control. The movement of each joint of the robotic arm is coordinated through dynamic compensation control to perform precise sampling operations.

[0015] Optionally, the mineral image data and the lidar ranging data are preprocessed to obtain mineral feature data, including:

[0016] Calculate the spatial distance and pixel value similarity of the mineral image data, and apply a bilateral filtering algorithm to perform weighted averaging on the neighborhood of each pixel to obtain the denoised mineral image data.

[0017] Calculate the cumulative distribution function of the denoised mineral image data, remap the pixel values ​​according to the cumulative distribution function, and apply an adaptive histogram equalization algorithm to enhance the local contrast of the denoised mineral image data to obtain contrast-enhanced mineral image data.

[0018] Based on the contrast-enhanced mineral image data and the lidar ranging data, statistical outlier removal and voxel downsampling are performed on the lidar point cloud data, followed by pixel-level annotation and data enhancement to obtain the mineral feature data.

[0019] Optionally, pixel-level annotation and data augmentation are performed, including:

[0020] Label the bounding boxes and semantic segmentation masks for different mineral types to obtain the initial labeled dataset;

[0021] For the initial labeled dataset, geometric transformation operation, lighting simulation operation, noise addition operation and special effect simulation operation are applied in sequence to expand the initial labeled dataset to more than 3 times the original size, resulting in an expanded dataset;

[0022] The distribution of the number of various mineral samples in the expanded dataset is statistically analyzed. The number of the first sample is increased by oversampling technology, and the distribution of various mineral samples is balanced by weight adjustment technology to obtain the mineral feature data.

[0023] Optionally, a deep convolutional neural network architecture for multi-scale feature fusion is designed and trained in multiple stages, including:

[0024] An initial deep convolutional neural network was constructed, and basic feature learning was trained for 100 rounds with an initial learning rate of 0.001. The deep convolutional neural network was trained to identify basic mineral categories, and a basic feature learning model was obtained.

[0025] The learning rate of the basic feature learning model was reduced to 0.0001 and fine-grained classification training was performed for 50 rounds to optimize the recognition accuracy of sulfide and oxide ores, thus obtaining a fine-grained classification model.

[0026] Based on the fine-grained classification model, a cosine annealing learning rate strategy is used for end-to-end joint training for 30 rounds to optimize the overall performance of object detection and semantic segmentation, thereby obtaining the mineral recognition model.

[0027] Optionally, the mineral identification model is used to perform target detection on the mineral feature data, assigning a unique ID to the detected target mineral and extracting its depth and appearance features. A Kalman filter is used to establish a target motion model and perform prediction. A Hungarian algorithm is then used for multi-target association matching to obtain the motion trajectory data of the target mineral, including:

[0028] Based on the mineral identification model, target detection is performed on the mineral feature data. A unique ID is assigned to the detected target mineral, and its location, size, category, and appearance features are recorded to establish an initial target state set.

[0029] Based on the initial target state set, a 128-dimensional feature vector for each target is calculated using a pre-trained feature extraction network to obtain the target depth feature set;

[0030] Based on the target depth feature set and the target position history data of the previous N frames, a target motion model is established using a Kalman filter to predict the possible location region of the target in the next frame. The target's velocity and acceleration parameters are updated through the state transition matrix to obtain the target predicted state set.

[0031] Based on the target prediction state set and the detection results of the new frame, the weighted values ​​of appearance feature similarity and spatial distance are calculated. The current detection results are then matched with existing trajectories using the Hungarian algorithm to handle target appearance, disappearance, and occlusion, thereby obtaining the motion trajectory data of the target mineral.

[0032] Optionally, the method further includes:

[0033] The loss function of the deep convolutional neural network is modified from the L2 norm to the L1 norm. For pixel-level segmentation tasks, the loss function is defined as the sum of the absolute differences between the predicted probability map and the true label. For object detection tasks, the bounding box regression loss is modified to L1 loss, resulting in the L1 loss function configuration.

[0034] Based on the L1 loss function configuration, random noise conforming to a Gaussian distribution is introduced into the input image during the training process. The noise parameters are set to mean μ=0 and standard deviation σ=5-15 pixel values. The label remains unchanged, forcing the model to learn to extract essential features from the noise data, thus obtaining noise-enhanced training data.

[0035] Based on the noise-enhanced training data, the sum of absolute deviations is minimized in the local neighborhood, and the pixel value that best represents the real mineral surface features is found through iterative optimization. While preserving the mineral edge and texture features, noise interference is suppressed, and L1 optimized feature data is obtained.

[0036] For the L1 optimized feature data, a penalty term for the sum of the absolute values ​​of the weights is added to the loss function. The regularization strength parameter λ ranges from 0.001 to 0.01, guiding the network to automatically select the most representative mineral features to obtain sparse feature representation.

[0037] Based on the sparse feature representation, the basic feature learning training is performed for 100 rounds with an initial learning rate of 0.001, followed by 50 rounds of fine-grained classification training with the learning rate reduced to 0.0001, and then 30 rounds of end-to-end joint training using a cosine annealing learning rate strategy to obtain the mineral recognition model.

[0038] Optionally, a deep convolutional neural network architecture for multi-scale feature fusion is designed, including:

[0039] Based on the mineral dataset and the physical properties of mineral spectral reflectance, crystal structure and chemical composition, mineralogy laws are encoded as constraints of the network structure. A physical awareness module and a data-driven module are designed, and the two parts are coupled through a physical consistency loss function to obtain a physical awareness hybrid network architecture.

[0040] Using the aforementioned physical-informed hybrid network architecture, the high-dimensional input is decomposed into multiple one-dimensional functions, and N parallel sub-networks are designed to be responsible for the extraction of color, texture and morphological features respectively. The outputs of the N parallel sub-networks are integrated through adaptive weighted fusion to obtain a parallel Kolmogorov-Arnold network structure, where N is a positive integer greater than 1.

[0041] Based on the parallel Kolmogorov-Arnold network structure, subdomains are divided according to mineral categories, including sulfide minerals, oxide minerals, and silicate minerals, and physical properties, including reflectivity, color, and hardness. A dedicated subnetwork is configured for each subdomain, and information exchange between subdomains is maintained through overlapping regions and boundary conditions, resulting in a domain decomposition network configuration.

[0042] Based on the domain decomposition network configuration, a multi-objective loss function is designed, which includes standard classification loss, physical consistency loss, inter-domain consistency loss and uncertainty estimation loss. By balancing data-driven learning and physical law constraints, a physical constraint loss function is obtained.

[0043] Based on the physical constraint loss function, an improved ResNet-50 is used as the backbone network and integrated with a feature pyramid network structure for multi-stage training to obtain the mineral identification model.

[0044] Optionally, based on the velocity and displacement data, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned to reconstruct the three-dimensional spatial position of the target mineral. A nonlinear dynamic model is established, and an extended Kalman filter is used to fuse the multi-sensor data and predict the target mineral's position at future moments to obtain target spatial coordinate data, including:

[0045] Based on the velocity and displacement data, a set of nonlinear partial differential equations describing the motion of minerals on the conveyor belt is constructed. The dynamic system is decomposed into principal terms and perturbation terms. The principal terms adopt Newton's equations of motion, and the perturbation terms are learned from the measured data through a neural network to capture complex nonlinear effects that the theoretical model fails to describe, thus obtaining the perturbation PINN dynamic model.

[0046] Based on the perturbation PINN dynamics model, a hierarchical neural network architecture is designed. Different layers of the network are responsible for modeling the physical processes at different time and space scales, from millisecond-level particle collisions to second-level overall motion. The physical consistency between models at each scale is ensured through loss function design, resulting in a multi-scale PINN model.

[0047] Based on the multi-scale PINN model, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned, and the target mineral in the two-dimensional image is reconstructed into its position and volume information in three-dimensional space through projection mapping relationship, thus obtaining a three-dimensional spatial representation of the target mineral.

[0048] Based on the three-dimensional spatial representation of the target mineral, a nonlinear dynamic model considering the acceleration, deceleration, vibration, and steering factors of the mining truck is established to describe the motion law of the target mineral relative to the sampling robotic arm, and the dynamic model parameter set is obtained.

[0049] Based on the aforementioned dynamic model parameter set, an extended Kalman filter is used to fuse visual measurements, laser ranging, and IMU data. When the statistical estimate deviates from the physical laws, it is automatically corrected to ensure that the state estimate conforms to the basic physical laws. The position of the target mineral is predicted within the next 0.5 to 1 second, thus obtaining the target spatial coordinate data.

[0050] Optionally, based on the motion trajectory data, feature points of consecutive image frames are extracted using an improved FAST corner detection algorithm, feature matching is performed using ORB descriptors and the RANSAC algorithm, and the pixel-level displacement vector field is calculated using the pyramid Lucas-Kanade optical flow algorithm to obtain the speed and displacement data of the mining truck, including:

[0051] Based on the motion trajectory data and the continuous image frames, feature points in the image are extracted using the improved FAST corner detection algorithm, and feature representations are calculated using ORB descriptors to obtain feature description data.

[0052] For the feature description data, the similarity between feature points is calculated using Hamming distance, and the feature corresponding points are selected by the RANSAC algorithm to obtain the feature matching point set;

[0053] An image pyramid is constructed, and the optical flow of the feature matching point set is calculated layer by layer from coarse to fine scale. The Lucas-Kanade optical flow algorithm is applied to calculate the pixel-level displacement vector field between consecutive frames to achieve sub-pixel accuracy motion estimation and obtain the optical flow vector field.

[0054] Based on the optical flow vector field, the camera's rotation matrix R and translation vector t are recovered by solving the fundamental matrix or homography matrix, and the transformation relationship between the world coordinate system and the camera coordinate system is established to obtain the camera attitude parameters.

[0055] Based on the camera attitude parameters and system timestamps, displacement accumulation calculations are performed between consecutive frames. Combined with wheel encoder data, fusion correction is performed to eliminate accumulated errors, thereby obtaining the speed and displacement data of the mining truck.

[0056] Optionally, based on the target spatial coordinate data, a joint spatial trajectory is generated using a fifth-order polynomial interpolation method and optimized for time-optimized control. Dynamic compensation control coordinates the movement of each joint of the robotic arm, performing precise sampling operations, including:

[0057] Based on the target space coordinate data and the DH parameters and joint constraints of the robotic arm, forward and inverse kinematic models are established, the mapping relationship between the end effector position and the joint angle is calculated, the workspace and singular positions are defined, and the kinematic model of the robotic arm is obtained.

[0058] Based on the aforementioned robotic arm kinematic model, fifth-order polynomial interpolation is used to interpolate joint angles, velocities, and accelerations to generate smooth joint spatial trajectories and obtain initial trajectory parameters.

[0059] For the initial trajectory parameters, a time-optimal control algorithm is applied to optimize the trajectory execution time while satisfying joint velocity and acceleration constraints, thereby obtaining the optimal trajectory parameters;

[0060] Based on the optimal trajectory parameters, a dynamic model is established considering the mass, inertia tensor, and friction coefficient of each joint of the robotic arm. Feedforward compensation control is implemented to reduce tracking errors during high-speed motion. An adaptive impedance control strategy is used to cope with changes in contact force during sampling, and joint torque control commands are obtained.

[0061] Based on the joint torque control command, the movement of each joint of the robotic arm is coordinated and controlled by a high-performance servo driver and a real-time control system to achieve precise positioning of the end effector. When the target mineral reaches the predetermined position, the sampler opening and closing, depth and sampling amount are controlled to perform the precise sampling operation.

[0062] This application also provides a deep learning-based mineral dynamic identification and precise sampling system, including:

[0063] The acquisition module is used to acquire mineral image data and lidar ranging data on the mine car in the underground environment, and to preprocess the mineral image data and lidar ranging data to acquire mineral feature data.

[0064] The training module is used to design a deep convolutional neural network architecture for multi-scale feature fusion and perform multi-stage training to determine the mineral recognition model.

[0065] The target detection module is used to perform target detection on the mineral feature data using the mineral recognition model, assign a unique ID to the detected target mineral and extract depth and appearance features, establish a target motion model using a Kalman filter and make predictions, and perform multi-target association matching using the Hungarian algorithm to obtain the motion trajectory data of the target mineral.

[0066] The feature matching module is used to extract feature points of continuous image frames based on the motion trajectory data using the improved FAST corner detection algorithm, perform feature matching through ORB descriptor and RANSAC algorithm, and apply the pyramid Lucas-Kanade optical flow algorithm to calculate the pixel-level displacement vector field to obtain the speed and displacement data of the mine car.

[0067] The prediction module is used to perform spatiotemporal alignment of multi-sensor data from the camera, lidar, and IMU based on the velocity and displacement data, reconstruct the three-dimensional spatial position of the target mineral, establish a nonlinear dynamic model, use an extended Kalman filter to fuse multi-sensor data and predict the position of the target mineral at future moments, and obtain target spatial coordinate data.

[0068] The sampling module is used to generate joint spatial trajectories based on the target spatial coordinate data using a fifth-order polynomial interpolation method and to optimize time-optimized control. It coordinates the movement of each joint of the robotic arm through dynamic compensation control and performs precise sampling operations.

[0069] The beneficial effects of this invention are:

[0070] 1. High-precision mineral identification: Through a deep convolutional neural network with multi-scale feature fusion and a multi-stage training strategy, this invention can achieve high-precision identification of different types of minerals in complex downhole environments, with an identification accuracy rate of over 95%.

[0071] 2. Dynamic target tracking: The multi-target tracking technology, which combines Kalman filtering and Hungarian algorithm, can stably track multiple mineral targets while the mining truck is moving at high speed, and can maintain tracking stability even under partial obstruction.

[0072] 3. High-precision motion compensation: By using visual odometry, optical flow method and multi-sensor fusion, the motion parameters of the mining truck are accurately estimated and the future position of the minerals is predicted. The prediction accuracy can reach ±2mm, laying the foundation for accurate sampling.

[0073] 4. Real-time dynamic sampling: Employing fifth-order polynomial interpolation and dynamic compensation control, the robotic arm can accurately follow the moving target and perform sampling operations without stopping, significantly improving production efficiency.

[0074] 5. Strong system adaptability: Through deep learning and adaptive optimization mechanisms, the system can adapt to different working conditions and mineral types, and has strong environmental adaptability and anti-interference ability. Attached Figure Description

[0075] Figure 1 is an overall flowchart of the method of the present invention;

[0076] Figure 2 is a detailed flowchart of S1;

[0077] Figure 3 is a detailed flowchart of S2;

[0078] Figure 4 shows the architecture of a deep convolutional neural network for multi-scale feature fusion. Detailed Implementation

[0079] As shown in Figure 1, this invention proposes a deep learning-based method for dynamic mineral identification and precise sampling. This method includes six main steps: mineral data acquisition and preprocessing, deep learning mineral identification model construction, multi-target dynamic tracking, visual odometry and motion estimation, multi-sensor fusion motion compensation prediction, and intelligent robotic arm trajectory planning and control.

[0080] Step S1: Acquire mineral image data and lidar ranging data from the mine car in the underground environment, preprocess the mineral image data and lidar ranging data to obtain mineral feature data.

[0081] In this step, high-resolution industrial cameras (2048×1536 pixels) and lidar (measurement accuracy ±2mm) deployed underground are first used to acquire images and ranging data of the minerals on the mining truck. For the acquired mineral image data, the spatial distance and pixel value similarity of the images are calculated. A bilateral filtering algorithm is applied to perform weighted averaging on the neighborhood of each pixel to obtain denoised mineral image data. Then, the cumulative distribution function of the denoised mineral image data is calculated, and the pixel values ​​are remapped according to the cumulative distribution function. An adaptive histogram equalization algorithm is applied to enhance the local contrast of the denoised mineral image data, resulting in contrast-enhanced mineral image data.

[0082] For the lidar ranging data, statistical outlier removal and voxel downsampling are performed, and the point cloud data is registered with the image data. Finally, based on the contrast-enhanced mineral image data and the processed lidar ranging data, pixel-level annotation and data augmentation are performed to obtain mineral feature data.

[0083] Pixel-level annotation and data augmentation processing includes: labeling the bounding boxes and semantic segmentation masks of different mineral types (such as sulfide minerals, oxide minerals, waste rock, etc.) to obtain the initial annotation dataset; applying geometric transformation operations (such as rotation, scaling, translation), lighting simulation operations (brightness and contrast adjustment), noise addition operations (Gaussian noise, salt-and-pepper noise), and special effect simulation operations (blurring, dust occlusion simulation) to the initial annotation dataset in sequence to expand the initial annotation dataset to more than 3 times the original size to obtain the expanded dataset; statistically analyzing the quantity distribution of various mineral samples in the expanded dataset, increasing the number of rare samples through oversampling techniques, and balancing the distribution of various mineral samples through weight adjustment techniques to obtain the final mineral feature data.

[0084] Step S2: Design a deep convolutional neural network architecture for multi-scale feature fusion and perform multi-stage training to determine the mineral recognition model.

[0085] In this step, a deep convolutional neural network architecture for multi-scale feature fusion is designed based on the mineral dataset and the physical properties of mineral spectral reflectance, crystal structure, and chemical composition. Specifically, this includes: encoding mineralogical laws as constraints for the network structure; designing a physically informed module and a data-driven module; coupling the two parts through a physical consistency loss function to obtain a physically informed hybrid network architecture; using this architecture, decomposing the high-dimensional input into multiple one-dimensional functions; designing N parallel sub-networks responsible for extracting color, texture, and morphological features respectively; and integrating the outputs of these parallel sub-networks through adaptive weighted fusion to obtain a parallel Kolmogorov-Arnold network structure.

[0086] Furthermore, subdomains are divided according to mineral categories (including sulfide minerals, oxide minerals, and silicate minerals) and physical properties (including reflectivity, color, and hardness). A dedicated subnetwork is configured for each subdomain, and information exchange between subdomains is maintained through overlapping regions and boundary conditions, resulting in a domain decomposition network configuration. Based on this configuration, a multi-objective loss function is designed, which includes standard classification loss, physical consistency loss, inter-domain consistency loss, and uncertainty estimation loss, balancing data-driven learning with physical constraints, resulting in a physical constraint loss function.

[0087] During training, the loss function of the deep convolutional neural network is modified from the L2 norm to the L1 norm. For pixel-level segmentation tasks, the loss function is defined as the sum of the absolute differences between the predicted probability map and the true label. For object detection tasks, the bounding box regression loss is modified to L1 loss, resulting in the L1 loss function configuration. Based on this configuration, random noise conforming to a Gaussian distribution is introduced into the input images during training. The noise parameters are set to mean μ=0 and standard deviation σ=5-15 pixel values. The labels are kept unchanged, forcing the model to learn to extract essential features from the noisy data, resulting in noise-enhanced training data.

[0088] Based on the noise-enhanced training data, the sum of absolute biases is minimized in the local neighborhood. Through iterative optimization, the pixel values ​​that best represent the real mineral surface features are found. While preserving the mineral edge and texture features, noise interference is suppressed, resulting in L1 optimized feature data. For this data, a penalty term for the sum of the absolute values ​​of the weights is added to the loss function. The regularization intensity parameter λ ranges from 0.001 to 0.01, guiding the network to automatically select the most representative mineral features, resulting in sparse feature representation.

[0089] Finally, a multi-stage training strategy was adopted: First, a basic feature learning training was conducted for 100 rounds with an initial learning rate of 0.001 to train a deep convolutional neural network to identify basic mineral categories, thus obtaining a basic feature learning model; then, the learning rate was reduced to 0.0001 for 50 rounds of fine-grained classification training, focusing on optimizing the identification accuracy of sulfide and oxide minerals, thus obtaining a fine-grained classification model; finally, based on the fine-grained classification model, a cosine annealing learning rate strategy was used for 30 rounds of end-to-end joint training to optimize the overall performance of object detection and semantic segmentation, thus obtaining the final mineral identification model.

[0090] Step S3: Use the mineral identification model to perform target detection on the mineral feature data, assign a unique ID to the detected target mineral and extract its depth and appearance features, use a Kalman filter to establish a target motion model and make predictions, and use the Hungarian algorithm to perform multi-target association matching to obtain the motion trajectory data of the target mineral.

[0091] In this step, target detection is performed on the real-time input mineral feature data based on the trained mineral recognition model. A unique ID is assigned to each detected target mineral, and its location, size, category, and appearance features are recorded to establish an initial target state set. Then, a pre-trained feature extraction network is used to calculate a 128-dimensional feature vector for each target, resulting in a target depth feature set. Based on the target depth feature set and historical target position data from the previous N frames, a Kalman filter is used to establish a target motion model, predicting the possible location region of the target in the next frame. The target's velocity and acceleration parameters are updated using a state transition matrix to obtain the target predicted state set.

[0092] Based on the target prediction state set and the detection results of the new frame, a weighted value of appearance feature similarity and spatial distance is calculated. The Hungarian algorithm is then used to associate and match the current detection result with existing trajectories, handling target appearance, disappearance, and occlusion. Specifically, when the similarity between a newly detected target and any existing trajectory is below a threshold, a new trajectory is created for it; when an existing trajectory has no matching detection results in multiple consecutive frames, it is marked as disappeared; for temporarily occluded targets, the system continues to use the prediction results of the Kalman filter to maintain the trajectory until the target reappears and is matched. In this way, the system can maintain stable target trajectories in complex environments, ultimately obtaining complete motion trajectory data of the target mineral.

[0093] Step S4: Based on the motion trajectory data, feature points of continuous image frames are extracted using the improved FAST corner detection algorithm, feature matching is performed using ORB descriptors and the RANSAC algorithm, and the pixel-level displacement vector field is calculated using the pyramid Lucas-Kanade optical flow algorithm to obtain the speed and displacement data of the mine car.

[0094] In this step, feature points in the image are first extracted using an improved FAST corner detection algorithm based on the target mineral's motion trajectory data and consecutive image frames. This improved algorithm enhances corner stability while maintaining detection efficiency through adaptive threshold selection and non-maximum suppression optimization. Then, feature representations are calculated using the ORB (Oriented FAST and Rotated BRIEF) descriptor to obtain feature description data.

[0095] For the feature description data, Hamming distance is used to calculate the similarity between feature points, and RANSAC (Random Sample Consensus) algorithm is used to remove erroneous matches, resulting in a high-quality feature matching point set. Next, an image pyramid is constructed, and the optical flow of the feature matching point set is calculated layer by layer from coarse to fine scale. The Lucas-Kanade optical flow algorithm is applied to calculate the pixel-level displacement vector field between consecutive frames, achieving sub-pixel accuracy motion estimation and obtaining the optical flow vector field.

[0096] Based on the optical flow vector field, the camera's rotation matrix R and translation vector t are recovered by solving the fundamental matrix or homography matrix, establishing the transformation relationship between the world coordinate system and the camera coordinate system, and obtaining the camera attitude parameters. Finally, based on the camera attitude parameters and system timestamps, displacement accumulation calculations are performed between consecutive frames, and combined with wheel encoder data for fusion correction to eliminate accumulated errors, obtaining the mine car's speed and displacement data.

[0097] Step S5: Based on the velocity and displacement data, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned and the three-dimensional spatial position of the target mineral is reconstructed. A nonlinear dynamic model is established, and the multi-sensor data is fused using an extended Kalman filter to predict the future position of the target mineral, thereby obtaining the target spatial coordinate data.

[0098] In this step, based on the speed and displacement data of the mining truck, a set of nonlinear partial differential equations describing the motion of the minerals on the conveyor belt is first constructed, decomposing the dynamic system into principal terms and perturbation terms. The principal terms adopt Newton's equations of motion, while the perturbation terms are learned from measured data through a neural network to capture complex nonlinear effects that theoretical models fail to describe, resulting in a perturbation PINN (Physics-Informed Neural Networks) dynamic model.

[0099] Based on this dynamic model, a hierarchical neural network architecture is designed. Different layers of the network are responsible for modeling the physical processes at different time and space scales, from millisecond-level particle collisions to second-level overall motion. The physical consistency between models at different scales is ensured through loss function design, resulting in a multi-scale PINN model.

[0100] Then, the multi-sensor data from the camera, LiDAR, and IMU are spatiotemporally aligned. Time synchronization of the sensor data is achieved through hardware triggering and software timestamp calibration, and the data from different sensors are aligned to a unified coordinate system using a calibrated rigid body transformation matrix. The target mineral in the two-dimensional image is reconstructed into its position and volume information in three-dimensional space through projection mapping, resulting in a three-dimensional spatial representation of the target mineral.

[0101] Based on the three-dimensional spatial representation of the target mineral, a nonlinear dynamic model considering factors such as acceleration / deceleration, vibration, and steering of the mining truck is established to describe the motion of the target mineral relative to the sampling robotic arm, thus obtaining the parameter set of the dynamic model. Finally, an extended Kalman filter is used to fuse visual measurements, laser ranging, and IMU data. When the statistical estimate deviates from the physical laws, automatic corrections are made to ensure that the state estimate conforms to basic physical laws, predicting the position of the target mineral within the next 0.5 to 1 second, thus obtaining the target spatial coordinate data.

[0102] Step S6: Based on the target spatial coordinate data, the joint spatial trajectory is generated using the fifth-order polynomial interpolation method and optimized for time-optimized control. The movement of each joint of the robotic arm is coordinated through dynamic compensation control to perform precise sampling operations.

[0103] In this step, firstly, based on the target space coordinate data and the robot arm's DH parameters and joint constraints, forward and inverse kinematic models are established. The mapping relationship between the end effector position and joint angles is calculated, and the workspace and singular positions are defined to obtain the robot arm's kinematic model. Based on this model, fifth-order polynomial interpolation is used to interpolate the joint angles, velocities, and accelerations to generate a smooth joint space trajectory, thus obtaining the initial trajectory parameters.

[0104] Then, a time-optimal control algorithm is applied to the initial trajectory parameters to optimize the trajectory execution time while satisfying joint velocity and acceleration constraints, thus obtaining the optimal trajectory parameters. Based on the optimal trajectory parameters, a dynamic model is established considering the mass, inertia tensor, and friction coefficient of each joint of the robotic arm. Feedforward compensation control is implemented to reduce tracking errors during high-speed motion, and an adaptive impedance control strategy is used to address changes in contact force during sampling, thereby obtaining joint torque control commands.

[0105] Finally, a high-performance servo drive and a real-time control system coordinate the movement of each joint of the robotic arm to achieve precise positioning of the end effector. When the target mineral reaches the predetermined position, the system controls the sampler's opening and closing, depth, and sampling volume to perform precise sampling. The system dynamically adjusts sampling parameters (such as opening and closing force and sampling depth) to adapt to the physical properties of different types of minerals, ensuring the representativeness and integrity of the samples.

[0106] As shown in Figure 2, the mineral image data and the lidar ranging data are preprocessed to obtain mineral feature data, including:

[0107] S1.1 Calculate the spatial distance and pixel value similarity of the mineral image data, and apply a bilateral filtering algorithm to perform weighted averaging on the neighborhood of each pixel to obtain the denoised mineral image data;

[0108] In methods for dynamic mineral identification and precise sampling, the preprocessing of mineral image data and lidar ranging data is a fundamental step in the entire system, directly affecting the accuracy of subsequent identification and sampling. This invention employs a complete preprocessing workflow to ensure the acquisition of high-quality mineral feature data in complex downhole environments.

[0109] The first step (S1.1) of preprocessing is to denoise the mineral image data. Underground environments often suffer from insufficient lighting and dust interference, resulting in raw images containing significant noise. This invention first calculates the spatial distance and pixel value similarity between each pixel and its neighboring pixels, and then applies a bilateral filtering algorithm. The unique feature of bilateral filtering is that it simultaneously considers spatial distance and pixel value differences, performing a weighted average of the neighborhood of each pixel. Specifically, when a neighboring pixel is spatially close to the center pixel and has similar pixel values, that pixel contributes more to the filtering result; conversely, its contribution is smaller. This method effectively removes noise while preserving important details such as image edges and textures, making it particularly suitable for processing the rich surface textures and boundary information in mineral images. In practical applications, the spatial domain parameter σs of the bilateral filter is typically set to 5-10 pixels, and the value domain parameter σr is set to 20-30 pixel values, which can be adjusted according to the specific noise level.

[0110] S1.2 Calculate the cumulative distribution function of the denoised mineral image data, remap the pixel values ​​according to the cumulative distribution function, and apply the adaptive histogram equalization algorithm to enhance the local contrast of the denoised mineral image data to obtain contrast-enhanced mineral image data.

[0111] The second step of preprocessing (S1.2) is to enhance image contrast and improve the distinguishability of mineral features. First, the cumulative distribution function (CDF) of the denoised mineral image is calculated, which describes the statistical distribution characteristics of pixel values. Based on the CDF, the image pixel values ​​are remapped to achieve global histogram equalization, expanding the dynamic range of the image. However, global histogram equalization may lead to the loss of local details. Therefore, this invention further applies an adaptive histogram equalization (AHE) algorithm to enhance the local contrast of the image. The AHE algorithm divides the image into multiple small regions (usually 8×8 or 16×16 grids), applies histogram equalization independently to each region, and then fuses the results of adjacent regions through bilinear interpolation to avoid artificial traces at region boundaries. To prevent excessive noise amplification, a contrast limiting strategy (CLAHE) is typically used to uniformly redistribute portions of the histogram exceeding a set threshold. This locally adaptive enhancement method effectively improves the local details and texture features of mineral images in low-light downhole environments, making the characteristics of different types of minerals more prominent.

[0112] S1.3 Based on the contrast-enhanced mineral image data and the lidar ranging data, statistical outlier removal and voxel downsampling processing are performed on the lidar point cloud data, followed by pixel-level annotation and data enhancement processing to obtain the mineral feature data.

[0113] In S1.3, pixel-level annotation and data augmentation are performed, including:

[0114] Label the bounding boxes and semantic segmentation masks for different mineral types to obtain the initial labeled dataset;

[0115] For the initial labeled dataset, geometric transformation operation, lighting simulation operation, noise addition operation and special effect simulation operation are applied in sequence to expand the initial labeled dataset to more than 3 times the original size, resulting in an expanded dataset;

[0116] The distribution of the number of various mineral samples in the expanded dataset is statistically analyzed. The number of the first sample is increased by oversampling technology, and the distribution of various mineral samples is balanced by weight adjustment technology to obtain the mineral feature data.

[0117] The third step of preprocessing (S1.3) involves integrating image data and LiDAR ranging data, followed by annotation and enhancement. First, statistical outlier removal is performed on the point cloud data acquired by the LiDAR. By analyzing the distance distribution between each point and its neighbors, outliers are identified and removed. These outliers are typically caused by sensor errors or environmental interference (such as dust reflection). Then, voxel downsampling is performed to convert the original high-density point cloud into a uniformly distributed low-density point cloud, significantly reducing the data volume while preserving key geometric information. The downsampled point cloud is then registered with the enhanced image data to establish the correspondence between two-dimensional pixels and three-dimensional points, laying the foundation for subsequent 3D reconstruction and spatial localization.

[0118] In the annotation and data augmentation phase, a team of professional geological experts first performs pixel-level annotation on the preprocessed image data, marking the bounding boxes and semantic segmentation masks for different mineral types (such as sulfide minerals, oxide minerals, and waste rock), resulting in an initial labeled dataset. This process is typically completed using specialized annotation tools to ensure the accuracy and consistency of the annotations. Then, the initial labeled dataset undergoes systematic data augmentation processing, including four types of operations: geometric transformation operations (rotation, scaling, translation, flipping, etc.) to simulate the observation of minerals at different angles and distances; lighting simulation operations (brightness adjustment, contrast changes, tone conversion, etc.) to simulate the appearance of minerals under different lighting conditions; noise addition operations (Gaussian noise, salt-and-pepper noise, etc.) to improve the model's robustness to noise; and special effects simulation operations (blurring, dust occlusion simulation, etc.) to simulate the complex interference factors of the actual downhole environment. Through this series of augmentation operations, the initial labeled dataset is expanded to more than three times its original size, significantly increasing the diversity and quantity of training samples.

[0119] Finally, statistical analysis of the distribution of mineral samples across different classes in the augmented dataset reveals that certain mineral types (such as rare minerals) typically have significantly fewer samples than others. To address this imbalance, oversampling techniques are employed to increase the number of samples in the first class (the rare class), including random duplication and SMOTE (Synthetic Minority Oversampling). Simultaneously, the weights of each class in the loss function are adjusted to balance their contributions, ensuring that the rare class receives sufficient attention during training. After these processing steps, the resulting mineral feature data contains rich visual and spatial information and exhibits good class balance and diversity, providing a high-quality data foundation for subsequent deep learning model training.

[0120] In the process of constructing a deep learning mineral identification model, this invention adopts a multi-scale feature fusion convolutional neural network architecture and a three-stage training strategy, which effectively solves the problems of fine-grained classification and environmental adaptability in mineral identification.

[0121] As shown in Figures 3 and 4, a deep convolutional neural network architecture for multi-scale feature fusion is designed and trained in multiple stages, including:

[0122] S2.1 Construct an initial deep convolutional neural network and train it for 100 rounds with an initial learning rate of 0.001 to learn basic features. Train the deep convolutional neural network to identify basic mineral categories and obtain a basic feature learning model.

[0123] The first stage (S2.1) is the basic feature learning stage. First, an initial deep convolutional neural network is constructed. The network architecture is based on an improved ResNet-50 as the backbone network (composed of C2-C5, where C represents different ResNet layers), and an integrated Feature Pyramid Network (FPN, composed of P2-P5, where P represents the number of layers in the pyramid network) structure is used to achieve the fusion of features at different scales. To improve the model's ability to represent complex mineral features, channel attention and spatial attention mechanisms are added in addition to the standard convolutional layers, enabling the network to adaptively focus on key feature regions. During training, a relatively large initial learning rate of 0.001 is used, allowing the model to quickly explore the parameter space and learn basic mineral texture, color, and shape features. Training lasts for 100 epochs, optimized using mini-batch stochastic gradient descent with a batch size of 32, and the momentum parameter is set to 0.9. In this stage, the model mainly learns the ability to distinguish basic mineral categories (such as metallic minerals versus non-metallic minerals, sulfide minerals versus oxide minerals), forming a basic feature learning model. This stage can typically improve the model's accuracy to around 85%, laying the foundation for subsequent refined training.

[0124] S2.2 Reduce the learning rate of the basic feature learning model to 0.0001 and perform 50 rounds of fine-grained classification training to optimize the recognition accuracy of sulfide minerals and oxide minerals, thereby obtaining a fine-grained classification model;

[0125] The second stage (S2.2) is the fine-grained classification optimization stage. Based on the basic feature learning model trained in the first stage, the learning rate is reduced to 0.0001, transitioning to fine-grained training. The reduced learning rate allows the model to be fine-tuned based on the established basic feature representations, capturing more subtle differences in mineral features. This stage focuses on optimizing the identification accuracy of key minerals such as sulfide and oxide ores, which have higher economic value in actual production and require more accurate identification. Training lasts for 50 rounds, using the same batch size and optimizer parameters. To enhance the model's ability to distinguish similar minerals, a hard sample mining strategy is introduced, intentionally increasing the proportion of difficult-to-classify samples in each mini-batch. Simultaneously, Focal Loss is applied instead of standard cross-entropy loss to further enhance attention to difficult-to-classify samples. Through this stage of training, the model can accurately distinguish between mineral subclasses with similar appearances, such as pyrite and chalcopyrite, hematite and limonite, with an identification accuracy exceeding 93%, forming a fine-grained classification model.

[0126] S2.3 Based on the fine-grained classification model, a cosine annealing learning rate strategy is used to perform end-to-end joint training for 30 rounds to optimize the overall performance of target detection and semantic segmentation, thereby obtaining the mineral recognition model.

[0127] The third stage (S2.3) is the end-to-end joint optimization stage. Based on the fine-grained classification model of the second stage, a cosine annealing learning rate strategy is used for end-to-end joint training. The cosine annealing learning rate starts from 0.0001 and changes periodically according to the cosine function, gradually decreasing to 0.00001. At the end of each cycle, the learning rate is briefly increased to help the model escape local optima. This stage continues training for 30 rounds, while optimizing the overall performance of object detection and semantic segmentation. In terms of loss function design, a multi-task joint loss is adopted, combining classification loss, bounding box regression loss, and pixel-level segmentation loss to ensure that the model can accurately locate the target position and boundary while identifying mineral types. In addition, domain adaptation technology is introduced to enable the model to better adapt to the differences in data distribution in different mining areas and working conditions. Through this stage of training, the overall performance of the model in complex downhole environments is significantly improved, with a recognition accuracy of over 98%, a target localization accuracy of better than ±5 pixels, and a segmentation IoU (Intersection over Union) of over 0.9, ultimately forming a high-performance mineral recognition model.

[0128] The three-stage training strategy described above fully considers the specific needs of mineral identification. Through a progressive learning process from coarse to fine and from basic to professional, the model not only masters the ability to represent universal mineral features but also possesses the ability to accurately identify key mineral types, while maintaining adaptability to complex environments. This training method is more effective than single-stage training, achieving higher recognition accuracy and generalization ability under limited training data conditions, providing a reliable recognition foundation for subsequent multi-target dynamic tracking and precise sampling.

[0129] Specifically, the mineral identification model is used to perform target detection on the mineral feature data, a unique ID is assigned to the detected target mineral and depth and appearance features are extracted, a Kalman filter is used to establish a target motion model and perform prediction, and a Hungarian algorithm is used for multi-target association matching to obtain the motion trajectory data of the target mineral, including:

[0130] S3.1 Based on the mineral identification model, target detection is performed on the mineral feature data, a unique ID is assigned to the detected target mineral, and its location, size, category and appearance features are recorded to establish an initial target state set;

[0131] S3.2 Based on the initial target state set, a 128-dimensional feature vector for each target is calculated using a pre-trained feature extraction network to obtain the target depth feature set;

[0132] S3.3 Based on the target depth feature set and the target position history data of the previous N frames, a target motion model is established using a Kalman filter to predict the possible location region of the target in the next frame. The target's velocity and acceleration parameters are updated through the state transition matrix to obtain the target prediction state set.

[0133] S3.4 Based on the target prediction state set and the detection results of the new frame, calculate the weighted value of appearance feature similarity and spatial distance, and use the Hungarian algorithm to associate and match the current detection results with the existing trajectory, handle the appearance, disappearance and occlusion of the target, and obtain the motion trajectory data of the target mineral.

[0134] In deep learning-based methods for dynamic mineral identification and precise sampling, a multi-target tracking system is a crucial link connecting identification and sampling. This system achieves stable tracking of multiple mineral targets on a moving mining truck through four progressive steps, maintaining high accuracy in target trajectory prediction even when the truck is moving at high speed and minerals partially obscure the target.

[0135] The first step (S3.1) is target detection and initialization. The system uses a pre-trained mineral recognition model to detect targets in the input mineral feature data, identifying mineral targets in the image and outputting the bounding box coordinates, category identifier, and confidence score for each target. To ensure detection quality, the system only processes detection results with a confidence score exceeding a preset threshold (usually 0.75). For each newly detected target mineral, the system assigns a globally unique ID identifier, using a combination of timestamp and serial number to ensure that the ID is not duplicated. Simultaneously, key information about the target is recorded, including the center position coordinates (x, y), bounding box dimensions (width w, height h), mineral category (e.g., sulfide mineral, oxide mineral, etc.), and preliminary appearance features (e.g., average color, texture histogram, etc.). This information constitutes the initial target state set, serving as the starting point for target tracking. In practical applications, the system can simultaneously track 15-20 mineral targets, meeting the tracking needs of dense minerals on mining trucks.

[0136] The second step (S3.2) is deep feature extraction. The simple appearance features in the initial state set are insufficient to handle target matching challenges in complex environments. Therefore, the system utilizes a pre-trained feature extraction network to compute high-dimensional deep features for each target. This network uses ResNet-50 as the backbone, but removes the final fully connected classification layer and adds a global average pooling and feature mapping layer after the last convolutional layer, outputting a 128-dimensional feature vector. The network pre-training employs a combination of ID supervision and triplet loss, ensuring that the feature vector distances for the same target are small across different frames, while the feature vector distances for different targets are large, resulting in strong discriminative ability. During feature extraction, the system first crops and scales the detected target regions to a uniform size (typically 224×224 pixels), then feeds them into the feature network for forward computation to obtain the target's deep feature set. These features are robust, remaining stable even when the target rotates, partially occludes, or experiences changes in illumination, providing a reliable basis for subsequent target association.

[0137] The third step (S3.3) is motion prediction. Based on the target depth feature set and historical trajectory data, the system builds a Kalman filter model for each target to predict its possible location in the next frame. The Kalman filter is a recursive estimator that updates the target state with minimum mean square error by combining the current state estimate and new measurements. In this system, the target state vector is defined as an 8-dimensional vector. ,in These are the coordinates of the target center. It's the aspect ratio. It's about height. These are the corresponding velocity components. The system uses a uniform motion model as the state transition equation: ,in It is the state transition matrix, which defines the relationships between the components of the state vector. It is process noise, assumed to follow a multivariate Gaussian distribution with a mean of zero, X(t) v ) for t v The state transition equation at time t, X(t) v +1) is t v The state transition equation at time +1. State transition matrix. The design ensures that position and dimensional parameters vary linearly according to their respective velocity components. The system will consider the preceding... Frame (usually) Historical data is used to adjust the Kalman filter's prediction model, making predictions more accurate. The process noise covariance matrix is ​​typically set as a diagonal matrix, with the main diagonal elements determined according to the uncertainty of the target's motion, assigning different weights to the position and velocity components. The measurement noise covariance matrix is ​​also a diagonal matrix, set according to the detector's accuracy. Through these parameter settings, the Kalman filter can predict the target's next state at each time step, generating a target predicted state set, providing prior information for subsequent target association.

[0138] The fourth step (S3.4) is multi-target association matching. Based on the target prediction state set and the detection results of the new frame, the system needs to determine which newly detected targets correspond to existing trajectories, which are newly appearing targets, and which existing trajectories do not have corresponding detection results in the current frame. This is a typical assignment problem, and the system uses the Hungarian algorithm (also known as the Kuhn-Munkres algorithm) to solve for the optimal match. First, the association cost between each pair of predicted targets and detection results is calculated. The cost function comprehensively considers two factors: one is spatial distance, usually represented by the intersection-over-union ratio (IoU) of the bounding boxes or the Euclidean distance between the center points; the other is appearance similarity, obtained by calculating the cosine distance between the depth feature vectors. The comprehensive cost calculation formula is:

[0139]

[0140] in and These are weight parameters (usually) , ), It is a prediction box and detection box The intersection and union ratio, It is the cosine distance of the feature vectors. All possible (prediction, detection) pairs constitute the cost matrix, and the Hungarian algorithm solves for the optimal matching of this matrix, that is, the allocation scheme with the minimum total cost.

[0141] During the association matching process, the system needs to handle three special cases: First, when a new target appears, if the minimum cost of the new detection result and all existing trajectories is greater than the preset threshold T1 (usually 0.8), the system creates a new trajectory for it, but requires detection for 3 consecutive frames to be confirmed as a stable trajectory to avoid noise interference; Second, when the target disappears, if there are no matching detection results for a trajectory for M' consecutive frames (M' is usually 30), the system marks it as disappeared and deletes it from the active trajectory set; Third, when there is temporary occlusion, if there are no matching results for a trajectory for a short period of time (less than M' frames), the system continues to use the prediction results of the Kalman filter to maintain the trajectory until the target reappears or the termination condition is met.

[0142] Furthermore, the system implements an ID switching prevention mechanism, monitoring and correcting potential ID swapping errors through temporal consistency checks of feature similarity. For example, when two trajectories separate after approaching each other, the system compares the changes in feature similarity before and after separation to determine if an ID swap has occurred and corrects it if necessary. Through this series of processes, the system can maintain a stable target trajectory in complex mining truck environments and obtain complete target mineral movement trajectory data. Experiments show that the tracking system maintains a tracking accuracy of over 95% even when the mining truck speed reaches 2 m / s, with an average ID swapping rate of less than 3%, meeting the requirements for accurate sampling.

[0143] Specifically, based on the motion trajectory data, an improved FAST corner detection algorithm is used to extract feature points from consecutive image frames. Feature matching is performed using the ORB descriptor and RANSAC algorithm. The Lucas-Kanade optical flow algorithm is applied to calculate the pixel-level displacement vector field, thereby obtaining the speed and displacement data of the mining truck, including:

[0144] S4.1 Based on the motion trajectory data and the continuous image frames, feature points in the image are extracted using the improved FAST corner detection algorithm, and feature representations are calculated using ORB descriptors to obtain feature description data;

[0145] To ensure stable tracking of the target mineral, the system needs to accurately estimate the motion parameters of the mining truck itself, providing crucial information for subsequent motion compensation and precise sampling. This process is achieved through visual odometry technology, which recovers the motion trajectory of the camera (i.e., the system) from a continuous sequence of images, thereby obtaining the speed and displacement data of the mining truck.

[0146] The first step (S4.1) is feature point extraction and description. Based on the previously obtained mineral motion trajectory data and continuous image frames, the system uses an improved FAST (Features from Accelerated Segment Test) corner detection algorithm to extract feature points from the image. The standard FAST algorithm determines corner points by detecting consecutive similar pixels on a ring around a pixel, which is fast but has limited stability. This system improves upon it in two aspects: first, it introduces an adaptive threshold selection mechanism, dynamically adjusting the judgment threshold based on the contrast of local image regions, making feature points more evenly distributed across the image; second, it adds a non-maximum suppression step to ensure the selection of the strongest corner response and avoid excessive clustering of feature points. The improved algorithm significantly improves the stability and distribution uniformity of corner points while maintaining detection efficiency (processing time per frame <10ms).

[0147] For detected feature points, the system uses the ORB (Oriented FAST and Rotated BRIEF) descriptor to calculate feature representations. The ORB descriptor combines an improved FAST corner descriptor with a rotation-invariant BRIEF descriptor to generate a 256-bit binary feature vector. The calculation process first determines the principal orientation of the feature point (by calculating the gray-level centroid of pixels surrounding the corner point), then adjusts the sampling mode according to the principal orientation, calculates the gray-level relationship between specific point pairs, and forms a binary description vector. This description method is computationally efficient (processing time for a single feature point <1ms) and has a certain degree of invariance to rotation, scale changes, and illumination changes, making it particularly suitable for real-time feature matching requirements in mining truck environments.

[0148] S4.2 For the feature description data, the similarity between feature points is calculated using Hamming distance, and the feature corresponding points are selected by the RANSAC algorithm to obtain the feature matching point set;

[0149] The second step (S4.2) is feature matching. For the extracted feature description data, the system uses Hamming distance to calculate the similarity between feature points. Hamming distance is defined as the number of bits that differ between corresponding bits in two binary vectors; it is computationally efficient and suitable for binary descriptors such as ORB. For each feature point, the system finds the two candidate matching points with the smallest Hamming distance and applies a nearest neighbor ratio test (the ratio threshold is usually set to 0.75) to initially screen matching pairs. That is, if the distance ratio between the best and second-best matching is less than the threshold, the match is accepted.

[0150] The initial matching results may still contain mismatches, so the system further applies the RANSAC (Random Sample Consensus) algorithm for geometric verification. The RANSAC algorithm searches for the largest consistent set that satisfies the geometric model (such as the fundamental matrix or homography matrix) through repeated random sampling. Specific implementation parameters include: an interior point threshold of 2-3 pixels, a maximum number of iterations of 1000, and a confidence level of 0.99. This process effectively eliminates mismatches that do not conform to the overall motion pattern, improving matching quality. After RANSAC screening, the system typically obtains 200-500 high-quality feature matching points. These matching points constitute a feature matching point set, providing a reliable foundation for subsequent optical flow calculations.

[0151] S4.3 Construct an image pyramid, calculate the optical flow of the feature matching point set layer by layer from coarse scale to fine scale, apply the Lucas-Kanade optical flow algorithm to calculate the pixel-level displacement vector field between consecutive frames, achieve sub-pixel accuracy motion estimation, and obtain the optical flow vector field;

[0152] The third step (S4.3) is optical flow calculation. To improve the accuracy and robustness of motion estimation, the system constructs a four-layer image pyramid (scaling factor of 0.5), calculating the optical flow of the feature matching point set layer by layer from coarse to fine scale. The pyramid structure can handle large displacement problems and adapts to scenarios with fast-moving mining trucks. On each layer of the pyramid, the system applies the Lucas-Kanade optical flow algorithm to calculate the precise displacement of the feature points. This algorithm is based on the assumption of invariant gray levels within local regions and obtains the feature point displacements by solving the optical flow equations:

[0153]

[0154] Where Ix and Iy are the image spatial gradients, It is the temporal gradient, and [u,v] is the displacement vector to be determined. In the actual implementation, an 11×11 or 15×15 local window is selected for each feature point, and the optical flow equations of all pixels within the window are calculated to form an overdetermined system of equations, which is then solved using the weighted least squares method. The weights are usually Gaussian functions to maximize the contribution of the central pixel. The algorithm also employs an iterative optimization strategy, typically iterating 3-5 times until the displacement update is less than a preset threshold (e.g., 0.01 pixels).

[0155] The calculation begins at the top of the pyramid (the coarsest scale), and the resulting displacement vector is magnified and used as the initial value for the next layer. The calculation is refined layer by layer, ultimately obtaining a sub-pixel precision optical flow vector field on the original resolution image. This coarse-to-fine strategy not only handles large displacements but also reduces local minima traps and improves computational stability. The resulting optical flow vector field describes the precise displacement of each feature point in the image from the previous frame to the current frame, with an accuracy reaching the sub-pixel level (typically within 0.1 pixels).

[0156] S4.4 Based on the optical flow vector field, the camera's rotation matrix and translation vector are recovered by solving the fundamental matrix or homography matrix, and the transformation relationship between the world coordinate system and the camera coordinate system is established to obtain the camera attitude parameters;

[0157] The fourth step (S4.4) is camera pose recovery. Based on the calculated optical flow vector field, the system uses epipolar geometry to recover the camera's motion parameters. First, an appropriate geometric model is adaptively selected according to scene characteristics: when the scene depth changes significantly, a fundamental matrix model is used; when the scene is approximately planar, a homography matrix model is used. The system compares the applicability of the two models by calculating the reprojection error and selects the model with the smaller error.

[0158] For the fundamental matrix model, the system uses the eight-point method or the normalized eight-point method for solution. After obtaining the fundamental matrix F, it is further decomposed to obtain the essential matrix E, and then the camera rotation matrix R and translation vector t are decomposed from E (there are at most four possible solutions, and the correct solution is selected by checking the depth sign of the triangulation points). For the homography matrix model, the system uses the direct linear transformation method for solution, and then obtains R and t through SVD decomposition. Regardless of the model used, the system performs nonlinear optimization on the initially recovered attitude, typically using the Levenberg-Marquardt algorithm to minimize the reprojection error and improve the attitude estimation accuracy.

[0159] The final recovered rotation matrix R (3×3) and translation vector t (3×1) together constitute the camera pose parameters, describing the rigid body transformation of the camera from the previous frame to the current frame. R represents the rotation angle and direction, and t represents the translation distance and direction. Together, they define the transformation relationship between the world coordinate system and the camera coordinate system.

[0160] S4.5 Based on the camera attitude parameters and system timestamp, perform displacement accumulation calculation between consecutive frames, and combine the wheel encoder data for fusion correction to eliminate accumulated errors, thereby obtaining the speed and displacement data of the mine car.

[0161] The fifth step (S4.5) is velocity and displacement calculation. Based on the recovered camera attitude parameters and system timestamps, the system calculates and accumulates the displacement between consecutive frames to obtain the complete trajectory of the mine car. Specifically, for each pair of consecutive frames in the time series... Calculate the relative displacement of the camera and relative rotation Then according to the time interval Calculate instantaneous velocity and angular velocity .

[0162] A major problem with visual odometry calculations is the accumulation of errors. As the number of frames increases, small errors accumulate, leading to long-term trajectory drift. To address this issue, the system incorporates data from mine car wheel encoders for fusion correction. Wheel encoders, directly mounted on the mine car wheels, provide high-precision distance and speed information (typically ±1mm), but are susceptible to wheel slippage. The system employs an extended Kalman filter (EKF) framework to fuse visual odometry and wheel encoder data. The state vector includes position, orientation, linear velocity, and angular velocity, while the measurement vector includes visual estimation results and encoder readings.

[0163] This multi-sensor fusion approach effectively eliminates accumulated errors and improves the accuracy of long-term trajectory estimation. Experiments show that when a mining truck travels 100 meters at a speed of 1.5 m / s, the system's accumulated position error is less than 0.4%, and its speed estimation accuracy is better than ±0.05 m / s, fully meeting the accuracy requirements for mining truck motion parameters in subsequent precise sampling. The system outputs mining truck speed and displacement data including three-dimensional position coordinates (x, y, z), three-dimensional attitude angles (roll, pitch, yaw), linear velocity, and angular velocity, with a temporal resolution consistent with the camera frame rate (typically 25-30 Hz), laying the foundation for the next stage of multi-sensor fusion and motion prediction.

[0164] Specifically, based on the velocity and displacement data, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned to reconstruct the three-dimensional spatial position of the target mineral. A nonlinear dynamic model is established, and an extended Kalman filter is used to fuse the multi-sensor data and predict the future position of the target mineral, resulting in target spatial coordinate data, including:

[0165] S5.1 Based on the velocity and displacement data, a set of nonlinear partial differential equations describing the motion of minerals on the conveyor belt is constructed. The dynamic system is decomposed into principal terms and perturbation terms. The principal terms adopt Newton's equations of motion, and the perturbation terms are learned from the measured data through a neural network to capture complex nonlinear effects that the theoretical model fails to describe, thus obtaining the perturbation PINN dynamic model.

[0166] S5.2 Based on the aforementioned perturbation PINN dynamics model, a hierarchical neural network architecture is designed. Different layers of the network are responsible for modeling the physical processes at different time and space scales, from millisecond-level particle collisions to second-level overall motion. The physical consistency between models at each scale is ensured through loss function design, resulting in a multi-scale PINN model.

[0167] S5.3 Based on the multi-scale PINN model, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned, and the target mineral in the two-dimensional image is reconstructed into its position and volume information in three-dimensional space through projection mapping relationship, thus obtaining a three-dimensional spatial representation of the target mineral.

[0168] S5.4 Based on the three-dimensional spatial representation of the target mineral, a nonlinear dynamic model considering the acceleration, deceleration, vibration, and steering factors of the mining truck is established to describe the motion law of the target mineral relative to the sampling robotic arm, and the dynamic model parameter set is obtained;

[0169] S5.5 Based on the dynamic model parameter set, an extended Kalman filter is used to fuse visual measurement, laser ranging, and IMU data. When the statistical estimate deviates from the physical law, it is automatically corrected to ensure that the state estimate conforms to the basic physical law. The position of the target mineral in the next 0.5 to 1 second is predicted to obtain the target spatial coordinate data.

[0170] Multi-sensor fusion and spatial coordinate prediction are the core components for achieving accurate sampling. This process fuses data from different sensors to establish an accurate mineral dynamics model, predicts the future location of minerals, and provides accurate target coordinates for robotic arm sampling. The entire process consists of five key steps, overcoming the limitations of traditional methods in complex dynamic environments through physical information neural networks and multi-sensor fusion technology.

[0171] The first step (S5.1) is to establish a perturbation PINN dynamic model. Based on the previously obtained mine car velocity and displacement data, the system first constructs a set of nonlinear partial differential equations describing the motion of the minerals. Traditional dynamic models are difficult to accurately describe the complexity of the mineral motion on the mine car; therefore, an innovative Physical-Informed Neural Networks (PINN) method is adopted to decompose the dynamic system into principal terms and perturbation terms. The principal terms describe the basic motion laws under ideal conditions based on Newton's equations of motion, such as the basic trajectory of the mineral moving with the mine car under ideal conditions, expressed as:

[0172]

[0173] Where X'' represents acceleration. It is a force. It refers to mass. The perturbation term, on the other hand, is learned from measured data through a neural network. It is used to capture complex nonlinear effects that theoretical models fail to describe, such as mine car vibration, mineral interactions, and the effects of irregular tracks. The neural network for the perturbation term adopts a five-layer fully connected structure with hidden layer nodes of [64, 128, 256, 128, 64] and uses GELU as the activation function, enabling it to learn complex nonlinear mapping relationships.

[0174] The complete perturbation PINN model can be represented as:

[0175]

[0176] in Predicted values ​​calculated based on physical laws This is the output of the neural network. The model training uses a hybrid loss function:

[0177]

[0178] in Ensure the model conforms to the physical equations (physical consistency loss). To ensure that model predictions match observed data (data fitting loss), weight parameters and To balance the contributions of the two losses, it is typically set to... , The training data includes several hours of mineral movement trajectories recorded by high-precision sensors, covering various operating conditions. Training uses the Adam optimizer with an initial learning rate of 0.001, and a cosine annealing strategy is used to gradually reduce the learning rate. A total of 10,000 training rounds are completed, which takes approximately 8 hours on an RTX 3090 GPU.

[0179] The advantage of the perturbation PINN method lies in its ability to maintain the interpretability and generalization capability of the physical model while utilizing neural networks to capture complex nonlinear dynamic characteristics. This makes it more suitable for describing the complex dynamic behavior of minerals on mining trucks than purely physical or purely data-driven models. Experiments show that this model can reduce prediction errors by more than 40%, especially under nonlinear conditions such as acceleration, deceleration, and turning of the mining truck.

[0180] The second step (S5.2) is to construct a multi-scale PINN model. The movement of minerals on the mining cart involves physical phenomena at multiple temporal and spatial scales, ranging from millisecond-level particle collisions to second-level overall motion. To capture these multi-scale characteristics, the system further designs a hierarchical neural network architecture based on the perturbation PINN model. This architecture contains three layers of networks: a micro-network (sampling rate 100Hz) responsible for modeling high-frequency dynamic characteristics such as particle collisions and local vibrations; a meso-network (sampling rate 30Hz) capturing mineral interactions and local environmental changes; and a macro-network (sampling rate 10Hz) modeling the overall motion trend and long-term scale changes.

[0181] Each layer of the network employs a similar basic structure, but differs in input features, network depth, and training data. To ensure physical consistency between models at different scales, the system designs a special cross-scale consistency loss function:

[0182]

[0183] L-scale ensures the consistency of predictions from models of different scales in overlapping regions, achieved by calculating the differences in prediction results from different scale models at a common time point. Weight parameters =1.0, =0.8, =0.5, which reflects the relative importance of physical consistency, data fit and cross-scale consistency.

[0184] The training strategy employs a coarse-to-fine approach: first, a macroscopic network is trained to capture overall motion characteristics; then, some parameters are frozen to train a mesoscopic network; and finally, a microscopic network is trained to capture high-frequency details. This ensures that each layer focuses on the dynamic characteristics at its own scale while maintaining overall consistency. The multi-scale PINN model can simultaneously predict mineral motion characteristics at different time scales, meeting varying prediction requirements in terms of accuracy and timeliness. It is particularly suitable for high-precision operations such as mineral sampling that require millimeter-level accuracy.

[0185] The third step (S5.3) is multi-sensor spatiotemporal alignment and 3D reconstruction. The system achieves time synchronization of multi-sensor data through hardware triggering and software timestamp calibration. Hardware triggering uses PPS (Pulse Per Second) signals to synchronize the acquisition time of the camera and LiDAR, with an accuracy of microseconds; software timestamps achieve cross-device time unification based on the CPU clock, and maintain clock synchronization of each device through NTP service, with time deviation controlled within 1ms.

[0186] Spatial alignment is achieved by determining the rigid body transformation relationships between sensors through extrinsic parameter calibration. Calibration utilizes a specially designed calibration board simultaneously observed by both the camera and LiDAR, calculating the transformation matrix by minimizing reprojection error. The calibration results show a rotation error of less than 0.5 degrees and a translation error of less than 5 mm. The calibration of the IMU and vision system involves simultaneously recording data in both static and dynamic states, and solving for the relative pose relationship using maximum likelihood estimation.

[0187] After spatiotemporal alignment, the system combines the target detection results from the camera and the depth measurement from the lidar to reconstruct the 3D position of the target mineral through projection mapping. Specifically, the detected target bounding box is projected onto the lidar point cloud, the point set of the corresponding region is extracted, and a clean target point cloud is obtained through Euclidean clustering and statistical outlier removal. Then, the geometric center, volume, and principal axis orientation of the point cloud are calculated to form a 3D spatial representation of the mineral. The spatial positioning accuracy of this process can reach ±2mm, fully meeting the sampling accuracy requirements.

[0188] The fourth step (S5.4) is to establish a nonlinear dynamic model. Based on the three-dimensional spatial representation of the target mineral, a nonlinear dynamic model considering the acceleration, deceleration, vibration, and steering factors of the mine car is established to describe the motion of the target mineral relative to the sampling robotic arm. This model comprehensively considers the following factors: the motion state of the mine car (speed, acceleration, steering angle, etc.), mineral characteristics (mass, shape, coefficient of friction, etc.), environmental factors (mine roadway slope, curvature, etc.), and interaction forces (collision and friction between minerals, etc.).

[0189] The model uses a system of differential equations to describe mineral motion, including state variables such as position, velocity, and acceleration, as well as various forces and constraints. For different types of minerals (such as massive ore and powdered ore), the system establishes different sets of dynamic parameters, including mass distribution, friction coefficient, and coefficient of restitution. These parameters are initially estimated through offline analysis of historical data, and then updated and optimized online to ensure the model can adapt to different mineral types and operating conditions. The accuracy of the dynamic model directly affects the prediction accuracy. Experiments show that the model can control the prediction error within ±1 mm, laying the foundation for subsequent accurate predictions.

[0190] The fifth step (S5.5) is multi-sensor fusion and position prediction. Based on the established dynamic model, the system uses an extended Kalman filter (EKF) to fuse visual measurements, laser ranging, and IMU data to predict the future position of the target mineral. The EKF state vector includes the mineral's three-dimensional position, velocity, acceleration, and attitude parameters. The state transition equation is based on the aforementioned dynamic model, while the observation equation is established based on the measurement characteristics of each sensor.

[0191] The system assigns different confidence weights to data from different sensors. For example, visual data has a higher weight in well-lit conditions, while laser data has a higher weight in poorly lit conditions but when the mineral surface is highly reflective. The EKF prediction step uses a dynamic model to predict the state at the next moment, and the update step incorporates new observation data to correct the prediction. To improve robustness, the system integrates physical constraints into the filtering framework. When the statistical estimate deviates significantly from physical laws (e.g., the predicted position does not conform to energy conservation or kinematic constraints), the system automatically corrects it to ensure that the state estimate conforms to fundamental physical laws.

[0192] The system can predict the mineral's location within 0.5 to 1 second. The prediction time span is dynamically adjusted based on the mine car's speed and system latency to ensure prediction accuracy meets sampling requirements. At a mine car speed of 1.5 m / s, the system's prediction error for the location 0.5 seconds later is less than ±1.5 mm, fully meeting the requirements for precise sampling. The entire prediction process runs on an edge computing platform (such as NVIDIA Jetson Xavier), with a computation latency of less than 15 ms, and can output prediction results at 60 Hz in real time. The final target spatial coordinate data includes the mineral's three-dimensional position, orientation, volume, and corresponding uncertainty estimates, providing accurate target information for subsequent robotic arm trajectory planning and control.

[0193] Based on the target spatial coordinate data, a fifth-order polynomial interpolation method is used to generate joint spatial trajectories and perform time-optimized control optimization. Dynamic compensation control coordinates the movement of each joint of the robotic arm, executing precise sampling operations, including:

[0194] S6.1 Based on the target space coordinate data and the DH parameters and joint constraints of the robotic arm, establish forward and inverse kinematic models, calculate the mapping relationship between the end effector position and the joint angle, define the workspace and singular position, and obtain the kinematic model of the robotic arm;

[0195] S6.2 Based on the kinematic model of the robotic arm, the joint angles, velocities and accelerations are interpolated using the fifth-order polynomial interpolation method to generate a smooth joint space trajectory and obtain the initial trajectory parameters;

[0196] S6.3 Apply the time-optimal control algorithm to the initial trajectory parameters to optimize the trajectory execution time while satisfying the joint velocity and acceleration constraints, and obtain the optimal trajectory parameters;

[0197] S6.4 Based on the optimal trajectory parameters, a dynamic model is established considering the mass, inertia tensor and friction coefficient of each joint of the robotic arm. Feedforward compensation control is implemented to reduce tracking errors during high-speed motion. An adaptive impedance control strategy is used to cope with changes in contact force during sampling and to obtain joint torque control commands.

[0198] S6.5 Based on the joint torque control command, the movement of each joint of the robotic arm is coordinated and controlled by a high-performance servo driver and a real-time control system to achieve precise positioning of the end effector. When the target mineral reaches the predetermined position, the sampler opening and closing, depth and sampling amount are controlled to perform the precise sampling operation.

[0199] Based on accurate target spatial coordinate data, the system needs to control the robotic arm to perform precise sampling operations. This process involves multiple stages such as robotic arm kinematic modeling, trajectory planning, and dynamic control. It requires finding the optimal balance between high speed and high precision to ensure that the target mineral sample is accurately obtained during the movement of the mining truck.

[0200] The first step (S6.1) is to establish the kinematic model of the robotic arm. Based on the target spatial coordinate data and the physical parameters of the robotic arm, the system uses the Denavit-Hartenberg (DH) parameter method to establish forward and inverse kinematic models. For a 6-DOF robotic arm, the DH parameters for each joint include link length (a), link torsion angle (α), link offset (d), and joint angle (θ). These parameters fully describe the spatial relationship between adjacent joints.

[0201] The forward kinematics model describes the mapping relationship between joint angles and end effector positions. By multiplying the transformation matrices of each joint, the position and orientation of the end effector under any joint configuration can be calculated.

[0202] T' = A1 · A2 · ... · A6

[0203] Where T' is a 4×4 homogeneous transformation matrix, representing the pose of the end effector relative to the base, and Ai is the transformation matrix of the i-th joint, determined by the corresponding DH parameters.

[0204] Inverse kinematics addresses the inverse problem, namely, given the target pose of the end effector, calculating the joint angles required to achieve that pose. This system employs a strategy combining analytical and numerical iterative methods: first, analytical solutions for the first three joints (primarily controlling position) are obtained through geometric analysis; then, the angles for the last three joints (primarily controlling attitude) are solved using the Jacobi iterative method. This hybrid approach ensures both computational efficiency (typically <2ms) and sufficient robustness to handle singular positions and multiple solutions.

[0205] The system also needs to define the workspace and identify singular positions. The workspace is the set of all positions reachable by the robotic arm's end effector, determined through Monte Carlo sampling and verification. Singular positions are special joint configurations that reduce the rank of the Jacobian matrix, causing the robotic arm to lose degrees of freedom in certain directions. The system identifies potential singular positions by analyzing the condition number of the Jacobian matrix and actively avoids these regions during trajectory planning, ensuring smooth and controllable motion.

[0206] The physical constraints of each joint of the robotic arm are also an important part of the model, including joint angle constraints (such as the angle range of the i-th joint). ), joint speed limit ( ) and joint acceleration limitation ( These limitations, determined by the physical structure of the robotic arm and the performance of the drive system, must be strictly adhered to during trajectory planning to ensure safe operation. Fifth-order polynomial trajectories possess continuous acceleration characteristics, effectively reducing shock and vibration during robotic arm movement, improving motion accuracy and the lifespan of the mechanical system. This smoothness is particularly important for precise operations such as sampling tasks. The system generates a trajectory for each joint separately, initially setting a uniform execution time T to obtain initial trajectory parameters.

[0207] The third step (S6.3) is trajectory time optimization. The system applies a time-optimal control algorithm to optimize the generated initial trajectory, aiming to minimize the trajectory execution time while satisfying joint velocity and acceleration constraints. This is a constrained optimization problem, expressed as:

[0208]

[0209] The actual solution uses a binary search method: starting from the theoretical shortest possible time... Start by calculating based on the maximum angular displacement and maximum velocity limits, then gradually increase the time until the shortest time that satisfies all constraints is found. For each candidate time The system recalculates the fifth-order polynomial coefficients and then uniformly samples the trajectory to check whether the velocity and acceleration of all joints meet the constraints.

[0210] The optimized trajectory not only completes the task in the shortest possible time but also ensures smooth and controllable joint movements without violating the physical limitations of the mechanical system. For typical sampling operations, the optimized execution time is usually in the range of 0.5-1 seconds, depending on the distance between the start and end points and the performance of the robotic arm.

[0211] Step four (S6.4) is the design of the dynamic control scheme. To improve tracking accuracy, a detailed dynamic model of the robotic arm is established, considering parameters such as the mass, inertia tensor, and friction coefficient of each joint. Based on the Lagrangian method, the dynamic equation is expressed as:

[0212]

[0213] in It is the inertia matrix. Represents the Coriolis force and centrifugal force terms. It is the gravity term. It is the friction term. This refers to the joint driving torque. These parameters are obtained through the robotic arm's CAD model, manufacturing parameters, or experimental calibration.

[0214] Based on the dynamic model, the system implements feedforward compensation control, calculates the theoretical torque required to achieve the desired trajectory, and reduces tracking errors during high-speed motion. The control law is designed as follows:

[0215]

[0216] in , and These are the desired position, velocity, and acceleration (obtained through trajectory planning). and This is the position and velocity feedback gain matrix (usually a diagonal matrix, determined by system characteristics). The first four terms constitute feedforward compensation, and the last two terms constitute error feedback, together ensuring that the robotic arm accurately tracks the desired trajectory. To address contact force changes during the sampling process, the system implements an adaptive impedance control strategy, dynamically adjusting control parameters based on the interaction force between the end effector and the environment.

[0217] The system adaptively adjusts control parameters based on real-time force sensor feedback to ensure both accuracy and safety in the sampling operation. The final output joint torque control command includes position, velocity, and force control components, providing precise drive commands for each joint.

[0218] The fifth step (S6.5) is to perform precise sampling. Based on joint torque control commands, the system coordinates the movement of each joint of the robotic arm through a high-performance servo driver and a real-time control system to achieve precise positioning of the end effector. The control system adopts EtherCAT bus technology with a control cycle of 1kHz, ensuring accurate real-time performance and low jitter control.

[0219] When the target mineral reaches the predetermined location, the system controls the sampler to perform the sampling operation, including precise control of the sampler's opening and closing angle, insertion depth, and sample volume. The sampler is designed as a dedicated mechanical gripping device, capable of accommodating mineral samples of different shapes and hardnesses. The gripping force is adjustable from 5-50N, adapting to various sampling needs from loose ores to hard ores. The insertion depth is calculated in real time based on the mineral volume and location, typically controlled within the range of 10-50mm to ensure the acquisition of representative mineral samples.

[0220] During sampling, the system continuously monitors the force sensor feedback and promptly detects any abnormalities. If excessive resistance or sudden changes in mineral position are detected, the system will automatically adjust its strategy or terminate the operation to prevent equipment damage. After sampling is complete, the system controls the robotic arm to transfer the sample to the sample collection area, ready for the next sampling operation.

[0221] The entire precision sampling process is highly automated, requiring no manual intervention from target identification to sample collection. It can achieve a sampling position accuracy of ±2mm when the mine truck is traveling at a speed of 1.5m / s, with a sampling success rate of 98%. Compared with traditional manual sampling or fixed-position mechanical sampling, this system significantly improves sampling efficiency and sample representativeness, providing an efficient and accurate intelligent sampling solution for mining production.

[0222] In one embodiment, the method further includes:

[0223] A1. Modify the loss function of the deep convolutional neural network from L2 norm to L1 norm. For pixel-level segmentation tasks, define the loss function as the sum of the absolute differences between the predicted probability map and the true label. For object detection tasks, modify the bounding box regression loss to L1 loss to obtain the L1 loss function configuration.

[0224] A2. Based on the L1 loss function configuration, random noise conforming to a Gaussian distribution is introduced into the input image during the training process. The noise parameters are set to mean μ=0 and standard deviation σ=5-15 pixel values. The label is kept unchanged to force the model to learn to extract essential features from the noise data, thus obtaining noise-enhanced training data.

[0225] Based on the noise-enhanced training data, A3 minimizes the sum of absolute deviations in the local neighborhood and finds the pixel value that best represents the real mineral surface features through iterative optimization. While preserving the mineral edge and texture features, it suppresses noise interference and obtains L1 optimized feature data.

[0226] A4 optimizes the L1 feature data by adding a penalty term for the sum of the absolute values ​​of the weights to the loss function. The regularization strength parameter λ ranges from 0.001 to 0.01, guiding the network to automatically select the most representative mineral features to obtain sparse feature representation.

[0227] Based on the sparse feature representation, A5 performs basic feature learning training for 100 rounds with an initial learning rate of 0.001, reduces the learning rate to 0.0001 for fine-grained classification training for 50 rounds, and performs end-to-end joint training for 30 rounds using a cosine annealing learning rate strategy to obtain the mineral recognition model.

[0228] In the optimization research of mineral identification models, researchers proposed two different implementation schemes to improve system performance from two directions: robust optimization and physically-informed deep learning. These methods offer innovative solutions to specific challenges in mining truck environments, such as noise interference, lighting variations, and the diversity of mineral properties, significantly improving the model's identification accuracy and generalization ability in complex environments.

[0229] The first approach employs L1 norm optimization and sparse feature representation techniques to improve the model's noise resistance from a statistical robustness perspective. Firstly (A1), researchers modified the loss function of the deep convolutional neural network from L2 norm (mean squared error) to L1 norm (mean absolute error). For pixel-level segmentation tasks, the loss function is defined as the sum of the absolute differences between the predicted probability map and the true label, expressed as:

[0230]

[0231] Where N is the total number of pixels. It is a probability plot predicted by the model. These are the true labels. For object detection tasks, the bounding box regression loss is modified from the traditional squared error to absolute error:

[0232]

[0233] in and These represent the coordinates of the predicted and ground truth bounding boxes, respectively. Compared to L2 loss, L1 loss is less sensitive to outliers and large errors, thus performing more robustly in noisy mineral images. Experiments show that this simple modification can improve the model's robustness in noisy environments by about 15%, especially when dealing with mineral samples with blurred boundaries and complex textures.

[0234] The second step (A2) is noise-enhanced training. Researchers introduce random noise conforming to a Gaussian distribution into the input images during the training process, with the noise parameter set to the mean. Standard deviation Pixel values. The key innovation is to keep the labels unchanged and force the model to learn to extract essential features from noisy data. This method simulates various noise sources in a real mining truck environment, such as sensor noise, dust interference, and changes in lighting. Noise intensity. The value is dynamically adjusted based on different mineral types and training stages; for example, a lower noise level is used for minerals with smooth surfaces. For minerals with complex textures, a higher noise level is used. The training process employs a gradual increase in noise strategy, starting with a low noise level and gradually increasing it to the target level, allowing the model to progressively adapt to the noisy environment. This strategy prevents the model from converging prematurely to a local optimum and improves its adaptability to noise.

[0235] The third step (A3) is L1 optimization feature extraction, which minimizes the sum of absolute biases within the local neighborhood and finds the pixel values ​​that best represent the true mineral surface features through iterative optimization. This method is based on the principle of median filtering, but is further extended to a weighted absolute bias minimization problem:

[0236]

[0237] in These are the optimized feature values. It is the pixel value within the neighborhood. The weights are determined by a similarity function. This process effectively suppresses noise interference while preserving mineral edge and texture features. A 5×5 or 7×7 layout is used in the implementation.

[0238]

[0239] in It is the original task loss (classification or detection). It is the L1 norm of the network weights. It is a constant.

[0240] Example 1:

[0241] As shown in Figure 1, this embodiment implements a deep learning-based method for dynamic mineral identification and precise sampling. The method first acquires mineral images and lidar ranging data from a mining truck in the underground environment, and obtains high-quality mineral feature data after preprocessing. Then, it performs mineral identification through a deep convolutional neural network that fuses multi-scale features. Next, it achieves multi-target dynamic tracking, visual odometry calculation, and multi-sensor fusion position prediction. Finally, it uses an intelligent robotic arm to perform precise sampling operations.

[0242] During the data acquisition phase, an industrial camera with a resolution of 2048×1536 pixels and a lidar with a measurement accuracy of ±2mm were used to collect mineral sample data from mining trucks in an underground environment (light conditions of 10-500 lux and dust concentration of 0.5-10 mg / m³). More than 10,000 mineral images were collected under different working conditions, covering various mineral types including sulfide ores, oxide ores, and waste rock.

[0243] For the acquired image data, the spatial distance and pixel value similarity of the images are first calculated, and then a bilateral filtering algorithm is applied to perform weighted averaging on the neighborhood of each pixel.

[0244] For the denoised image, the cumulative distribution function is calculated and pixel values ​​are remapped. Then, an adaptive histogram equalization algorithm is applied to enhance local contrast. For laser point cloud data, statistical outlier filtering is used to remove outliers, and then voxel downsampling is performed to reduce the data size. The voxel size is set to 0.01m.

[0245] The preprocessed data was pixel-level annotated by a team of professional geological experts, marking bounding boxes and semantic segmentation masks for different mineral types. To increase the diversity of training samples, the annotated data underwent augmentation, including: random rotation within ±30°, random scaling within ±20%, brightness and contrast adjustment within ±10%, and the addition of Gaussian noise (μ=0, σ=10). After data augmentation, the training set was expanded to 3.5 times its original size, comprising approximately 35,000 images.

[0246] Example 2:

[0247] Unlike S2.1-2.3, in another embodiment, the design of a deep convolutional neural network architecture for multi-scale feature fusion further includes:

[0248] B1. Based on the mineral dataset and the physical properties of mineral spectral reflectance, crystal structure and chemical composition, mineralogical laws are encoded as constraints of the network structure. A physical awareness module and a data-driven module are designed, and the two parts are coupled through a physical consistency loss function to obtain a physical awareness hybrid network architecture.

[0249] B2. Using the aforementioned physical-informed hybrid network architecture, the high-dimensional input is decomposed into multiple one-dimensional functions, and N parallel sub-networks are designed to be responsible for the extraction of color, texture and morphological features respectively. The outputs of the N parallel sub-networks are integrated through adaptive weighted fusion to obtain a parallel Kolmogorov-Arnold network structure, where N is a positive integer greater than 1.

[0250] B3. Based on the parallel Kolmogorov-Arnold network structure, subdomains are divided according to mineral categories including sulfide minerals, oxide minerals and silicate minerals, and physical properties including reflectivity, color and hardness. A dedicated subnetwork is configured for each subdomain, and information exchange between subdomains is maintained through overlapping regions and boundary conditions to obtain the domain decomposition network configuration.

[0251] B4. Based on the domain decomposition network configuration, design a multi-objective loss function that includes standard classification loss, physical consistency loss, inter-domain consistency loss and uncertainty estimation loss, and balance data-driven learning with physical law constraints to obtain the physical constraint loss function;

[0252] B5. Based on the physical constraint loss function, an improved ResNet-50 is used as the backbone network and integrated with a feature pyramid network structure for multi-stage training to obtain the mineral identification model.

[0253] This embodiment details the design and training process of a mineral identification model based on a physically informed hybrid parallel Kolmogorov-Arnold network architecture. This model combines mineralogical physical properties with the advantages of deep learning, significantly improving identification accuracy.

[0254] First, a physical-information hybrid network architecture was designed based on mineralogical knowledge, encoding the spectral reflectance, crystal structure, and chemical composition characteristics of minerals as constraints for the network structure. Specifically, two parallel modules were created: one is a physical-information module built based on the physical properties of minerals, and the other is a data-driven feature extraction module. These two modules are coupled through a physical consistency loss function to ensure that the model's prediction results conform to the fundamental laws of mineralogical science.

[0255] Based on the Kolmogorov-Arnold representation theorem, four parallel sub-networks were designed, each responsible for extracting the color, texture, morphology, and reflective properties of the minerals. Each sub-network employs a similar basic structure, including five convolutional layers and two fully connected layers, but differs in hyperparameters such as kernel size and activation function to accommodate different feature extraction requirements. Finally, the outputs of each sub-network are fused using an adaptive weighting mechanism, where the weights are determined through network self-learning.

[0256] Furthermore, problem domains are divided according to mineral type (sulfide minerals, oxide minerals, and silicate minerals) and physical properties (reflectivity, color, and hardness), with a dedicated sub-network configured for each sub-domain. Information exchange between domains is maintained by setting a 20% overlap area and sharing underlying features.

[0257] A three-stage training strategy was employed: the first stage involved 100 epochs of training with a learning rate of 0.001 to learn basic mineral categories; the second stage involved 50 epochs of training with the learning rate reduced to 0.0001 to optimize fine-grained classification of sulfide and oxide ores; and the third stage involved 30 epochs of end-to-end joint training using a cosine annealing learning rate to optimize overall performance. The training hardware consisted of four servers equipped with NVIDIA A100 GPUs, employing a distributed training framework, with each epoch taking approximately 20 minutes.

[0258] Testing showed that the model achieved a mineral recognition accuracy of 98.5% on the test set, an improvement of 8 percentage points compared to traditional CNN models, maintaining high accuracy even under complex lighting and partial occlusion conditions. The model is approximately 200MB in size and, after TensorRT optimization, can achieve an inference speed of 30fps on edge computing devices, meeting real-time processing requirements.

[0259] Example 3:

[0260] This embodiment describes the implementation process of a multi-target dynamic tracking system. This system can stably track multiple mineral targets while the mining truck is moving at high speed, providing accurate trajectory information for subsequent precise sampling.

[0261] First, the mineral recognition model trained in Example 2 is used to detect targets in the input video stream, identifying all mineral targets in the current frame. The detection results include the bounding box coordinates, category identifier, and confidence score of the target. The system assigns a unique ID to new detected targets with a confidence score exceeding 0.75 and records their location, size, category, and appearance features, establishing an initial target state set.

[0262] Next, a pre-trained feature extraction network (based on a ResNet-50 backbone network, with the last fully connected layer replaced by a 128-dimensional feature vector output layer) is used to compute depth appearance features for each detected target. These features have strong discriminative power and remain stable even when the target is partially occluded or the lighting changes.

[0263] To handle target appearance, disappearance, and occlusion, the system has the following rules set:

[0264] 1. New target processing: When the association cost between the detection result and any existing trajectory is greater than the threshold T1 (0.8), a new trajectory is created, but it needs to be detected for 3 consecutive frames to be confirmed as a stable trajectory;

[0265] 2. Trajectory Termination: When a trajectory has no matching detection result in M ​​consecutive frames (M=30), it is marked as disappeared and deleted;

[0266] 3. Occlusion handling: When there is no matching result for the trajectory within a short period of time (less than M frames), continue to use the prediction of the Kalman filter to maintain the trajectory until the target reappears.

[0267] The system also implements trajectory smoothing and ID swap detection mechanisms. By comparing the depth feature similarity between the current frame and historical frames, it detects and corrects possible ID swap errors. The final output target mineral motion trajectory data includes the ID, category, location history, and predicted location of each target, with a temporal resolution of 30Hz (consistent with the camera frame rate).

[0268] Experimental verification shows that the multi-target tracking system can maintain a tracking accuracy of over 95% even when the mine car speed reaches 2m / s, with an average ID exchange error rate of less than 3%, and can effectively handle complex scenarios such as dense targets, partial occlusion, and changes in lighting.

[0269] Example 4:

[0270] This embodiment describes in detail the implementation method of the visual odometry and motion estimation system, which accurately calculates the speed and displacement data of the mine car by analyzing the motion of feature points in continuous image frames.

[0271] First, based on the target mineral trajectory data obtained in Example 3, feature points in consecutive image frames are extracted using an improved FAST corner detection algorithm. The standard FAST algorithm determines corners by comparing the difference between a pixel and its 16 surrounding pixels. This example improves upon this by introducing an adaptive threshold selection mechanism that automatically adjusts the threshold based on local image contrast, and by adding a non-maximum suppression step to ensure the selection of the strongest corner response. In the experiment, for a 1920×1080 resolution image, approximately 1000-2000 feature points were extracted per frame, with a computation time of less than 10ms.

[0272] For the extracted feature points, the ORB (Oriented Fast and Rotated BRIEF) descriptor is used to calculate the feature representation. The ORB descriptor is a 256-bit binary vector, obtained by calculating the intensity relationship between the pixel pairs surrounding the feature point. It is computationally efficient and rotation-invariant. During feature calculation, the principal orientation of the feature point is first determined, and then the sampling mode is adjusted according to the principal orientation to ensure rotation invariance.

[0273] Based on the calculated optical flow vector field, the system uses epipolar geometry to solve for the fundamental matrix (when the scene depth changes significantly) or the homography matrix (when the scene is approximately planar), from which it recovers the camera's rotation matrix R and translation vector t. The specific algorithm is adaptively selected according to the scene characteristics, and the optimal model is determined by calculating the reprojection error.

[0274] Finally, based on camera attitude parameters and system timestamps, displacement accumulation calculations are performed between consecutive frames. To prevent cumulative errors, the system combines mine wheel encoder data (accuracy ±1mm) for fusion correction. The correction adopts the EKF (Extended Kalman Filter) framework, with the state vector including position, orientation, linear velocity, and angular velocity, and the measurement vector including visual odometry estimation and wheel encoder readings.

[0275] Testing showed that when the mining truck traveled 100 meters at a speed of 1.5 m / s, the system's cumulative position error was less than 0.4%, and its speed estimation accuracy was better than ±0.05 m / s, fully meeting the accuracy requirements for motion estimation in mineral sampling. Running on an Intel Core i7 processor, the system achieved a real-time performance of 25 fps when processing 1920×1080 resolution video.

[0276] Example 5:

[0277] This embodiment implements a multi-sensor fusion motion compensation prediction system, which accurately predicts the spatial location of target minerals in the future by fusing visual, lidar, and IMU data.

[0278] First, based on the mine car speed and displacement data obtained in Example 4, the system achieves time synchronization of multi-sensor data through hardware triggering and software timestamp calibration. Hardware triggering uses PPS (Pulse Per Second) signals to synchronize the acquisition time of the camera and lidar, with an accuracy of microseconds; software timestamps achieve cross-device time unification based on the CPU clock, and maintain clock synchronization of each device through NTP service, with time deviation controlled within 1ms.

[0279] For spatial alignment, the rigid body transformation relationships between the sensors are determined through extrinsic parameter calibration. The calibration employs the target plate method, where specially designed calibration plates are placed at different locations and simultaneously observed by both the camera and lidar. The transformation matrix is ​​calculated by minimizing the reprojection error. The calibration results show a rotation error of less than 0.5 degrees and a translation error of less than 5 mm.

[0280] After spatiotemporal alignment, the system combines the target detection results from the camera and the depth measurement from the lidar to reconstruct the three-dimensional position of the target mineral through projection mapping. The specific method is as follows: the detected target bounding box is projected onto the lidar point cloud, the point set of the corresponding region is extracted, and the accurate target point cloud is obtained through clustering and outlier removal. Then, the centroid position and size information are calculated.

[0281] Based on the aforementioned dynamic model, the system uses an extended Kalman filter (EKF) to fuse visual measurements, laser ranging, and IMU data to predict the future position of the target mineral. The EKF's state vector includes position, velocity, acceleration, and attitude parameters, while the process noise and observation noise covariance matrices are set according to the actual sensor performance. To improve robustness, the system integrates physical constraints into the filtering framework, automatically correcting for deviations in statistical estimates from physical laws.

[0282] The system can predict the target position within 0.5-1 second. At a mine car speed of 1.5 m / s, the prediction error is less than ±1.5 mm, fully meeting the requirements for high-precision sampling. Running on an edge computing platform equipped with Jetson Xavier NX, the system takes approximately 15 ms and can output prediction results at 60 Hz in real time.

[0283] Example 6:

[0284] This embodiment describes in detail the implementation method of the intelligent robotic arm trajectory planning and control system. The system controls the robotic arm to perform high-precision sampling operations based on accurate target spatial coordinate prediction.

[0285] First, based on the target space coordinate data obtained in Example 5, a kinematic model of the robotic arm is established. This embodiment uses a 6-DOF robotic arm, and a forward kinematic model is established using the DH parameter method to describe the mapping relationship between joint angles and end effector positions.

[0286] Inverse kinematics employs a strategy combining analytical and numerical iteration methods. First, analytical solutions for the first three joints are obtained through geometric analysis. Then, the Jacobi iteration method is used to solve for the angles of the last three joints. This method ensures both computational efficiency (average solution time) and... It has sufficient robustness to handle singular positions and multiple solutions.

[0287] coefficient to The trajectory is determined by satisfying the position, velocity, and acceleration boundary conditions at the initial and final moments. This fifth-order polynomial trajectory exhibits continuous acceleration characteristics, effectively reducing impact and vibration during the robotic arm's movement. For the generated initial trajectory...

[0288] Finally, the movement of each joint of the robotic arm is coordinated and controlled by a high-performance servo driver and a real-time control system to achieve precise positioning of the end effector. The control system adopts EtherCAT bus technology with a control cycle of 1kHz, ensuring accurate real-time performance. When the target mineral reaches the predetermined position, the system controls the sampler to perform sampling operations, including precise control of opening and closing, depth, and sampling volume, to ensure the acquisition of representative mineral samples.

[0289] Tests have shown that the system can achieve a sampling position accuracy of ±2mm when the mining truck is traveling at a speed of 1.5m / s, with a sampling success rate of 98%, which is significantly better than traditional sampling methods, providing an efficient and accurate intelligent sampling solution for mining production.

[0290] Example 7:

[0291] This embodiment demonstrates the application effect of the proposed method in a real mining production environment. The experiment was conducted in the main haulage roadway at the -800m level in a copper mine, where the environment was characterized by insufficient lighting (average 50 lux), high dust concentration (approximately 5 mg / m³), and high humidity (85%).

[0292] The equipment used in the experiment included: a 6-DOF industrial robotic arm (10kg payload), two industrial cameras (2048×1536 resolution, 30fps), a 16-line LiDAR (±2mm accuracy, 10Hz), a 9-axis IMU (200Hz), and an edge computing unit (NVIDIA Jetson AGX Orin). The system was installed at key nodes in the mine's transportation system to perform real-time mineral identification and sampling of passing mining trucks.

[0293] The experiment compared three methods: (1) the traditional fixed time interval sampling method; (2) the sampling method based on traditional computer vision; and (3) the deep learning dynamic recognition and accurate sampling method of the present invention. The evaluation indicators included: recognition accuracy, sampling position accuracy, sampling success rate, and system stability.

[0294] Experimental results show that the method of the present invention is significantly superior to the prior art in all aspects:

[0295] 1. Mineral identification accuracy: The method of this invention achieves 98.5%, while traditional computer vision methods achieve 85%, and fixed-interval methods do not involve identification;

[0296] 2. Sampling position accuracy: ±2mm for the method of this invention, ±15mm for the traditional computer vision method, and ±50mm for the fixed interval method;

[0297] 3. Sampling success rate (defined as the proportion of successful acquisition of target mineral samples): The success rate of this invention is 98%, while that of traditional computer vision methods is 75%, and that of fixed-interval methods is 45%.

[0298] 4. System stability (continuous stable running time): The method of this invention can run for more than 7 days without manual intervention. Traditional computer vision methods require a restart on average every 2 days. Although the fixed interval method has a simple structure, the mechanical parts wear out severely and require frequent maintenance.

[0299] Of particular note is the outstanding performance of the method of this invention under extreme conditions. Even when the mine car speed reaches 2 m / s and the dust concentration suddenly increases to 8 mg / m³, the method of this invention still maintains a sampling success rate of over 95%, while traditional methods almost fail to function properly.

[0300] Through a three-month long-term validation process, the method of this invention processed approximately 10,000 mining truck trips, collecting over 30,000 samples, averaging more than 100 trips per day. The high-quality sample data generated by the system significantly improved the accuracy of ore grade analysis, providing a reliable basis for mine production decisions, indirectly increasing the ore beneficiation recovery rate by 2.5 percentage points, and creating significant economic benefits.

[0301] Furthermore, the adaptability of the method of this invention has also been verified. When the type of ore mined in the mining area changes, the system can quickly adapt with only a small number of new samples (about 50) through the online learning mechanism, and the recognition accuracy can be restored to the normal level in a short time, which greatly reduces the workload of system maintenance and retraining.

[0302] Example 8:

[0303] To verify the versatility and scalability of the present invention, this embodiment applies the proposed method to the identification and sampling of different types of ores, including three typical ores: iron ore, copper ore, and gold ore.

[0304] First, the core modules of the method are adaptively adjusted: for the network architecture, the basic structure of the physically informed hybrid parallel Kolmogorov-Arnold network remains unchanged, but the parameters of the physically informed module are adjusted for different ore types, especially the weights of reflectivity, color, and structural features; for data preprocessing, the filtering parameters and enhancement algorithms are adjusted for the characteristics of different ores; for the multi-target tracking system, the Kalman filter parameters and the weights of the association cost function are adjusted according to the motion characteristics and morphological differences of different ores.

[0305] The experiment was conducted in three different mining areas, with approximately 5,000 ore images collected from each area for model training, followed by a two-week system test. Experimental results show that, after appropriate adjustments, the method of this invention performs excellently across different ore types: iron ore identification accuracy of 97.8%, copper ore 98.5%, and gold ore 96.2%; sampling success rates of 98%, 97%, and 96%, respectively.

[0306] This result verifies that the method of the present invention has good versatility and adaptability, and can quickly adapt to the identification and sampling needs of different ore types through limited parameter adjustments, providing an efficient and flexible solution for intelligent production in the mining industry.

[0307] Based on the results of the above embodiments, the deep learning-based dynamic mineral identification and precise sampling method of the present invention can achieve high-precision mineral identification and dynamic precise sampling in complex downhole environments, significantly improving the representativeness and efficiency of mineral sampling. This method has demonstrated significant technical advantages and economic benefits in engineering practice, providing important technical support for intelligent mining production.

[0308] This application also provides a deep learning-based mineral dynamic identification and precise sampling system, including:

[0309] The acquisition module is used to acquire mineral image data and lidar ranging data on the mine car in the underground environment, and to preprocess the mineral image data and lidar ranging data to acquire mineral feature data.

[0310] The training module is used to design a deep convolutional neural network architecture for multi-scale feature fusion and perform multi-stage training to determine the mineral recognition model.

[0311] The target detection module is used to perform target detection on the mineral feature data using the mineral recognition model, assign a unique ID to the detected target mineral and extract depth and appearance features, establish a target motion model using a Kalman filter and make predictions, and perform multi-target association matching using the Hungarian algorithm to obtain the motion trajectory data of the target mineral.

[0312] The feature matching module is used to extract feature points of continuous image frames based on the motion trajectory data using the improved FAST corner detection algorithm, perform feature matching through ORB descriptor and RANSAC algorithm, and apply the pyramid Lucas-Kanade optical flow algorithm to calculate the pixel-level displacement vector field to obtain the speed and displacement data of the mine car.

[0313] The prediction module is used to perform spatiotemporal alignment of multi-sensor data from the camera, lidar, and IMU based on the velocity and displacement data, reconstruct the three-dimensional spatial position of the target mineral, establish a nonlinear dynamic model, use an extended Kalman filter to fuse multi-sensor data and predict the position of the target mineral at future moments, and obtain target spatial coordinate data.

[0314] The sampling module is used to generate joint spatial trajectories based on the target spatial coordinate data using a fifth-order polynomial interpolation method and to optimize time-optimized control. It coordinates the movement of each joint of the robotic arm through dynamic compensation control and performs precise sampling operations.

Claims

1. A method for dynamic mineral identification and precise sampling based on deep learning, characterized in that, The process includes the following steps: acquiring mineral image data and lidar ranging data from a mining truck in the underground environment; preprocessing the mineral image data and lidar ranging data to obtain mineral feature data; designing a deep convolutional neural network architecture for multi-scale feature fusion and performing multi-stage training to determine a mineral recognition model. The design of the multi-scale feature fusion deep convolutional neural network architecture includes: encoding mineralogical laws into network structure constraints based on the mineral dataset and the physical properties of mineral spectral reflectance, crystal structure, and chemical composition; designing a physically informed module and a data-driven module; coupling the two parts through a physical consistency loss function to obtain a physically informed hybrid network architecture; and utilizing the physically informed hybrid network architecture to... The high-dimensional input is decomposed into multiple one-dimensional functions. N parallel sub-networks are designed to extract color, texture, and morphological features respectively. The outputs of the N parallel sub-networks are integrated through adaptive weighted fusion to obtain a parallel Kolmogorov-Arnold network structure, where N is a positive integer greater than 1. Based on the parallel Kolmogorov-Arnold network structure, subdomains are divided according to mineral categories (sulfide minerals, oxide minerals, and silicate minerals) and physical properties (reflectivity, color, and hardness). A dedicated sub-network is configured for each subdomain. Information exchange between subdomains is maintained through overlapping regions and boundary conditions to obtain a domain decomposition network configuration. Based on the domain decomposition network configuration, a system is designed that includes standard classification loss, physical consistency loss, inter-domain consistency loss, and uncertainty. A multi-objective loss function is estimated to balance data-driven learning and physical constraints, resulting in a physical constraint loss function. Based on this physical constraint loss function, an improved ResNet-50 is used as the backbone network and integrated with a feature pyramid network structure for multi-stage training to obtain the mineral recognition model. The mineral recognition model is then used to detect targets in the mineral feature data, assigning unique IDs to detected target minerals and extracting depth and appearance features. A Kalman filter is used to establish a target motion model and perform prediction. A Hungarian algorithm is used for multi-objective association matching to obtain the motion trajectory data of the target minerals. Based on this motion trajectory data, an improved FAST corner detection algorithm is used to extract feature points from consecutive image frames. Feature matching is performed using ORB descriptors and the RANSAC algorithm. The pixel-level displacement vector field is calculated using the Lucas-Kanade optical flow algorithm to obtain the speed and displacement data of the mining truck. Based on the speed and displacement data, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned to reconstruct the three-dimensional spatial position of the target mineral. A nonlinear dynamic model is established, and the multi-sensor data is fused using an extended Kalman filter to predict the future position of the target mineral, thus obtaining the target spatial coordinate data. Based on the target spatial coordinate data, the joint spatial trajectory is generated using a fifth-order polynomial interpolation method and optimized for time-optimized control. The movement of each joint of the robotic arm is coordinated through dynamic compensation control to perform precise sampling operations.

2. The method according to claim 1, characterized in that, The mineral image data and the lidar ranging data are preprocessed to obtain mineral feature data, including: calculating the spatial distance and pixel value similarity of the mineral image data; applying a bilateral filtering algorithm to perform weighted averaging on the neighborhood of each pixel to obtain denoised mineral image data; calculating the cumulative distribution function of the denoised mineral image data; remapping the pixel values ​​according to the cumulative distribution function; applying an adaptive histogram equalization algorithm to enhance the local contrast of the denoised mineral image data to obtain contrast-enhanced mineral image data; and based on the contrast-enhanced mineral image data and the lidar ranging data, performing statistical outlier removal and voxel downsampling on the lidar point cloud data, and performing pixel-level annotation and data augmentation to obtain the mineral feature data.

3. The method according to claim 2, characterized in that, Pixel-level annotation and data augmentation processing are performed, including: labeling bounding boxes and semantic segmentation masks for different mineral types to obtain an initial labeled dataset; For the initial labeled dataset, geometric transformation operation, lighting simulation operation, noise addition operation, and special effect simulation operation are applied sequentially to expand the initial labeled dataset to more than three times the original size, resulting in an expanded dataset; the quantity distribution of various mineral samples in the expanded dataset is statistically analyzed, the number of first samples is increased by oversampling technology, and the distribution of various mineral samples is balanced by weight adjustment technology to obtain the mineral feature data.

4. The method according to claim 1, characterized in that, Design a deep convolutional neural network architecture for multi-scale feature fusion and perform multi-stage training, including: constructing an initial deep convolutional neural network, training basic feature learning for 100 rounds with an initial learning rate of 0.001, training the initial deep convolutional neural network to identify basic mineral categories, and obtaining a basic feature learning model. The learning rate of the basic feature learning model is reduced to 0.0001 and fine-grained classification training is performed for 50 rounds to optimize the recognition accuracy of sulfide and oxide ores, resulting in a fine-grained classification model. Based on the fine-grained classification model, a cosine annealing learning rate strategy is used for end-to-end joint training for 30 rounds to optimize the overall performance of target detection and semantic segmentation, resulting in the mineral recognition model.

5. The method according to claim 1, characterized in that, The mineral identification model is used to perform target detection on the mineral feature data, assign a unique ID to the detected target mineral and extract its depth and appearance features, use a Kalman filter to establish a target motion model and make predictions, and use the Hungarian algorithm to perform multi-target association matching to obtain the motion trajectory data of the target mineral. The process includes: performing target detection on the mineral feature data based on the mineral identification model, assigning a unique ID to the detected target mineral and recording its position, size, category and appearance features, and establishing an initial target state set. Based on the initial target state set, a 128-dimensional feature vector for each target is calculated using a pre-trained feature extraction network to obtain a target depth feature set. Based on the target depth feature set and historical target position data from the previous N frames, a Kalman filter is used to establish a target motion model to predict the possible location region of the target in the next frame. The velocity and acceleration parameters of the target are updated using a state transition matrix to obtain a target prediction state set. Based on the target prediction state set and the detection results of the new frame, a weighted value of appearance feature similarity and spatial distance is calculated. The current detection results are then associated and matched with existing trajectories using a Hungarian algorithm to handle target appearance, disappearance, and occlusion, thereby obtaining the motion trajectory data of the target mineral.

6. The method according to claim 4, characterized in that, The method further includes: modifying the loss function of the initial deep convolutional neural network from L2 norm to L1 norm; defining the loss function as the sum of the absolute differences between the predicted probability map and the true label for pixel-level segmentation tasks; and modifying the bounding box regression loss to L1 loss for object detection tasks, thus obtaining the L1 loss function configuration; based on the L1 loss function configuration, introducing random noise conforming to a Gaussian distribution into the input image during the training process, with noise parameters set to mean μ=0 and standard deviation σ=5-15 pixel values, while keeping the labels unchanged and forcing the model to learn to extract essential features from the noisy data, thus obtaining noise-enhanced training data; Based on the noise-enhanced training data, the sum of absolute biases is minimized within the local neighborhood. Through iterative optimization, the pixel values ​​that best represent the real mineral surface features are found. While preserving the mineral edge and texture features, noise interference is suppressed, resulting in L1 optimized feature data. For the L1 optimized feature data, a penalty term for the sum of the absolute values ​​of the weights is added to the loss function. The regularization intensity parameter λ ranges from 0.001 to 0.01, guiding the network to automatically select the most representative mineral features, resulting in sparse feature representation. Based on the sparse feature representation, the basic feature learning training is performed for 100 rounds with an initial learning rate of 0.001, followed by 50 rounds of fine-grained classification training with the learning rate reduced to 0.0001, and then 30 rounds of end-to-end joint training using a cosine annealing learning rate strategy to obtain the mineral recognition model.

7. The method according to claim 1, characterized in that, Based on the velocity and displacement data, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned to reconstruct the three-dimensional spatial position of the target mineral. A nonlinear dynamic model is established, and an extended Kalman filter is used to fuse the multi-sensor data and predict the future position of the target mineral to obtain target spatial coordinate data. This includes: based on the velocity and displacement data, constructing a set of nonlinear partial differential equations describing the motion of the mineral on the conveyor belt; decomposing the dynamic system into principal terms and perturbation terms; the principal terms adopt Newton's equations of motion; and the perturbation terms are learned from measured data through a neural network to capture complex nonlinear effects that the theoretical model fails to describe, resulting in a perturbation PINN dynamic model. Based on the perturbation PINN dynamic model, a hierarchical neural network architecture is designed, with different layers of the network responsible for modeling the physical processes at different time and spatial scales, from millisecond-level particle collisions to second-level overall motion. The physical consistency between models at different scales is ensured through loss function design, resulting in a multi-scale PINN model. Based on the multi-scale PINN model, the multi-sensor data from the camera, lidar, and IMU are spatiotemporally aligned. Through projection mapping, the target mineral in the two-dimensional image is reconstructed into its position and volume information in three-dimensional space, resulting in a three-dimensional spatial representation of the target mineral. Based on this three-dimensional spatial representation, a nonlinear dynamic model considering the acceleration, deceleration, vibration, and steering factors of the mining truck is established to describe the motion of the target mineral relative to the sampling robotic arm, resulting in a set of dynamic model parameters. Based on this set of dynamic model parameters, an extended Kalman filter is used to fuse visual measurement, laser ranging, and IMU data. Automatic corrections are made when statistical estimates deviate from physical laws to ensure that the state estimation conforms to basic physical laws. The position of the target mineral within the next 0.5 to 1 second is predicted, resulting in the target spatial coordinate data.

8. The method according to claim 1, characterized in that, Based on the motion trajectory data, feature points of continuous image frames are extracted using an improved FAST corner detection algorithm. Feature matching is performed using ORB descriptors and the RANSAC algorithm. The pixel-level displacement vector field is calculated using the pyramid Lucas-Kanade optical flow algorithm to obtain the speed and displacement data of the mine car. This includes: based on the motion trajectory data and the continuous image frames, feature points in the image are extracted using an improved FAST corner detection algorithm; feature representations are calculated using ORB descriptors to obtain feature description data; for the feature description data, the similarity between feature points is calculated using Hamming distance; and feature matching point sets are obtained by filtering corresponding feature points using the RANSAC algorithm. An image pyramid is constructed, and the optical flow of the feature matching point set is calculated layer by layer from coarse to fine scale. The Lucas-Kanade optical flow algorithm is applied to calculate the pixel-level displacement vector field between consecutive frames, achieving sub-pixel accuracy motion estimation and obtaining the optical flow vector field. Based on the optical flow vector field, the camera's rotation matrix R and translation vector t are recovered by solving the fundamental matrix or homography matrix, and the transformation relationship between the world coordinate system and the camera coordinate system is established to obtain the camera attitude parameters. Based on the camera attitude parameters and the system timestamp, the displacement accumulation calculation between consecutive frames is performed, and the accumulated error is eliminated by fusion correction combined with wheel encoder data to obtain the speed and displacement data of the mining truck.

9. The method according to claim 1, characterized in that, Based on the target spatial coordinate data, a fifth-order polynomial interpolation method is used to generate joint spatial trajectories and perform time-optimized control optimization. Dynamic compensation control coordinates the movement of each joint of the robotic arm, performing precise sampling operations. This includes: establishing forward and inverse kinematic models based on the target spatial coordinate data, the robotic arm's DH parameters, and joint constraints; calculating the mapping relationship between the end effector position and joint angles; defining the workspace and singular positions to obtain the robotic arm kinematic model; based on the robotic arm kinematic model, using a fifth-order polynomial interpolation method to interpolate joint angles, velocities, and accelerations to generate smooth joint spatial trajectories, obtaining initial trajectory parameters; and applying time to the initial trajectory parameters. An optimal control algorithm optimizes trajectory execution time while satisfying joint velocity and acceleration constraints, yielding optimal trajectory parameters. Based on these optimal trajectory parameters, a dynamic model is established considering the mass, inertia tensor, and friction coefficient of each joint of the robotic arm. Feedforward compensation control is implemented to reduce tracking errors during high-speed motion. An adaptive impedance control strategy is used to address changes in contact force during sampling, resulting in joint torque control commands. Based on these joint torque control commands, a high-performance servo driver and a real-time control system coordinate the movement of each joint of the robotic arm, achieving precise positioning of the end effector. When the target mineral reaches the predetermined position, the sampler's opening and closing, depth, and sampling quantity are controlled to execute the precise sampling operation.

10. A deep learning-based system for dynamic mineral identification and precise sampling, characterized in that, Includes: an acquisition module, used to acquire mineral image data and lidar ranging data on a mining car in the underground environment, and to preprocess the mineral image data and lidar ranging data to acquire mineral feature data; The training module is used to design a deep convolutional neural network architecture for multi-scale feature fusion and perform multi-stage training to determine the mineral recognition model. The design of the multi-scale feature fusion deep convolutional neural network architecture includes: encoding mineralogical laws into network structure constraints based on the physical properties of mineral datasets and mineral spectral reflectance, crystal structure, and chemical composition; designing a physically informed module and a data-driven module; coupling the two parts through a physical consistency loss function to obtain a physically informed hybrid network architecture; using this physically informed hybrid network architecture, decomposing the high-dimensional input into multiple one-dimensional functions; designing N parallel sub-networks responsible for extracting color, texture, and morphological features respectively; integrating the outputs of the N parallel sub-networks through adaptive weighted fusion to obtain a parallel Kolmogorov-Arnold network structure, where N is a positive integer greater than 1; based on the parallel Kolmogorov-Arnold network structure, the model is categorized by mineral type, including sulfide minerals, oxide minerals, and silicate minerals, as well as physical properties... The mineral identification model is divided into subdomains based on reflectivity, color, and hardness. A dedicated subnetwork is configured for each subdomain, and information exchange between subdomains is maintained through overlapping regions and boundary conditions, resulting in a domain decomposition network configuration. Based on this configuration, a multi-objective loss function is designed, incorporating standard classification loss, physical consistency loss, inter-domain consistency loss, and uncertainty estimation loss. This balances data-driven learning with physical constraints, resulting in a physical constraint loss function. Based on this loss function, an improved ResNet-50 is used as the backbone network, integrated with a feature pyramid network structure, and subjected to multi-stage training to obtain the mineral identification model. A target detection module is used to perform target detection on the mineral feature data using the mineral identification model. A unique ID is assigned to the detected target mineral, and depth and appearance features are extracted. A Kalman filter is used to establish a target motion model and perform prediction. A Hungarian algorithm is used for multi-target association matching to obtain the motion trajectory data of the target mineral. The feature matching module is used to extract feature points from continuous image frames based on the motion trajectory data using an improved FAST corner detection algorithm, perform feature matching using ORB descriptors and the RANSAC algorithm, and calculate the pixel-level displacement vector field using the pyramid Lucas-Kanade optical flow algorithm to obtain the speed and displacement data of the mining truck. The prediction module is used to perform spatiotemporal alignment of multi-sensor data from cameras, lidar, and IMU based on the speed and displacement data, reconstruct the three-dimensional spatial position of the target mineral, establish a nonlinear dynamic model, fuse multi-sensor data using an extended Kalman filter, and predict the future position of the target mineral to obtain the target spatial coordinate data. The sampling module is used to generate joint spatial trajectories based on the target spatial coordinate data using a fifth-order polynomial interpolation method and perform time-optimized control optimization, coordinate the movement of each joint of the robotic arm through dynamic compensation control, and perform precise sampling operations.

Images