Round stockyard intelligent material piling and taking control method based on multi-dimensional sensor fusion

Abstract

The invention relates to a circular stockyard intelligent material piling and taking control method based on multi-dimensional sensor fusion, and the method comprises a circular stockyard intelligent material piling and taking control system based on multi-dimensional sensor fusion, and also comprises the steps: collecting multi-dimensional data in real time through a multi-dimensional sensor network; inputting the preprocessed multi-dimensional data into a multi-modal fusion deep neural network of an edge calculation processing unit, and generating environment perception information and equipment state evaluation information in real time; the intelligent control decision module receives the environment perception information and the equipment state evaluation information, generates and optimizes a self-adaptive material piling and taking control strategy through a three-layer architecture, and issues the self-adaptive material piling and taking control strategy to the material piling and taking machine for execution; the predictive maintenance system calculates the equipment health index and predicts the remaining service life based on the equipment state evaluation information, and formulates a maintenance scheme in combination with the operation plan. According to the invention, all-around perception, intelligent control decision and predictive maintenance of the material yard environment are realized, and the efficiency, the safety and the economical efficiency of material piling and taking operation are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of circular stockyard stacking and reclaiming technology, specifically to a smart stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion. Background Technology

[0002] Currently, with the transformation of port material handling towards intelligentization, circular stockyards, as important bulk material storage facilities for ports and industrial enterprises, directly impact material handling efficiency and operating costs due to the high level of automation in their stacking and reclaiming operations. Currently, traditional circular stockyard stacking and reclaiming control systems mainly suffer from the following problems:

[0003] First, the sensing capabilities are limited. Existing systems often use only one type of sensor, such as lidar or vision sensors, which cannot comprehensively perceive the complex environment of the material yard. This results in insufficient information for control decisions, affecting operational accuracy and safety. Second, control strategies are rigid. Traditional control systems use preset, fixed control modes and cannot dynamically adjust operational strategies based on environmental changes, material characteristics, and equipment status, making it difficult to adapt to changing operating environments and demands. Furthermore, current technology separates maintenance from operation, treating equipment maintenance and operation scheduling as independent issues. This lack of unified optimization leads to conflicts between maintenance plans and operational needs, impacting overall efficiency. Finally, there is a lack of predictive capabilities. Traditional systems are mostly reactive, lacking the ability to predict environmental changes and equipment status, and thus cannot adjust strategies in advance or prevent malfunctions.

[0004] Therefore, there is an urgent need for a smart stacking and reclaiming control method for circular material yards that integrates multi-dimensional data to achieve intelligent and efficient stacking and reclaiming control. Summary of the Invention

[0005] To address the problems existing in current circular material yard stacking and reclaiming control methods, a smart stacking and reclaiming control method based on multi-dimensional sensor fusion is proposed for circular material yards. This method enables comprehensive perception of the material yard environment, intelligent control decision-making, and predictive maintenance, significantly improving the efficiency, safety, and economy of stacking and reclaiming operations.

[0006] To achieve the above objectives, this invention proposes a smart stacking and reclaiming control method for circular material yards based on multi-dimensional sensor fusion, including a smart stacking and reclaiming control system for circular material yards based on multi-dimensional sensor fusion, comprising a multi-dimensional sensor network, an edge computing processing unit, an intelligent control decision module, a predictive maintenance system, a digital twin visualization platform, and a communication network module.

[0007] It also includes step 1: real-time acquisition of lidar point cloud data, temperature field data, vibration time series data and visual image data through a multi-dimensional sensor network, and preprocessing them;

[0008] Step 2: Input the preprocessed multi-dimensional data into the multimodal fusion deep neural network of the edge computing processing unit to generate environmental perception information and equipment status assessment information in real time;

[0009] Step 3: The intelligent control decision module receives environmental perception information and equipment status assessment information, generates and optimizes the adaptive stacker-reclaimer control strategy through a three-layer architecture, and sends it to the stacker-reclaimer for execution;

[0010] Step 4: The predictive maintenance system calculates the equipment health index and predicts the remaining service life based on equipment condition assessment information, and formulates a maintenance plan in conjunction with the work plan;

[0011] Step 5: Monitor the operation process and equipment status in real time through the digital twin visualization platform, and adjust the control strategy and maintenance plan according to the actual situation.

[0012] Furthermore, multiple lidar sensors are evenly arranged around the material yard, providing 360-degree full coverage three-dimensional point cloud data. An array thermal imager is installed in the material yard to monitor the temperature distribution of the material. High-precision accelerometers are installed on the bearings and reducers of the stacker-reclaimer to monitor the vibration characteristics of the equipment in real time. High-definition vision sensors with night vision capabilities are deployed around the material yard to obtain visual information about the environment and equipment.

[0013] The preprocessing described in step 1 includes: removing outliers from the lidar data using voxel filtering and RANSAC-Ground, followed by surface reconstruction to output a simplified point cloud;

[0014] Temperature data were obtained by using image enhancement CLAHE and U-Net-based region segmentation to obtain temperature zone boundaries, and then normalized to [0,1].

[0015] Vibration data are divided into multi-resolution frequency bands in Hertz units. A two-dimensional time-frequency graph is obtained by using short-time Fourier transform (STFT), and then the main frequency peak features are extracted by 1D-FFT.

[0016] Visual data is used to perform multi-scale template matching for target detection, extracting ROIs containing key components of the device, and performing color correction and distortion correction.

[0017] By combining targeted multimodal data preprocessing algorithms, redundant noise is effectively filtered and core features are extracted, providing high-quality data support for subsequent fusion analysis and decision-making. The preprocessed simplified point cloud and normalized temperature data significantly reduce the processing load of the edge computing unit while ensuring data accuracy and avoiding insufficient operational precision caused by interference from the original data, thus laying a data foundation for precise control of stacking and reclaiming operations.

[0018] Furthermore, the multimodal fusion deep neural network includes an MM-FusionNet neural network, which includes a feature extraction layer, a cross-modal attention layer, and a temporal modeling layer;

[0019] The feature extraction layer uses PointNet++ to extract local-global geometric features f_P for lidar point cloud data, 2D-CNN to extract thermal distribution texture features f_T for temperature data, 1D-CNN-FFT combined with self-supervised contrastive learning to obtain spectral features f_A for vibration data, and ReaNet-50 to obtain semantic features f_I for visual images.

[0020] The MM-FusionNet neural network employs a three-layer core architecture and utilizes dedicated feature extraction algorithms for different modalities of data to accurately extract multi-dimensional information such as lidar geometric features and temperature thermal distribution characteristics. Compared to traditional general feature extraction methods, this design fully leverages the unique value of each modal data, enhancing the relevance and effectiveness of feature representation. This provides high-quality feature input for subsequent cross-modal fusion and dynamic prediction, ensuring the accuracy of environmental perception and equipment status assessment.

[0021] Furthermore, the edge computing processing unit sequentially executes the static fusion analysis module and the dynamic fusion analysis module;

[0022] The static fusion analysis module performs feature alignment, cross-modal attention weighting, and state decoding on the multimodal feature tensor {f_P^t,f_T^t,f_A^t,f_I^t} at the current time t to obtain the environment state vector s^t and the device state vector e^t.

[0023] Feature alignment: The geometric features f_P^t of the LiDAR are mapped to the visual coordinate system through KNN-based geometric alignment, and the visual semantic features f_I^t are projected to the point cloud coordinate system according to the extrinsic matrix. The temperature texture features f_T^t and the vibration spectrum features f_A^t are increased in dimension to a unified 256-dimensional embedding space through 1×1 convolution.

[0024] Cross-modal attention: A multi-head attention mechanism is adopted, and the aligned embedding vector is mapped to query Q, key K, and value V. The attention weight calculation formula is as follows:

[0025] α_ij=softmax\big((Q_iK_j^T) / sqrt(d)\big) (1);

[0026] Where d = 256, and the fusion feature F^t = ∑_jα_ijV_j;

[0027] State decoding: F^t is decoded by a fully connected layer + LayerNorm, outputting s^t and e^t;

[0028] The dynamic fusion analysis module performs time-series modeling on the static fusion vector sequence H = [F^{t-l+1},…,F^{t}] of the most recent l = 30 frames;

[0029] The specific implementation is a stacked two-layer bidirectional LSTM with 512 hidden units and dropout 0.2.

[0030] The dynamic fusion analysis module outputs a global context vector c^t = BiLSTM(H), which is then used for prediction via a multi-task regression head.

[0031] Future k = 10 frames of stockpile height field \hat{H}^{t+1:t+k};

[0032] Key performance indicators for future equipment: \hat{e}^{t+1:t+k};

[0033] During the training phase, a weighted sum of L2 loss and SmoothL1 loss is used for end-to-end optimization to achieve dynamic prediction of stockpile morphology and equipment health status.

[0034] The static fusion analysis module achieves accurate fusion and state decoding of multimodal data at the current moment, while the dynamic fusion analysis module captures the evolution trend of historical data and predicts future states, forming a dual combination of real-time perception and dynamic prediction. The design of bidirectional LSTM and multi-task regression head can not only accurately output the current state of the environment and equipment, but also predict changes in the stockpile shape and potential equipment anomalies in advance, providing forward-looking support for control strategy adjustment and fault prevention, and significantly improving the system's adaptability and risk avoidance capabilities.

[0035] Further, step 3 includes:

[0036] The three-layer architecture includes a state evaluation layer, a decision generation layer, and an optimization search layer;

[0037] The state assessment layer receives the environmental state vector s^t, the device state vector e^t, and the job requirement vector r^t. It generates a fused representation z^t through two layers of 8-head TinyFormer, with each layer having d_model = 256. The inference latency is ≤0.5ms.

[0038] The decision generation layer consists of a safety control submodule and an efficiency optimization submodule. The safety submodule matches z^t with the rule table R_safe and outputs the control command a_safe that must be executed in real time. The efficiency submodule takes (z^t, r^t) as the state input.

[0039] The Actor network is used to output continuous actions a_eff, and the Critic network is used to evaluate the Q value and update it offline.

[0040] The optimized search layer performs simulated annealing local search on a_safe⊕a_eff, with the objective function being:

[0041] J=ω1·E_prod-ω2·E_energy-ω3·E_wear (2);

[0042] ω1, ω2, and ω3 can be configured, and the final control vector a*^t is sent to the stacker-reclaimer.

[0043] The three-layer architecture enables intelligent decision-making throughout the entire process, including state fusion, dual-track decision-making, and multi-objective optimization. The state assessment layer efficiently integrates multi-dimensional information, the decision generation layer balances safety baselines with efficiency optimization, and the optimization search layer achieves multi-objective balance through simulated annealing. Compared to traditional fixed decision-making models, this architecture ensures operational safety through the safety control submodule and reduces energy consumption and wear through the efficiency optimization submodule. The final output control vector balances safety, efficiency, and economy, significantly improving the overall benefits of stacking and reclaiming operations.

[0044] Furthermore, the rule table R_safe ∈ the rule base, the rule base includes a safety control rule base and an optimization suggestion rule base, the optimization suggestion rule base uses a five-element membership function μ_i(x) to perform fuzzy reasoning on five categories of indicators: energy consumption, efficiency, wear, noise, and emissions, and outputs a weight vector ω, which is used to dynamically adjust the objective function J;

[0045] The knowledge acquisition process for the rule base includes offline expert knowledge distillation and online reinforcement learning adaptation.

[0046] By integrating safety control and optimization suggestions through a rule base, the rule table R_safe ensures rapid matching and execution of safety instructions, while the optimization suggestion rule base provides a basis for dynamic adjustment of the objective function. It inherits the reliability of expert experience while adapting to dynamic changes in the operating environment. Furthermore, the hot-loading function supports flexible manual adjustments, further enhancing the system's scenario adaptability and decision-making flexibility.

[0047] Furthermore, the predictive maintenance system includes a health index calculation module, a remaining life prediction module, and a maintenance scheduling optimization module:

[0048] The health index calculation module uses multi-kernel support vector regression (MK-SVR). The radial basis kernel κ1 and the linear kernel κ2 regress the health status h1 and h2 on 34-dimensional vibration characteristics, 12-dimensional temperature trend, and 8-dimensional visual defect characteristics, respectively. The final health index h = 0.6·h1 + 0.4·h2.

[0049] The remaining lifetime prediction module uses a Seq2Seq-LSTM network to predict the RUL distribution for the next 720 hours, and uses Pinball loss during training to reduce the risk of underestimation.

[0050] The maintenance scheduling optimization module is based on an improved genetic algorithm, which integrates work window, spare parts inventory and human resource constraints to output maintenance work order schedules. The improved genetic algorithm has a population of 50, an iteration of 100, a crossover rate of 0.8 and a mutation rate of 0.1.

[0051] The predictive maintenance system employs a three-module collaborative approach: the MK-SVR algorithm accurately calculates equipment health indices, the Seq2Seq-LSTM network effectively improves the accuracy of remaining lifetime prediction, and the genetic algorithm optimizes maintenance scheduling. Compared to traditional periodic maintenance, this system achieves proactive prevention, both predicting equipment failures in advance and reducing failure rates, and optimizing maintenance time without affecting work plans, thus minimizing work interruption losses.

[0052] Furthermore, the digital twin visualization platform is used to provide a three-dimensional visualization interface to display the three-dimensional model of the material yard, the distribution of material piles, the location of equipment and the operating status in real time. The digital twin visualization platform visualizes sensor data and fusion results, and provides a human-machine interface for parameter adjustment and strategy configuration.

[0053] Furthermore, the communication network module employs high-speed wireless communication technology to achieve real-time transmission of sensor data, low-latency issuance of control commands, and remote monitoring of system status.

[0054] The beneficial effects of the present invention through the above technical solution are as follows:

[0055] 1. This invention proposes a complete system encompassing multi-dimensional sensor data acquisition, AI fusion processing and analysis, intelligent decision-making, and adaptive control. Compared to traditional single-sensor control systems, this system achieves four-dimensional data fusion (LiDAR + temperature + vibration + vision), significantly enhancing information acquisition capabilities and greatly improving the accuracy and comprehensiveness of environmental perception. It integrates all-around environmental perception of the material yard, dynamic generation of control strategies, and predictive maintenance.

[0056] 2. This invention employs a multimodal fusion neural network based on an attention mechanism. It adaptively allocates importance weights for different modal data through a cross-modal attention fusion layer and captures dynamic change trends through a time-series modeling layer. Compared with traditional fixed control strategies, its adaptability is greatly improved.

[0057] 3. This invention proposes a method for unified optimization of predictive maintenance and job scheduling, realizing a shift from reactive maintenance to predictive maintenance. Compared with the traditional periodic maintenance model, it can effectively improve maintenance efficiency, significantly reduce equipment failure rate, and increase equipment availability.

[0058] 4. The intelligent control decision module of this invention can dynamically adjust the control strategy according to environmental changes, material characteristics, and equipment status, achieving true adaptive control. Through intelligent path planning and parameter optimization, it can effectively improve operational efficiency and reduce system energy consumption.

[0059] 5. This invention employs high-speed wireless communication technology and an edge computing architecture to achieve real-time control response, meeting the high-precision control requirements in complex operating environments. The digital twin visualization platform provides an intuitive monitoring and operation interface, greatly improving the system's operability. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the steps of a smart stacking and reclaiming control method for a circular material yard based on multi-dimensional sensor fusion, as described in this invention.

[0061] Figure 2 This is a schematic diagram of the structure of a smart stacking and reclaiming control system for a circular material yard based on multi-dimensional sensor fusion according to the present invention;

[0062] Figure 3 This is a multimodal fusion deep neural network framework diagram for a smart stacking and reclaiming control method for circular material yards based on multidimensional sensor fusion, as described in this invention. Detailed Implementation

[0063] Example 1

[0064] like Figure 1-3 As shown, a smart stacking and reclaiming control method for circular material yards based on multi-dimensional sensor fusion includes a smart stacking and reclaiming control system for circular material yards based on multi-dimensional sensor fusion, comprising a multi-dimensional sensor network, an edge computing processing unit, an intelligent control decision module, a predictive maintenance system, a digital twin visualization platform, and a communication network module.

[0065] It also includes step 1: real-time acquisition of lidar point cloud data, temperature field data, vibration time series data and visual image data through a multi-dimensional sensor network, and preprocessing them;

[0066] Step 2: Input the preprocessed multi-dimensional data into the multimodal fusion deep neural network of the edge computing processing unit to generate environmental perception information and equipment status assessment information in real time;

[0067] Step 3: The intelligent control decision module receives environmental perception information and equipment status assessment information, generates and optimizes the adaptive stacker-reclaimer control strategy through a three-layer architecture, and sends it to the stacker-reclaimer for execution;

[0068] Step 4: The predictive maintenance system calculates the equipment health index and predicts the remaining service life based on equipment condition assessment information, and formulates a maintenance plan in conjunction with the work plan;

[0069] Step 5: Monitor the operation process and equipment status in real time through the digital twin visualization platform, and adjust the control strategy and maintenance plan according to the actual situation.

[0070] like Figure 2 As shown, multiple lidar sensors are evenly distributed around the material yard, providing 360-degree full-coverage three-dimensional point cloud data. An array-type thermal imager (infrared temperature sensor) is installed in the material yard to monitor the temperature distribution of the materials. High-precision accelerometers (vibration sensors) are installed on the bearings and reducers of the stacker-reclaimer to monitor the vibration characteristics of the equipment in real time. High-definition vision sensors with night vision capabilities are deployed around the material yard to obtain visual information about the environment and equipment. In addition, dust concentration sensors (to monitor air quality) and airflow sensors (to monitor ventilation status) are also installed.

[0071] The preprocessing described in step 1 includes: removing outliers from the lidar data using voxel filtering and RANSAC-Ground, followed by surface reconstruction to output a simplified point cloud;

[0072] Temperature data were obtained by using image enhancement CLAHE and U-Net-based region segmentation to obtain temperature zone boundaries, and then normalized to [0,1].

[0073] Vibration data are divided into multi-resolution frequency bands in Hertz units. A two-dimensional time-frequency graph is obtained by using short-time Fourier transform (STFT), and then the main frequency peak features are extracted by 1D-FFT.

[0074] Visual data is used to perform multi-scale template matching for target detection, extracting ROIs containing key components of the device, and performing color correction and distortion correction.

[0075] After the above preprocessing, the modal features are simultaneously fed into the MM-FusionNet neural network to complete fusion inference, thereby obtaining accurate stockpile morphology parameters and equipment status features.

[0076] like Figure 3As shown, the multimodal fusion deep neural network includes an MM-FusionNet neural network (trained via end-to-end gradient backpropagation, with an inference latency of less than 35ms on a typical industrial PC (Intel i7 + RTX 4060), meeting the real-time control requirements of stacking and reclaiming operations). The MM-FusionNet neural network includes a feature extraction layer, a cross-modal attention layer, and a temporal modeling layer.

[0077] The feature extraction layer uses PointNet++ to extract local-global geometric features f_P from lidar point cloud data, 2D-CNN to extract thermal distribution texture features f_T from infrared temperature thermal image data, 1D-CNN-FFT combined with self-supervised contrastive learning to obtain spectral features f_A from vibration spectrum data, and ResNet-50 to obtain semantic features f_I from visual image data, combined with environmental data.

[0078] The edge computing processing unit executes the Static Fusion Analysis (SFA) module and the Dynamic Fusion Analysis (DFA) module in sequence.

[0079] The static fusion analysis module performs feature alignment, cross-modal attention weighting, and state decoding on the multimodal feature tensor {f_P^t,f_T^t,f_A^t,f_I^t} at the current time t to obtain the environment state vector s^t and the device state vector e^t.

[0080] Feature alignment: The geometric features f_P^t of the LiDAR are mapped to the visual coordinate system through KNN-based geometric alignment, and the visual semantic features f_I^t are projected to the point cloud coordinate system according to the extrinsic matrix. The temperature texture features f_T^t and the vibration spectrum features f_A^t are increased in dimension to a unified 256-dimensional embedding space through 1×1 convolution.

[0081] Cross-modal attention: A multi-head attention mechanism is adopted, and the aligned embedding vector is mapped to query Q, key K, and value V. The attention weight calculation formula is as follows:

[0082] α_ij=softmax\big((Q_iK_j^T) / sqrt(d)\big) (1);

[0083] Where d = 256, and the fusion feature is: F^t = ∑_jα_ijV_j;

[0084] State decoding: F^t is decoded by a fully connected layer + LayerNorm, outputting s^t (128-dimensional environment state) and e^t (64-dimensional device state);

[0085] The dynamic fusion analysis module performs time-series modeling on the static fusion vector sequence H = [F^{t-l+1},…,F^{t}] of the most recent l = 30 frames;

[0086] The specific implementation is a stacked two-layer bidirectional LSTM with 512 hidden units and dropout 0.2.

[0087] The dynamic fusion analysis module outputs a global context vector c^t = BiLSTM(H), which is then used for prediction via a multi-task regression head.

[0088] Future k = 10 frames of stockpile height field \hat{H}^{t+1:t+k};

[0089] Key performance indicators for future equipment: \hat{e}^{t+1:t+k};

[0090] During the training phase, a weighted sum of L2 loss and SmoothL1 loss is used for end-to-end optimization to achieve dynamic prediction of stockpile morphology and equipment health status.

[0091] The three-layer architecture includes a state evaluation layer, a decision generation layer, and an optimization search layer;

[0092] The state assessment layer receives the environmental state vector s^t, the equipment state vector e^t, and the job requirement vector r^t (from the static fusion analysis module and the dynamic fusion analysis module). It generates the fusion representation z^t through two layers of 8-head TinyFormer, with each layer having d_model = 256, and the inference latency is ≤0.5ms.

[0093] The decision generation layer consists of a safety control submodule (based on expert rules) and an efficiency optimization submodule (based on deep reinforcement learning DDPG). The safety submodule matches z^t with the rule table R_safe and outputs the control instruction a_safe that must be executed in real time. The efficiency submodule takes (z^t, r^t) as the state input.

[0094] An Actor network (3×256) is used to output continuous actions a_eff, and a Critic network (3×256) is used to evaluate the Q value and update it offline.

[0095] The optimized search layer performs simulated annealing local search on a_safe⊕a_eff, with the objective function being:

[0096] J=ω1·E_prod-ω2·E_emergy-ω3·E_wear (2);

[0097] ω1, ω2, and ω3 can be configured, and the final control vector a*^t is sent to the stacker-reclaimer.

[0098] The rule table R_safe is located in the rule base, which includes a security control rule base and an optimization suggestion rule base. An example of the security control rule base R_safe (58 IF-THEN rules) is shown below.

[0099] IF dust concentration >4 mg·m -3 AND wind direction = west-southwest THEN {reduce rotation speed by 30% and start the spray dust removal system.}

[0100] The optimization suggestion rule base R_opt (based on fuzzy logic) uses a five-element membership function μ_i(x) to perform fuzzy reasoning on five categories of indicators: energy consumption, efficiency, wear, noise, and emissions, and outputs a weight vector ω to dynamically adjust the objective function J.

[0101] The rule base supports hot reloading; through a visual interface, schedulers can add / edit rules and have them take effect immediately.

[0102] The knowledge acquisition process for the rule base includes offline expert knowledge distillation and online reinforcement learning adaptation.

[0103] In the offline phase, 400 manually annotated historical task records were collected, and 221 expert rules were extracted using an ID3 decision tree as R0. In the online phase, the DDPG submodule writes the top-5 reward experiences into the replay buffer and incrementally updates the network every 1000 interaction steps. When the global average of the Critic... Compared with historical average When the difference exceeds 5%, knowledge transfer is triggered, the latest strategy is distilled into an interpretable rule and added to the optimization suggestion rule base, realizing human-machine collaborative evolution.

[0104] To adapt to various operational scenarios, the system has a built-in ScenarioTemplateLibrary (STL) in the management platform, currently containing 18 templates across three categories: coal, ore, and grain. Each template consists of a quadruple (R_safe, R_opt, weight set Ω, equipment parameter table Π). The platform employs a collaborative filtering-based recommendation algorithm: calculating the cosine similarity between the current operation's feature vector and 2000 historical cases, automatically recommending the Top-3 templates for dispatchers to load with a single click, and allowing online fine-tuning of Ω and Π.

[0105] As shown, the predictive maintenance system includes a Health Index (HIC) module, a Remaining Life Prediction (RULP) module, and a Maintenance Scheduling Optimization (MSO) module:

[0106] The health index calculation module uses multi-kernel support vector regression (MK-SVR). The radial basis kernel κ1 and the linear kernel κ2 regress the health status h1 and h2 on 34-dimensional vibration characteristics, 12-dimensional temperature trend, and 8-dimensional visual defect characteristics, respectively. The final health index h = 0.6·h1 + 0.4·h2.

[0107] The remaining lifetime prediction module uses a Seq2Seq-LSTM (Encoder2×128, Decoder2×128) network to predict the RUL distribution for the next 720 hours. During training, Pinball loss is used to reduce the risk of underestimation.

[0108] The maintenance scheduling optimization module is based on an improved genetic algorithm, which integrates the constraints of the work window, spare parts inventory, and human resources to output the maintenance work order schedule. The improved genetic algorithm has a population of 50, an iteration of 100, a crossover rate of 0.8, and a mutation rate of 0.1. It integrates the constraints of the work window, spare parts inventory, and human resources to output the maintenance work order schedule.

[0109] Training and Deployment: HIC was subjected to 5-fold cross-validation on 18,000 hours of full lifecycle data, with a mean squared error of 2.3; RULP achieved a P50 error of 6.1 hours on the independent test set; MSO reduced the average annual downtime from 320 hours to 190 hours. PMS was deployed on an edge server (DualXeon6326+RTXA4500), with a single health check and RULP inference time of <40ms, meeting the requirements for real-time maintenance scheduling.

[0110] The digital twin visualization platform is used to provide a three-dimensional visualization interface to display the three-dimensional model of the material yard, the distribution of material piles, the location of equipment and the operating status in real time. The digital twin visualization platform visualizes sensor data and fusion results, and provides a human-machine interface for parameter adjustment and strategy configuration.

[0111] The communication network module uses high-speed wireless communication technology to achieve real-time transmission of sensor data, low-latency issuance of control commands, and remote monitoring of system status.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, all equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included within the scope of the present invention.

Claims

1. A smart stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion, characterized in that, It includes a smart stacking and reclaiming control system for circular stockyards based on multi-dimensional sensor fusion, comprising a multi-dimensional sensor network, an edge computing processing unit, an intelligent control decision module, a predictive maintenance system, a digital twin visualization platform, and a communication network module; It also includes step 1: real-time acquisition of lidar point cloud data, temperature field data, vibration time series data and visual image data through a multi-dimensional sensor network, and preprocessing them; Step 2: Input the preprocessed multi-dimensional data into the multimodal fusion deep neural network of the edge computing processing unit to generate environmental perception information and equipment status assessment information in real time; Step 3: The intelligent control decision module receives environmental perception information and equipment status assessment information, generates and optimizes the adaptive stacker-reclaimer control strategy through a three-layer architecture, and sends it to the stacker-reclaimer for execution; Step 4: The predictive maintenance system calculates the equipment health index and predicts the remaining service life based on equipment condition assessment information, and formulates a maintenance plan in conjunction with the work plan; Step 5: Monitor the operation process and equipment status in real time through the digital twin visualization platform, and adjust the control strategy and maintenance plan according to the actual situation.

2. The intelligent stacking and reclaiming control method for circular material yards based on multi-dimensional sensor fusion as described in claim 1, characterized in that, Multiple lidar sensors are evenly distributed around the material yard, providing 360-degree full coverage three-dimensional point cloud data. An array thermal imager is installed in the material yard to monitor the temperature distribution of the material. High-precision accelerometers are installed on the bearings and reducers of the stacker-reclaimer to monitor the vibration characteristics of the equipment in real time. High-definition vision sensors with night vision capabilities are deployed around the material yard to obtain visual information about the environment and equipment. The preprocessing described in step 1 includes: removing outliers from the lidar data using voxel filtering and RANSAC-Ground, followed by surface reconstruction to output a simplified point cloud; Temperature data were obtained by using image enhancement CLAHE and U-Net-based region segmentation to obtain temperature zone boundaries, and then normalized to [0,1]. Vibration data are divided into multi-resolution frequency bands in Hertz units. A two-dimensional time-frequency graph is obtained by using short-time Fourier transform (STFT), and then the main frequency peak features are extracted by 1D-FFT. Visual data is used to perform multi-scale template matching for target detection, extracting ROIs containing key components of the device, and performing color correction and distortion correction.

3. The intelligent stacking and reclaiming control method for circular material yards based on multi-dimensional sensor fusion as described in claim 1, characterized in that, The multimodal fusion deep neural network includes an MM-FusionNet neural network, which includes a feature extraction layer, a cross-modal attention layer, and a temporal modeling layer. The feature extraction layer uses PointNet++ to extract local-global geometric features f_P for lidar point cloud data, 2D-CNN to extract thermal distribution texture features f_T for temperature data, 1D-CNN-FFT combined with self-supervised contrastive learning to obtain spectral features f_A for vibration data, and ResNet-50 to obtain semantic features f_I for visual images.

4. The intelligent stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion as described in claim 3, characterized in that, The edge computing processing unit executes the static fusion analysis module and the dynamic fusion analysis module in sequence; The static fusion analysis module performs feature alignment, cross-modal attention weighting, and state decoding on the multimodal feature tensor {f_P^t,f_T^t,f_A^t,f_I^t} at the current time t to obtain the environment state vector s^t and the device state vector e^t. feature Alignment: The geometric features f_P^t of the LiDAR are mapped to the visual coordinate system through KNN-based geometric alignment, and the visual semantic features f_I^t are projected to the point cloud coordinate system according to the extrinsic matrix. The temperature texture features f_T^t and the vibration spectrum features f_A^t are increased in dimension to a unified 256-dimensional embedding space through 1×1 convolution. Cross-modal attention: A multi-head attention mechanism is adopted, and the aligned embedding vector is mapped to query Q, key K, and value V. The attention weight calculation formula is as follows: α_ij=softmax\big((Q_iK_j^T) / sqrt(d)\big) (1); Where d = 256, and the fusion feature F^t = ∑_jα_ijV_j; State decoding: F^t is decoded by a fully connected layer + LayerNorm, outputting s^t and e^t; The dynamic fusion analysis module performs time-series modeling on the static fusion vector sequence H = [F^{t-l+1},…,F^{t}] of the most recent l = 30 frames; The specific implementation is a stacked two-layer bidirectional LSTM with 512 hidden units and dropout 0.

2. The dynamic fusion analysis module outputs a global context vector c^t = BiLSTM(H), which is then used for prediction via a multi-task regression head. Future k = 10 frames of stockpile height field \hat{H}^{t+1:t+k}; Key performance indicators for future equipment: \hat{e}^{t+1:t+k}; During the training phase, a weighted sum of L2 loss and SmoothL1 loss is used for end-to-end optimization to achieve dynamic prediction of stockpile morphology and equipment health status.

5. The intelligent stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion according to claim 4, characterized in that, Step 3 includes: The three-layer architecture includes a state evaluation layer, a decision generation layer, and an optimization search layer; The state assessment layer receives the environmental state vector s^t, the device state vector e^t, and the job requirement vector r^t. It generates a fused representation z^t through two layers of 8-head TinyFormer, with each layer having d_model = 256. The inference latency is ≤0.5ms. The decision generation layer consists of a safety control submodule and an efficiency optimization submodule. The safety submodule matches z^t with the rule table R_safe and outputs the control command a_safe that must be executed in real time. The efficiency submodule takes (z^t, r^t) as the state input. The Actor network is used to output continuous actions a_eff, and the Critic network is used to evaluate the Q value and update it offline. The optimized search layer performs simulated annealing local search on a_safe⊕a_eff, with the objective function being: J=ω1·E_prod-ω2·E_energy-ω3·E_wear (2); ω1, ω2, and ω3 can be configured, and the final control vector a*^t is sent to the stacker-reclaimer.

6. The intelligent stacking and reclaiming control method for circular material yards based on multi-dimensional sensor fusion as described in claim 5, characterized in that, The rule table R_safe ∈ the rule base, which includes a safety control rule base and an optimization suggestion rule base. The optimization suggestion rule base uses a five-element membership function μ_i(x) to perform fuzzy reasoning on five categories of indicators: energy consumption, efficiency, wear, noise, and emissions, and outputs a weight vector ω to dynamically adjust the objective function J. The knowledge acquisition process for the rule base includes offline expert knowledge distillation and online reinforcement learning adaptation.

7. The intelligent stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion according to claim 1, characterized in that, The predictive maintenance system includes a health index calculation module, a remaining life prediction module, and a maintenance scheduling optimization module. The health index calculation module uses multi-kernel support vector regression (MK-SVR). The radial basis kernel κ1 and the linear kernel κ2 regress the health status h1 and h2 on 34-dimensional vibration characteristics, 12-dimensional temperature trend, and 8-dimensional visual defect characteristics, respectively. The final health index h = 0.6·h1 + 0.4·h2. The remaining lifetime prediction module uses a Seq2Seq-LSTM network to predict the RUL distribution for the next 720 hours, and uses Pinball loss during training to reduce the risk of underestimation. The maintenance scheduling optimization module is based on an improved genetic algorithm, which integrates work window, spare parts inventory and human resource constraints to output maintenance work order schedules. The improved genetic algorithm has a population of 50, an iteration of 100, a crossover rate of 0.8 and a mutation rate of 0.

1.

8. The intelligent stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion according to claim 1, characterized in that, The digital twin visualization platform is used to provide a three-dimensional visualization interface to display the three-dimensional model of the material yard, the distribution of material piles, the location of equipment and the operating status in real time. The digital twin visualization platform visualizes sensor data and fusion results, and provides a human-machine interface for parameter adjustment and strategy configuration.

9. The intelligent stacking and reclaiming control method for circular stockyards based on multi-dimensional sensor fusion according to claim 1, characterized in that, The communication network module uses high-speed wireless communication technology to achieve real-time transmission of sensor data, low-latency issuance of control commands, and remote monitoring of system status.

Images