Multi-mode prediction type intelligent straightening system and method for fully mechanized coal mining face of coal mine

Abstract

The invention discloses a coal mine fully mechanized coal mining face multi-mode prediction type intelligent straightening system and method.The system comprises a multi-mode sensing network, a data fusion prediction module and a control optimization module, and thermal imaging double cameras, pressure / dip angle / stroke / distance measuring sensors and laser scanners are arranged on hydraulic supports and inspection equipment; the comprehensive sensing of the working environment of the fully mechanized coal mining face is realized; fusing visible light, infrared, point cloud and state parameter data, and performing support attitude estimation and trend prediction by adopting Kalman filtering and an LSTM neural network; according to the intelligent support control system and the control method thereof, self-adaptive dynamic straightening control is executed based on a prediction result, remote monitoring and a closed-loop optimization mechanism are combined, the support control precision and the operation safety are improved, the automation level and the response capability of fully-mechanized coal mining operation are improved, and the intelligent support control system is suitable for complex mine pressure conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mine fully-mechanized mining automation control, and in particular to a multi-modal predictive intelligent straightening system and method for a fully-mechanized coal mine working face. Background Art

[0002] Currently, during tunneling and support alignment in fully mechanized coal mining faces, monitoring support posture and coal wall morphology primarily relies on manual experience or a single sensing method. For example, some systems use only pressure sensors to monitor the operating pressure of hydraulic supports, ignoring key information such as support displacement, inclination, and distance from the coal wall. Other visual inspection solutions based on visible light or infrared imaging suffer from image quality degradation due to the high dust environment underground, making continuous and stable monitoring difficult. These single-modality detection methods often suffer from data delays, measurement blind spots, and insufficient anti-interference capabilities, making them difficult to meet the needs of continuous and precise dynamic alignment of fully mechanized coal mining faces.

[0003] In addition, most existing straightening controls use preset threshold triggers or simple PID closed-loop control, which lack the ability to predict and adaptively adjust future support posture development trends. When a support exhibits "large foot misalignment" and overall deformation, manual or automatic compensation can often only be performed after the failure occurs, resulting in impaired operational continuity, increased equipment fatigue, and even safety hazards. Moreover, due to the asynchronous timing of data from various sensors, simple post-processing fusion can easily cause decision delays, making it impossible to achieve coordinated control and overall optimization of the support group. Therefore, it is very necessary to design a multi-modal predictive intelligent straightening system and method for coal mine fully mechanized mining working faces that can perform predictive drive, adaptive regulation, and multi-modal information fusion. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal predictive intelligent straightening system and method for a fully mechanized coal mining face, so as to solve the problems raised in the above-mentioned background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a multi-modal predictive intelligent straightening system and method for a fully mechanized coal mining face, comprising the following steps:

[0006] (1) Deploy multimodal sensing equipment on hydraulic supports and inspection vehicles, including dual thermal imaging cameras, pressure sensors, travel sensors, tilt sensors, distance sensors, and 3D laser scanners, to build a sensing network;

[0007] (2) Perform response calibration, geometric calibration, and time synchronization on the above sensors to ensure data consistency in space and time;

[0008] (3) Upload all types of collected data to the ground control center through CAN bus and industrial Ethernet for unified summary processing;

[0009] (4) Perform extended Kalman filtering on the image, point cloud and state parameter data to achieve noise suppression and state estimation;

[0010] (5) Use long short-term memory neural network (LSTM) to model and predict the time series data of hydraulic support posture;

[0011] (6) When the deviation trend of the support posture exceeds a preset threshold, a straightening control instruction is generated and the support actuator is driven to adjust the posture;

[0012] (7) Collect control results in real time and compare them with the target posture to form error feedback;

[0013] (8) Remote adaptive optimization of fuzzy controller parameters and filter gain coefficients based on feedback error;

[0014] (9) Display three-dimensional status, heat map and control log in a graphical manner in the centralized control center to achieve remote supervision;

[0015] (10) When control anomalies or risks are detected, the system switches to manual control mode and triggers an alarm mechanism to ensure system safety.

[0016] According to the above technical solution, the thermal imaging dual camera in step (1) includes infrared and visible light channels, and is deployed at a frequency of one for every four brackets. Inclination, stroke and pressure sensors are installed on each bracket, and one ranging sensor is shared by every 12 brackets.

[0017] According to the above technical solution, in step (2), the IEEE 1588 protocol or GPS clock is used to synchronize the time of all acquisition devices to ensure that the multi-source data alignment error does not exceed ±10ms.

[0018] According to the above technical solution, in step (4), an extended Kalman filter or an unscented Kalman filter is used to filter and estimate state quantities such as the displacement stroke, the support inclination, and the coal wall distance, and output a three-dimensional posture matrix.

[0019] According to the above technical solution, in step (5), the LSTM model performs posture trend prediction every 10 seconds, the prediction time window is 30 seconds in the future, and the prediction output is the "Bigfoot dislocation risk probability".

[0020] According to the above technical solution, in step (6), if the predicted risk probability exceeds 60%, the straightening action is automatically started, and the adjacent brackets are linked to coordinate adjustments.

[0021] According to the above technical solution, in step (7), the error feedback includes the bracket inclination residual, the displacement deviation and the coal machine trajectory fitting error, which serve as subsequent optimization inputs.

[0022] According to the above technical solution, in step (8), the fuzzy controller parameters are jointly optimized by gradient descent and genetic algorithm to adapt to different support stiffness and coal wall disturbance conditions.

[0023] According to the above technical solution, in step (10), when the control response time exceeds the set upper limit, the error continues to deviate, or the number of iterations exceeds the limit, the system enters a safety protection mode and outputs an audible and visual alarm signal.

[0024] According to the above technical solution, the system includes:

[0025] The perception and data acquisition module is used to collect multimodal raw data in real time at the fully mechanized mining face, including visible light images, infrared thermal images, 3D point clouds, and hydraulic support pressure, displacement, inclination, and ranging information. It also achieves high-precision transmission and synchronization of multi-source data through a unified time synchronization and communication network.

[0026] The data fusion and prediction module is used to perform spatiotemporal alignment and fusion processing on multimodal data, improve data accuracy through filtering algorithms, and dynamically predict the attitude change trend of hydraulic supports based on neural network models, providing data support for early warning and intelligent control;

[0027] The control and optimization module is used to automatically perform support attitude straightening control based on the fusion and prediction results, and perform remote monitoring, error feedback analysis and adaptive optimization of control parameters through the ground control center to ensure the stability of the support attitude and the safety of system operation.

[0028] Compared with the existing technology, the beneficial effects achieved by the present invention are as follows: the present invention constructs a high-precision multimodal perception network by providing a thermal imaging dual camera integrating visible light and infrared imaging, a three-dimensional laser scanner and a variety of state sensors (including pressure, displacement, inclination and ranging sensors); and combines a unified time synchronization mechanism with an extended Kalman filter algorithm to realize the spatiotemporal fusion and dynamic state estimation of multi-source data, effectively improving the monitoring reliability and data accuracy in high dust, low light and strong interference underground environments.

[0029] At the same time, the present invention introduces a support posture evolution prediction model based on long short-term memory neural network (LSTM), which can identify the risk of "big foot dislocation" in advance and trigger adaptive straightening control; through the remote closed-loop optimization mechanism, the fuzzy controller and filter parameters are dynamically adjusted to achieve continuous compression of posture errors and continuous optimization of control effects, greatly improving the intelligent control level, operation stability and system robustness of the hydraulic support of the comprehensive mining working face. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0031] In the attached figure:

[0032] Figure 1 A flowchart of a multi-modal predictive intelligent straightening method for a fully mechanized coal mining face provided in the first embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the modules of a multi-modal predictive intelligent straightening system for a fully mechanized coal mining face provided in the second embodiment of the present invention;

[0034] Figure 3 A multi-sensor data fusion flow chart provided in Example 1 of the present invention;

[0035] Figure 4 A logic diagram of the support straightening control provided in the first embodiment of the present invention;

[0036] Figure 5 This is a timing diagram of the predictive straightening strategy provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] Example 1: Figure 1 This is a flow chart of a multi-modal predictive intelligent straightening method for a fully-mechanized coal mining face provided in the first embodiment of the present invention. This embodiment is applied to a fully-mechanized coal mining face. This method can be executed by a multi-modal predictive intelligent straightening system for a fully-mechanized coal mining face provided in the second embodiment of the present invention. Figure 1 As shown, the method specifically includes the following steps:

[0039] Step 1: Deployment and installation of multimodal sensing network;

[0040] For example, a pan-tilt thermal imaging dual camera is installed on every four hydraulic supports in the fully mechanized mining face. The pan-tilt thermal imaging dual camera is equipped with visible light and infrared sensors, which can monitor the equipment status in real time through visible light and infrared thermal imaging technology in a dusty environment. On each hydraulic support, three pressure sensors, two displacement sensors and two inclination sensors are integrated. The pressure sensor is used to monitor the working pressure of the hydraulic system in real time, the displacement sensor is used to detect the displacement of the support, and the inclination sensor is used to collect the inclination angle of the support in real time. At the same time, one distance measuring sensor is configured for every 12 supports to measure the distance to the coal wall. The sensor needs to be installed at the top of the support and can measure the real-time distance of the coal wall through it, with an accuracy requirement of ±5mm;

[0041] For example, a three-dimensional laser scanner is installed on the inspection vehicle to scan the working surface and generate a three-dimensional model. The laser scanner should be installed on the top of the vehicle to ensure that its scanning angle can cover the entire working surface, and regularly scan the working surface to generate high-precision three-dimensional point cloud data for subsequent posture calibration and straightening control. All sensors, cameras and laser scanners need to be connected to the power system through cables to ensure that the equipment can obtain power normally and operate stably. For the pan-tilt thermal imaging dual camera and other sensors, a standard signal interface (such as RS-485, CAN bus, etc.) is used, and a dedicated cable is used to connect to the data acquisition module to ensure stable data transmission. The data of each sensor should be able to be fed back to the data acquisition unit in real time, and transmitted to the ground control center through the data bus for real-time monitoring;

[0042] For example, a hybrid architecture of CAN bus and industrial Ethernet is used to achieve data interconnection and remote data communication between devices. The CAN bus is used for local data transmission between various sensors and actuators to ensure real-time data synchronization between local devices; industrial Ethernet serves as the backbone network for data aggregation and transmission, responsible for transmitting the data collected by various modules (such as coal mining machines, hydraulic supports, thermal imaging equipment, etc.) to the ground control center. A dedicated data acquisition module is installed on each support and inspection vehicle, and the collected signals are transmitted to the ground control center via the CAN bus. The ground control center is equipped with unified network equipment, including switches, routers, etc., to ensure smooth data transmission and achieve interconnection with other equipment.

[0043] Step 2: Sensor calibration and synchronization;

[0044] For example, the two channels of the thermal imaging dual camera, namely the visible light channel and the infrared channel, are geometrically calibrated separately to ensure that the alignment accuracy of the two channel images in the spatial coordinate system meets the fusion requirements. Using a standard checkerboard calibration plate, visible light and infrared images are collected at multiple distances and angles. The intrinsic parameter matrix (such as focal length, principal point position) and distortion parameters are obtained through image feature extraction and matching algorithms. The relative pose transformation matrix between the two channels is further calculated through extrinsic parameters.

[0045] For example, based on the aforementioned calibration parameters, image space alignment is performed and the alignment accuracy is verified. Through manual calibration points and automatic image fusion detection, the spatial alignment error is ensured to be controlled within ±5mm, thereby meeting the precise monitoring requirements of the equipment in complex coal mine environments. In terms of radiation calibration, a standard heat source blackbody radiator is used to set a multi-level temperature gradient, collect the correspondence between the infrared image and the actual temperature, and construct an infrared image grayscale-temperature mapping curve to achieve quantitative temperature expression of the infrared thermal image;

[0046] Exemplarily, a standard weight loading method is used to sequentially load multiple standard weights of known mass to the sensor interface or the simulated pressure interface, and the sensor output voltage / current signal value is recorded at each loading point. A mapping relationship between pressure and output is established, and a linear or nonlinear calibration curve is fitted to generate a response coefficient matrix of the pressure sensor.

[0047] Stroke sensor calibration: By controlling the bracket push device to move on the standard scale guide rail, the electrical signal output values ​​corresponding to different strokes are recorded, the stroke-signal relationship is fitted, and the stroke calibration curve and error compensation coefficient are obtained.

[0048] Tilt sensor calibration: Use a high-precision tilt reference plate to test the sensor output signal at known tilt angles (such as 0°, 5°, 10°, 15°, etc.). By comparing the actual angle with the sensor output, a mapping is established, and offset errors and nonlinear responses are corrected to obtain the tilt sensor calibration matrix.

[0049] Distance sensor calibration: Using a standard distance measurement platform (such as a sliding track device), multiple standard distance measurement points are set. The sensor's output values ​​at different distances are recorded and used to fit a mathematical model between distance and signal. Considering the presence of interference factors such as dust and smoke in actual coal mine environments, the sensor's anti-interference performance should be tested simultaneously under the corresponding interference background and compensation corrections should be performed.

[0050] After all sensors are calibrated, all calibration data (including internal parameter matrices, response coefficients, mapping functions, etc.) is organized and formatted into a parameter dictionary, which is then uploaded to the system's intelligent decision-making unit. Upon receiving the calibration parameters, the intelligent decision-making unit updates the configuration of each sensor, ensuring data correction and error compensation based on actual calibration values ​​during operation. This download process utilizes the Industrial Ethernet protocol, and the configuration data packets are encrypted to prevent tampering or loss of calibration parameters during transmission.

[0051] In addition to spatial calibration and response calibration, the system also requires time synchronization calibration for sensor data acquisition. By deploying a unified time synchronization server (or using the GPS time reference / IEEE 1588 protocol), all sensor data acquisition timestamps are corrected to a unified time coordinate system, eliminating multi-source information fusion failures caused by data delays or synchronization errors.

[0052] Step 3: Data collection triggered by the shearer position;

[0053] In this embodiment, to achieve accurate perception and dynamic modeling of the fully mechanized mining face operating environment, a data acquisition mechanism, triggered by changes in the shearer's position, links dual thermal imaging cameras, various status sensors, and a laser scanning device on an inspection vehicle to simultaneously collect multimodal information from key areas of the coal mining operation.

[0054] For example, the position sensor (such as an encoder or laser ranging module) integrated on the coal mining machine detects its movement displacement on the guide rail or working surface in real time; when the sensor detects that the displacement increment compared to the last recorded position exceeds a set threshold, such as 50mm, the system determines that the position of the coal mining machine has changed significantly and triggers a data collection process; the displacement threshold can be dynamically adjusted according to the actual working conditions to take into account the collection frequency and data volume control. Subsequently, the system automatically identifies the current position of the coal mining machine and determines the three adjacent pan-tilt thermal imaging dual cameras in front and behind it as the target equipment for this collection; each thermal imaging dual camera adjusts its direction through the electric pan-tilt, automatically aligns with the area where the coal mining machine is located, and performs coordinated steering; after completing the target lock, the video stream collection tasks of the visible light channel and the infrared thermal imaging channel are simultaneously started; the video collection content includes the coal mining machine operating status, coal wall morphology, surrounding support heat distribution, etc. The fused image is used for subsequent fault detection, temperature anomaly analysis and visual recognition tasks. The collection trigger signal is sent to the data synchronization control unit, which uniformly controls the following data synchronization collection:

[0055] The pressure sensor on each hydraulic support collects and records the actual pressure value in the current hydraulic cylinder;

[0056] Tilt sensor: collects information about the spatial posture change of the bracket, including lateral and longitudinal tilt angles;

[0057] Stroke sensor: records the current stroke value of the telescopic component of the bracket and reflects its actual displacement;

[0058] Distance measuring sensor (one for every 12 supports): obtains the real-time distance from the front of the support to the coal wall or other obstacles;

[0059] All sensor data is reported to the edge intelligent processing unit in real time via the CAN bus or industrial Ethernet, and is marked with a unified timestamp to ensure data time alignment. In parallel with the sensor synchronous acquisition process, the 3D laser scanner deployed on the track inspection vehicle starts after receiving the acquisition trigger command; the vehicle moves to the area around the coal mining machine based on a preset path or autonomous positioning method, and collects the current working face spatial morphology in the form of a high-density point cloud; after the scan data is filtered and aligned, a 3D model of the current working face is generated, which visually reflects the progress of the coal wall, the spatial layout of the equipment, and the environmental structure;

[0060] This model is used to support subsequent applications such as mine pressure prediction, equipment interference analysis, assisted driving, and inspection task path planning. All collected multimodal data is initially fused and processed in the edge intelligent device to generate the following structured output:

[0061] Thermal imaging / visible light image fusion map;

[0062] Pressure-displacement-posture coordinated change curve;

[0063] Real-time 3D reconstruction of the working surface;

[0064] It can also be reported in real time to the ground control center or cloud decision-making platform through the industrial network, supporting higher-level data analysis, model training and remote monitoring command generation.

[0065] Through this data collection strategy based on coal mining machine position triggering, the target focus, data timeliness and processing efficiency of the fully mechanized mining working face perception system are significantly improved, providing a solid perception foundation for subsequent intelligent decision-making.

[0066] Step 4: Spatiotemporal fusion and filter calibration of multi-source data;

[0067] In this embodiment, to achieve dynamic modeling and accurate state restoration of the fully mechanized mining face operation process, the thermal imaging video, 3D laser point cloud, and multiple sensor data will be further spatiotemporally aligned and multi-source data fused. A filtering algorithm will also be introduced for real-time calibration and noise suppression to improve the system's accuracy in sensing the coal machine operating status and hydraulic support posture.

[0068] For example, various data acquisition modules (including thermal imaging dual cameras, 3D laser scanners, and pressure, inclination, travel, and distance sensors) are integrated with high-precision clock modules, and a unified standard timestamp is applied to the collected data. The system sorts and interpolates data streams from different sources according to timestamps to achieve temporal alignment of multimodal data. At the same time, the point cloud data obtained by 3D laser scanning undergoes coordinate registration and distortion correction during preprocessing to establish a mapping relationship with the spatial coordinates of the thermal imaging image. Finally, the thermal imaging image frame (visible light + infrared fusion), point cloud model, and state sensor data are fused to construct a unified 3D state map, which reflects the position of the coal machine, the coal wall morphology, and the support posture in real time.

[0069] For example, to address issues such as jitter, noise, and drift that may exist in the raw sensor data, the system constructs a state estimation model based on the extended Kalman filter (EKF) algorithm or the unscented Kalman filter (UKF) algorithm;

[0070] The state vector is constructed using the coal wall distance (obtained by the laser ranging sensor and point cloud fitting), the support movement stroke, and the inclination data as observation quantities. The Kalman filter dynamically adjusts the estimated value based on the residual between the state prediction at the previous moment and the current observation quantity, and outputs a high-confidence fusion result after eliminating outliers. During this process, the filtering system can also combine auxiliary inputs of acceleration or angular velocity (such as those from the inertial measurement unit (IMU)) to further improve the accuracy of attitude estimation.

[0071] The state estimation result of the filter output is obtained:

[0072] The real-time attitude matrix of each hydraulic support in three-dimensional space (including position, inclination, and displacement);

[0073] The trajectory curve and direction vector of the coal machine at the current working face in the coordinate system;

[0074] The system overlays these structured status data onto the three-dimensional model map to achieve real-time fusion visualization of multi-source information; at the same time, the filtered status parameters are synchronously sent to the edge intelligent processing unit and the ground control center to provide accurate input for the next step of intelligent decision-making, predictive analysis, and equipment control.

[0075] Step 5: Bracket posture prediction and dynamic straightening control;

[0076] In this embodiment, a multi-frame linkage dynamic straightening strategy is combined to achieve early warning and adaptive correction control of the "large foot misalignment" problem of the hydraulic support;

[0077] For example, the system regularly collects historical status data of each group of hydraulic supports (12 supports per group), including inclination angle change value, displacement stroke, distance error between adjacent supports, coal machine motion trajectory, support posture adjustment frequency, etc.; a long-term and short-term memory neural network is used to construct a time series model to capture the nonlinear evolution law of the support posture during continuous operation; the model training phase is based on labeled learning based on a large number of typical working condition samples, and the training output variable is the probability of "big foot misalignment event", that is, the risk indicator of the accumulation of inclination errors of a group of supports due to factors such as poor coordination and coal wall disturbance;

[0078] For example, during actual operation, the intelligent decision-making unit periodically (e.g., every 10 seconds) calls the deployed LSTM model to predict the posture deviation trend of the current support group within the next N seconds (e.g., 30 seconds). If the model outputs a "large foot misalignment probability" exceeding a set threshold (e.g., 60%), the system determines that the support group is at risk of significant posture inconsistency and triggers an active intervention mechanism. Simultaneously, the system combines the posture trends of neighboring supports to determine whether the deviation is a local disturbance or a global deformation trend, thereby optimizing subsequent straightening strategies. After each control, the system automatically evaluates the prediction accuracy and control effectiveness, using the feedback to fine-tune the model and gradually improve prediction stability and control precision. Key indicators (e.g., misalignment prediction rate, control frequency, and average corrected stroke) are uploaded to the ground control center as a basis for scheduling strategy evaluation. This enables early detection of hydraulic support posture change trends and establishes a comprehensive, predictive-driven, and adaptive dynamic straightening mechanism, effectively reducing "large foot misalignment" and improving operational continuity and safety in the fully mechanized mining face, significantly enhancing the level of intelligence.

[0079] Step 6: Remote monitoring and closed-loop optimization.

[0080] For example, in the hydraulic support posture straightening control system for a fully mechanized mining face, this embodiment establishes a closed-loop control mechanism based on remote monitoring and adaptive parameter optimization to further improve control accuracy and stability. This mechanism leverages a ground-based centralized control center to achieve real-time perception of system status, error feedback analysis, and intelligent parameter adjustment. Specifically, the ground-based centralized control center is connected to the underground hybrid communication network via Industrial Ethernet. This hybrid network comprises a CAN bus and Ethernet segments, ensuring high-speed and stable transmission of various data types.

[0081] For example, the system continuously receives multi-source heterogeneous data from the work surface, including visible light and infrared image streams transmitted by dual pan-tilt thermal imaging cameras, work surface point cloud models and spatial annotations generated by 3D laser scanning equipment, data collected by hydraulic support pressure sensors, tilt sensors, travel sensors, and ranging sensors, and feedback signals from the completion of each support actuator. This information is dynamically visualized in a layered manner on the ground-side interface, supporting the generation of situational heat maps by support group, individual support control log tracing, and global status trend analysis.

[0082] For example, the ground control system periodically analyzes the collected data and compares the actual posture of the hydraulic support after straightening with the desired posture. Key error metrics are extracted: the support angle residual (i.e., the difference between the current actual tilt angle and the desired angle), the stroke correction error (the deviation between the completed stroke value and the target value), and the coal machine trajectory fitting error (the spatial fitting offset between the laser point cloud and the coal machine's historical trajectory). All error values ​​are integrated into a set of error state vectors, which are used as input parameters for the control strategy optimization algorithm at a set period (e.g., every 5 minutes).

[0083] For example, the system supports remote dynamic optimization of the core algorithm parameters in the control module. This includes adjusting the fuzzy rule weights and membership function boundaries of the fuzzy controller to adapt to the hydraulic support stiffness and coal wall deformation characteristics under different operating conditions. It also supports adaptive correction of the Kalman filter gain coefficient to dynamically adjust the fusion weight between measured and predicted values, thereby improving the accuracy of attitude estimation. The optimization algorithm combines gradient descent with genetic algorithms, balancing local response speed with global search capabilities, achieving optimal control strategy adjustment while ensuring stable system operation.

[0084] For example, through multiple iterations of the closed-loop optimization mechanism described above, the support posture alignment system can control the residual inclination error to within ±0.5°, the displacement error to within ±10mm, and the coal machine trajectory fitting error to within ±30mm. This significantly shortens the posture stabilization time, effectively reducing the frequency of manual intervention and lowering system fatigue. The parameters obtained from each round of optimization are archived in a versioned manner to facilitate subsequent model migration and system reproduction under different operating conditions.

[0085] For example, to ensure the safety of the control system, the ground control center has set up manual intervention permissions, which can immediately terminate the optimization process and switch to manual control mode if abnormal error fluctuations or sensor failures are detected. At the same time, the system has multiple safety alarm mechanisms. If the number of optimization iterations exceeds the limit, the control response delay exceeds the safety threshold, or the attitude residual continuously deviates from the set range, the system will automatically trigger an audible and visual alarm and lock the control process to prevent further risk.

[0086] In summary, this embodiment provides a closed-loop attitude control system covering remote perception, error analysis, adaptive optimization and safety intervention functions, which significantly improves the response efficiency, control accuracy and system robustness in the hydraulic support straightening control process, and provides key technical guarantees for intelligent comprehensive mining operations.

[0087] Embodiment 2: Embodiment 2 of the present invention provides a multi-modal predictive intelligent straightening system for a fully mechanized coal mining face. Figure 2 A schematic diagram of the module composition of a multi-modal predictive intelligent straightening system for a fully mechanized coal mining face provided in the second embodiment of the present invention is shown in FIG. Figure 2 As shown, the system includes:

[0088] The perception and data acquisition module is used to collect multimodal raw data in real time at the fully mechanized mining face, including visible light images, infrared thermal images, 3D point clouds, and hydraulic support pressure, displacement, inclination, and ranging information. It also achieves high-precision transmission and synchronization of multi-source data through a unified time synchronization and communication network.

[0089] The data fusion and prediction module is used to perform spatiotemporal alignment and fusion processing on multimodal data, improve data accuracy through filtering algorithms, and dynamically predict the attitude change trend of hydraulic supports based on neural network models, providing data support for early warning and intelligent control;

[0090] The control and optimization module is used to automatically perform support attitude straightening control based on the fusion and prediction results, and conduct remote monitoring, error feedback analysis, and adaptive optimization of control parameters through the ground control center to ensure support attitude stability and system operation safety;

[0091] In some embodiments of the present invention, the perception and data acquisition module includes:

[0092] The sensor deployment and installation module is used to arrange pan-tilt thermal imaging dual cameras, pressure / tilt / travel / range sensors, and 3D laser scanners at predetermined intervals on the fully mechanized mining face to acquire multimodal raw data such as visible light, infrared, point cloud, and mechanical state data.

[0093] The communication network architecture module is used to adopt a hybrid network of CAN bus and industrial Ethernet, and collect the signals of various sensors to the ground control center through a dedicated data acquisition module, ensuring the real-time transmission and stable interconnection of multi-source data;

[0094] The calibration and time synchronization module is used to perform geometric, response, and radiometric calibration on the imaging channel, mechanical sensor, and laser scanner, and implements unified system-wide timestamps based on IEEE 1588 / GPS to eliminate calibration bias and timing errors.

[0095] In some embodiments of the present invention, the data fusion and prediction module includes:

[0096] The spatiotemporal alignment and multi-source fusion module is used to interpolate and match image frames, point cloud data, and sensor readings according to timestamps, construct a unified 3D state map, and connect space-time information.

[0097] The filtering and noise suppression module is used to dynamically estimate and correct residuals of the original observation values ​​based on the extended / unscented Kalman filter algorithm, eliminate jitter and interference, and output high-confidence attitude and environmental status;

[0098] The posture evolution prediction module is used to use the long short-term memory (LSTM) neural network to model the historical support posture time series, predict the risk of "big foot dislocation" or group imbalance in advance, and provide an early warning basis for subsequent control.

[0099] In some embodiments of the present invention, the control and optimization module includes:

[0100] A real-time straightening control module is used to automatically generate a straightening instruction when the predicted probability or real-time error exceeds a threshold, and to adaptively correct the posture of the bracket through the hydraulic actuator to restore the desired state;

[0101] Remote monitoring and closed-loop optimization module, used by the ground control center to compare actual and target postures, extract error vectors, and perform online parameter optimization (such as fuzzy rule weights and filter gains), iteratively updating the control strategy;

[0102] The safety assurance and emergency intervention module is used to monitor the number of optimization iterations, response delays, and residual fluctuations, triggering audible and visual alarms or switching to manual mode to ensure that automatic parameter adjustment can be quickly terminated and on-site personnel and equipment can be protected when an abnormality occurs.

[0103] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0104] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-modal predictive intelligent straightening method for a fully mechanized coal mining face, characterized by: The steps include: (1) Deploy multimodal sensing equipment on hydraulic supports and inspection vehicles, including dual thermal imaging cameras, pressure sensors, travel sensors, tilt sensors, distance sensors, and 3D laser scanners, to build a sensing network; (2) Perform response calibration, geometric calibration, and time synchronization on the above sensors to ensure data consistency in space and time; (3) Upload all types of collected data to the ground control center through CAN bus and industrial Ethernet for unified summary processing; (4) Perform extended Kalman filtering on the image, point cloud and state parameter data to achieve noise suppression and state estimation; (5) Use long short-term memory neural network (LSTM) to model and predict the time series data of hydraulic support posture; (6) When the deviation trend of the support posture exceeds a preset threshold, a straightening control instruction is generated and the support actuator is driven to adjust the posture; (7) Collect control results in real time and compare them with the target posture to form error feedback; (8) Remote adaptive optimization of fuzzy controller parameters and filter gain coefficients based on feedback error; (9) Display three-dimensional status, heat map and control log in a graphical manner in the centralized control center to achieve remote supervision; (10) When control anomalies or risks are detected, the system switches to manual control mode and triggers an alarm mechanism to ensure system safety.

2. A multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: The thermal imaging dual camera described in step (1) includes infrared and visible light channels, and is deployed at a frequency of one for every four brackets. The inclination, travel and pressure sensors are installed on each bracket, and the distance sensor is shared by one for every 12 brackets.

3. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (2), the IEEE 1588 protocol or GPS clock is used to synchronize the time of all acquisition devices to ensure that the multi-source data alignment error does not exceed ±10ms.

4. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (4), an extended Kalman filter or an unscented Kalman filter is used to filter and estimate the displacement stroke, support inclination, and coal wall distance, and a three-dimensional posture matrix is ​​output.

5. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (5), the LSTM model performs posture trend prediction every 10 seconds, with the prediction time window being the next 30 seconds, and the prediction output is the “risk probability of Bigfoot dislocation”.

6. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (6), if the predicted risk probability exceeds 60%, the straightening action is automatically started, and the adjacent brackets are coordinated and adjusted.

7. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (7), the error feedback includes the bracket inclination residual, the displacement deviation and the coal machine trajectory fitting error, which serve as the subsequent optimization input.

8. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (8), the fuzzy controller parameters are jointly optimized by gradient descent and genetic algorithm to adapt to different support stiffness and coal wall disturbance conditions.

9. The multi-modal predictive intelligent straightening method for a fully mechanized coal mining face according to claim 1, characterized in that: In step (10), when the control response time exceeds the set upper limit, the error continues to deviate, or the number of iterations exceeds the limit, the system enters the safety protection mode and outputs an audible and visual alarm signal.

10. A multi-modal predictive intelligent straightening system for fully mechanized coal mining working faces, characterized by: The system includes: The perception and data acquisition module is used to collect multimodal raw data in real time at the fully mechanized mining face, including visible light images, infrared thermal images, 3D point clouds, and hydraulic support pressure, displacement, inclination, and ranging information. It also achieves high-precision transmission and synchronization of multi-source data through a unified time synchronization and communication network. The data fusion and prediction module is used to perform spatiotemporal alignment and fusion processing on multimodal data, improve data accuracy through filtering algorithms, and dynamically predict the attitude change trend of hydraulic supports based on neural network models, providing data support for early warning and intelligent control; The control and optimization module is used to automatically perform support attitude straightening control based on the fusion and prediction results, and perform remote monitoring, error feedback analysis and adaptive optimization of control parameters through the ground control center to ensure the stability of the support attitude and the safety of system operation.

Images