Coal mine production equipment control method and system based on remote body interaction

Abstract

The invention relates to the technical field related to intelligent production control, in particular to a coal mine production equipment control method and system based on remote body interaction, and the method comprises the steps: coal mine production equipment uploads parameters, analyzes the state, configures a twin architecture, and carries out local decision abnormity early warning. A remote end is communicated with an interaction device, biological characteristics are reflected as a pose instruction, time delay compensation and track deviation prediction are embedded, a multi-mode feedback unit triggers smooth control according to the pose instruction, and tactile feedback is enhanced in combination with a vibration spectrum. The technical problems that the network condition of the underground coal mine environment is poor, obvious communication delay exists in remote control, operation instruction lagging exists, and the response speed and operation safety of equipment are affected are solved, dynamic time delay is predicted through the LSTM network, track deviation is compensated, the real-time performance and stability of remote control are improved, accurate execution of instructions is ensured, and the service life of the equipment is prolonged. And the remote operation efficiency and reliability of the coal mine equipment are improved through high-precision pose control and real-time compensation.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent production control, and particularly to a control method and system for coal mine production equipment based on remote embodied interaction. Background Art

[0002] The coal mine production environment is complex and highly uncertain. With the development of intelligent technologies, remote control has become an important direction for coal mine equipment management. However, the underground coal mine environment is harsh (such as high dust, low light, and strong vibration). At present, the remote control of coal mine equipment mainly relies on video monitoring + traditional remote operation. Operators manually control by observing the video feedback images, which has problems such as high latency, inaccurate operation, and lack of tactile feedback.

[0003] In summary, in the prior art, there are technical problems such as poor network conditions in the underground coal mine environment, significant communication delays in remote control, lagging operation instructions, and affecting the equipment response speed and operation safety. Summary of the Invention

[0004] The present application provides a control system for coal mine production equipment based on remote embodied interaction, aiming to solve the technical problems in the prior art, such as poor network conditions in the underground coal mine environment, significant communication delays in remote control, lagging operation instructions, and affecting the equipment response speed and operation safety.

[0005] In view of the above problems, the technical solution of the present application is as follows: On the one hand, the present application provides a control method for coal mine production equipment based on remote embodied interaction. The method includes: The coal mine production equipment uploads basic technical parameters; analyzes the equipment operation status based on the basic technical parameters, configures the equipment operation twin architecture, and makes local decisions through the equipment historical operation instances in the normal operation state; automatically marks and generates warning information using the abnormal data confirmed by the local decision; after the remote control end receives the warning information, it connects to the wearable interaction device; maps the biometric indicators uploaded by the wearable interaction device into the pose control instructions of the coal mine production equipment; embeds a time delay compensation factor in the remote embodied interaction channel for transmitting the pose control instructions, and predicts the equipment motion trajectory deviation within the dynamic time interval using an LSTM network; uses the multi-modal feedback unit of the wearable interaction device to trigger a judgment of a smooth control signal according to the pose control instructions and the equipment motion trajectory deviation; if the smooth control signal is triggered, synchronously enhance the tactile feedback intensity of the wearable interaction device in combination with the vibration spectrum of the coal mine production equipment.

[0006] On the other hand, the present application provides a control system for coal mine production equipment based on remote embodied interaction. The system includes: A decision-making module is used for coal mine production equipment to upload basic technical parameters; analyze the equipment operation status based on the basic technical parameters, configure the equipment operation twin architecture, and make local decisions through the equipment historical operation instances in the normal operation state; an automatic marking module is used to automatically mark and generate warning information using the abnormal data confirmed by the local decision; after receiving the warning information, a remote control terminal connects to a wearable interaction device; an embedding module is used to map the biometric indicators uploaded by the wearable interaction device into pose control instructions for the coal mine production equipment; a time delay compensation factor is embedded in the remote embodied interaction channel for transmitting the pose control instructions to predict the equipment motion trajectory deviation within a dynamic time period interval using an LSTM network; a trigger judgment module is used to use the multimodal feedback unit of the wearable interaction device to make a trigger judgment on a smooth control signal according to the pose control instructions and the equipment motion trajectory deviation; a synchronization enhancement module is used to, if the smooth control signal is triggered, synchronously enhance the tactile feedback intensity of the wearable interaction device in combination with the vibration spectrum of the coal mine production equipment.

[0007] In summary, in one or more technical solutions provided in this application, an LSTM network is used to predict dynamic time delay and compensate for trajectory deviation, improving the real-time performance and stability of remote control, ensuring accurate execution of instructions, and achieving the technical effects of improving the efficiency and reliability of remote operation of coal mine equipment with high-precision pose control and real-time compensation. Description of the Drawings

[0008] Figure 1 This is a schematic flowchart of a method for controlling coal mine production equipment based on remote embodied interaction provided by this application; Figure 2 This is a schematic structural diagram of a control system for coal mine production equipment based on remote embodied interaction provided by this application.

[0009] Description of the reference numerals: Decision-making module M100, automatic marking module M200, embedding module M300, trigger judgment module M400, synchronization enhancement module M500. Detailed Description of the Embodiments

[0010] Embodiment 1

[0011] The following specifically describes this application with reference to the drawings. As Figure 1 shown, this application provides a method for controlling coal mine production equipment based on remote embodied interaction, where the method includes: S1: Coal mine production equipment uploads basic technical parameters; analyzes the equipment operation status based on the basic technical parameters, configures the equipment operation twin architecture, and makes local decisions through the historical operation instances of the equipment in the normal operation state; S2: Automatically marks and generates warning information using the abnormal data confirmed by local decisions; after receiving the warning information, the remote control terminal connects to the wearable interaction device.

[0012] Specifically, in the field of coal mine remote control, the coal mine production equipment uploading basic technical parameters means that the equipment uploads its own operation parameters (such as power, temperature, vibration frequency, etc.) to the remote control system; the equipment operation status analysis means judging the health status of the equipment by analyzing these parameters; configuring the equipment operation twin architecture is to construct a virtual equipment model using the uploaded parameters to simulate the operation status of the real equipment; local decision-making means making intelligent decisions based on historical operation data and the current status; automatically marking and generating warning information means identifying abnormal data and sending out alarms in a timely manner; connecting to the wearable interaction device means connecting the remote control terminal to the interaction device (such as VR / AR helmets, gloves, etc.) worn by the operator.

[0013] In a feasible implementation, the equipment collects operation data (such as temperature, pressure, etc.) in real time through a sensor network and uploads it to the remote control center server on the ground with a millisecond-level delay through the underground wireless communication module (such as Wi-Fi 6 or 5G); the control center analyzes the uploaded parameters using big data analysis tools (such as Spark or Hadoop) to construct a digital twin model of the equipment. For example, by analyzing parameters such as the cutting head speed, traction speed, and load current of a shearer, a virtual model synchronized with the real equipment is generated, and this model can simulate the response of the equipment under different working conditions.

[0014] Based on machine learning algorithms (such as decision trees or neural networks), compare and analyze the uploaded data and historical data. If an abnormality is detected (such as the equipment vibration frequency exceeding 3 times the standard deviation of the normal range), automatically mark the abnormal point and generate warning information; after receiving the warning at the remote control terminal, establish a connection with the operator's interaction device through a dedicated network (such as the coal mine 5G private network). For example, in the control center on the ground, the AR glasses worn by the operator are connected to the shearer control system through the 5G network, and the network delay is controlled within 50 ms to ensure the real-time transmission of operation instructions.

[0015] From data collection to intelligent decision-making and then to real-time control, through accurate data analysis and low-latency remote control, the safety and efficiency of remote operation of coal mine equipment have been effectively improved.

[0016] S3: Map the biometric metrics uploaded by the wearable interaction device to pose control instructions for the coal mine production equipment; embed a time delay compensation factor in the remote embodied interaction channel for transmitting the pose control instructions to predict the deviation of the equipment movement trajectory within a dynamic time period using an LSTM network; S4: Use the multi-modal feedback unit of the wearable interaction device to trigger a judgment of a smooth control signal based on the pose control instructions and the equipment movement trajectory deviation; S5: If the smooth control signal is triggered, synchronously enhance the tactile feedback intensity of the wearable interaction device in combination with the vibration spectrum of the coal mine production equipment.

[0017] Specifically, biometric metrics refer to the physiological and behavioral data generated by an operator through a wearable interaction device (such as a VR / AR headset, smart gloves, etc.), including eye-tracking data (i.e., the operator's fixation points, fixation durations, etc.) and gesture recognition data (such as hand movements, gesture trajectories, etc.). Pose control instructions refer to commands for controlling the posture and position of coal mine production equipment converted based on biometric metrics. The remote embodied interaction channel refers to the communication link connecting the operator's interaction device and the remote control end for real-time transmission of control instructions, and its time delay compensation factor is a compensation measure for signal transmission delay. The LSTM network is a long short-term memory neural network used to predict the deviation of the equipment movement trajectory. The multi-modal feedback unit refers to a component in the interaction device that integrates multiple feedback forms (such as visual, auditory, tactile, etc.) and can provide comprehensive feedback based on the equipment status and control instructions. The triggering judgment of the smooth control signal refers to judging whether to start the smooth control algorithm based on specific conditions to optimize the equipment movement control process. The vibration spectrum refers to the frequency and amplitude distribution characteristics of the vibration signals generated by the coal mine production equipment during operation, reflecting the vibration state of the equipment. The tactile feedback intensity refers to the intensity of the tactile signal fed back to the operator by the interaction device after being adjusted according to the equipment vibration spectrum, enabling the operator to more intuitively perceive the vibration situation of the equipment.

[0018] In a feasible implementation, the operator wears a wearable interaction device, such as AR glasses and smart gloves. Its eye-tracking module captures the operator's fixation points at a sampling frequency of 0.01 seconds. When the fixation point stays in a specific area of the virtual device interface for more than 0.5 seconds, it is parsed as a selection operation on a certain part of the equipment. At the same time, hand movement data is collected. Based on a pre-trained gesture semantic library, a grasping gesture is mapped to a lowering instruction for the shearer cutting head, and an opening gesture is mapped to a raising instruction, etc. In this way, the operator's natural biometric features are directly converted into precise equipment control commands, improving the intuitiveness and efficiency of remote operation.

[0019] In the remote embodied interaction channel, the signal transmission delay usually fluctuates between 30 and 100 milliseconds. The system monitors the delay change in real time through the delay compensation algorithm and dynamically adjusts the compensation parameters. At the same time, based on the device operation data collected in the past 10 seconds (including the control instruction sequence and the actual operation trajectory once every 0.1 second), the LSTM network can predict the deviation of the device motion trajectory after training. This process effectively reduces the control lag caused by network delay, improves the response speed of the device, and significantly enhances the real-time performance and stability of remote control.

[0020] The multi-modal feedback unit integrates the visual, auditory, and tactile feedback information of the device. When the change rate of the pose control instruction between two consecutive moments (T-1 and T moments, with a time interval of 0.1 second) exceeds the preset threshold (such as an angular change rate exceeding 10 degrees / second and a position change rate exceeding 0.5 meters / second), and the cumulative trajectory deviation amount predicted by the LSTM exceeds the deviation amount threshold (such as a cumulative deviation exceeding 5 millimeters), the fuzzy logic controller makes inferences based on the fuzzy rule base and the membership function. After the smooth control algorithm is started, the smoothness of the device movement is improved, effectively reducing the device jitter caused by sudden changes in the control signal, and at the same time improving the operation accuracy.

[0021] The tactile feedback unit of the wearable interaction device generates different levels of vibration feedback modes according to the vibration spectrum of the coal mine equipment (the frequency range is usually between 0 and 200 Hz, and the amplitude is between 0.1 and 5 mm). For example, when the shearer cuts hard rock and the vibration frequency reaches 80 Hz and the amplitude reaches 3 mm, the tactile feedback unit simulates the equipment vibration with the corresponding intensity (such as the vibration motor speed reaching 1200 revolutions per minute) to sense the force condition of the equipment in real time. When the smooth control signal is triggered, the tactile feedback intensity is synchronously increased by 20% - 50%, such as increasing the vibration motor speed to 1800 revolutions per minute, to more clearly feel the change of the equipment state, so as to adjust the operation strategy in time, improve the accuracy and safety of remote control, and reserve a key time window for taking countermeasures.

[0022] Furthermore, the method of this application includes: The biometric indicators include eye tracking data and gesture recognition data; the pose control instruction of the coal mine production equipment is determined through the eye tracking data and the first device control operation, and the gesture recognition data and the second device control operation.

[0023] Specifically, eye tracking data refers to the information obtained by monitoring the operator's eye movements through equipment (such as AR glasses), including the location of the gaze point, the duration of gaze, etc.; gesture recognition data refers to the recognition results of the operator's gesture movements through sensors (such as smart gloves), covering hand postures and movement trajectories, etc. In this method, the operator wears an interactive device, and his eye and gesture movements correspond to different device control operations. The system will identify and analyze these biometric indicators and convert them into specific posture control instructions.

[0024] In a feasible implementation, the operator wears AR glasses and smart gloves to perform remote operation; the eye movements are captured to accurately locate the gaze point. When the gaze point stays in a certain area of the virtual device interface for more than a time constraint, it is determined as a selection operation on the device part; the inertial measurement unit (IMU) built into the gloves collects hand motion data, and based on the pre-trained gesture recognition model, it interprets specific gestures (such as grasping, opening, swinging, etc.) into control signals such as the rise, fall, and rotation of the device. For example, in the control of the coal mining machine, by looking at the area of the coal mining machine's cutting head and making a grasping gesture, a posture control instruction for the cutting head to descend can be generated.

[0025] The parsed first device control operation (such as eye gaze at the selected device part) and the second device control operation (such as the action type determined by the gesture) are fused and processed. Taking the coal mining machine as an example, when the operator gazes at the cutting head area (determined by eye tracking data) and makes a clockwise rotation gesture (determined by gesture recognition data), the system comprehensively determines that it generates a posture control instruction for the cutting head to rotate clockwise. This instruction is transmitted to the coal mining machine through the remote control channel to control it to perform the corresponding action, so that the equipment accurately responds to the operator's natural interaction intention, improves the flexibility and accuracy of remote control, and effectively improves the efficiency and safety of coal mine production operations.

[0026] Furthermore, through the eye tracking data and the first device control operation, the gesture recognition data and the second device control operation, the method of the present application includes: The eye tracking data in the biometric indicator is used to analyze the operator's gaze point, and the stay position and duration on the virtual device interface are synchronously circled and converted into a first equipment control operation of the coal mine production equipment. The hand movement trajectory is extracted by the gesture recognition data in the biometric indicator, and mapped to a second equipment control operation of the coal mine production equipment based on a pre-trained gesture semantic library.

[0027] Specifically, in the control of coal mine equipment for remote embodied interaction, the parsing of eye-tracking data refers to monitoring the eye movements of the operator through devices (such as AR glasses) to determine information such as the position of the fixation point and the fixation duration; synchronously delineating the stay position and duration refers to real-time marking of the area where the operator's gaze stays and its duration on the virtual device interface; converting it into the first device control operation refers to mapping this gaze-related data into control instructions for specific parts of the coal mine production equipment. And the extraction of gesture recognition data refers to capturing the hand movement trajectory through sensors, and the mapping based on the pre-trained gesture semantic library refers to using a pre-trained model to establish a correspondence between the hand movement and the device control operation, thereby generating the second device control operation.

[0028] The operator wears AR glasses, and its eye-tracking module captures eye movements at a sampling frequency of 240 Hz, accurately locates the fixation point, and analyzes the fixation point in real time. When the fixation point stays in a certain area of the virtual device interface for more than a fixed time limit, it automatically delineates the stay position, and converts it into a control operation for the corresponding device part according to the stay duration and fixation intensity (such as changes in blink frequency). In the control of the roadheader, when the operator's gaze stays in the area of the roadheader cutting head, it is determined as an instruction to start the cutting head; if the fixation stay duration continues to increase, it is determined as an instruction to increase the cutting power, enabling the operator to efficiently control the key parts of the equipment through natural eye movements.

[0029] The inertial measurement unit (IMU) built into the intelligent glove collects hand movement data, including gesture postures and movement trajectories. Based on the pre-trained gesture semantic library, it maps the hand movement trajectory into specific control operations of the device. For example, in the control of the shearer, when the operator makes a grasping gesture (fingers closed, palm facing inwards) and the trajectory moves downward, the system parses it as an instruction for the shearer cutting head to lower; if the operator makes an opening gesture (fingers open, palm facing outwards) and the trajectory moves to the right, it is parsed as an instruction for the shearer to move to the right. Through this gesture recognition method, the operator can control the device with intuitive and natural gesture movements, effectively improving the efficiency and accuracy of remote control.

[0030] Furthermore, to predict the deviation of the device movement trajectory within a dynamic time period using an LSTM network, the method of this application includes: Collecting the device operation data, control instruction sequences, and actual operation trajectories within multiple historical interaction time periods, and configuring a time series data set; setting up an LSTM network including an input layer, a hidden layer, and an output layer, combining the time series data set, and performing adaptive training using the mean square error loss function; determining the optimal network parameters through cross-validation, and deploying the adaptively trained LSTM network on the edge computing node to predict the deviation of the device movement trajectory under a dynamic time period, where the dynamic time period is dynamically adjusted according to the network delay.

[0031] Specifically, in the control of coal mine equipment for remote embodied interaction, the equipment operation data in multiple historical interaction periods refers to the operation status information of the equipment in multiple historical time periods, including the pose, speed, acceleration, etc. of the equipment; the control instruction sequence refers to the control instructions sent by the operator during these historical periods; the actual operation trajectory refers to the movement path actually executed by the equipment according to the control instructions; the time-series data set refers to the data set formed by integrating these operation data, control instructions, and actual trajectories in chronological order. The LSTM network of the input layer, hidden layer, and output layer refers to a long short-term memory neural network with a specific structure. The input layer receives the time-series data, the hidden layer processes the time-dependent relationships in the data, and the output layer predicts the deviation of the equipment movement trajectory. The mean squared error loss function is a function used to measure the difference between the predicted value and the actual value, and is used to guide the adaptive training of the network. Cross-validation is a method for evaluating the performance of a model. By dividing the data set into multiple subsets for multiple training and validation, the optimal network parameters are determined. The edge computing node refers to a computing device close to the data source, which is used to process and analyze data in real time to reduce data transmission latency. The dynamic time period interval refers to the prediction time range adjusted in real time according to the network latency.

[0032] In a feasible implementation, the system collects the operation data of the equipment under different working conditions in historical periods from the historical database, including the equipment pose (position and attitude), speed, acceleration recorded every 0.1 second, as well as the operator control instruction sequence (such as ascending, descending, rotating instructions) and the actual operation trajectory of the equipment at the corresponding time. These data are integrated into a time-series data set in chronological order, covering the operation status of the equipment under different loads and different environmental conditions, ensuring the diversity and representativeness of the data.

[0033] Construct an LSTM network including an input layer, a hidden layer (using a gated recurrent unit to capture long-term dependencies), and an output layer. Use the mean squared error loss function to adaptively train the network through the backpropagation algorithm; adopt a 5-fold cross-validation method to optimize the hyperparameters of the network (such as the learning rate, the number of neurons in the hidden layer, etc.) to determine the optimal network parameters. Deploy the trained LSTM network on the edge computing node in the coal mine (such as an industrial-grade server with powerful computing capabilities), and control the communication latency between this node and the equipment control system and the remote control end within a certain time limit. In the prediction of the dynamic time period interval, the system monitors the network latency in real time. When the latency increases, the dynamic time period interval is adjusted to ensure the real-time and accuracy of the prediction.

[0034] By constructing and training a high-precision LSTM prediction model, the system can predict the deviation of the device motion trajectory in advance, providing a basis for compensation in remote control; the deployment of edge computing nodes effectively reduces data transmission latency and improves the real-time performance of prediction. The adjustment of the dynamic time period ensures prediction accuracy under different network conditions, making the remote control of the device more stable and reliable, and thus significantly improving the safety and efficiency of the remote operation of coal mine equipment.

[0035] Furthermore, the method of this application further includes: The hidden layer of the LSTM network uses gated recurrent units to capture long-term dependencies in the time series data set; the time series data set is divided into an input sequence and a target sequence according to a time step; the input sequence includes historical control instructions and device state parameters, and the target sequence is the device motion trajectory; using the input sequence and the target sequence, an adaptive training is performed using the mean squared error loss function.

[0036] Specifically, the gated recurrent unit (GRU) is a simplified version of the long short-term memory unit, which is used to capture long-term dependencies in time series data, can effectively alleviate the problem of gradient disappearance, and enhance the network's learning ability for long time series. Dividing the time series data set according to a time step means dividing the collected continuous time series data into an input sequence and a target sequence according to a fixed time interval, where the input sequence includes historical control instructions and device state parameters and is used as the input of the neural network, and the target sequence is the actual motion trajectory of the device and is used as the target output for training. The mean squared error loss function is an index used to measure the difference between the predicted value and the actual value. By calculating the mean of the sum of the squares of the errors between the predicted trajectory and the actual trajectory, it guides the adaptive training process of the neural network, enabling the network to automatically adjust parameters to minimize the prediction error.

[0037] In a feasible implementation, when constructing the LSTM network, the hidden layer uses gated recurrent units to replace traditional RNN units. The gated recurrent unit determines how to update the hidden state through the control of the reset gate and the update gate, thereby effectively capturing long-term dependencies in the time series data set. For example, in the control of coal mine equipment, the motion trajectory of the equipment is not only affected by the current control instruction but is also closely related to the operation history within the past 10 seconds. The gated recurrent unit can remember these long-term dependencies, enabling the network to consider information within a longer time range during prediction, thereby improving the prediction accuracy.

[0038] The collected set of time-series data is divided into an input sequence and a target sequence according to a time step. The input sequence contains historical control instructions and device status parameters for the past 10 time steps (each time step is 0.1 seconds), a total of 100 data points; the target sequence is the device motion trajectory for the next 1 time step (0.1 seconds). The mean squared error loss function is used to adaptively train the network. After multiple training cycles, the mean squared error corresponding to the validation set gradually converges, indicating that the network can accurately predict the device motion trajectory deviation, can accurately predict the future state of the device based on historical data, discover potential trajectory deviations in advance, provide a compensation basis for remote control, and significantly improve the stability and safety of the control system.

[0039] By using an LSTM network with gated recurrent units, the long-term dependencies during device operation can be effectively captured, and the adaptability to complex dynamic environments can be improved. The reasonable division of time-series data and the use of the mean squared error loss function ensure the efficiency of network training and the accuracy of prediction, control the average error of device motion trajectory prediction within a certain range, significantly improve the real-time performance and accuracy of remote control, and provide a strong guarantee for the efficient and safe remote operation of coal mine equipment.

[0040] Furthermore, according to the pose control instruction and the device motion trajectory deviation, a trigger judgment of a smooth control signal is performed. The method of the present application includes: Using the change rate of the pose control instruction at time T-1 and time T, taking the change rate exceeding the change rate threshold as the first prerequisite; at the same time, based on the device motion trajectory deviation predicted by the LSTM network, accumulating the device motion trajectory deviation within the cumulative dynamic time interval, and taking the accumulated deviation amount exceeding the deviation amount threshold as the second prerequisite; performing fuzzy logic control with the first prerequisite and the second prerequisite to determine the trigger probability under the trigger judgment of the smooth control signal.

[0041] Specifically, in the control of coal mine equipment for remote embodied interaction, the change rate of the pose control instruction at time T-1 and time T refers to the change amplitude of the device control instruction between two adjacent time points (T-1 and T), reflecting the operation intensity and urgency; the change rate threshold is a preset upper limit of the change rate, used to judge whether the operation is violent; the device motion trajectory deviation within the cumulative dynamic time interval refers to the sum of the deviations between the device motion trajectory predicted by the LSTM network and the actual trajectory within the dynamically adjusted time period; the deviation amount threshold is a preset upper limit of the total deviation, used to judge whether the trajectory deviation exceeds the allowable range. Fuzzy logic control is a control method based on fuzzy rules and membership functions, used to process uncertain and fuzzy information, and can make a comprehensive judgment according to multiple conditions to determine the trigger probability of the smooth control signal.

[0042] In a feasible implementation, the change rates of the pose control instructions at time T-1 and time T are monitored in real time, including the device position change rate (unit: meters per second) and the attitude change rate (unit: radians per second). For example, when the position change rate exceeds 0.5 meters per second or the attitude change rate exceeds 0.2 radians per second, the first prerequisite is triggered. In the rapid turning operation of the shearer, if the attitude change rate reaches 0.3 radians per second at a certain moment, exceeding the set threshold, it is determined that the first prerequisite is established. This judgment mechanism can timely identify the drastic changes in the operation and provide a basis for subsequent smooth control.

[0043] After the LSTM network predicts the deviation of the device motion trajectory, the system accumulates these deviation values within a dynamic time period (such as the past 1 second). For example, if the accumulated deviation exceeds the preset deviation threshold (such as the position deviation exceeds 0.1 meter or the attitude deviation exceeds 0.05 radians), the second prerequisite is triggered. During the straight tunneling process of the roadheader, if the accumulated position deviation within the past 1 second reaches 0.12 meters, exceeding the threshold, it is determined that the second prerequisite is established, which can effectively identify the accumulated deviation of the device motion trajectory and provide data support for smooth control.

[0044] The fuzzy logic controller receives the first prerequisite (change rate state) and the second prerequisite (accumulated deviation amount state) as inputs. Through the fuzzy rule base (including rules such as if the change rate is high and the accumulated deviation amount is large, then the triggering probability is high) and the membership function (defining the fuzzy sets of the input states), fuzzyfication processing, reasoning, and defuzzification processing are performed, and finally the triggering probability of the smooth control signal is determined. This fuzzy logic-based control method can comprehensively consider various uncertain factors, dynamically adjust the triggering probability of the smooth control signal, make the device motion more stable, and effectively reduce the device vibration and impact caused by the sudden change of the control signal and the trajectory deviation.

[0045] Furthermore, the method of this application includes: Input the change rate and the accumulated deviation amount into the fuzzy logic controller, perform fuzzyfication processing through the fuzzy rule base and the membership function; perform reasoning based on the fuzzy sets to which the input change rate and accumulated deviation amount belong, obtain the fuzzy output of the triggering probability, and perform defuzzification processing.

[0046] Specifically, in the control of coal mine equipment for remote embodied interaction, a fuzzy logic controller is a tool that uses fuzzy set theory for reasoning and decision-making and can handle uncertain and fuzzy information. The fuzzy rule base refers to a set containing a series of fuzzy rules that define the mapping relationship between different input conditions and output results. The membership function is a function used to determine the degree to which a specific value belongs to a certain fuzzy set, and its value range is between 0 and 1. The fuzzification process is to convert the precise numerical input into a fuzzy set, while the defuzzification process (also known as crispification) is to convert the fuzzy inference result into a precise numerical output for actual control operations.

[0047] In a feasible implementation, the change rate of the pose control instruction and the deviation of the equipment motion trajectory within the cumulative dynamic time interval are used as input variables. The fuzzy sets are defined as three levels: low, medium, and high, corresponding to different ranges of the change rate and the cumulative deviation. Calculate the membership degrees for each fuzzy set through the membership function; perform reasoning according to the pre-defined fuzzy rule base; determine the excitation intensity of each rule based on the membership degrees of the input variables and the fuzzy rules, and use the weighted average method for defuzzification. Take the center point of each fuzzy output set as the candidate value, and the weight is the excitation intensity of the corresponding rule; start the smooth control to optimize the equipment motion control process, reduce the vibration and impact of the equipment caused by sudden changes in control signals and trajectory deviations, and improve the stability and safety of remote control. Through fuzzy logic control, the system can more intelligently handle complex and changeable scenarios of remote control of coal mine equipment, effectively improving control accuracy and reliability.

[0048] Furthermore, in combination with the vibration spectrum of the coal mine production equipment, synchronously enhance the tactile feedback intensity of the wearable interaction device. The method of the present application includes: The tactile feedback unit of the wearable interaction device generates different levels of vibration feedback modes according to the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment; if the smooth control signal is triggered, perform the tactile feedback enhancement configuration in different levels of vibration feedback modes.

[0049] Specifically, in the control of coal mine equipment for remote embodied interaction, the tactile feedback unit refers to the components in the wearable interaction device (such as smart gloves or tactile vests) that provide tactile feedback to the operator, usually including vibration motors, pneumatic tactors, etc. The vibration spectrum refers to the frequency and amplitude distribution characteristics of the vibration signals generated by coal mine production equipment during operation, reflecting the vibration state of the equipment. The frequency characteristics refer to the proportion of different frequency components and the dominant frequency in the vibration signal, and the amplitude characteristics refer to the intensity of the vibration signal. Different levels of vibration feedback modes refer to various tactile feedback methods preset according to the vibration characteristics of the equipment, such as vibration signals with different frequencies, intensities, or modes. The smooth control signal refers to the control signal used to optimize the equipment motion control process to make it more stable. The tactile feedback enhancement configuration refers to adjusting the output mode of the tactile feedback unit when the smooth control signal is triggered to enhance the operator's perception of changes in the equipment state.

[0050] In a feasible implementation, in the coal mine remote control scenario, the interaction device worn by the operator includes a tactile feedback unit, which is connected to the remote control system; the vibration spectrum data of the coal mine production equipment is obtained in real time, including frequency characteristics (such as the dominant frequencies are 25Hz, 50Hz, 100Hz, etc.) and amplitude characteristics (such as the amplitude is between 0.1mm and 5mm); the tactile feedback unit generates different levels of vibration feedback modes based on these characteristics. For example, when the equipment vibration frequency is 25Hz and the amplitude is 1mm, the tactile feedback unit gives feedback in a low-frequency light vibration mode (vibration frequency 30Hz, intensity 30%); when the frequency rises to 100Hz and the amplitude reaches 3mm, it switches to a high-frequency strong vibration mode (vibration frequency 120Hz, intensity 80%). This mode switching enables the operator to intuitively perceive the vibration state of the equipment and enhances the immersion and accuracy of remote operation.

[0051] When the smooth control signal is triggered, the tactile feedback unit enhances the feedback intensity according to the preset rules. The enhanced tactile feedback enables the operator to more clearly perceive changes in the equipment state, such as the minute vibration adjustments during the execution of smooth control by the equipment, so as to make more accurate operation decisions in a timely manner, reduce misoperations caused by perception delays, and improve the safety and efficiency of remote control.

[0052] Furthermore, according to the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment, the method of this application includes: Deploy a multi-spectral explosion-proof camera group to obtain the working face image information by penetrating dust interference; set a visual attention model with the working face image information to identify the contours of the equipment moving parts; generate a depth compensation map through laser speckle projection with the contours of the equipment moving parts to determine the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment.

[0053] Specifically, the multi-spectral explosion-proof camera unit refers to a special camera device that can work normally in harsh environments such as high dust and low light in coal mines. It has multiple spectral imaging capabilities and can penetrate dust interference to obtain clear images. The visual attention model is an algorithm model used to extract key information from complex images and can guide the system to focus on important regions in the image. Laser speckle projection is a technology that measures the depth information of an object's surface by projecting a speckle pattern with a laser. The depth compensation map refers to the image data used to correct the depth information of an image and can more accurately reflect the three-dimensional structure of an object. The frequency characteristics and amplitude characteristics in the vibration spectrum refer to the key characteristics such as the frequency components and amplitude sizes of the device vibration signal and are used to analyze the vibration state of the device.

[0054] In a feasible implementation, the multi-spectral explosion-proof camera unit is deployed at key positions around the device. The working wavelength range covers visible light (400 - 700nm) and near-infrared light (700 - 1000nm), and it can penetrate dust to obtain clear images of the working face. The image information of the working face obtained is input into the visual attention model based on deep learning. This model is trained with a large amount of coal mine equipment image data and can quickly locate and identify the contours of the moving parts of the device, such as the cutting head of the shearer and the scraper of the roadheader, etc. It can accurately identify the contour of the cutting head and mark its position and shape, providing accurate regional positioning for subsequent depth measurement and vibration analysis.

[0055] For the identified contour area of the moving part, a laser speckle of a specific pattern is projected through the laser speckle projection module. The laser wavelength is 850nm, and the projection power is stable at 50mW to ensure that a clear depth feature is formed by the speckle pattern on the device surface. The depth sensor collects the image after the reflection of the speckle pattern and generates a depth compensation map through algorithm processing. This map represents the depth information with gray values, and the gray resolution can reach 16 bits, which can accurately reflect the minute deformation and vibration conditions of the device surface.

[0056] According to the depth compensation map, analyze the depth changes of the moving parts of the device at different time points, and calculate the frequency characteristics and amplitude characteristics of the vibration spectrum. For example, through Fourier transform analysis of the depth compensation map, obtain its main vibration frequency and vibration amplitude. These vibration characteristic data provide accurate input for the tactile feedback unit, and the vibration state of the device can be perceived in real time by wearing the interaction device, enhancing the intuitiveness and safety of remote operation and effectively improving the efficiency and reliability of remote control.

[0057] In summary, the beneficial effects of the embodiments of this application are: Due to the adoption of uploading basic technical parameters of coal mine production equipment; analyzing the equipment operation status with the basic technical parameters, configuring the equipment operation twin architecture, and making local decisions through the historical operation instances of the equipment in the normal operation state; using the abnormal data confirmed by the local decision to automatically mark and generate warning information; after the remote control end receives the warning information, connecting the wearable interaction device; mapping the biometric indicators uploaded by the wearable interaction device into the pose control instructions of the coal mine production equipment; embedding a time delay compensation factor in the remote embodied interaction channel for transmitting the pose control instructions, and predicting the equipment motion trajectory deviation in the dynamic time period interval with an LSTM network; using the multi-modal feedback unit of the wearable interaction device to make a trigger judgment on the smooth control signal according to the pose control instructions and the equipment motion trajectory deviation; if the smooth control signal is triggered, combining the vibration spectrum of the coal mine production equipment to synchronously enhance the tactile feedback intensity of the wearable interaction device. By providing a control method and system for coal mine production equipment based on remote embodied interaction, this application predicts the dynamic time delay through an LSTM network and compensates for the trajectory deviation, improving the real-time performance and stability of remote control, ensuring the accurate execution of instructions, and achieving the technical effects of improving the efficiency and reliability of remote operation of coal mine equipment with high-precision pose control and real-time compensation.

[0058] Embodiment 2

[0059] Based on the same inventive concept as the method for controlling coal mine production equipment based on remote embodied interaction in the foregoing embodiment, as Figure 2 shown, this embodiment of the application provides a control system for coal mine production equipment based on remote embodied interaction, wherein the system includes: A decision-making module M100, configured to upload basic technical parameters of coal mine production equipment; analyze the equipment operation status with the basic technical parameters, configure the equipment operation twin architecture, and make local decisions through the historical operation instances of the equipment in the normal operation state.

[0060] An automatic marking module M200, configured to use the abnormal data confirmed by the local decision to automatically mark and generate warning information; after the remote control end receives the warning information, connect the wearable interaction device.

[0061] An embedding module M300, configured to map the biometric indicators uploaded by the wearable interaction device into the pose control instructions of the coal mine production equipment; embed a time delay compensation factor in the remote embodied interaction channel for transmitting the pose control instructions, and predict the equipment motion trajectory deviation in the dynamic time period interval with an LSTM network.

[0062] A trigger judgment module M400, configured to use the multi-modal feedback unit of the wearable interaction device to make a trigger judgment on the smooth control signal according to the pose control instructions and the equipment motion trajectory deviation.

[0063] The synchronous enhancement module M500 is used to synchronously enhance the tactile feedback intensity of the wearable interactive device in combination with the vibration spectrum of the coal mine production equipment if the smooth control signal is triggered.

[0064] Furthermore, the embedded module M300 is used to execute the following method: The biometric indicators include eye tracking data and gesture recognition data; the position and posture control instructions of the coal mine production equipment are determined through the eye tracking data and the first device control operation, the gesture recognition data and the second device control operation.

[0065] Furthermore, the embedded module M300 is also used to execute the following method: The eye tracking data in the biometric indicator is used to analyze the operator's gaze point, and the stay position and duration on the virtual device interface are synchronously circled and converted into a first equipment control operation of the coal mine production equipment. The hand movement trajectory is extracted by the gesture recognition data in the biometric indicator, and mapped to a second equipment control operation of the coal mine production equipment based on a pre-trained gesture semantic library.

[0066] Furthermore, the embedded module M300 is used to execute the following method: The equipment operation data, control instruction sequence and actual operation trajectory within multiple historical interaction periods are collected to configure a time series data set; an LSTM network including an input layer, a hidden layer and an output layer is set, and the mean square error loss function is used for adaptive training in combination with the time series data set; the optimal network parameters are determined through cross-validation, and the LSTM network after adaptive training is deployed on the edge computing node to predict the equipment motion trajectory deviation under dynamic time period intervals, and the dynamic time period intervals are dynamically adjusted according to network delays.

[0067] Furthermore, the embedded module M300 is also used to execute the following method: The hidden layer of the LSTM network uses a gated recurrent unit to capture long-term dependencies in a time series data set; the time series data set is divided into an input sequence and a target sequence according to the time step; the input sequence includes historical control instructions and device state parameters, and the target sequence is the device motion trajectory; the input sequence and the target sequence are used to perform adaptive training using a mean square error loss function.

[0068] Furthermore, the trigger determination module M400 is used to execute the following method: Taking the change rate of the pose control instruction at time T-1 and time T, using the change rate exceeding the change rate threshold as the first prerequisite; at the same time, based on the deviation of the device motion trajectory predicted by the LSTM network, accumulating the deviation of the device motion trajectory within the dynamic time period interval, and using the accumulated deviation amount exceeding the deviation amount threshold as the second prerequisite; performing fuzzy logic control with the first prerequisite and the second prerequisite to determine the trigger probability under the trigger judgment of the smoothing control signal.

[0069] Further, the trigger judgment module M400 is further configured to execute the following method: Input the change rate and the accumulated deviation amount into a fuzzy logic controller, perform fuzzy processing through a fuzzy rule base and a membership function; perform reasoning according to the fuzzy sets to which the input change rate and accumulated deviation amount belong, obtain the fuzzy output of the trigger probability, and perform defuzzification processing.

[0070] Further, the synchronization enhancement module M500 is configured to execute the following method: The tactile feedback unit of the wearable interaction device generates different levels of vibration feedback modes according to the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment; if the smoothing control signal is triggered, perform tactile feedback enhancement configuration with different levels of vibration feedback modes.

[0071] Further, the synchronization enhancement module M500 is further configured to execute the following method: Deploy a multi-spectral explosion-proof camera group to obtain working face image information by penetrating dust interference; set a visual attention model with the working face image information to identify the outline of the device moving parts; generate a depth compensation map through laser speckle projection with the outline of the device moving parts to determine the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment.

[0072] In summary, any step can be stored as computer instructions or programs in an unrestricted computer memory and can be called and recognized by an unrestricted computer processor, without further limitation here.

[0073] Further, the above technical solutions only reflect the preferred technical solutions of the technical solutions of the embodiments of the present application. Some changes that those skilled in the art may make to some parts thereof all reflect the principles of the new type of the embodiments of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application.

Claims

1. A control method for coal mine production equipment based on remote embodied interaction, characterized in that, The method comprises: Coal mine production equipment uploads basic technical parameters; uses the basic technical parameters to analyze the equipment operation status, configures the equipment operation twin architecture, and makes localized decisions based on historical operation instances of equipment in normal operation; Using the abnormal data confirmed by the localized decision, automatically marking and generating warning information; after the remote control terminal receives the warning information, it connects to the wearable interactive device; The biometric indicators uploaded by the wearable interactive device are mapped into the posture control instructions of the coal mine production equipment; the delay compensation factor is embedded in the remote embodied interaction channel for transmitting the posture control instructions, and the LSTM network is used to predict the deviation of the equipment motion trajectory within the dynamic time period; Using the multimodal feedback unit of the wearable interactive device, triggering judgment of the smooth control signal is performed according to the posture control instruction and the device motion trajectory deviation; If the smooth control signal is triggered, combined with the vibration spectrum of the coal mine production equipment, the tactile feedback intensity of the wearable interactive device is synchronously enhanced.

2. The control method of coal mine production equipment based on remote embodied interaction according to claim 1, characterized in that, The biometric indicators include eye tracking data and gesture recognition data; The position and posture control instructions of the coal mine production equipment are determined through the eye tracking data and the first device control operation, the gesture recognition data and the second device control operation.

3. The control method for coal mine production equipment based on remote embodied interaction according to claim 2, wherein By using the eye tracking data and the first device control operation, the gesture recognition data and the second device control operation, the method includes: The eye tracking data in the biometric indicator is used to analyze the operator's gaze point, and the stay position and duration on the virtual device interface are simultaneously circled, and converted into the first device control operation of the coal mine production equipment part; The hand motion trajectory is extracted using the gesture recognition data in the biometric indicator, and is mapped to a second device control operation of the coal mine production equipment based on a pre-trained gesture semantic library.

4. The control method for coal mine production equipment based on remote embodied interaction according to claim 1, wherein, The method for predicting the deviation of the movement trajectory of the equipment within the dynamic time interval by using the LSTM network includes: Collect equipment operation data, control instruction sequences and actual operation trajectories in multiple historical interaction periods, and configure time series data sets; Setting an LSTM network including an input layer, a hidden layer, and an output layer, combining the time series data set, and using a mean square error loss function for adaptive training; The optimal network parameters are determined through cross-validation, and the adaptively trained LSTM network is deployed on the edge computing node to predict the device motion trajectory deviation under dynamic time periods, which are dynamically adjusted according to network delays.

5. The control method for coal mine production equipment based on remote embodied interaction according to claim 4, characterized in that, The hidden layer of the LSTM network uses a gated recurrent unit to capture long-term dependencies in a time series data set; Dividing the time series data set into an input sequence and a target sequence according to the time step; the input sequence includes historical control instructions and device state parameters, and the target sequence is the device motion trajectory; Adaptive training is performed using the input sequence and the target sequence using a mean square error loss function.

6. The method for controlling coal mine production equipment based on remote embodied interaction according to claim 1, wherein, The method comprises: determining the triggering of a smooth control signal according to the posture control instruction and the device motion trajectory deviation; Taking the change rate of the posture control command between time T-1 and time T as the first prerequisite, the change rate exceeds the change rate threshold; Meanwhile, based on the deviation of the device motion trajectory predicted by the LSTM network, the deviation of the device motion trajectory within the cumulative dynamic time period is accumulated, and taking that the accumulated deviation amount exceeds the deviation amount threshold as the second prerequisite; Using the first prerequisite and the second prerequisite for fuzzy logic control to determine the trigger probability under the trigger judgment of the smoothing control signal.

7. The method for controlling coal mine production equipment based on remote embodied interaction according to claim 6, characterized in that, Inputting the change rate and the accumulated deviation amount into the fuzzy logic controller, and performing fuzzy processing through the fuzzy rule base and the membership function; Reasoning according to the fuzzy sets to which the input change rate and accumulated deviation amount belong, obtaining the fuzzy output of the trigger probability, and performing defuzzification processing.

8. The method for controlling coal mine production equipment based on remote embodied interaction according to claim 7, characterized in that, Combining with the vibration spectrum of the coal mine production equipment to synchronously enhance the tactile feedback intensity of the wearable interaction device, the method includes: The tactile feedback unit of the wearable interaction device generates different levels of vibration feedback modes according to the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment; If the smoothing control signal is triggered, execute the tactile feedback enhancement configuration with different levels of vibration feedback modes.

9. The control method of coal mine production equipment based on remote embodied interaction according to claim 8, wherein, According to the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment, the method includes: Deploying a multi-spectral explosion-proof camera group to obtain the working face image information by penetrating dust interference; Setting a visual attention model with the working face image information to identify the outline of the device moving parts; Generating a depth compensation map through laser speckle projection with the outline of the device moving parts to determine the frequency characteristics and amplitude characteristics in the vibration spectrum of the coal mine production equipment.

10. The control system for coal mine production equipment based on remote embodied interaction is characterized in that, For implementing the coal mine production equipment control method based on remote embodied interaction according to any one of claims 1-9, the system includes: A decision-making module, which is used for the coal mine production equipment to upload basic technical parameters; analyzing the equipment operation state with the basic technical parameters, configuring the equipment operation twin architecture, and making local decisions through the equipment historical operation instances in the normal operation state; An automatic marking module, which is used to automatically mark and generate early warning information using the abnormal data confirmed by the local decision; after receiving the early warning information, the remote control terminal connects to the wearable interaction device; An embedding module, which is used to map the biometric indexes uploaded by the wearable interaction device into the pose control instructions of the coal mine production equipment; embedding a time delay compensation factor in the remote embodied interaction channel for transmitting the pose control instructions, and predicting the deviation of the device motion trajectory within the dynamic time period with an LSTM network; A trigger judgment module, which is used to use the multi-modal feedback unit of the wearable interaction device to make a trigger judgment on the smoothing control signal according to the pose control instructions and the device motion trajectory deviation; A synchronous enhancement module, which is used to synchronously enhance the tactile feedback intensity of the wearable interaction device in combination with the vibration spectrum of the coal mine production equipment if the smoothing control signal is triggered.

Images