Underground equipment remote control system for MR

Abstract

The invention relates to the field of mine intellectualization, and discloses an underground equipment remote control system for MR, and the system comprises a collection and transmission module which is used for collecting multi-modal data of underground equipment, and carrying out the anti-interference transmission of all data; the processing fusion module is used for performing time synchronization preprocessing on the multi-modal data, performing feature fusion through tensor decomposition based on mutual information feature screening, and outputting a fusion feature vector containing multi-modal associated information; and the intelligent decision-making module is used for constructing a Markov decision-making process by adopting a reinforcement learning algorithm on the basis of the fusion feature vector in combination with an underground physical field stress constraint model constructed through a finite element method. Dynamic visual events, attitude vibration, structural strain and environmental parameters of equipment are obtained in real time, time sequence deviation of multi-source data is eliminated through time synchronization calibration, strong correlation modes are calculated and screened through mutual information, and low-dimensional fusion feature vectors containing the visual and environmental information coupling relation are extracted through regularization tensor decomposition.

Description

Technical Field

[0001] The present invention relates to the field of intelligent mine technology, and in particular to an underground equipment remote control system for MR. Background Art

[0002] As mining transforms towards unmanned and intelligent operations, underground equipment (such as tunnel boring machines and coal mining machines) needs to be precisely controlled in complex geological environments. The core of this requires efficient fusion of multimodal perception data – by integrating equipment motion posture, structural stress, environmental parameters and visual information to build a comprehensive working condition recognition model.

[0003] Traditional remote control systems for downhole equipment used in MR generally directly concatenate visual image pixel values ​​(dimension > 10^4), inertial parameters (6 dimensions), and pressure data (1 dimension) into high-dimensional feature vectors. The simple feature splicing or weighted average fusion method does not quantify the information interaction between modalities (for example, the mutual information between visual event streams and strain data is not calculated), resulting in key coupling information (such as the impact of motion posture changes on structural stress) being submerged in redundant data. The input quality of the decision model is poor, resulting in the existing system's one-sided understanding of the working conditions of downhole equipment, the control strategy is unable to adapt to complex dynamic scenarios, and the equipment operation efficiency is low. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a remote control system for downhole equipment for MR, which solves the problem that traditional remote control systems for downhole equipment for MR cannot adapt to complex dynamic scenarios, resulting in low equipment operation efficiency.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A downhole equipment remote control system for MR, comprising:

[0006] Acquisition and transmission module: used to collect multimodal data from downhole equipment and transmit the data in an anti-interference manner;

[0007] Processing and fusion module: used to perform time synchronization preprocessing on multimodal data, perform feature fusion through tensor decomposition based on mutual information feature screening, and output a fused feature vector containing multimodal correlation information;

[0008] Intelligent decision-making module: Based on the fused feature vectors and the stress constraint model of the downhole physical field constructed using the finite element method, the module uses a reinforcement learning algorithm to construct a Markov decision process and generate a control strategy that includes adjustments to equipment operating parameters.

[0009] MR interactive control module: This module is used to achieve 3D visualization of device status parameters, digital twin models, and path planning information through a mixed reality terminal based on control strategies. It also receives user intervention commands through gesture recognition and voice recognition engines for human-computer interaction.

[0010] Device execution module: used to receive and execute control strategies or user intervention instructions, adjust device operating parameters and feedback execution status.

[0011] By adopting the above technical solutions, dynamic visual events, posture vibrations, structural strains and environmental parameters of the equipment can be acquired in real time. Time synchronization calibration is used to eliminate the timing deviation of multi-source data. Mutual information calculation is used to screen strongly correlated modes. Low-dimensional fusion feature vectors containing the coupling relationship between visual, mechanical and environmental information are extracted through regularized tensor decomposition. The fusion feature vectors are combined with the von Mises stress output by the finite element physical field model to construct a reinforcement learning decision model with physical constraints. A control strategy that takes into account production efficiency, safety boundaries and energy consumption optimization is generated. The equipment status and path planning are visualized in three dimensions through the digital twin model, and natural intervention is achieved by combining gesture / voice interaction. Efficient fusion of multimodal data of downhole equipment, intelligent decision-making under physical constraints, immersive human-computer interaction and execution control are achieved. This solves the problem that traditional downhole equipment remote control systems used for MR cannot adapt to complex dynamic scenarios, resulting in low equipment operation efficiency.

[0012] Preferably, the acquisition and transmission module includes a data acquisition unit and an anti-interference transmission unit. The data acquisition unit is used to collect multimodal data of downhole equipment through an acquisition device. The acquisition device includes an event camera, a nine-axis inertial measurement unit, a fiber Bragg grating sensor, and a gas sensor. The event camera is used to collect dynamic visual event streams containing pixel coordinates, timestamps, and polarization information. The nine-axis inertial measurement unit is used to collect posture vibration data of equipment acceleration, angular velocity, and magnetic field strength. The fiber Bragg grating sensor is used to collect strain and temperature data of key components of the equipment. The gas sensor is used to collect environmental parameters of downhole gas concentration and dust concentration. The anti-interference transmission unit transmits various data in an anti-interference manner through a composite communication architecture including a 5G ultra-reliable low-latency communication link, an orthogonal frequency division multiple access wireless communication link, and a dynamic frequency hopping spread spectrum module.

[0013] Preferably, the processing fusion module includes a data preprocessing unit, a visual feature extraction unit, and a mutual information tensor decomposition unit. The data preprocessing unit is used to perform time synchronization calibration on the multimodal data and synthesize the event stream into a pseudo grayscale image through exponential decay accumulation. The visual feature extraction unit is used to perform real-time detection of the pseudo grayscale image through a lightweight target detection model and output visual features containing the position and category of the obstacle. The mutual information tensor decomposition unit is used to construct the multimodal data containing visual features and environmental parameters after time synchronization into a three-dimensional tensor. By calculating the mutual information between modalities I(X i ;X j ) screens the effective modal combinations and extracts the fusion feature vector z through regularized canonical polyadic decomposition.

[0014] Preferably, the calculation of the inter-modal mutual information is Among them, p(x i ,x j ) is the joint probability distribution of the i-th and j-th modal data, p(x i )、p(x j ) is the marginal probability distribution;

[0015] The regularization paradigm polyadic is Among them, N is the number of samples, M is the number of modes, K is the feature dimension, R is the decomposition rank, λ is the mutual information regularization parameter, A, B, and C are the feature matrices after decomposition, ||·|| F is the Frobenius norm.

[0016] Preferably, the intelligent decision-making module includes a physical field modeling unit and a reinforcement learning decision-making unit. The physical field modeling unit is used to construct a three-dimensional geological model of the well based on the finite element method, input rock mechanics parameters and solve the stress equilibrium equation, and output the von Mises stress σ of key components of the equipment. vonMises The reinforcement learning decision unit is used to construct the state space of the Markov decision process by fusing the eigenvector and the von Mises stress, and to train the policy network by improving the deep deterministic policy gradient algorithm to output a control strategy including the equipment operating speed adjustment Δv and the hydraulic pressure adjustment Δp.

[0017] Preferably, the stress balance equation Where σ is the stress tensor, f is the body force, and includes equipment load and formation gravity;

[0018] The improved deep deterministic policy gradient algorithm includes an action clipping mechanism that introduces physical field constraints. When σ von Mises >0.8σ threshWhen , the gradient of the action output by the policy network is truncated, and the regularization term of the state value function is designed to minimize the value difference of adjacent states. The formula is in, is the experience replay buffer, λ smooth is the smoothing coefficient.

[0019] Preferably, the improved deep deterministic policy gradient algorithm trains the policy network and optimizes the control policy using the following reward function: Among them, α+β+γ=1 is the weight coefficient, Q prod is the real-time output, Q max is the rated output, c gas is the gas concentration, E cons is the real-time energy consumption, E avg is the average energy consumption, and 1(·) is the safety status indicator function.

[0020] Preferably, the MR interactive control module includes a three-dimensional visualization unit and a natural interaction unit. The three-dimensional visualization unit is used to render the device digital twin model, real-time status parameter heat map and path planning guidance through the mixed reality terminal according to the control strategy and device status data, and output them to the natural interaction unit. The natural interaction unit is used to convert user intervention instructions into control signals through the gesture recognition engine and the voice recognition engine, and send them to the device execution module or the intelligent decision-making module. The user intervention instructions include emergency shutdown, parameter adjustment, and mode switching.

[0021] Preferably, the device execution module includes a drive control unit and a state feedback unit. The drive control unit is used to control the device to adjust according to the control strategy or user intervention instructions, including adjusting the device operating speed through a servo motor and adjusting the hydraulic pressure through a hydraulic valve group. The state feedback unit is used to collect the real-time operating speed of the device through an encoder and the real-time hydraulic pressure through a pressure transmitter to form execution status data including speed v and pressure p, and feed it back to the acquisition and transmission module and the MR interactive control module.

[0022] A method for remotely controlling downhole equipment for MR, applied to the above-mentioned remote control system for downhole equipment for MR, comprises the following steps:

[0023] Acquisition and transmission: Collect multimodal data from downhole equipment and transmit the data in an anti-interference manner;

[0024] Processing and fusion: Perform time synchronization preprocessing on multimodal data, perform feature fusion through tensor decomposition based on mutual information feature screening, and output a fused feature vector containing multimodal correlation information;

[0025] Intelligent decision-making: Based on the fused feature vectors and the stress constraint model of the downhole physical field constructed using the finite element method, a reinforcement learning algorithm is used to construct a Markov decision process to generate a control strategy that includes adjustments to equipment operating parameters.

[0026] MR interactive control: Based on the control strategy, the mixed reality terminal realizes the three-dimensional visualization of equipment status parameters, digital twin models and path planning information, and receives user intervention instructions through gesture recognition and voice recognition engines to conduct human-computer interaction;

[0027] Device execution: Receives and executes control strategies or user intervention instructions, adjusts device operating parameters, and provides feedback on execution status.

[0028] The present invention provides a remote control system for downhole equipment used in MR, which has the following beneficial effects:

[0029] 1. The present invention obtains dynamic visual events, posture vibrations, structural strains, and environmental parameters of the equipment in real time, eliminates timing deviations of multi-source data through time synchronization calibration, uses mutual information calculation to screen strongly correlated modes, and extracts low-dimensional fused feature vectors containing the coupling relationship between visual, mechanical, and environmental information through regularized tensor decomposition. The fused feature vectors are combined with the von Mises stress output by the finite element physical field model to construct a reinforcement learning decision model with physical constraints. This generates a control strategy that takes into account production efficiency, safety margins, and energy consumption optimization. The digital twin model is used to visualize the equipment status and path planning in three dimensions, and natural intervention is achieved through gesture / voice interaction. This achieves efficient fusion of multimodal data of downhole equipment, intelligent decision-making under physical constraints, immersive human-computer interaction, and execution control. This solves the problem that traditional remote control systems for downhole equipment used in MR are unable to adapt to complex dynamic scenarios, resulting in low equipment operation efficiency.

[0030] 2. The present invention eliminates the spatiotemporal mismatch and redundant information of multi-source data through a three-level processing flow of time synchronization calibration, mutual information feature screening and regularized tensor decomposition, mines the deep correlation semantics between modalities, and compresses the original data containing hundreds of thousands of dimensions into low-dimensional fusion feature vectors. While reducing the computational complexity, it retains key decision-making information, provides accurate working condition representation input for the intelligent decision-making module, and avoids the decision-making model misjudgment problem caused by traditional shallow fusion methods.

[0031] 3. By combining finite element physical field stress constraints with reinforcement learning algorithms, constructing a state space including von Mises stress, designing a multi-objective reward function, and introducing an action clipping mechanism, the present invention generates a control strategy that can not only meet the mechanical safety boundaries of equipment operation, but also take into account production efficiency and energy consumption optimization, thereby improving the robustness and practicality of the control strategy in complex geological environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a system architecture diagram of a downhole equipment remote control system for MR proposed by the present invention;

[0033] Figure 2 This is a flow chart of a method for remotely controlling downhole equipment for MR proposed by the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] Please see the attached Figure 1 The embodiment of the present invention provides a downhole equipment remote control system for MR, including:

[0036] Acquisition and transmission module: used to collect multimodal data of downhole equipment and transmit the data in an anti-interference manner; the acquisition and transmission module includes a data acquisition unit and an anti-interference transmission unit. The data acquisition unit is used to collect multimodal data of downhole equipment through acquisition equipment. The acquisition equipment includes an event camera, a nine-axis inertial measurement unit, a fiber Bragg grating sensor, and a gas sensor. The event camera is used to collect dynamic visual event streams containing pixel coordinates, timestamps, and polarization information. The nine-axis inertial measurement unit is used to collect equipment acceleration, angular velocity, and attitude vibration data of magnetic field strength. The fiber Bragg grating sensor is used to collect strain and temperature data of key components of the equipment. The gas sensor is used to collect environmental parameters such as gas concentration and dust concentration downhole. The anti-interference transmission unit transmits the data in an anti-interference manner through a composite communication architecture including a 5G ultra-reliable low-latency communication link, an orthogonal frequency division multiple access wireless communication link, and a dynamic frequency hopping spread spectrum module.

[0037] Specifically, the acquisition and transmission module serves as the data entry point for the underground equipment remote control system. It utilizes a three-level architecture combining distributed multi-sensor collaborative acquisition with a composite communication architecture. This architecture encompasses physical layer perception, signal layer processing, and network layer transmission, enabling real-time, reliable acquisition and transmission of multimodal data in complex underground environments. Regarding physical layer perception, the data acquisition unit utilizes a heterogeneous sensor network that combines event-driven and traditional frame-driven approaches. Based on dynamic visual sensing principles, the event camera generates an event stream by asynchronously detecting pixel-level light intensity changes. Its response speed reaches microseconds, and its data volume is only 1% of that of traditional cameras. This makes it particularly suitable for dynamic monitoring of fast-moving equipment underground, such as the cutting head of a roadheader. The event stream is represented by a four-tuple e(x,y,t,p), where x and y are pixel coordinates, t is the timestamp, and p∈{-1,+1} represents the polarity of the light intensity change. Events within a time window are accumulated exponentially to form a pseudo-grayscale image, providing low-latency input for subsequent visual feature extraction.

[0038] The nine-axis inertial measurement unit (IMU) utilizes a MEMS process to integrate a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. It uses an extended Kalman filter algorithm to calculate the device's six-degree-of-freedom (6DOF) posture parameters in real time. The unit employs an adaptive sampling rate mechanism, increasing the sampling frequency during the device's startup and shutdown phases (acceleration rate > 5g / s) and reducing it during stable operation, balancing data accuracy and power consumption. Fiber Bragg grating (FBG) sensors achieve distributed measurement of strain and temperature by measuring the offset of the reflected light's central wavelength. Using wavelength division multiplexing, they integrate 16 sensing points on a single optical fiber, covering key stress-bearing areas of the device (such as the base of the cantilever beam and the hinge points of the hydraulic support). The gas sensor array utilizes a hybrid sensing technology combining catalytic combustion and infrared absorption to provide real-time monitoring of gas concentration within a range of 0-4%, with a response time of <15s. An adaptive baseline calibration algorithm eliminates zero-point drift caused by dust particle adhesion.

[0039] For signal layer processing, the data acquisition unit utilizes a two-stage preprocessing architecture. The first stage of preprocessing is performed locally at the sensor node and includes event stream spatiotemporal denoising (based on a spatiotemporal consistency filtering algorithm), IMU data Kalman filtering, FBG signal wavelength demodulation, and gas concentration and temperature compensation. The second stage of preprocessing is performed at the edge computing unit. Through a high-precision clock synchronization mechanism based on the PTP (IEEE1588) protocol, multi-source heterogeneous data is unified to microsecond timestamps, forming a time-aligned multimodal dataset.

[0040] In terms of network layer transmission, the anti-interference transmission unit adopts a triple-redundant communication architecture. The 5G URLLC link (3.5GHz frequency band, 20MHz bandwidth) is responsible for transmitting control instructions and high-priority status data, and ensures end-to-end delay <20ms and packet loss rate <10^-5 through HARQ retransmission mechanism and slice isolation technology. The orthogonal frequency division multiple access wireless link (2.4GHz ISM frequency band) adopts OFDMA+SC-FDMA hybrid access mode, supports 64 terminals to connect concurrently, and realizes efficient transmission of non-real-time data (such as historical status and environmental parameters). The dynamic frequency hopping spread spectrum module adopts FHSS technology, presets 32 channels in the 2.4-2.4835GHz frequency band, and achieves a frequency hopping rate of ≥1000 times / second through a pseudo-random sequence generator to avoid narrowband interference caused by underground electromechanical equipment.

[0041] The acquisition and transmission module is used to achieve high-fidelity acquisition and reliable transmission of multimodal data in complex underground environments, providing high-quality input data with time synchronization, spatial alignment, and semantic association for subsequent processing and fusion modules, thereby improving the system's perception accuracy and response speed of underground equipment status.

[0042] Processing and fusion module: used to perform time synchronization preprocessing on multimodal data, perform feature fusion through tensor decomposition based on mutual information feature screening, and output a fusion feature vector containing multimodal correlation information; the processing and fusion module includes a data preprocessing unit, a visual feature extraction unit, and a mutual information tensor decomposition unit. The data preprocessing unit is used to perform time synchronization calibration on multimodal data and synthesize a pseudo grayscale image by exponential decay accumulation of event streams. The visual feature extraction unit is used to perform real-time detection of the pseudo grayscale image through a lightweight target detection model and output visual features containing the position and category of obstacles. The mutual information tensor decomposition unit is used to construct the multimodal data containing visual features and environmental parameters after time synchronization into a three-dimensional tensor. By calculating the mutual information between modalities I(X i ;X j ) screens the effective modal combinations and extracts the fusion feature vector z through regularized canonical polyadic decomposition.

[0043] The calculation of the mutual information between modalities is: Among them, p(x i ,x j ) is the joint probability distribution of the i-th and j-th modal data, p(x i )、p(x j ) is the marginal probability distribution;

[0044] Regularization paradigm polyadic is Among them, N is the number of samples, M is the number of modes, K is the feature dimension, R is the decomposition rank, λ is the mutual information regularization parameter, A, B, and C are the feature matrices after decomposition, ||·|| F is the Frobenius norm.

[0045] Specifically, the processing and fusion module connects the acquisition and transmission module with the intelligent decision-making module. It focuses on the collaborative processing and deep fusion of multimodal data, and provides accurate correlation information for intelligent decision-making through a three-level processing process of time series calibration, feature extraction, and tensor decomposition.

[0046] This module receives multimodal data output by the acquisition and transmission module. This data includes the dynamic visual event stream generated by the event camera, attitude and vibration data collected by the nine-axis inertial measurement unit, strain and temperature data obtained by the fiber Bragg grating sensor, and environmental parameters monitored by the gas sensor. The data preprocessing unit first initiates time synchronization calibration. Based on the microsecond timestamp of the PTP protocol in the acquisition and transmission module, it constructs a global time base for the multimodal data to eliminate timing deviations caused by sensor response delays and transmission path differences. For the asynchronous event stream output by the event camera, the data preprocessing unit uses an exponential decay accumulation algorithm, using the polarity of the light intensity change of the event within the time window as the weight, and accumulates the pseudo-grayscale image pixel by pixel. This not only preserves the high timeliness of dynamic vision, but also adapts to the requirements of subsequent frame-based visual algorithms.

[0047] The pre-processed multimodal data enters the visual feature extraction unit. This unit deploys a lightweight target detection model. Aiming at the low redundancy and high dynamic characteristics of the pseudo-grayscale image, it optimizes the model's feature pyramid structure, strengthens the recognition ability of small targets and fast-moving objects, and accurately outputs the position coordinates of obstacles in the image coordinate system, category labels and other visual features. These visual features are input into the mutual information tensor decomposition unit together with the time-synchronized posture vibration data, strain temperature data, and environmental parameters to construct a three-dimensional tensor. Where N corresponds to the number of samples, M represents the modal type (such as vision, inertia, stress, environment, etc.), and K is the characteristic dimension of each modal.

[0048] In the mutual information tensor decomposition unit, the mutual information I(X i ;X j) by quantifying the degree of correlation between the joint probability distribution and the marginal probability distribution, it screens out modal combinations containing strongly correlated information and discards redundant and noisy modalities. Building on this, a regularized canonical polyadic decomposition is introduced to minimize tensor reconstruction error. Combined with a mutual information regularization term to constrain the decomposition process, this approach reduces feature dimensionality while preserving the semantics of multimodal data associations. The decomposed feature matrices A, B, and C are fused to produce a fused feature vector z containing multimodal association information, providing high-quality input for the intelligent decision-making module with both dimensionality reduction and semantic association.

[0049] For different modal data, such as taking visual feature modality X i and inertial attitude mode X j , using the formula Calculate the joint probability distribution p(x i ,x j ) and the marginal probability distribution p(x i )、p(x j ) to measure the correlation between visual features and inertial posture data, screen out effective modal combinations with strong correlation, and discard redundant and noisy modalities. Then, a three-dimensional tensor is constructed based on these modal data. , using the formula When minimizing tensor reconstruction error, a mutual information regularization term is introduced to constrain the decomposition process, resulting in the decomposition of feature matrices A, B, and C, which are then fused to form a fused feature vector. These two formulas enable effective multimodal data association mining and dimensionality compression, ensuring that the output fused feature vector retains the semantics of multimodal associations, providing high-quality input for the intelligent decision-making module and supporting the development of control strategies tailored to the underground environment.

[0050] Through the above process, the processing and fusion module realizes the leap from "heterogeneous acquisition" to "correlated fusion" of multimodal data, uses time synchronization calibration to ensure the temporal consistency of data, and lays the foundation for subsequent fusion; lightweight detection and mutual information screening accurately extract effective features and avoid redundant calculations; regularized tensor decomposition reduces the dimension while retaining the associated semantics, improving the efficiency and accuracy of intelligent decision-making. The final output fusion feature vector can support the intelligent decision-making module to build a control strategy that is more suitable for the complex underground environment.

[0051] Intelligent decision-making module: Based on the fusion feature vector and the downhole physical field stress constraint model constructed by the finite element method, the reinforcement learning algorithm is used to construct a Markov decision process to generate a control strategy that includes the adjustment of equipment operating parameters. The intelligent decision-making module includes a physical field modeling unit and a reinforcement learning decision unit. The physical field modeling unit is used to construct a downhole three-dimensional geological model based on the finite element method, input rock mechanical parameters and solve the stress balance equation, and output the von Mises stress σ of key components of the equipment.von Mises The reinforcement learning decision unit is used to construct the state space of the Markov decision process by fusing the eigenvector and the von Mises stress. The policy network is trained by improving the deep deterministic policy gradient algorithm to output a control strategy containing the equipment operating speed adjustment Δv and the hydraulic pressure adjustment Δp.

[0052] Stress equilibrium equation Where σ is the stress tensor, f is the body force, and includes equipment load and formation gravity;

[0053] Improvements to the deep deterministic policy gradient algorithm include introducing a physical field constrained action clipping mechanism. When σ von Mises >0.8σ thresh When , the gradient of the action output by the policy network is truncated, and the regularization term of the state value function is designed to minimize the value difference of adjacent states. The formula is in, is the experience replay buffer, λ smooth is the smoothing coefficient.

[0054] Improve the deep deterministic policy gradient algorithm to train the policy network and optimize the control policy using the following reward function: Among them, α+β+γ=1 is the weight coefficient, Q prod is the real-time output, Q max is the rated output, c gas is the gas concentration, E cons is the real-time energy consumption, E avg is the average energy consumption, and 1(·) is the safety status indicator function.

[0055] Specifically, the intelligent decision-making module takes over the fusion feature vector output by the processing fusion module, associates the multimodal data attributes fed back by the acquisition and transmission module, and builds a decision-making system of "physical field constraint coupling reinforcement learning optimization". The physical field modeling unit relies on the finite element method to integrate the strain data of the fiber Bragg grating sensor in the acquisition and transmission module and the equipment posture information of the nine-axis inertial measurement unit, and incorporates the rock mechanics parameters obtained from the underground geological survey to build a three-dimensional underground geological-equipment coupling model, and numerically solve the stress balance equation. (where σ is the stress tensor, f aggregates the equipment operating load and the formation gravity), and the von Mises stress σ of the key load-bearing components of the equipment is solved in real time von Mises , define the physical security boundary for control strategy.

[0056] The reinforcement learning decision unit will process the fusion feature vector of the fusion module and the von Mises stress output by the physical field modeling unit and incorporate them into the state space of the Markov decision process to achieve the fusion mapping of multimodal semantic information and physical constraints. In order to improve the deep deterministic policy gradient algorithm, on the one hand, the action clipping mechanism of physical field constraints is introduced: when σ von Mises Exceeding 80% of the safety threshold (0.8σ thresh ), the gradient of the equipment operating parameter adjustment action output by the strategy network is truncated to avoid high stress risk conditions; on the other hand, the regularization term of the design state value function Using the Experience Replay Buffer The state samples s and s′ obtained by processing the multimodal data (such as device posture, environmental parameters, etc.) fed back by the collection and transmission module are stored in the formula to calculate the regularization term of the state value function, thereby reducing the difference in the value function between adjacent states, making the policy training more stable, and avoiding the instability of the policy network training due to large state fluctuations. The smoothing coefficient λ is used to calculate the regularization term of the state value function. smooth Constrain the value function fluctuations to further improve the stability of strategy training.

[0057] In the strategy optimization stage, the reinforcement learning decision unit combines the gas concentration c fed back by the collection and transmission module gas , the real-time output Q sent back by the equipment execution module prod and energy consumption E cons , through the reward function Quantify the decision benefits and balance production efficiency, safety constraints and energy consumption optimization goals with α+β+γ=1. The policy network trained by the improved algorithm outputs a control strategy that includes the equipment speed adjustment Δv and the hydraulic pressure adjustment Δp. It not only meets the stress safety requirements of the underground physical field, but also adapts to the changes in working conditions represented by multimodal data, providing a precise control basis for the equipment execution module, supporting the stable operation of underground equipment in complex environments. For example: the gas concentration c collected by the transmission module gas , Real-time output Q fed back by the equipment execution module prod and real-time energy consumption E cons As input, combined with the rated output Q max , average energy consumption E avg And the weight coefficients α, β, γ (satisfying α+β+γ=1), calculate the reward value r. The reward value is used to optimize the reinforcement learning strategy network so that the trained strategy can ensure the safety of gas concentration (c gas Under the premise of triggering the safety status indication function 1(·) when it is less than 1%, balance the production efficiency (output ratio ) and energy consumption (energy consumption as a percentage of ), and the final output includes the control strategy of the equipment operating speed adjustment Δv and the hydraulic pressure adjustment Δp, allowing the equipment to operate stably, efficiently and safely in the complex environment underground.

[0058] MR interactive control module: used to realize three-dimensional visualization presentation of equipment status parameters, digital twin models and path planning information through mixed reality terminals based on control strategies, and receive user intervention instructions through gesture recognition and voice recognition engines for human-computer interaction; the MR interactive control module includes a three-dimensional visualization unit and a natural interaction unit. The three-dimensional visualization unit is used to render the equipment digital twin model, real-time status parameter heat map and path planning guidance through the mixed reality terminal according to the control strategy and equipment status data, and output them to the natural interaction unit. The natural interaction unit is used to convert user intervention instructions into control signals through gesture recognition engines and voice recognition engines, and send them to the equipment execution module or intelligent decision-making module. User intervention instructions include emergency shutdown, parameter adjustment, and mode switching.

[0059] Specifically, the MR interactive control module serves as the human-machine interface connecting control strategies and operators, creating an immersive control environment through 3D visualization and natural interaction technologies. The 3D visualization unit receives control strategies output by the intelligent decision-making module and real-time status data from the equipment execution module. Based on the Unity3D engine, it constructs a digital twin model of the equipment, whose geometric parameters and physical properties directly mirror the physical equipment downhole. The model utilizes hierarchical detail rendering technology, dynamically switching LOD (Level of Detail) at different viewing distances to ensure efficient real-time rendering of complex mechanical structures.

[0060] To target key equipment parameters, the 3D visualization unit uses heat map visualization technology: physical quantities such as temperature and strain acquired by the acquisition and transmission module are mapped to color codes, and dynamic color transitions on the model surface are achieved through shader programming. For example, strain data collected by the fiber Bragg grating sensor is processed and overlaid on the surface of the equipment's digital twin model in a red-yellow-green gradient. The color intensity corresponds to the stress value, and areas exceeding the threshold automatically flash as a warning. Path planning guidance is based on the optimal trajectory generated by the intelligent decision-making module and rendered in space as a translucent light band. The width and curvature of the light band reflect the equipment's operating speed and steering angle in real time.

[0061] The natural interaction unit adopts a multimodal fusion human-computer interaction architecture. The gesture recognition engine is based on the Microsoft Azure Kinect depth camera, builds a palm bone point tracking system, and recognizes specific gesture semantics through a spatiotemporal convolutional network. For example, the fist-clenching action is mapped to an emergency stop command, sliding the palm left and right corresponds to parameter adjustment, and opening the arms triggers a mode switch. The speech recognition engine adopts a hybrid architecture of offline wake-up + cloud recognition: a lightweight keyword detection model (such as "system wake-up") is deployed locally, and after wake-up, the voice stream is transmitted to the cloud ASR service through a 5G link. The Transformer architecture is used to realize speech recognition in complex environmental noise underground, and the recognition results are parsed into control intentions through NLP processing.

[0062] To ensure real-time interactive responses, predictive rendering technology is employed. Based on the device's current state and control strategy, the twin model's posture changes for the next 200ms are rendered in advance. Combined with HoloLens 3's eye tracking capabilities, the rendering accuracy of the focal area of ​​the viewing cone is dynamically adjusted. When a user issues an intervention command, the natural interaction unit distributes it to the device's execution module or intelligent decision-making module via a message bus. Emergency stop commands directly trigger the execution module's safety braking sequence; parameter adjustment commands are evaluated by the intelligent decision-making module to generate a new control strategy; and mode switch commands reconfigure the parameter configuration of the entire decision-making-execution closed loop.

[0063] This module presents the status of underground equipment in an intuitive three-dimensional form through an interactive interface that integrates virtual and real elements. The operator's natural interactive commands are seamlessly integrated into the control process, enhancing the immersion and decision-making efficiency of remote operations.

[0064] Device Execution Module: This module is used to receive and execute control strategies or user intervention instructions, adjust device operating parameters, and provide feedback on execution status. The device execution module includes a drive control unit and a state feedback unit. The drive control unit is used to control the device according to the control strategy or user intervention instructions, including adjusting the device's operating speed through a servo motor and adjusting the hydraulic pressure through a hydraulic valve group. The state feedback unit is used to collect the device's real-time operating speed through an encoder and the real-time hydraulic pressure through a pressure transmitter. This generates execution status data containing speed v and pressure p, which is fed back to the acquisition and transmission module and the MR interactive control module.

[0065] Specifically, the equipment execution module serves as the execution terminal and state feedback hub of the control instructions, taking over the control strategy output by the intelligent decision-making module and the user intervention instructions forwarded by the MR interactive control module, and building a closed-loop link of "instruction analysis-drive regulation-state feedback". The drive control unit receives the equipment operating speed adjustment Δv, hydraulic pressure adjustment Δp in the control strategy, as well as user emergency shutdown, parameter adjustment and other instructions: For the servo motor, based on the encoder data of the state feedback unit, a speed closed-loop control is built, and the motor drive pulse is dynamically adjusted through the PID algorithm to make the actual speed of the equipment fit the target speed (v target =v current +Δv); for the hydraulic valve group, according to the real-time pressure feedback from the pressure transmitter, the valve core opening is controlled by the proportional solenoid, and the pressure adjustment demand is linearly responded (p target =p current +Δp). When the emergency stop command is triggered, the drive control unit immediately cuts off the power circuit, activates the mechanical brake mechanism, and simultaneously feeds back the emergency stop status to the intelligent decision module.

[0066] The state feedback unit uses an encoder to collect the real-time operating speed of the equipment, and a pressure transmitter to collect the real-time pressure of the hydraulic system with a time accuracy that matches that of the acquisition and transmission module. After anti-aliasing filtering, it generates execution status data including speed v and pressure p. On the one hand, this data is transmitted back to the acquisition and transmission module, where it participates in the timing calibration and feature association of the processing fusion module as a core component of multimodal data. On the other hand, it is transmitted to the MR interactive control module to provide a real-time basis for the three-dimensional visualization unit to update the state heat map of the equipment digital twin model and dynamically correct path planning. Through the coordination of precise command execution and real-time status feedback, the equipment execution module not only ensures the accuracy of intelligent decision-making and user intervention, but also provides real-world working condition support for the closed-loop control of the system, helping underground equipment to operate stably and dynamically adapt in complex environments.

[0067] The acquisition and transmission module uses multimodal sensors such as event cameras and nine-axis inertial measurement units to obtain dynamic visual events, posture vibrations, structural strains and environmental parameters of the equipment in real time. URLLC and dynamic frequency hopping spread spectrum technologies ensure interference-resistant data transmission, providing the system with high-quality raw data aligned in time and space. The processing and fusion module eliminates timing deviations in multi-source data through time synchronization calibration, screens strongly correlated modes using mutual information calculation, and extracts low-dimensional fusion feature vectors that incorporate the coupling relationship between visual, mechanical, and environmental information through regularized tensor decomposition, addressing the redundancy and lack of correlation issues inherent in traditional shallow fusion. The intelligent decision-making module combines the fused feature vectors with the von Mises stress output from the finite element physics model to construct a reinforcement learning decision-making model with physical constraints. An improved deep deterministic policy gradient algorithm is used to generate control strategies that balance production efficiency, safety margins, and energy optimization, thus avoiding the safety risks of purely data-driven decisions. The MR interactive control module uses a digital twin model to three-dimensionally visualize equipment status and path planning, combining gesture / voice interaction for natural intervention, enhancing operators' spatial perception and decision-making efficiency in complex working conditions. The equipment execution module precisely executes control commands based on servo motors and hydraulic valve groups, and provides real-time operating status feedback through encoders and pressure transmitters, forming a closed control loop. It achieves efficient fusion of multimodal data of downhole equipment, intelligent decision-making under physical constraints, immersive human-computer interaction and precise execution control, and solves the problem that traditional downhole equipment remote control systems for MR are unable to adapt to complex dynamic scenarios due to insufficient multimodal perception fusion and lack of decision-making security, resulting in low equipment operation efficiency.

[0068] Please see the attached Figure 2 A remote control method for downhole equipment for MR is applied to the above-mentioned remote control system for downhole equipment for MR, comprising the following steps:

[0069] Acquisition and transmission: Collect multimodal data from downhole equipment and transmit the data in an anti-interference manner;

[0070] Processing and fusion: Perform time synchronization preprocessing on multimodal data, perform feature fusion through tensor decomposition based on mutual information feature screening, and output a fused feature vector containing multimodal correlation information;

[0071] Intelligent decision-making: Based on the fused feature vectors and the stress constraint model of the downhole physical field constructed using the finite element method, a reinforcement learning algorithm is used to construct a Markov decision process to generate a control strategy that includes adjustments to equipment operating parameters.

[0072] MR interactive control: Based on the control strategy, the mixed reality terminal realizes the three-dimensional visualization of equipment status parameters, digital twin models and path planning information, and receives user intervention instructions through gesture recognition and voice recognition engines to conduct human-computer interaction;

[0073] Device execution: Receives and executes control strategies or user intervention instructions, adjusts device operating parameters, and provides feedback on execution status.

[0074] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A remote control system for downhole equipment used in MR, characterized in that: include: Acquisition and transmission module: used to collect multimodal data from downhole equipment and transmit the data in an anti-interference manner; Processing and fusion module: used to perform time synchronization preprocessing on multimodal data, perform feature fusion through tensor decomposition based on mutual information feature screening, and output a fused feature vector containing multimodal correlation information; Intelligent decision-making module: Based on the fused feature vectors and the stress constraint model of the downhole physical field constructed using the finite element method, the module uses a reinforcement learning algorithm to construct a Markov decision process and generate a control strategy that includes adjustments to equipment operating parameters. MR interactive control module: This module is used to achieve 3D visualization of device status parameters, digital twin models, and path planning information through a mixed reality terminal based on control strategies. It also receives user intervention commands through gesture recognition and voice recognition engines for human-computer interaction. Device execution module: used to receive and execute control strategies or user intervention instructions, adjust device operating parameters and feedback execution status.

2. A downhole equipment remote control system for MR according to claim 1, characterized in that: The acquisition and transmission module includes a data acquisition unit and an anti-interference transmission unit. The data acquisition unit is used to collect multimodal data of downhole equipment through acquisition equipment. The acquisition equipment includes an event camera, a nine-axis inertial measurement unit, a fiber Bragg grating sensor, and a gas sensor. The event camera is used to collect dynamic visual event streams containing pixel coordinates, timestamps, and polarization information. The nine-axis inertial measurement unit is used to collect posture vibration data of equipment acceleration, angular velocity, and magnetic field strength. The fiber Bragg grating sensor is used to collect strain and temperature data of key components of the equipment. The gas sensor is used to collect environmental parameters such as downhole gas concentration and dust concentration. The anti-interference transmission unit transmits various data in an anti-interference manner through a composite communication architecture including a 5G ultra-reliable low-latency communication link, an orthogonal frequency division multiple access wireless communication link, and a dynamic frequency hopping spread spectrum module.

3. The downhole equipment remote control system for MR according to claim 1, characterized in that: The processing fusion module includes a data preprocessing unit, a visual feature extraction unit, and a mutual information tensor decomposition unit. The data preprocessing unit is used to perform time synchronization calibration on the multimodal data and synthesize the event stream into a pseudo grayscale image through exponential decay accumulation. The visual feature extraction unit is used to perform real-time detection of the pseudo grayscale image through a lightweight target detection model and output visual features containing the position and category of the obstacle. The mutual information tensor decomposition unit is used to construct the multimodal data containing visual features and environmental parameters after time synchronization into a three-dimensional tensor. By calculating the mutual information between modalities I(X i ;X j ) screens the effective modal combinations and extracts the fusion feature vector z through regularized canonical polyadic decomposition.

4. A downhole equipment remote control system for MR according to claim 3, characterized in that: The calculation of the mutual information between modalities is: Among them, p(x i ,x j ) is the joint probability distribution of the i-th and j-th modal data, p(x i )、p(x j ) is the marginal probability distribution; The regularization paradigm polyadic is Among them, N is the number of samples, M is the number of modes, K is the feature dimension, R is the decomposition rank, λ is the mutual information regularization parameter, A, B, and C are the feature matrices after decomposition, ||·|| F is the Frobenius norm.

5. The downhole equipment remote control system for MR according to claim 1, characterized in that: The intelligent decision-making module includes a physical field modeling unit and a reinforcement learning decision-making unit. The physical field modeling unit is used to construct a three-dimensional geological model of the well based on the finite element method, input rock mechanical parameters and solve the stress balance equation, and output the von Mises stress σ of key components of the equipment. vonMises The reinforcement learning decision unit is used to construct the state space of the Markov decision process by fusing the eigenvector and the von Mises stress, and to train the policy network by improving the deep deterministic policy gradient algorithm to output a control strategy including the equipment operating speed adjustment Δv and the hydraulic pressure adjustment Δp.

6. A downhole equipment remote control system for MR according to claim 5, characterized in that: The stress balance equation Where σ is the stress tensor, f is the body force, and includes equipment load and formation gravity; The improved deep deterministic policy gradient algorithm includes an action clipping mechanism that introduces physical field constraints. When σ vonMises >0.8σ thresh When , the gradient of the action output by the policy network is truncated, and the regularization term of the state value function is designed to minimize the value difference of adjacent states. The formula is in, is the experience replay buffer, λ smooth is the smoothing coefficient.

7. The downhole equipment remote control system for MR according to claim 5, characterized in that: The improved deep deterministic policy gradient algorithm trains the policy network and optimizes the control policy using the following reward function: Among them, α+β+γ=1 is the weight coefficient, Q prod is the real-time output, Q max is the rated output, c gas is the gas concentration, E cons is the real-time energy consumption, E avg is the average energy consumption, and 1(·) is the safety status indicator function.

8. The downhole equipment remote control system for MR according to claim 1, characterized in that: The MR interactive control module includes a three-dimensional visualization unit and a natural interaction unit. The three-dimensional visualization unit is used to render the device digital twin model, real-time status parameter heat map and path planning guidance through the mixed reality terminal according to the control strategy and device status data, and output them to the natural interaction unit. The natural interaction unit is used to convert user intervention instructions into control signals through the gesture recognition engine and the voice recognition engine, and send them to the device execution module or the intelligent decision-making module. The user intervention instructions include emergency shutdown, parameter adjustment, and mode switching.

9. The downhole equipment remote control system for MR according to claim 1, characterized in that: The device execution module includes a drive control unit and a state feedback unit. The drive control unit is used to control the device to adjust according to the control strategy or user intervention instructions, including adjusting the device operating speed through a servo motor and adjusting the hydraulic pressure through a hydraulic valve group. The state feedback unit is used to collect the real-time operating speed of the device through an encoder and the real-time hydraulic pressure through a pressure transmitter to form execution status data including speed v and pressure p, and feed it back to the acquisition and transmission module and the MR interactive control module.

10. A method for remotely controlling downhole equipment for MR, characterized in that: A downhole equipment remote control system for MR as claimed in any one of claims 1 to 9 comprises the following steps: Acquisition and transmission: Collect multimodal data from downhole equipment and transmit the data in an anti-interference manner; Processing and fusion: Perform time synchronization preprocessing on multimodal data, perform feature fusion through tensor decomposition based on mutual information feature screening, and output a fused feature vector containing multimodal correlation information; Intelligent decision-making: Based on the fused feature vectors and the stress constraint model of the downhole physical field constructed using the finite element method, a reinforcement learning algorithm is used to construct a Markov decision process to generate a control strategy that includes adjustments to equipment operating parameters. MR interactive control: Based on the control strategy, the mixed reality terminal realizes the three-dimensional visualization of equipment status parameters, digital twin models and path planning information, and receives user intervention instructions through gesture recognition and voice recognition engines to conduct human-computer interaction; Device execution: Receives and executes control strategies or user intervention instructions, adjusts device operating parameters, and provides feedback on execution status.

Images