Hydraulic support XR virtual-real interaction and hybrid intelligent cooperative regulation and control method and system

Abstract

The invention belongs to the technical field of hydraulic support intelligent control and decision making, and provides a hydraulic support XR virtual-real interaction and hybrid intelligent cooperative regulation and control method and system in order to solve the problem that related technologies cannot adapt to complex working conditions. A hydraulic support XR virtual-real interaction and hybrid intelligent cooperative regulation and control system composed of a multi-modal data acquisition module, a hybrid decision model, a dynamic confidence evaluation module, an AR / VR cooperative control interface and a man-machine mutual feedback optimization module is constructed, and multi-modal data real-time acquisition and interconnection, feature matrix construction and hybrid decision generation are executed. According to the regulation and control method for cooperative decision execution and optimization, XR virtual-real interaction, knowledge and data dual-drive decision and a dynamic confidence mechanism are combined, and decision reliability and man-machine cooperation efficiency of the hydraulic support under complex working conditions are improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control and decision-making technology for hydraulic supports, and specifically relates to a method and system for XR virtual-reality interaction and hybrid intelligent collaborative control of hydraulic supports. Background Art

[0002] With the advent of the smart mine concept, intelligent decision-making technology for hydraulic supports has gradually evolved from single-sensor data analysis to multimodal fusion and adaptive learning. Simultaneously, the application of extended reality (XR) technology in the coal mining sector has rapidly grown. Through interactive methods such as augmented reality (AR) and virtual reality (VR), XR enables remote control of coal mining faces and collaborative virtual-reality interaction, providing new insights into intelligent decision-making for hydraulic supports. Combining XR's immersive interactive capabilities with intelligent decision-making technology to build efficient human-machine collaborative decision-making approaches has become a key area of ​​focus for intelligent decision-making in hydraulic supports.

[0003] The goal of the public paper "Exploration and Practice of Intelligent Operation Mode of Human-Machine Collaboration in Comprehensive Mining Faces Driven by 'Human-Oriented Intelligent Manufacturing and XR+'" is to improve the level of intelligent human-machine collaboration in coal mine comprehensive mining faces through the concept of "human-oriented intelligent manufacturing". It uses XR (extended reality) technology and a multi-functional parallel system to build a virtual simulation scene with deep integration of "man-machine-environment". Through key technologies such as parameterized virtual miner models, real-time human-machine linkage, and inspection gesture interaction, it realizes intelligent deduction and collaborative decision-making between inspection workers, centralized control workers, and maintenance workers and virtual systems. It effectively improves the operating efficiency and operational reliability of remote monitoring, intelligent inspection, and equipment maintenance of working faces, and provides a new technical path for human-machine collaborative decision-making for the intelligent transformation of coal mines. However, the technology disclosed in this paper still has the problem of insufficient adaptability under complex working conditions, and the control lags under dynamic working conditions, resulting in low control efficiency. Summary of the Invention

[0004] In order to solve at least one of the above-mentioned technical problems existing in the prior art, the present invention provides a method and system for XR virtual-reality interaction and hybrid intelligent collaborative control of a hydraulic support.

[0005] According to the first aspect, a hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control method includes the following steps: Step 1: Real-time collection and interconnection of multimodal data; Step 1.1: Acquire first operating data of a virtual hydraulic support controlled by a control instruction in real time through a virtual scene on an AR end, wherein the first operating data includes posture parameters and interaction events of the virtual hydraulic support; Step 1.2: Simultaneously obtain sensor data of the actual hydraulic support during operation mapped to the AR end, wherein the sensor data includes pressure, inclination angle, and displacement parameters; Step 1.3: Acquire second operation data when the VR terminal controls the real hydraulic support, wherein the second operation data includes a handle input signal and physical support movement feedback; Step 1.4: Interconnect the data obtained from the AR and VR terminals in real time to obtain a multimodal dataset of the hydraulic support in virtual and real scenes; Step 2: Feature matrix construction and hybrid decision generation; Step 2.1: Denoising, standardizing, and feature extracting the multimodal dataset, constructing a multimodal feature matrix, and generating a training set; Step 2.2: Using the training set to train the XGBoost decision model, outputting the predicted posture data of the hydraulic support, and generating a data-driven first control decision recommendation based on the prediction result; Step 2.3: Parallel call the rules in the preset expert decision knowledge base, match the current working conditions through the Drools decision rule engine, and generate rule-driven second control decision recommendations; Step 3: Collaborative decision execution and optimization: Step 3.1: Fusing the first control decision suggestion and the second control decision suggestion to obtain a fused decision suggestion, and calculating a joint confidence score in combination with a preset confidence function; Step 3.2: When the confidence reaches a preset threshold, the control instruction is automatically executed; otherwise, the fusion decision recommendation is pushed to the terminal device; Step 3.3: Drive the XGBoost decision model to perform incremental learning and update the Drools decision rule engine.

[0006] Preferably, step 1.4 includes: The TCP dual-channel protocol is used to achieve real-time mapping of the pose data of the virtual hydraulic support on the AR side and the real hydraulic support on the VR side. At the same time, the virtual-reality consistency score is calculated based on the three-dimensional coordinate deviation and dynamically updated to the multimodal dataset.

[0007] Preferably, the feature extraction in step 2.1 includes: Extract motion acceleration and spatial coordinate sequences from the AR-side gesture trajectory in the multimodal dataset, parse frequency and semantic labels from the voice commands in the multimodal dataset, and align them with the sensor time series data in the multimodal dataset to construct a time series correlation feature matrix.

[0008] Preferably, the preset credibility function in step 3.1 is: Where, is the weight coefficient of the XGBoost decision model; is the weight coefficient of the Drools decision rule engine; The weight coefficient for the consistency score between the virtual hydraulic support on the AR side and the real hydraulic support on the VR side; is the predicted probability of the XGBoost decision model; The decision rule matching degree of the Drools decision rule engine, that is, the ratio of the number of triggered rules to the total number of rules; Score the consistency between the virtual hydraulic support in the AR scene and the real hydraulic support in the VR scene.

[0009] According to a second aspect, a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system is capable of executing a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control method as described in the first aspect and any preferred embodiment, comprising: a multimodal data acquisition module, a hybrid decision model, a dynamic confidence assessment module, an AR / VR collaborative control interface, and a human-machine mutual feedback optimization module; The multimodal data acquisition module is used to collect and interconnect multimodal data in real time and store the collected multimodal data; The hybrid decision model is used to integrate the XGBoost decision model and the Drools decision rule engine to construct a feature matrix and generate a hybrid decision; The dynamic confidence evaluation module is used to calculate the joint confidence according to a preset confidence function and dynamically allocate control instructions according to the calculation results; The AR / VR collaborative control interface is used to achieve bidirectional control and data synchronization of virtual and real scenes between the AR and VR ends through the TCP dual-channel protocol; The human-machine mutual feedback optimization module is used to update the XGBoost decision model and the Drools decision rule engine in real time.

[0010] Preferably, the multimodal data acquisition module includes an XR data interaction unit, a sensor data unit, and a recording unit; The XR data interaction unit is used to capture AR gesture trajectories and voice commands through the MRTK module of the Unity3D engine, and to achieve real-time data sharing and display with the VR scene of the HTC Vive; The sensor data unit is used to obtain the hydraulic support pressure sensor data, tilt sensor data, and displacement sensor data, and perform data preprocessing, and simultaneously obtain environmental parameter data including at least downhole visibility, dust concentration, temperature, and humidity; The recording unit is used to encode the control action instructions sent by the terminal device into an OP-Code sequence, associate it with the working condition tag and store it in the SQL Server database.

[0011] Preferably, the hybrid decision model includes: Perform decision conflict arbitration. When the first regulation decision suggestion generated by the XGBoost decision model conflicts with the second regulation decision suggestion generated by the Drools decision rule engine, the operation instruction issued by the terminal device is executed first, and the rule weight is reversely optimized based on feature importance analysis.

[0012] Preferably, the XGBoost decision model adopts dynamic weight optimization to update new feature parameters based on the weights of historically important features; the Drools decision rule engine retains a copy of the old version when it is updated, and if the new rule causes the control error rate to increase, it will go back to the previous stable version; the Drools decision rule engine and the XGBoost decision model realize bidirectional parameter optimization.

[0013] Preferably, the AR / VR collaborative control interface includes: an anti-accidental touch unit, an environment adaptation unit, and a multi-person collaborative protocol module; The anti-accidental touch unit is used to control the movement of the virtual hydraulic support by setting a voice-gesture dual control mode; The environmental adaptation unit is used to dynamically adjust the transparency of the AR end display interface and support voice control mode based on the underground environmental parameter data; The multi-person collaborative protocol module is used to allocate control rights based on role authority priority when multiple users operate the same virtual hydraulic support, and low-authority users can only view but not operate.

[0014] Preferably, the human-machine mutual feedback optimization module includes a terminal device data recording module, a feedback learning unit and a rule generation unit; The terminal device data recording module is used to record multimodal data when operating according to the control instructions sent by the terminal device and store it in the conflict sample library in the SQL Server; The feedback learning unit is used to train the XGBoost decision model based on the newly added data samples and retain the historical version; The rule generation unit is used to analyze the control instructions sent by the terminal device through the Apriori algorithm and generate candidate rules, and synchronously update them to the Drools decision rule engine.

[0015] Compared with the prior art, the present invention has the following beneficial effects: Compared with the existing model, this method proposes a method for constructing multimodal data acquisition, which effectively solves the problem that the training of the decision model is overly dependent on the single-dimensional operating data of the three fully mechanized mining machines, which leads to insufficient adaptability of the system to complex working conditions.

[0016] The method proposed in this paper proposes a hybrid decision-making model that integrates machine learning and decision-making rules for intelligent decision-making in hydraulic supports, breaking through the limitations of traditional single static decision-making models. By integrating the XGBoost decision-making model with the Drools decision-making rule engine, a hybrid decision-making model that combines data-driven and knowledge-driven collaboration is constructed, solving the problem of model prediction and expert experience in traditional systems. The method also introduces decision-conflict arbitration logic, giving priority to responding to manual operation instructions sent by terminal devices based on XR scenarios when AI suggestions conflict with rules, and reversely optimizing the knowledge base weights.

[0017] In the construction of the decision-making authority allocation mechanism, the method of the present invention proposes an authority allocation method based on dynamic confidence. By fusing multi-source data such as the XGBoost model prediction probability, the decision rule matching degree, and the consistency of the virtual-real system interaction, the joint confidence is calculated in real time. Based on the pre-set threshold, an elastic authority switching mechanism is selected for automatic execution or manual confirmation to achieve dynamic adaptation of control authority and working conditions. Based on the AR interface, the voice interaction mode is automatically switched under complex working conditions such as excessive dust concentration to ensure operational reliability in harsh environments.

[0018] In the multimodal interaction of XR scenarios, the method of the present invention proposes a multimodal collaborative verification method. By integrating gesture + voice dual interaction mode verification, it effectively ensures the prevention of accidental touch in AR / VR scenarios. At the same time, it develops a multi-person collaborative priority protocol, intelligently allocates control rights according to job permissions, and solves the problem of multi-user operation conflicts.

[0019] This method, developed for intelligent decision-making feedback verification of hydraulic supports, proposes a method for the co-evolution of decision models and knowledge-based decision rules. By collecting real-time snapshots of multimodal data from manual corrections in XR scenarios, it drives online learning and updates of machine learning models, forming a closed-loop "operation-feedback-optimization" process. A flexible weight allocation method is used to retain important historical features. Multimodal data mining combined with manual operations automatically generates candidate decision rules, which are then dynamically updated in the Drools decision rule engine after manual review. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is a schematic diagram of the architecture of a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 2This is a schematic diagram of a hydraulic support XR virtual simulation scene in a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 3 This is a schematic diagram of a multimodal data acquisition module in a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 4 This is a schematic diagram of the construction of a hybrid decision-making module in the hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 5 This is a schematic diagram of the construction of a dynamic confidence assessment module in the hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 6 This is a schematic diagram of the construction of a human-machine mutual feedback optimization module of a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 7 This is a schematic diagram of the construction of an AR / VR collaborative control interface module in the hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by the present invention; Figure 8 This is a flow chart of a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control method provided by the present invention. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present invention are clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other implementations derived by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0023] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention. It should be noted that in this specification, relational terms such as first and second are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0024] The existing technologies related to hydraulic support automated decision-making models still have the following defects: First, the traditional electro-hydraulic control interface lacks three-dimensional visualization and real-time data mapping functions, making it difficult for operators to quickly perceive the coordinated status of the bracket group, and there is a cognitive gap in human-machine interaction; Second, the training of the decision-making model overly relied on single-dimensional operating data of the three fully mechanized mining machines, failing to effectively integrate multimodal data such as underground environmental parameters, personnel operation information, and XR interaction information, resulting in insufficient adaptability of the system to complex working conditions. Third, existing hydraulic support decision-making systems are mostly based on static rules, with rigid modal selection and a lack of real-time two-way mapping between multimodal sensor data and digital twin models. Furthermore, the knowledge-driven and data-driven decision-making processes are separated, leading to delayed control under dynamic working conditions. Fourth, the existing methods in the field of hydraulic support control lack dynamic decision-making authority allocation mechanism and safety mechanism, resulting in an imbalance between control efficiency and safety.

[0025] To solve the above problems, an embodiment of the present invention provides a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control method and system, which combines XR virtual-reality interaction, knowledge and data dual-driven decision-making and dynamic confidence mechanism to achieve improved decision-making reliability and human-machine collaboration efficiency of the hydraulic support under complex working conditions.

[0026] In order to more clearly introduce the above-mentioned objects, features and advantages of the present invention, further detailed description is given below with reference to the accompanying drawings and specific embodiments.

[0027] In this embodiment of the present invention, sensor data is obtained using the following sensors as examples: a SICK LMS511 laser displacement sensor with a ranging accuracy of ±1 cm and a scanning frequency of 50 Hz; a Bosch Rexroth pressure sensor with a range of 0-400 bar and an accuracy of ±0.05% FS; and a TE Connectivity FX29 inclination sensor with a measurement range of ±90°. The backbone network uses an industrial-grade gigabit fiber optic ring network.

[0028] Virtual reality devices required for XR scenarios: Microsoft HoloLens 2, HTC Vive Focus3, and VR controller.

[0029] like Figure 1 Figure 1 shows a schematic diagram of the structure of a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system provided by an embodiment of the present invention, including: a multimodal data acquisition module, a hybrid decision model, a dynamic confidence assessment module, an AR / VR collaborative control interface, and a human-machine mutual feedback optimization module; The multimodal data acquisition module is used to collect and interconnect multimodal data in real time and store the collected multimodal data; The hybrid decision model is used to integrate the XGBoost decision model and the Drools decision rule engine to construct a feature matrix and generate a hybrid decision; The dynamic confidence evaluation module is used to calculate the joint confidence according to a preset confidence function and dynamically allocate control instructions according to the calculation results; The AR / VR collaborative control interface is used to achieve bidirectional control and data synchronization of virtual and real scenes between the AR and VR ends through the TCP dual-channel protocol; The human-machine mutual feedback optimization module is used to update the XGBoost decision model and the Drools decision rule engine in real time.

[0030] Optionally, the multimodal data acquisition module includes an XR data interaction unit, a sensor data unit and a recording unit; the XR data interaction unit is used to capture the gesture trajectory and voice commands of the AR end through the MRTK module of the Unity3D engine, and realize real-time data sharing and display with the scene of the VR end of the HTC Vive; the sensor data unit is used to obtain the hydraulic support pressure sensing data, tilt sensing data, displacement sensing data, and perform data preprocessing, and at the same time obtain environmental parameter data including at least underground visibility, dust concentration, temperature, and humidity; the recording unit is used to encode the control action instructions sent by the terminal device into an OP-Code sequence, and at the same time associate the working condition label and store it in the SQL Server database.

[0031] Optionally, the hybrid decision model includes: performing decision conflict arbitration. When the first regulation decision suggestion generated by the XGBoost decision model conflicts with the second regulation decision suggestion generated by the Drools decision rule engine, the operation instruction issued by the terminal device is executed first, and the rule weight is reversely optimized based on feature importance analysis.

[0032] Optionally, the XGBoost decision model uses dynamic weight optimization to update new feature parameters based on the weights of historically important features. The Drools decision rule engine retains a copy of the old version when it is updated. If the new rule causes the control error rate to increase, it will go back to the previous stable version. The Drools decision rule engine and the XGBoost decision model implement bidirectional parameter optimization.

[0033] Optionally, the AR / VR collaborative control interface includes: an anti-accidental touch unit, an environmental adaptation unit and a multi-person collaborative protocol module; the anti-accidental touch unit is used to control the action of the virtual hydraulic support by setting a voice-gesture dual control mode; the environmental adaptation unit is used to dynamically adjust the transparency of the AR-end display interface, and at the same time support the voice control mode based on the downhole environmental parameter data; the multi-person collaborative protocol module is used to allocate control rights based on role authority priority when multiple users operate the same virtual hydraulic support, and low-authority users can only view but not operate.

[0034] Optionally, the human-computer mutual feedback optimization module includes a terminal device data recording module, a feedback learning unit and a rule generation unit; the terminal device data recording module is used to record multimodal data when operating according to the control instructions sent by the terminal device and store it in the conflict sample library in SQL Server; the feedback learning unit is used to train the XGBoost decision model based on the newly added data samples and retain historical versions; the rule generation unit is used to analyze the control instructions sent by the terminal device through the Apriori algorithm and generate candidate rules, and synchronously update them to the Drools decision rule engine.

[0035] In this embodiment, Figure 2 The figure shows a schematic diagram of the hydraulic support XR virtual simulation scene, in which the collected sensor data is preprocessed by denoising, outlier processing, filling missing data, data standardization, and feature extraction.

[0036] Build a 3D model assembly of the hydraulic support in Solidworks. After converting the format to ".fbx", in Unity 3D, right-click the folder in the project window and select "Import New Asset" to import the converted file. The model includes components such as the base, front and rear connecting rods, piston, and hydraulic cylinder. Change the position and size of the model based on the "Transform" component in Unity.

[0037] The imported hydraulic support model is based on Unity's "Rigidbody" and "Collider" components, and its mass, density resistance, angular resistance, inertia and other parameter values ​​are defined to ensure that the component has real physical properties and can collide with other components, and add parent-child relationships between the components.

[0038] Analyze the motion relationship between the various components of the hydraulic support, and add joint components such as Hinge Joint and Configureable Joint to connect the various components of the hydraulic support, and define parameters such as spring stiffness and damping of the joint components.

[0039] Based on Unity's "UGUI" module, a virtual control panel for the hydraulic support's electro-hydraulic control, a physical sensor data display panel, and a decision-making recommendation panel are created. The action script for the hydraulic support is written in C# to give the support basic actions.

[0040] In this embodiment, Figure 3 The figure below shows a schematic diagram of the multimodal data acquisition module. Enable XR plug-in management in the Unity project. Set the project to support Windows Mixed Reality by clicking "XR Plug-in Management" under "Edit," "Project Settings," and then "XR Plug-in Management." Download and install the MRTK plug-in from the Unity Asset Store or GitHub to enable gesture and voice control of the hydraulic support and spatial positioning in AR scenarios based on the HoloLens 2.

[0041] Open the "Input" configuration file of MRTK, enable the gesture recognition module, and set gestures such as grabbing, rotating, and making a fist to control the virtual twin. Set the response behavior of each gesture in the "Hand Tracking Profile". Enable the "Speech" voice recognition function in MRTK so that the operator can control the movement of the virtual hydraulic support through voice commands. Create a C# script in the Unity scene to receive gestures and voice commands and convert them into movements of the virtual support.

[0042] Based on the TCP communication protocol, a TCP communication script is written in Unity and connected to the physical sensor data server of the hydraulic support, so that the AR end can receive the sensor data of the real hydraulic support in real time. Use "System.Net.Sockets" to establish a TCP connection; create a C# script to update the hydraulic support sensor data display panel. The sensor data is displayed in real time through the "Text" component and updated on the UI data display panel in the AR scene.

[0043] Install the "SteamVR plug-in" in Unity and configure "SteamVR Input" to ensure that Unity can recognize the HTC Vive input device. Set the handle buttons and joysticks to map to the hydraulic support's electro-hydraulic controller. Write a C# script to convert the HTC Vive handle input data into hydraulic support control signals and send them to the actual electro-hydraulic control system.

[0044] The hydraulic support's position changes are fed back into the virtual twin model. Using the TCP communication protocol, physical sensor data is transmitted to the AR scene, and the virtual support's status is updated in real time. After receiving the sensor data, the AR end updates the virtual twin's position, angle, pressure, and other information to ensure synchronization between the virtual and real supports.

[0045] Based on the SQL Server knowledge base data hierarchy architecture, multimodal data collection and storage are realized. The system simultaneously receives data from multiple input sources, including XR interaction data such as gesture position, rotation angle, acceleration; voice frequency and volume; user control of the bracket position through VR handles or helmets, action records, interaction events, etc.

[0046] The collected sensor data is processed and denoised, and the statistical method "Z-Score" is used to detect and handle data outliers. The sensor data with different sampling frequencies are aligned to the same time scale using the interpolation method. Features are extracted based on XR interaction data, sensor data, and downhole environmental data, and the features are standardized using "Z-Score". "Min-Max" normalization is performed, a feature matrix is ​​constructed, and a training set is generated.

[0047] Define the hydraulic support posture data as the target variable of the XG Boost decision model, and use the "train_test_split" function based on Python code to divide the dataset into a training set and a test set, with a division ratio of 80% for the training set and 20% for the test set; select the XG Boost regression model; import the divided training set "(X_train, Y_train)" into the XGBoost decision model for training, and optimize the parameters of the XG Boost decision model; perform cross-validation and decision tree hyperparameter search based on "GridSearchCV", and set the optimal hyperparameter value based on "K-fold cross validation"; perform model training, and then view the relative importance of each feature based on the "plot_importance" function.

[0048] Create a decision rule knowledge base based on environmental condition rules, sensor data anomaly rules, and bracket action conflict rules. Create a rule file based on the "Drools rule engine" and save it in the "**DRL" format. Load the Drools rule file into the Drools engine and instantiate it based on the "KieContainer". Input the collected real-time data into the Drools rule engine based on the "Java" code and match it.

[0049] Build a knowledge- and data-driven collaborative decision-making framework in Java, and establish a conflict judgment and resolution mechanism. Optimize the decision weights of Drools decision rules based on feature importance analysis of the XG Boost decision model. Adjust the priority and judgment conditions of Drools decision rules based on conflict analysis results. Adjust hyperparameters or feature selection in the XGBoost decision model based on data fed back by the rule engine. Adjust the conditions in the Drools decision rules based on feature importance analysis of the XGBoost decision model.

[0050] In this embodiment, based on the "Apache Flink" real-time data stream processing engine, input data from various sensors, AR / VR devices, etc. is processed in real time; the hybrid decision model is integrated: the "Python Web" framework is used to encapsulate the XGBoost decision model into an API interface, and the Drools decision rule engine can run independently as a Java server; based on "Kafka", data is transmitted to the hybrid decision model in real time, and the data is transmitted to the decision engine through the stream processing framework to execute the decision process such as Figure 4 As shown in the figure, based on the fusion decision results, the control signal can be sent to the electro-hydraulic controller through the TCP communication protocol to drive the hydraulic support to perform the action. The position information of the hydraulic support will be fed back to the system in real time by the sensor as the input for the next decision. Based on the decision results, the decision weights of the XGBoost decision model and the Drools rule engine are dynamically adjusted to optimize the decision confidence.

[0051] In this embodiment, the decision suggestion is displayed in the form of a UI panel based on the AR scene through HoloLens 2 and manually confirmed. The operation process is as follows: Figure 5 All manual operation records sent by terminal devices are written into the SQL Server database and synchronized with the training set and rule engine.

[0052] In this embodiment, Figure 6 The figure shows a schematic diagram of the human-machine mutual feedback optimization module. The "low confidence + manual intervention" operation is recorded as a "conflict sample database". In the AR / VR operation interface, the operator's rejection of the control suggestion automatically generated by the system is defined as a "manual correction operation event". The actual posture parameters after the manual operation are stored in the "conflict sample database" in the "OP-Code" format.

[0053] In this embodiment, an operation closed-loop optimization mechanism based on the XR scenario is constructed: after a manual correction operation occurs in the AR / VR scenario, new decision rules are discovered based on the manual operation data, and they are automatically recorded and stored in the knowledge base to achieve incremental learning - the XGBoost decision model loads new operation data samples of the XR scenario and performs learning updates, thereby optimizing feature weights; candidate rules are generated through the Apriori algorithm and updated to the Drools decision rule engine; the digital twin replays historical operation sequences to verify the effectiveness of rules and model iterations, thereby forming a decision closed loop.

[0054] The XGBoost decision model learns and updates features based on the conflict sample database. The Apriori algorithm is used to analyze the relationship between the OP-Code and the working condition label in the conflict sample library, generate candidate decision rules, and manually review them. The candidate rules are updated to the Drools decision rule engine, and the recommended actions are executed based on the XR scenario and compared with the actual execution actions after manual correction for playback verification.

[0055] In this embodiment, Figure 7 The figure shows a schematic diagram of the AR / VR collaborative control interface structure. Based on the "Edge Highlight + Shadow Contour" algorithm, the edge contour of the bracket model is dynamically enhanced in AR mode. A dual bracket adjustment mechanism using voice and gestures is set up through a C# script. Two-way communication is established between the AR / VR end and the electro-hydraulic control PLC system through the "TCP dual-channel protocol". The virtual-real deviation is determined based on the virtual hydraulic bracket state and sensor posture data. The code for dynamic permission allocation is written in Python.

[0056] In this embodiment, an adaptive multimodal interaction mechanism is developed underground: the AR side supports enhancing the outline of the virtual hydraulic support and the ambient brightness under complex working conditions and supports voice control; a voice-gesture dual control mode is set to prevent accidental touch operations under complex working conditions; AR and VR synchronize the virtual and real states of the hydraulic support through the TCP protocol to support collaborative operations.

[0057] like Figure 8 The figure shows a flow chart of a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control method, which includes the following steps: Step 1: Real-time collection and interconnection of multimodal data; Step 1.1: Acquire first operating data of a virtual hydraulic support controlled by a control instruction in real time through a virtual scene on an AR end, wherein the first operating data includes posture parameters and interaction events of the virtual hydraulic support; Step 1.2: Simultaneously obtain sensor data of the actual hydraulic support during operation mapped to the AR end, wherein the sensor data includes pressure, inclination angle, and displacement parameters; Step 1.3: Acquire second operation data when the VR terminal controls the real hydraulic support, wherein the second operation data includes a handle input signal and physical support movement feedback; Step 1.4: Interconnect the data obtained from the AR and VR ends in real time to obtain a multimodal dataset of the hydraulic support in virtual and real scenes.

[0058] Optionally, step 1.4 includes: The TCP dual-channel protocol is used to achieve real-time mapping of the pose data of the virtual hydraulic support on the AR side and the real hydraulic support on the VR side. At the same time, the virtual-reality consistency score is calculated based on the three-dimensional coordinate deviation and dynamically updated to the multimodal dataset.

[0059] In this embodiment, a data interconnection system across XR devices is constructed based on the Unity3D engine and its MRTK module. On the AR side, the operator wears the terminal device HoloLens glasses and then controls the virtual hydraulic support in the Unity virtual scene through gestures, voice and other control commands, and simultaneously receives the real hydraulic support sensor data and maps it to the AR data interface in real time; on the VR side, another operator remotely controls the real hydraulic support through the HTC Vive handle, and its posture data is transmitted to the electro-hydraulic control system via the TCP communication protocol, while realizing the feedback of the real hydraulic support to drive the virtual hydraulic support model to move. AR / VR dual-end data interconnection realizes the synchronization and two-way control of the virtual and real scene states, providing multi-modal data input for the hybrid decision-making model.

[0060] Step 2: Feature matrix construction and hybrid decision generation; Step 2.1: Denoise, standardize, and extract features from the multimodal dataset, construct a multimodal feature matrix, and generate a training set.

[0061] Optionally, feature extraction includes: extracting motion acceleration and spatial coordinate sequence from the AR-end gesture trajectory in the multimodal dataset, parsing frequency and semantic labels from the voice instructions in the multimodal dataset, and aligning them with the sensor time series data in the multimodal dataset to construct a time series correlation feature matrix.

[0062] Step 2.2: Using the training set to train the XGBoost decision model, outputting the predicted posture data of the hydraulic support, and generating a data-driven first control decision recommendation based on the prediction result; Step 2.3: Parallel call the rules in the preset expert decision knowledge base, match the current working conditions through the Drools decision rule engine, and generate rule-driven second control decision recommendations.

[0063] In this embodiment, a collaborative decision-making mechanism integrates the XGBoost decision model and the Drools decision rule engine. A feature matrix is ​​constructed using multimodal data (AR gestures, voice frequency, VR control records, sensor data time series, and environmental parameters) to generate a training set. The XGBoost decision model is trained to predict the bracket's position data and generate decision recommendations and confidence levels. Expert decision rules are pre-installed in the knowledge base, and the Drools engine matches working conditions in real time to achieve rule-driven decision-making. When AR / VR manual operations conflict with the decision model's recommendations, an arbitration mechanism is triggered, and the Drools decision rule engine's decision weights are reversely optimized based on XGBoost feature importance analysis, achieving a two-way enhancement of data-driven and knowledge-driven decision-making.

[0064] Step 3: Collaborative decision execution and optimization: Step 3.1: Fusing the first control decision suggestion and the second control decision suggestion to obtain a fused decision suggestion, and calculating a joint confidence score in combination with a preset confidence function; Step 3.2: When the confidence reaches a preset threshold, the control instruction is automatically executed; otherwise, the fusion decision recommendation is pushed to the terminal device; Step 3.3: Drive the XGBoost decision model to perform incremental learning and update the Drools decision rule engine.

[0065] In this embodiment, the preset credibility function is: Where, is the weight coefficient of the XGBoost decision model; is the weight coefficient of the Drools decision rule engine; The weight coefficient for the consistency score between the virtual hydraulic support on the AR side and the real hydraulic support on the VR side; is the predicted probability of the XGBoost decision model; The decision rule matching degree of the Drools decision rule engine, that is, the ratio of the number of triggered rules to the total number of rules; Score the consistency between the virtual hydraulic support in the AR scene and the real hydraulic support in the VR scene.

[0066] Calculate according to the preset trust function. When using AR / VR, the real hydraulic support is directly driven by the operation. When the user is prompted to make a decision, the system freezes manual control of the terminal device and activates the XGBoost decision model and Drools decision rule engine to make the decision. Simultaneously, the system pushes the fused decision suggestions on the AR interface, which are then confirmed and executed by the terminal operator. By setting confidence thresholds, hierarchical control is achieved, balancing underground operation efficiency and safety.

[0067] In this embodiment, an operation closed-loop optimization mechanism based on the XR scenario is constructed: after a manual correction operation occurs in the AR / VR scenario, new decision rules are discovered based on the manual operation data of the terminal device, and they are automatically recorded and stored in the knowledge base to achieve incremental learning - the XGBoost decision model loads the new operation data samples of the XR scenario and performs learning updates, thereby optimizing the feature weights; candidate rules are generated through the Apriori algorithm and updated to the Drools decision rule engine; the digital virtual hydraulic support model replays the historical operation sequence to verify the effectiveness of the rules and model iterations, thereby forming a decision closed loop.

[0068] The foregoing description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control method, characterized in that: The steps include: Step 1: Real-time collection and interconnection of multimodal data; Step 1.1: Acquire first operating data of a virtual hydraulic support controlled by a control instruction in real time through a virtual scene on an AR end, wherein the first operating data includes posture parameters and interaction events of the virtual hydraulic support; Step 1.2: Simultaneously obtain sensor data of the actual hydraulic support during operation mapped to the AR end, wherein the sensor data includes pressure, inclination angle, and displacement parameters; Step 1.3: Acquire second operation data when the VR terminal controls the real hydraulic support, where the second operation data includes a handle input signal and physical support movement feedback; Step 1.4: Interconnect the data obtained from the AR and VR terminals in real time to obtain a multimodal dataset of the hydraulic support in virtual and real scenes; Step 2: Feature matrix construction and hybrid decision generation; Step 2.1: Denoising, standardizing, and feature extracting the multimodal dataset, constructing a multimodal feature matrix, and generating a training set; Step 2.2: Using the training set to train the XGBoost decision model, outputting the predicted posture data of the hydraulic support, and generating a data-driven first control decision recommendation based on the prediction result; Step 2.3: Parallel call the rules in the preset expert decision knowledge base, match the current working conditions through the Drools decision rule engine, and generate rule-driven second control decision recommendations; Step 3: Collaborative decision execution and optimization: Step 3.1: Fusing the first control decision suggestion and the second control decision suggestion to obtain a fused decision suggestion, and calculating a joint confidence score in combination with a preset confidence function; Step 3.2: When the confidence reaches a preset threshold, the control instruction is automatically executed; otherwise, the fusion decision recommendation is pushed to the terminal device; Step 3.3: Drive the XGBoost decision model to perform incremental learning and update the Drools decision rule engine.

2. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control method according to claim 1 is characterized in that: Step 1.4 includes: The TCP dual-channel protocol is used to achieve real-time mapping of the pose data of the virtual hydraulic support on the AR side and the real hydraulic support on the VR side. At the same time, the virtual-reality consistency score is calculated based on the three-dimensional coordinate deviation and dynamically updated to the multimodal dataset.

3. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control method according to claim 1 is characterized in that: Feature extraction in step 2.1 includes: Extract motion acceleration and spatial coordinate sequences from the AR-side gesture trajectory in the multimodal dataset, parse frequency and semantic labels from the voice commands in the multimodal dataset, and align them with the sensor time series data in the multimodal dataset to construct a time series correlation feature matrix.

4. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control method according to claim 1 is characterized in that: The default belief function in step 3.1 is: Where, is the weight coefficient of the XGBoost decision model; is the weight coefficient of the Drools decision rule engine; The weight coefficient for the consistency score between the virtual hydraulic support on the AR side and the real hydraulic support on the VR side; is the predicted probability of the XGBoost decision model; The decision rule matching degree of the Drools decision rule engine, that is, the ratio of the number of triggered rules to the total number of rules; Score the consistency between the virtual hydraulic support in the AR scene and the real hydraulic support in the VR scene.

5. A hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control system, capable of executing a hydraulic support XR virtual-reality interaction and hybrid intelligent collaborative control method according to any one of claims 1 to 4, characterized in that: include: Multimodal data acquisition module, hybrid decision model, dynamic confidence assessment module, AR / VR collaborative control interface, and human-machine mutual feedback optimization module; The multimodal data acquisition module is used to collect and interconnect multimodal data in real time and store the collected multimodal data; The hybrid decision model is used to integrate the XGBoost decision model and the Drools decision rule engine to construct a feature matrix and generate a hybrid decision; The dynamic confidence evaluation module is used to calculate the joint confidence according to a preset confidence function and dynamically allocate control instructions according to the calculation results; The AR / VR collaborative control interface is used to achieve bidirectional control and data synchronization of virtual and real scenes between the AR and VR ends through the TCP dual-channel protocol; The human-machine mutual feedback optimization module is used to update the XGBoost decision model and the Drools decision rule engine in real time.

6. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control system according to claim 5 is characterized in that: The multimodal data acquisition module includes an XR data interaction unit, a sensor data unit and a recording unit; The XR data interaction unit is used to capture AR gesture trajectories and voice commands through the MRTK module of the Unity3D engine, and to achieve real-time data sharing and display with the VR scene of the HTC Vive; The sensor data unit is used to obtain the hydraulic support pressure sensor data, tilt sensor data, and displacement sensor data, and perform data preprocessing, and simultaneously obtain environmental parameter data including at least downhole visibility, dust concentration, temperature, and humidity; The recording unit is used to encode the control action instructions sent by the terminal device into an OP-Code sequence, associate it with the working condition tag and store it in the SQL Server database.

7. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control system according to claim 5 is characterized in that: The hybrid decision model includes: Perform decision conflict arbitration. When the first regulation decision suggestion generated by the XGBoost decision model conflicts with the second regulation decision suggestion generated by the Drools decision rule engine, the operation instruction issued by the terminal device is executed first, and the rule weight is reversely optimized based on feature importance analysis.

8. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control system according to claim 7 is characterized in that: The XGBoost decision model uses dynamic weight optimization to update new feature parameters based on the weights of historically important features; When the Drools decision rule engine is updated, a copy of the old version is retained. If the new rules cause the control error rate to increase, it will be rolled back to the previous stable version; The Drools decision rule engine and the XGBoost decision model implement bidirectional parameter optimization.

9. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control system according to claim 5 is characterized in that: The AR / VR collaborative control interface includes: an anti-accidental touch unit, an environment adaptation unit and a multi-person collaborative protocol module; The anti-accidental touch unit is used to control the movement of the virtual hydraulic support by setting a voice-gesture dual control mode; The environmental adaptation unit is used to dynamically adjust the transparency of the AR end display interface and support voice control mode based on the underground environmental parameter data; The multi-person collaborative protocol module is used to allocate control rights based on role authority priority when multiple users operate the same virtual hydraulic support, and low-authority users can only view but not operate.

10. The hydraulic support XR virtual-reality interaction and hybrid intelligent coordinated control system according to claim 5 is characterized in that: The human-machine mutual feedback optimization module includes a terminal device data recording module, a feedback learning unit and a rule generation unit; The terminal device data recording module is used to record multimodal data when operating according to the control instructions sent by the terminal device and store it in the conflict sample library in the SQL Server; The feedback learning unit is used to train the XGBoost decision model based on the newly added data samples and retain the historical version; The rule generation unit is used to analyze the control instructions sent by the terminal device through the Apriori algorithm and generate candidate rules, and synchronously update them to the Drools decision rule engine.

Images