Hydraulic support group cloud measurement method based on AR calibration matching

Abstract

The invention discloses a hydraulic support group cloud measurement method based on AR calibration matching. A physical system, a remote cloud data processing system and a VR system are included. In the physical system, AR glasses provide an interactive interface for virtual-real interaction and data display, and an inspector performs manual matching and dragging adjustment through an MRTK toolkit to complete detection. In the remote cloud data processing system, the cloud service computing system carries out data storage and preliminary analysis, the centralized control worker and the GPT decision making system send a mixed decision making result to the AR glasses through secondary data processing to make a detection data supplement request, and finally the field inspection worker completes measurement. In the VR system, a twin hydraulic support group model is constructed and corrected, an overall working space is constructed, and pose reconstruction precision verification is carried out, so that virtual pose real-time measurement of the hydraulic support group is realized.

Description

Technical Field

[0001] The invention relates to a method for measuring the position and posture of a hydraulic support group in a fully mechanized mining working face, and in particular to a method for measuring the cloud of a hydraulic support group based on AR calibration and matching. Background Art

[0002] Intelligent coal mines refer to the use of advanced information technology and automation technology to digitize and automate the coal mine production and management process, and realize intelligent management and optimization of the production process. By applying intelligent technology, coal mines can improve safety, increase production efficiency, reduce costs, and achieve efficient use of resources.

[0003] As an important supporting equipment for the fully mechanized mining face, the hydraulic support group plays a decisive role in whether the working face can work normally and continuously. Therefore, the real-time support posture detection of the hydraulic support has become the key to the intelligentization of the fully mechanized mining face. The complexity of the support group posture measurement is reflected in its linkage characteristics of "internal posture" (movable joints composed of four-link, top beam and column) and "external posture" (the positioning base point of the entire support is the support base).

[0004] In the hydraulic support group, the single-blade AR dynamic measurement method refers to a method that uses augmented reality technology and dynamic measurement methods to measure and analyze the relative position and posture of each support in the hydraulic support group. This method combines the virtual superposition capability of AR technology and the real-time data collection of dynamic measurement methods, and can accurately and quickly obtain the relative position and posture information between each support in the hydraulic support group.

[0005] The invention patent with the publication number CN111442759A discloses a unified monitoring system for the posture of equipment in a fully mechanized mining face, which relates to the field of coal mining face monitoring and automatic control technology, and can monitor the posture information of equipment in a fully mechanized mining face in a unified and integrated manner. It includes: a hydraulic support posture monitoring subsystem, which monitors the posture information of the hydraulic support and the relative position between adjacent hydraulic supports; a coal mining machine posture monitoring subsystem, which monitors the posture information of the coal mining machine and the relative position between the coal mining machine, the hydraulic support, and the scraper conveyor; a scraper conveyor posture monitoring subsystem, which is used to monitor the posture information of a single chute of the scraper conveyor and the relative position between adjacent chutes and between the chute and the hydraulic support; a three-machine overall posture monitoring subsystem, which is used to monitor the relative spatial position of the hydraulic support, coal mining machine, and scraper conveyor with the coal mining face during the coal mining process; and a posture solving module, which is used to convert the above data into a unified coordinate system and solve the spatial posture of the three machines.

[0006] The invention patent with the publication number CN115371560A discloses a method for perceiving and describing the state of a hydraulic support base group on a working surface, which includes S1. Taking the end hydraulic support as the overall reference, the subsequent hydraulic support is used as the target support relative to the preceding adjacent hydraulic support; S2. The hydraulic support base is provided with three monitoring points D1, D2 and D3 on one side of the camera facing the preceding adjacent hydraulic support, D1, D2 and D3 are not collinear and cannot form an isosceles triangle, and the line connecting D1D2 is parallel to the edge line of the base; S3. The camera shoots the monitoring points of the subsequent adjacent hydraulic support and transmits the captured image to the processor; S4. The processor parses the coordinates of each monitoring point to obtain the spatial state of the target hydraulic support base. The present invention obtains the position of each support base in turn by parsing the coordinates of the monitoring points of the adjacent hydraulic support bases, and can visually observe the uniformity in all directions. The model and mechanical structure are simple, the description result is highly reliable, and it is not limited by the number and space of hydraulic support installations.

[0007] The invention patent with the publication number CN116188570A discloses a method for detecting the position and posture of a hydraulic support based on binocular vision, which includes establishing a camera, a hydraulic support, a top beam, and a marker point coordinate system, and determining the relative posture relationship between the coordinate systems, performing target detection on the marker point based on a convolutional neural network, and performing binarization and morphological processing, calculating the position coordinates and three-dimensional coordinates of the center point of the marker point, and solving the posture matrix of a single hydraulic support coordinate system in the camera coordinate system and the posture matrix between different hydraulic support coordinate systems. The present invention can simultaneously realize posture detection inside a hydraulic support and between different hydraulic supports, accurately measure the posture of the top beam in space in a non-contact manner, and accurately obtain the relative posture between different hydraulic supports in the same binocular camera field of view, which is beneficial to the straightening and deviation correction of a hydraulic support group.

[0008] The invention patent with publication number CN117090617A discloses a method for self-adaptive adjustment of hydraulic support posture based on digital twin model. The method uses digital twin technology to establish a three-dimensional model of hydraulic support, and adopts a hydraulic support posture prediction model fused with an improved LSTM-GRU two-layer neural network and a genetic algorithm to realize self-adaptive adjustment of hydraulic support posture. The method includes the following steps: (1) establishing a three-dimensional model of hydraulic support; (2) obtaining measurement information of relative posture of hydraulic support; (3) training network model to obtain the optimal solution of prediction model; (4) constructing display information in UGUI component in Unity3D; (5) building a decision control platform, using decision tree algorithm and training set to build classification model, and using test set to verify the decision control effect of classification model. The method adopts digital twin technology and deep learning neural network algorithm based on attention mechanism to realize self-adaptive adjustment of hydraulic support posture.

[0009] The above patents respectively measure the posture of the entire comprehensive mining working face through sensor measurement, adjacent bracket posture analysis, binocular camera posture matrix acquisition, and a fusion algorithm posture prediction model based on digital twins. However, the above methods generally have the following defects.

[0010] (1) A large number of sensors need to be installed on the hydraulic support. The sensors are expensive, the installation is complicated and tedious, the maintenance cost is increased, the sensor data processing volume is too large, which increases the complexity of the system, the sensor accuracy is unstable or failures affect the accuracy of data collection, and the sensor network communication transmission is stable. These factors will increase the failure rate of the system and the difficulty of maintenance.

[0011] (2) Over-reliance on analytical supports to obtain posture will result in decreased accuracy, data transmission problems, posture drift, error accumulation, sensor interference, increased system complexity and cost, which will affect the accuracy of posture analysis results and the stability of the system, and lack comprehensive consideration of posture measurement of the fully mechanized mining working face.

[0012] (3) When using a depth camera to obtain the hydraulic support posture matrix, there are challenges such as matching difficulties, occlusion effects, accuracy limitations, and high environmental requirements. The hydraulic support surface with few textures or uneven lighting may cause matching difficulties, occlusion problems will affect the acquisition of posture information, and the camera's measurement accuracy and environmental requirements will also affect the accuracy of the results.

[0013] (4) The construction of models and algorithm prediction methods to measure the hydraulic support posture has problems such as control response delay and lack of data information feedback. The failure to update the hydraulic support posture measurement in real time may lead to outdated data information, difficulty in real-time monitoring, and limited task planning, which will affect the system's operational efficiency, safety, real-time and reliability.

[0014] (5) Only the posture measurement of the fully-mechanized mining face is considered, and the convenience of data reception by the inspection personnel is ignored, which makes the data transmission and communication method too complicated. The inspection personnel cannot obtain the posture measurement data in time, which affects their real-time monitoring and decision-making capabilities of the fully-mechanized mining face status, increases potential risks, and affects the practicality and operation efficiency of the posture measurement system of the fully-mechanized mining face. Summary of the invention

[0015] The purpose of the present invention is to provide a hydraulic support group cloud measurement method based on AR calibration and matching. The method uses AR to supplement the insufficient information of the current comprehensive mining working face measurement, can display real-time information, enhance safety and work efficiency, and effectively improve the work experience.

[0016] To achieve the above object, the technical solution provided by the present invention is:

[0017] A cloud measurement method for hydraulic support groups based on AR calibration and matching, comprising a physical system, a remote cloud data processing system and a VR system;

[0018] The physical system includes an inspector, AR glasses and a physical hydraulic support group. The inspector wearing AR glasses needs to walk and measure in the hydraulic support group and stop as needed. The detection process includes:

[0019] (1) Taking the movable joint of a single hydraulic support with internal posture as the detection object, the closed-loop deduction model of the single support posture is solved;

[0020] Establish multiple detail feature bodies inside the hydraulic support, and the detail feature bodies are selected on the movable joints of a single hydraulic support with internal posture; integrate the hydraulic support group and its internal structure with AR glasses in a unified coordinate system to build an AR-working surface-hydraulic support group three-level coordinate system; split the movable joints of a single hydraulic support from the internal perspective of the inspector, and select the three-dimensional feature bodies of the corresponding moving parts respectively; use the virtual simulation function and MRTK module in Unity3D software to complete the virtual-real model matching of the three-dimensional feature body, and display the movable joint posture data in real time on the AR interface;

[0021] (2) Taking the entire support with external posture and the support base as the positioning base as the detection object, the posture model deviation between adjacent hydraulic support frames is detected;

[0022] The posture of the entire bracket group is disassembled into the relative posture of two adjacent brackets as the research object, and the adjacent bracket model of the detection bracket and the reference bracket is established; the entire bracket with external posture and the bracket base as the positioning base point is selected from the external perspective of the inspector, and the bracket base posture information is obtained according to the relevant three-dimensional feature body of the base component to establish a virtual envelope cuboid, and the relative posture matrix of the hydraulic bracket is established according to the three-level coordinate system. The MRTK module in the Unity3D software is used to match the virtual envelope cuboid with the entire bracket; the six-point positioning principle is used to realize the relative posture detection between brackets, and the posture deviation relative to the reference bracket is displayed in real time on the AR interface;

[0023] (3) Component identification and inter-rack bidirectional matching verification;

[0024] The iterative verification of the internal posture widget group is completed by model training and dividing the training data, verification data and test data; the iterative verification of each large joint of the hydraulic support is completed by seamless linkage analysis of a single support; the iterative verification of different support groups of the hydraulic support group is completed by bidirectional matching verification between supports; the above verification methods are synchronized and verified;

[0025] The remote cloud data processing system includes a cloud service computing system, a GPT decision system and a centralized control worker. The cloud service computing system performs data storage and preliminary analysis. The centralized control worker and the GPT decision system send the mixed decision results to the AR glasses after secondary data processing to request additional test data. Finally, the on-site inspection worker completes the measurement.

[0026] The VR system constructs and corrects the twin hydraulic support group model through the real-time support posture data of the remote cloud data processing system, constructs the overall workspace and performs numbering, sorting and calibrating, and finally verifies the posture reconstruction accuracy; predicts the next-cut posture of the physical support group through the virtual support group, and supplements and corrects the data of the virtual support group through the physical support group, so as to achieve synchronization of the posture of the virtual and real hydraulic support groups.

[0027] As a preferred embodiment, the detailed feature body is selected from a base, a rear connecting rod, a front connecting rod, a column, a shield beam, a top beam, and a front beam having an internal posture.

[0028] As a preferred implementation, the virtual enveloping cuboid is obtained by modeling the various sub-components of the hydraulic support and embedding them into the enveloping cuboid.

[0029] As a preferred embodiment, the three-level coordinate system includes a coordinate system {O A}'s AR coordinate system, the coordinate system is {O H The unified coordinate system of the hydraulic support group of} is The local coordinate system of the i-th hydraulic support, wherein the unified coordinate system of the hydraulic support group is used as a sub-coordinate system of the AR coordinate system, and the local coordinate system of the hydraulic support is used as a sub-coordinate system of the unified coordinate system of the hydraulic support group.

[0030] As a preferred embodiment, the neighboring frame model is constructed in In the coordinate system, the i-th hydraulic support is selected as the reference support, the i+1-th hydraulic support is selected as the detection support, and the support base center O is defined. c is the coordinate origin of the reference bracket, and the center of the bracket base is O d To detect the coordinate origin of the bracket, the advancing direction of the working face is the positive direction of the x-axis, the direction from the coal seam bottom plate to the coal seam roof is the positive direction of the z-axis, and the y-axis direction is determined by the right-hand rule; the entire detection bracket is regarded as a model in which the base is fixed with the front beam, top beam, shield beam, front and rear connecting rods and other movable parts following the telescopic movement of the column.

[0031] As a preferred implementation method, the virtual-real feature matching and local feature calibration of the three-dimensional model are that the inspector manually matches the virtual three-dimensional feature body model with the real hydraulic support, imports and configures MRTK in Unity, deploys the Unity project to the HoloLens device, creates a scene in Unity, and imports the feature body model to be matched in the hydraulic support model database, and sets the position and direction of the model according to the actual situation; imports OpenXR and MR related configurations, creates multiple C# scripts to implement the functions of anchor initialization, adding anchors, deleting anchors, saving anchors and loading anchors in turn, and finally completes the design of the anchor shape through scene settings; during the operation of the inspector, ensure that the dynamic anchor is aligned with the real hydraulic support component, and align the virtual model with the anchor through visual guidance and interactive operations.

[0032] As a preferred implementation, for the identification and solution of small parts in the movable large joints, the inspection data is first preprocessed, and after the manual matching is completed, duplicate and invalid data records are removed and it is distinguished which data is the trainable data of the small part feature block and which is other redundant information; then the features of each key component are extracted and the features are reduced in dimension; by constructing a Random Forest model suitable for feature parsing and data modeling, the training data is used for model training, so that these models can predict the status or parameters of other related small parts according to the small part feature block data, and the redundant data that are not involved in the model training are processed, and the relevant information that can be used for verification is extracted as verification data; the data obtained by the parsing model is compared and verified with the relevant information in the verification data to check the consistency. If inconsistency or abnormality is found in the comparison, it is determined whether it is a detection error or there is actually a problem with the component; according to the analysis results, the detection process is adjusted to complete the iterative optimization verification; after the model training and parameter optimization are completed, the model is applied to the test data, and the performance is evaluated again, and the prediction results and feature importance analysis are recorded; the parsing results of the test data are compared with the results of the verification data part, and the consistency of the feature importance and the model prediction results is checked to show the consistency of the model on different data sets.

[0033] As a preferred implementation, in order to detect redundancy of large movable joints with internal postures, the known large movable joint monitoring posture data obtained by virtual-real matching of a single bracket is combined with the structural parameter Li of different models of brackets to obtain the unknown posture data of the bracket and verify the known posture data of the bracket. The inspection personnel perform feedback verification through the verification posture data, and realize accurate matching of virtual and real models through multi-faceted information fusion.

[0034] As a preferred implementation, in view of the possible errors in the neighboring frame models in the hydraulic support group, the AR measurement method is used to measure the support height difference, column telescopic length difference, pitch angle difference, roll angle difference, and yaw angle difference of the head and tail supports of a specific support group to verify the actual height and telescopic length of each support determined in the detection support coordinate system in different neighboring frame models of this group, as well as the actual attitude angle of each support; the nonlinear least squares method is used to obtain the minimum error between the measured data and the theoretical data, and after removing the invalid data, the Levenberg-Marquardt optimization algorithm is used to gradually feed back these deviations to each support and perform virtual-real model correction and adjustment. After multiple iterative verifications, the error is kept within a certain range to achieve two-way matching verification between frames.

[0035] As a preferred implementation, the remote cloud data processing system stores the cloud model database, establishes a hydraulic support model database by digitally managing the hydraulic support model and related data, selects SQL Server as the hydraulic support database management system, and selects mobile devices or hardware with explosion-proof functions as the on-site work platform for inspection personnel.

[0036] The present invention provides a hydraulic support group posture cloud measurement method of "local feature calibration-component identification solution-inter-frame bidirectional matching verification" for comprehensive mining AR inspectors. A stable and efficient UWP platform customer service environment is built through the virtual reality software Unity3D, and the fusion experience of physical models and digital information is realized through HoloLens, especially the manual overlap of virtual and real information realized by the MRTK toolkit, and the inter-frame closed-loop model of the internal and external displacement fusion of the support is solved, finally realizing the real-time measurement of the virtual posture of the hydraulic support group.

[0037] Compared with the prior art, the technical effects of the hydraulic support group cloud measurement method based on AR calibration matching described in the present invention include:

[0038] (1) The use of AR measurement technology can accurately obtain the position and posture information of the hydraulic support without installing a large number of sensors, avoiding the problems of high sensor costs, complex installation, and increased maintenance costs. At the same time, AR measurement technology can increase certain on-site computing capabilities and improve the adaptive iterative optimization capabilities of the physical system.

[0039] (2) The use of AR measurement technology can effectively solve the problems of posture drift and error accumulation faced by over-reliance on traditional analytical brackets to obtain posture, and effectively simplify the system structure and maintenance costs. AR measurement technology comprehensively considers the actual problems of posture measurement of the fully mechanized mining face, combines seamless linkage analytical methods to verify the movable large joint model, and provides a more intelligent and reliable posture analysis solution.

[0040] (3) The use of AR measurement technology can effectively solve the problems of matching difficulties, occlusion effects, and accuracy limitations encountered when using depth cameras to obtain the hydraulic support posture matrix. AR measurement technology can achieve high-precision matching by combining sensor data and computer vision algorithms. At the same time, it is less constrained by the environment and does not require additional special equipment. It improves the flexibility of inspection work through wireless communication.

[0041] (4) AR measurement technology can effectively solve the problems of response delay and lack of feedback in measuring the support posture by building models and algorithm prediction methods. Through efficient data processing of remote cloud data processing systems, real-time updates of hydraulic support posture measurement data information can be achieved. 5G data transmission and VR system trajectory prediction can significantly improve the system's real-time monitoring capabilities and task planning flexibility.

[0042] (5) AR measurement technology can effectively improve the convenience of data reception for patrol personnel. Through the augmented reality function of AR glasses, patrol personnel can obtain and feedback posture measurement data online. AR glasses and VR systems have significantly improved the real-time monitoring and decision-making capabilities of the status of the comprehensive mining face. The interactivity of AR technology has effectively reduced the potential risks of patrol operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall framework diagram of the present invention;

[0044] Figure 2 It is the coordinate system definition diagram of the present invention;

[0045] Figure 3 is an illustration of the physical system detection method of the present invention;

[0046] Figure 4 It is a three-system communication principle diagram of the present invention. DETAILED DESCRIPTION

[0047] Based on the problems of inconvenient installation and precision limitation of hydraulic support sensors or depth cameras in the prior art, the present invention realizes innovation by introducing the concept of augmented reality on the comprehensive mining working face, and realizes comprehensive consideration of the comprehensive mining working face posture measurement by combining the process of posture analysis of a single support, adjacent supports, support groups and comprehensive mining working face. From the perspective of patrol personnel, the present invention considers the environmental and safety problems caused by high temperature and humidity, perception difficulties caused by dark light and small space, communication difficulties with the ground command center, and difficulties in collecting and processing patrol data that patrol personnel may encounter at work. A set of feasible posture detection methods including real-time information display, enhanced safety and work efficiency, and effective improvement of work experience are proposed, and AR is used to supplement the insufficient information of the current comprehensive mining working face measurement.

[0048] like Figure 1As shown in the figure, the present invention designs a hydraulic support group posture cloud measurement method of "local feature calibration-component identification solution-bidirectional matching verification between frames" for comprehensive mining AR inspectors. This method generates three major systems, including a physical system, a remote cloud data processing system, and a VR system.

[0049] The physical system is the core system of the measurement method, including inspectors, AR glasses, physical hydraulic support groups and other elements. AR glasses provide information enhancement and operation guidance for inspectors. When the measurement information is insufficient, data verification is required for the data request sent by the remote cloud data processing system. At the same time, inspectors can also make independent decisions and provide data feedback in AR glasses. Inspectors wearing AR glasses need to measure while walking in the hydraulic support group and stop as they mine. AR glasses provide an interactive interface for virtual-real interaction and data display, and inspectors use the MRTK toolkit to manually match and drag and adjust to complete the inspection.

[0050] The remote cloud data processing system includes three elements: cloud service computing system, GPT decision system and centralized control worker. The cloud service computing system performs data storage and preliminary analysis, and the centralized control worker and GPT decision system send the mixed decision results to AR glasses after secondary data processing to request additional test data, and finally the on-site inspectors complete the measurement.

[0051] The VR system constructs and corrects the twin hydraulic support group model through the real-time support posture data of the remote cloud data processing system, then constructs the overall workspace and performs numbering, sorting and calibration, and finally verifies the accuracy of posture reconstruction. The virtual support group predicts the next cut posture of the physical support group, and the physical support group supplements and corrects the data of the virtual support group, so as to achieve synchronization of the posture of the virtual and real hydraulic support groups.

[0052] As attached Figure 2 As shown, the present invention establishes a three-level coordinate system and constructs a neighboring frame model. The integrated hydraulic support group and the internal structure are unified with the AR glasses in one coordinate system, and the AR glasses virtual monitoring data are linked with the real data of the physical support group, and finally the AR-hydraulic support group-hydraulic support three-level coordinate system and the neighboring frame model coordinate system are established.

[0053] The detailed introduction of each level of coordinate system is shown in Table 1, including the AR coordinate system with coordinate system {OA}, the unified coordinate system of the hydraulic support group with coordinate system {OH}, and the local coordinate system of the first hydraulic support with coordinate system {O}. Among them, the unified coordinate system of the hydraulic support group, as a sub-coordinate system of the AR coordinate system, can express the dynamic posture information of the hydraulic support group under the real-time detection of AR glasses; the local coordinate system of the hydraulic support, as a sub-coordinate system of the unified coordinate system of the hydraulic support group, can accurately express the position, direction and posture of each component inside the hydraulic support group relative to the entire support group. This three-level coordinate system can realize the adjustment of different coordinate axes in the virtual scene or operate the position, rotation angle, scaling ratio, etc. of the working surface and the hydraulic support group to meet the relevant needs in different simulation scenes.

[0054] Table 1 Introduction to the three-level coordinate system

[0055]

[0056] The construction of the adjacent frame model is to advance the hydraulic support group in the coal seam as an organic whole. All hydraulic supports need to move forward with the working face. Therefore, the unified coordinate system of the hydraulic support group is constructed. H In order to more accurately and effectively analyze and optimize the structure and performance of the hydraulic support group, the posture of the entire support group is decomposed into the relative posture of two adjacent supports as the research object, that is, In the coordinate system, the i-th hydraulic support is selected as the reference support, and the i+1-th hydraulic support is selected as the test support. Define the support base center O c is the coordinate origin of the reference bracket, and the center of the bracket base is O d The coordinate origin of the detection bracket is the positive direction of the x-axis, the direction from the coal seam floor to the coal seam roof is the positive direction of the z-axis, and the y-axis direction is determined by the right-hand rule. The entire detection bracket is regarded as a model in which the front beam, top beam, shield beam, front and rear connecting rods and other movable parts of the base are fixed and follow the telescopic movement of the column. The movement relationship of this model is simple. A virtual rectangular envelope model can be set on the detection bracket to match the virtual and real models and perform real-time parameter detection, so as to construct a neighboring frame model.

[0057] For the adjacent frame model obtained by disassembly, the relative position relationship of the two coordinate systems is determined based on different support working conditions such as squeezed frame, bitten frame, crooked frame and overturned frame formed by adjacent frames. The parameters measured in the detection support coordinate system are the length of the column extension and the height H of the hydraulic support, and the parameters measured in the reference support coordinate system are the pitch angle, roll angle and yaw angle.

[0058] In physical systems, constructs such as Figure 3The physical system measurement method shown in the figure includes two detection paths: the closed-loop deduction model solution of a single support attitude and the detection of the deviation of the posture model between adjacent hydraulic supports. It includes two detection objects: a movable joint composed of a base, a four-link, a top beam and a column with an internal posture, and a whole support with an external posture and a support base as the positioning base point, and ensures that the detection path corresponds to the detection object one by one. According to the detection process, the physical system includes the establishment of a three-level coordinate system and the construction of an adjacent frame model, the establishment of a detailed feature body and a virtual envelope cuboid from an internal perspective, the matching of virtual and real features of a three-dimensional model and the calibration of local features, the display of local posture data and the posture deviation of a specific support on a visual interface, and the identification and solution of components and the verification of bidirectional matching between frames. The closed-loop deduction model solution of a single support attitude and the posture model and posture deviation detection between adjacent hydraulic supports of the physical system are combined to form a closed-loop solution model between frames with internal and external displacement fusion of the support. The physical system, the remote cloud data processing system and the VR system realize the interactive mapping of virtual and real scenes, the coexistence of virtual and real space twins, and the closed-loop conduction of global data in the overall operation.

[0059] Among them, the closed-loop deduction model of the posture of a single support is solved by first establishing several detailed feature bodies inside the hydraulic support. These feature bodies are selected based on the movable joints composed of the base, four-link, top beam and column with internal posture. The feature body is required to accurately reflect the real shape and relative position relationship of the moving parts to ensure that the described information is completely accurate. At the same time, the local structure and internal details of the movable joint should be displayed to achieve the structural coupling between the internal posture and the main part of the moving parts, and the motion trajectory and interaction between the parts should be dynamically displayed. When selecting, the movable joints of the single hydraulic support are split from the internal perspective of the inspector, and the three-dimensional feature bodies of the corresponding moving parts are selected respectively. The virtual simulation function and MRTK module in Unity3D software are used to complete the matching of the virtual and real models of the three-dimensional feature bodies, and the movable joint posture data is displayed in real time on the AR interface; the telescopic length and height of the single hydraulic support during the movement process are analyzed by the seamless linkage method of the single support. This process realizes the solution of the closed-loop deduction model process of the posture of the single support.

[0060] Among them, the posture model deviation detection between adjacent hydraulic supports is aimed at various complex relative postures such as squeezed, bitten, crooked and overturned frames caused by the roof and floor of the coal seam. First, the posture of the entire support group should be disassembled into relative postures with two adjacent supports as the research object, and the adjacent frame model of the detection support and the reference support should be established; the entire support with external posture and the support base as the positioning base point should be selected from the external perspective of the inspector, and the support base posture information should be obtained according to the three-dimensional feature body related to the base component, and the corresponding right-handed coordinate system should be established with the base center of the reference support as the coordinate origin; the detection support envelope cuboid is set, and the virtual envelope cuboid is matched with the entire support by using the MRTK module in Unity3D software; the relative posture detection between frames is realized by using the six-point positioning principle, and the posture deviation relative to the reference support is displayed in real time on the AR interface; the posture model and posture deviation detection between adjacent hydraulic supports are realized by the relative posture description and analysis method between frames.

[0061] Among them, the six-point positioning principle is a geometric method for determining the position and posture of an object in three-dimensional space, which is particularly suitable for scenes that require accurate detection of the relative posture of an object. It is based on the spatial position measurement of six different points, and uses the geometric relationship of these points to derive the six degrees of freedom of the object. Specifically, the six-point positioning principle for relative posture detection between hydraulic supports is a three-dimensional virtual space measurement technology based on geometry, which is widely used in the posture detection of hydraulic supports on mining working faces, especially the real-time monitoring of the relative posture between supports. According to the measurement principle of the six-point positioning of hydraulic supports, the relative posture between frames is mainly reflected in the following six degrees of freedom, namely, the three translation parameters describing the displacement of the hydraulic support in space (translation along the x, y, and z axes), and the three rotation parameters describing the rotation state of the hydraulic support (rotation around the x, y, and z axes). For the implementation process, first calibrate the reference support, and fix the six measurement points on the detection support at different positions of the support. The initial positions of these points are known. When the relative position or posture between the supports changes, the relative distance or angle of the six points will change. By measuring these changes, the six-degree-of-freedom motion parameters of the support can be derived. One of the specific methods for constraining the six degrees of freedom of the hydraulic support is to arrange three support points in the xoy plane to constrain the two rotational degrees of freedom around the x and y axes and the translational degree of freedom along the z axis. Two support points are arranged in the xoz plane to constrain the rotational degree of freedom around the z axis and the translational degree of freedom along the y axis. One support point is arranged in the yoz plane to constrain the translational degree of freedom along the x axis. The spatial position changes of the six reference points are detected by the six-point positioning principle, and the relative position and posture of the hydraulic support are detected in real time to ensure the stability and safety of the support.

[0062] For the movable joints composed of the base, four-link, top beam and column with internal posture, the closed-loop deduction model of the single bracket posture is solved through the above-mentioned established three-level coordinate system, the movable joints are split and the "three-dimensional feature body" of the corresponding moving parts is selected. Then, the model database is imported for accurate matching of the virtual and real models of the "three-dimensional feature body". The movable joint posture data can be displayed in real time on the AR interface. AR glasses perform partial calculations of component identification and seamless linkage analysis. Iterative feedback verification of small components is performed through component identification and solution. After obtaining reasonable feedback verification results, feedback verification of movable large joints is performed through seamless linkage analysis. Self-correction makes the detection results more accurate. For the entire bracket with external posture and the bracket base as the positioning base point, a virtual envelope cuboid is established through the bracket base posture information obtained by the three-dimensional feature body. Then, the relative posture matrix of the hydraulic bracket is established according to the three-level coordinate system. The relative reference bracket posture deviation can be displayed in real time on the AR interface through virtual and real model matching. The inter-frame bidirectional matching verification makes the inter-frame posture deviation detection more accurate.

[0063] Establishing the detail feature body and virtual envelope cuboid of the internal perspective is the prerequisite for detection. The movable joints with internal posture are selected in the order of base, rear connecting rod, front connecting rod, column, shield beam, top beam and front beam. When selecting, the inspection personnel are in a specific position and perspective inside the hydraulic support; and for the entire support with external posture and the support base as the positioning base point, the inspection personnel are on the reference support to complete the virtual envelope cuboid matching of the detection support.

[0064] For the detailed feature bodies, it is necessary to construct local models of 7 movable joints. First, continue with the feature body selection. After the selection is completed, use UG to build a three-dimensional feature body model, select the UG model and export it into stl format, export the stl grid into FBX format in 3Dmax, and finally import the FBX model file in Unity 3d. Using feature bodies instead of movable large joints can effectively improve the matching accuracy. For the virtual envelope cuboid, it is necessary to model each subcomponent of the hydraulic support in detail and embed them into the envelope cuboid. After that, the corresponding cuboid model can be established according to the outer dimensions of the hydraulic support. According to the actual data of different supports, the virtual envelope cuboids can be numbered and the size of the envelope can be fine-tuned to ensure accuracy.

[0065] When selecting detail features, five feature bodies are selected for the base, namely, the left front base leg, the right front base leg, the front baffle of the column, the left pin shaft hole of the front connecting rod, and the right pin shaft hole of the front connecting rod. In terms of structure, the five local feature bodies are all part of the base components, fully demonstrating the local mechanical connection and working details of the five places. The left front base leg and the right front base leg are connected to the left and right sides of the base in turn, used to support the hydraulic support and form a middle gap connection and displacement mechanism. The front baffle of the column is connected to the top of the base to prevent objects or people from entering the base, and also to increase the structural strength of the base. The left pin shaft hole of the front connecting rod and the right pin shaft hole of the front connecting rod are connected to the left and right legs of the base in turn, playing the role of connecting and supporting the front seat and the front connecting rod. In terms of motion relationship, the five feature bodies are all feature bodies that move with the base, which fully guarantees that the base is constrained by three translational degrees of freedom and three rotational degrees of freedom in three-dimensional space. The motion trajectory of each feature is consistent with the motion trajectory of the base, but the relative position will change. The feature bodies of the remaining components are selected in a similar way to the base feature body. Finally, UG is used to construct a three-dimensional feature body model and complete the position constraint relationship between the single bracket assemblies.

[0066] The virtual envelope cuboid is constructed based on the neighboring frame model, and its coordinate system positioning base point is located at the center of the detection bracket base. To determine the position and posture of the detection bracket relative to the reference bracket, it is necessary to fully determine the spatial position and posture of each bracket and the adjacent bracket based on the six-point positioning principle.

[0067] When constructing the virtual envelope cuboid, it is necessary to model each subcomponent of the hydraulic support in detail and embed them into the envelope cuboid. Then, the corresponding cuboid model can be established according to the outer dimensions of the hydraulic support. The cuboid should completely cover the working range of the hydraulic support including the extreme position. According to the actual data of different supports, the virtual envelope cuboid is numbered and the size of the envelope is fine-tuned to ensure accuracy. The virtual envelope cuboid is constructed based on the neighboring frame model, and its coordinate system positioning base point is located at the center of the detection bracket base. To determine the position and posture of the detection bracket relative to the reference bracket, it is necessary to use the six-point positioning principle, that is, to completely position it in space, it is necessary to limit its six degrees of freedom in space. The most basic model is to use six points to limit the degrees of freedom of a rigid body in space. By measuring the linear motion displacement parameters of the detection bracket along the X, Y, and Z axes and the rotation angle parameters of the rigid body around the X, Y, and Z axes, the spatial position and posture of each bracket and the adjacent bracket can be completely determined.

[0068] The virtual-real feature matching and local feature calibration of the 3D model is that the inspectors manually match the virtual 3D feature body model with the real hydraulic support. It is necessary to import and configure MRTK in Unity to ensure that the project is compatible with the MixedReality device. After that, the Unity project is required to be deployed to the HoloLens device, that is, the PC and the HoloLens device are connected to the same wifi, select build setting under file in the Unity project, select to install the uwp platform, select am64 to generate the sln file and input the IP address to the HoloLens device to complete the deployment. After that, create a scene in Unity, import the feature body model that needs to be matched in the hydraulic support model database, and set the position and direction of the model according to the actual situation.

[0069] The spatial anchor point can anchor the position of the virtual object at the specified position in the real space. The anchor point can be stored persistently. The virtual object will still appear at the location of the anchor point the next time it is used, which can be used to achieve the effect of multi-person collaboration and displaying virtual objects at specific locations. To configure the spatial anchor point, you need to import OpenXR and MR related configurations under the corresponding project, create multiple C# scripts to implement the functions of anchor point initialization, adding anchor points, deleting anchor points, saving anchor points, and loading anchor points in turn, and finally complete the design of the anchor point shape through scene settings. During the operation of the inspection personnel, ensure that the dynamic anchor point is aligned with the real hydraulic support component, and align the virtual model with the anchor point through visual guidance and interactive operations.

[0070] By using the HandTracking function in the MRTK toolkit, the interactive response between the virtual feature body and the patrol personnel's gestures can be realized. First, select and configure the gesture recognizer, set the type and trigger conditions of the gesture, and define the corresponding event processing function for each gesture to respond to the triggering of the gesture. The NearInteractionGrabbable component is used for grabbable objects, and the NearInteractionTouchable component is used for touchable objects. Then associate the gesture recognizer with the virtual 3D feature body model that needs to interact to ensure that the gesture can be correctly recognized and interact with the feature body. According to the recognition results of the gesture, the interactive operation is realized in the corresponding C# script, and the button click, object movement, rotation and other operations are triggered according to the gesture. Finally, the patrol personnel can realize the direct matching of the virtual and real models by dragging the anchor points to align, and realize the precise matching of the virtual and real models by adjusting the local posture.

[0071] The visualization interface displays the local pose data and the specific support pose deviation by combining Unity with MRTK to create an application. The visualization interface content needs to design and arrange interface elements in the Unity scene. The interactive interface needs to use the UI components in MRTK to create interactive elements, ensuring that the size, position and trigger area of ​​the interactive elements are suitable for gesture operations on HoloLens. The visualization interface of the hydraulic support pose detection data is to create a new project in Unity and import the three-dimensional feature body model in the database, and then create an empty GameObject as an object for displaying the pose detection data. By creating and writing a C# script to process the hydraulic support pose detection data, the script defines a HydraulicPoseVisualization class to realize the visualization of the hydraulic support pose detection data. The script includes a 3D model of a hydraulic support three-dimensional feature body and a Vector3 variable that stores the pose detection data. In the Start method, the initial position of the three-dimensional feature body model is set to the value of the pose detection data. In the Update method, the position of the three-dimensional feature body model is updated, and the pose detection data is updated through gesture interaction to realize the real-time display of the dynamically updated pose data.

[0072] The local pose data needs to be mounted in the TMP project in Unity3D to implement the C# script for the three-level coordinate system conversion. By configuring the TMP project to the AR glasses used for inspection, direct data display from manual measurement to AR glasses can be realized. The purpose of the three-level coordinate system conversion is to directly display the local posture or overall posture measured in the AR first-level coordinate system as the corresponding pose data in the third-level local coordinate system of the i-th hydraulic support. The principle is to use a displacement matrix and a rotation matrix in three-dimensional space to achieve the pose data conversion from the AR coordinate system to the unified coordinate system of the hydraulic support group by designing the rotation and translation matrix.

[0073] The displacement matrix is Where dx, dy, and dz represent the displacements in the x-axis, y-axis, and z-axis directions of the AR coordinate system, respectively.

[0074] The rotation matrix is

[0075]

[0076] Where RxRyRz represents the rotation of the AR coordinate system around the x-axis, y-axis and z-axis respectively.

[0077] The transformation matrix composed of the displacement matrix and the rotation matrix can transform the AR coordinate system to the unified coordinate system of the hydraulic support at the specified position at one time.

[0078] The posture deviation of a specific support includes two parts. The first is to visualize the corresponding posture data of the detection support coordinate system in the adjacent support model, which is also the third-level local coordinate system of the i-th hydraulic support. Secondly, the posture data of the specific support in the second-level unified coordinate system of the hydraulic support group is displayed, so that the subsequent remote cloud data processing system can directly receive and send it to the VR system to complete the posture reconstruction of the virtual hydraulic support group. By establishing the relative posture matrix of the hydraulic support, the coordinate system of each support can be realized to the coordinate system of the first support (the unified coordinate system of the hydraulic support group). That is, the i-th hydraulic support is defined as the reference support, and the detection support is the i+1-th hydraulic support. Suppose the relative posture matrix of the i+1-th hydraulic support is i Q i+1 .

[0079]

[0080] The absolute position and posture of the i+1th hydraulic support in the unified coordinate system of the hydraulic support group 1 Q i+1 It can be calculated by the following formula.

[0081] i Q i+1 = 1 Q2 2 Q3…… i Q i+1 (i>=3).

[0082] Component identification and solution and inter-frame bidirectional matching verification are used to solve the problems of data processing and model accuracy caused by redundant detection information during the internal and external posture recognition of hydraulic supports. In view of the redundant detection information, the iterative verification of the internal posture small component group is completed by model training and dividing the training data, verification data and test data; the iterative verification of each large moving joint of the hydraulic support is completed by the seamless linkage analysis of a single support; and the iterative verification of different support groups of the hydraulic support group is completed by the inter-frame bidirectional matching verification.

[0083] First, complete the identification and solution of small local components of the movable large joint.

[0084] First, the inspection data is preprocessed. After the manual matching is completed, duplicate and invalid data records are removed and which data are trainable data of the small component feature block and which are other redundant information. Then the features of each key component are extracted to ensure that the motion state of the movable large joint can be effectively reflected. The features are reduced using dimensionality reduction techniques such as PCA to retain the most representative information to reduce the complexity of subsequent calculations. By building a RandomForest model suitable for feature parsing and data modeling, the training data is used for model training, so that these models can predict the state or parameters of other related small components based on the small component feature block data. The redundant data that is not involved in the model training is processed, and relevant information that can be used for verification is extracted as verification data. The data obtained by the parsing model is compared with the relevant information in the verification data for verification and consistency is checked. If inconsistencies or abnormalities are found in the comparison, it is determined whether it is a detection error or there is actually a problem with the component. According to the analysis results, feedback is given to adjust the detection process to complete the iterative optimization verification. After the model training and parameter optimization are completed, the model is applied to the test data, and the performance is evaluated again, and the prediction results and feature importance analysis are recorded. Compare the parsing results of the test data with the results of the validation data portion to check the consistency of feature importance and model prediction results to show the consistency of the model on different datasets.

[0085] The redundant detection information of movable large joints with internal postures is solved by a single bracket seamless linkage method. The monitoring posture data of known large moving joints obtained by virtual-real matching of a single bracket is combined with the structural parameters Li of different models of brackets to obtain the unknown posture data of the bracket and verify the known posture data of the bracket. The inspection personnel conduct feedback verification through the verified posture data to avoid the redundant information of each large moving joint of a single hydraulic bracket, and realize the accurate matching of virtual-real models through multi-directional information fusion.

[0086] Specifically, the analysis of the seamless linkage method of a single support mainly includes the analysis of the four-bar linkage of the hydraulic support, the coordinated analysis of the four-bar linkage and the top beam, and the coordinated analysis between the four-bar linkage, the top beam, and the front and rear columns. The monitoring posture data include the rear link inclination angle θ, the front link pin and the rear link connecting line and the base inclination angle φ, and the top beam inclination angle δ. The verification posture data include the front link inclination angle β, the shielding beam inclination angle γ, the angle between the front column of the base and the base η, and the angle between the rear column of the base and the base ε. The unknown posture data of the support include the length L of the column during this process and the height H of the hydraulic support. The structural parameters Li include the distance between the front and rear link pins on the shielding beam L1, the distance between the two pins of the front link L2, the distance between the two pins of the rear link L3, and the distance between the front and rear link pins on the base L4.

[0087] Through structural relationship analysis, we can know that:

[0088]

[0089] The solution is:

[0090] The intermediate variables a, b, and c are:

[0091]

[0092] Combined with the top beam inclination angle δ, the top beam and base structures are analyzed respectively to determine the coordinates of the front column pin point C on the top beam, the rear column pin point D on the top beam, the front column pin point A on the base, and the rear column pin point B on the base.

[0093]

[0094]

[0095] Through the seamless linkage analysis of the hydraulic support, feedback verification of the verification posture data including the front connecting rod inclination angle β, the shield beam inclination angle γ, the angle between the front column of the base and the base η, and the angle between the rear column of the base and the base ε is realized. According to the verification results, the posture deviation of the virtual and real models is controlled within a certain range, making the monitoring posture data more accurate and re-analyzed to obtain more accurate verification posture data. It is also possible to obtain the unknown posture data of the support, including the extension length L of the column during this process and the height H of the hydraulic support. The analytical data can also verify the parameters measured by the adjacent frame model.

[0096] The possible causes of the errors in the neighboring frame model in the hydraulic support group are the manual dragging errors of the inspectors during the detection process, the coincidence errors between the database model and the real support, and the accumulated errors of multiple groups of neighboring frame models. The AR measurement method is used to measure the height difference of the head and tail supports of a specific support group, the difference in the telescopic length of the column, the difference in the pitch angle, the difference in the roll angle, and the difference in the yaw angle to verify the actual height and telescopic length of each support determined in the detection support coordinate system in different neighboring frame models of this group, as well as the actual attitude angle of each support. The nonlinear least squares method is used to obtain the minimum error between the measured data and the theoretical data. After removing the invalid data, the Levenberg-Marquardt optimization algorithm is used to gradually feed back these deviations to each support and perform virtual and real model correction and adjustment. After multiple iterative verifications, the error is kept within a certain range to achieve two-way matching verification between frames.

[0097] The remote cloud data processing system is the center for information processing, data support and data request for the Department of Physics. The remote cloud data processing system includes three elements: cloud service computing system, GPT decision system and centralized control worker. Its operation mode is that AR glasses upload the bracket posture information to the cloud in real time, and the cloud service computing system performs storage and multimodal computing analysis. At the same time, the cloud service computing system that completes communication with the model database server imports the model into the AR glasses. After preliminary analysis, the cloud service computing system sends the bracket supplementary posture data to the centralized control worker, who gives full play to the ability of autonomous decision-making and then sends the required supplementary data to the GPT decision system. The trained GPT model sends the hybrid decision results to the AR glasses through comprehensive understanding and analysis through hybrid decision-making, and requests the on-site inspectors wearing AR glasses to complete the measurement of the required supplementary detection data. In this way, the closed-loop data flow between the physical system and the remote cloud data processing system is realized.

[0098] The remote cloud data processing system stores the cloud model database. By digitally managing the model and related data of the hydraulic support, a hydraulic support model database can be established, which is convenient for inspectors to export the database model for direct matching and effectively improve work efficiency. SQL Server with high security, high availability and easy management is selected as the hydraulic support database management system, and hardware that supports mobile devices or has explosion-proof functions is selected as the on-site work platform for inspectors. First, 3D model data is collected for the selected feature bodies on all moving parts of different types of hydraulic supports, including the rectangular envelope built in the adjacent frame model. At the same time, parts lists, operation manual tables, and maintenance record tables are designed to store data, and logical relationships and data consistency constraints between entities are set to ensure data integrity. The collected data is manually entered into the database, and outliers, duplicate values, and missing values ​​in the data are processed to ensure data accuracy. After that, a user interface program is built and an API interface is provided to allow other systems and applications to access the database so that inspectors can easily query, update, and use database information.

[0099] Considering the particularity of the underground working environment, such as poor network connection stability, inconvenience in carrying and operating equipment, large delays in data interaction, and possible safety hazards of the system, it is necessary to conduct comprehensive testing of the database and formulate a database security strategy, including access authorization, data backup, and fault recovery process, to protect data security, especially in explosive underground environments. The hydraulic support model database can be used to train inspection personnel on how to effectively use the database, including query operations, data input, and information interpretation, and continuously improve the functions and performance of the database based on the feedback of inspection personnel to ensure long-term adaptation to work needs.

[0100] The VR system is used to describe the overall information perception and data fusion of the hydraulic support group. The parameters that measure the relative posture between the frames, such as the top beam drop of adjacent supports, the left and right gaps of the top beams, the left and right gaps of the bases, the relative yaw angle, the relative roll angle and the center distance, can all be directly obtained through the workspace formed by the closed-loop solution model between the frames that is integrated internally and externally. The workflow of the VR system is to construct a data-driven virtual model of the support in sequence by accepting the information integrated from the remote cloud data processing data system. The overall workspace of the hydraulic support group formed by the combination of all support workspaces and the physical system inspection worker dynamically detect the single-knife inspection information, and complete the sorting, numbering, and calibration of the hydraulic support. Then, the key points of the support workspace are classified according to the marked positions through clustering algorithms or classification algorithms, and a support group fitting model is established based on the classification results to identify different types of supports and predict the operating status of the support group. Then, the established support group classification and fitting model is evaluated and the model is continuously optimized to improve the accuracy of classification and the precision of fitting. The VR system displays information including support straightness, working face mining space volume, and roof and floor morphology to the control personnel in the control center in real time through 5G base stations or 5G connection lines, and the control personnel will detect the virtual support information. Ultimately, the inspection workers can dynamically measure the relevant information of the hydraulic support group during the single-knife inspection process.

[0101] As attached Figure 4 As shown in the figure, the three-system communication can achieve efficient data transmission. The underground mine uses 5G base stations to interconnect through Ethernet to provide a full coverage unlimited network and to efficiently transmit information with the remote cloud data processing system. The remote cloud data processing system connects to the WEB server through the Http protocol and connects the application server and the model database server through the supported protocol. The application server deploys application software such as Unity3D. The PC accesses the SQLSever database through C# program code based on the unity3D software and uploads the model data to the PC. The physical system PC and AR require the same IP address. Before using AR glasses, the environment needs to be deployed through the PC, so that AR glasses can directly access the model database server and support online upload to the cloud. The WEB server can directly display the virtual bracket model data on the client of the centralized control center, and can also output it to the VR system through Ethernet at 5G speed in the cloud. The VR system and the centralized control center can also achieve wired data transmission through 5G network cables.

Claims

1. A method for measuring cloud groups of hydraulic supports based on AR calibration and matching, characterized in that: Including physical systems, remote cloud data processing systems and VR systems; The physical system includes an inspector, AR glasses and a physical hydraulic support group. The inspector wearing AR glasses needs to walk and measure in the hydraulic support group and stop as needed. The detection process includes: (1) Taking the movable joint of a single hydraulic support with internal posture as the detection object, the closed-loop deduction model of the single support posture is solved; Establish multiple detail feature bodies inside the hydraulic support, and the detail feature bodies are selected on the movable joints of a single hydraulic support with internal posture; integrate the hydraulic support group and its internal structure with AR glasses in a unified coordinate system to build an AR-working surface-hydraulic support group three-level coordinate system; split the movable joints of a single hydraulic support from the internal perspective of the inspector, and select the three-dimensional feature bodies of the corresponding moving parts respectively; use the virtual simulation function and MRTK module in Unity3D software to complete the virtual-real model matching of the three-dimensional feature body, and display the movable joint posture data in real time on the AR interface; (2) Taking the entire support with external posture and the support base as the positioning base as the detection object, the posture model deviation between adjacent hydraulic support frames is detected; The posture of the entire bracket group is disassembled into the relative posture of two adjacent brackets as the research object, and the adjacent bracket model of the detection bracket and the reference bracket is established; the entire bracket with external posture and the bracket base as the positioning base point is selected from the external perspective of the inspector, and the bracket base posture information is obtained according to the relevant three-dimensional feature body of the base component to establish a virtual envelope cuboid, and the relative posture matrix of the hydraulic bracket is established according to the three-level coordinate system. The MRTK module in the Unity3D software is used to match the virtual envelope cuboid with the entire bracket; the six-point positioning principle is used to realize the relative posture detection between brackets, and the posture deviation relative to the reference bracket is displayed in real time on the AR interface; (3) Component identification and inter-rack bidirectional matching verification; The iterative verification of the internal posture widget group is completed by model training and dividing the training data, verification data and test data; the iterative verification of each large joint of the hydraulic support is completed by seamless linkage analysis of a single support; the iterative verification of different support groups of the hydraulic support group is completed by bidirectional matching verification between supports; the above verification methods are synchronized and verified; The remote cloud data processing system includes a cloud service computing system, a GPT decision system and a centralized control worker. The cloud service computing system performs data storage and preliminary analysis. The centralized control worker and the GPT decision system send the mixed decision results to the AR glasses after secondary data processing to request additional test data. Finally, the on-site inspection worker completes the measurement. The VR system constructs and corrects the twin hydraulic support group model through the real-time support posture data of the remote cloud data processing system, constructs the overall workspace and performs numbering, sorting and calibrating, and finally verifies the posture reconstruction accuracy; predicts the next-cut posture of the physical support group through the virtual support group, and supplements and corrects the data of the virtual support group through the physical support group, so as to achieve synchronization of the posture of the virtual and real hydraulic support groups.

2. The method according to claim 1, characterized in that : The detailed feature bodies are selected on the base, rear connecting rod, front connecting rod, column, shield beam, top beam and front beam with internal posture.

3. The method according to claim 2, characterized in that The virtual enveloping cuboid is used to model the various subcomponents of the hydraulic support and embed them into the enveloping cuboid.

4. The method according to claim 1 or 3, characterized in that : The three-level coordinate system includes a coordinate system {O A }'s AR coordinate system, the coordinate system is {O H } hydraulic support group unified coordinate system, coordinate system is {O' H } the local coordinate system of the i-th hydraulic support, wherein the unified coordinate system of the hydraulic support group serves as a sub-coordinate system of the AR coordinate system, and the local coordinate system of the hydraulic support serves as a sub-coordinate system of the unified coordinate system of the hydraulic support group.

5. The method according to claim 4, characterized in that : The neighbor frame model is constructed in {O' H In the coordinate system, the i-th hydraulic support is selected as the reference support, the i+1-th hydraulic support is selected as the detection support, and the support base center O is defined. c is the coordinate origin of the reference bracket, and the center of the bracket base is O d To detect the coordinate origin of the bracket, the advancing direction of the working face is the positive direction of the x-axis, the direction from the coal seam bottom plate to the coal seam roof is the positive direction of the z-axis, and the y-axis direction is determined by the right-hand rule; the entire detection bracket is regarded as a model in which the base is fixed with the front beam, top beam, shield beam, front and rear connecting rods and other movable parts following the telescopic movement of the column.

6. The method according to claim 1 or 5, characterized in that : The virtual-real feature matching and local feature calibration of the three-dimensional model is that the inspector manually matches the virtual three-dimensional feature body model with the real hydraulic support, imports and configures MRTK in Unity, deploys the Unity project to the HoloLens device, creates a scene in Unity, and imports the feature body model that needs to be matched in the hydraulic support model database, and sets the position and direction of the model according to the actual situation; imports OpenXR and MR related configurations, creates multiple C# scripts to implement the functions of anchor initialization, adding anchors, deleting anchors, saving anchors and loading anchors in turn, and finally completes the design of the anchor shape through scene settings; during the operation of the inspector, ensure that the dynamic anchor is aligned with the real hydraulic support components, and align the virtual model with the anchor through visual guidance and interactive operations.

7. The method according to claim 6, characterized in that :For the identification and solution of small parts in the movable large joints, the inspection data is preprocessed first. After the manual matching is completed, duplicate and invalid data records are removed and the data are distinguished as the trainable data of the small parts feature blocks and other redundant information; then the features of each key component are extracted and the features are reduced in dimension; By constructing a Random Forest model suitable for feature parsing and data modeling, the training data is used for model training, so that these models can predict the status or parameters of other related widgets based on the widget feature block data, process the redundant data that are not involved in the model training, and extract relevant information that can be used for verification as verification data; compare and verify the data obtained by the parsing model with the relevant information in the verification data to check the consistency. If inconsistencies or abnormalities are found in the comparison, determine whether it is a detection error or an actual problem with the component; based on the analysis results, feedback is given to adjust the detection process to complete iterative optimization verification; after the model training and parameter optimization are completed, the model is applied to the test data, and the performance is evaluated again, and the prediction results and feature importance analysis are recorded; the parsing results of the test data are compared with the results of the verification data part, and the consistency of the feature importance and model prediction results is checked to show the consistency of the model on different data sets.

8. The method according to claim 7, characterized in that : Aiming at the redundancy of detection information of large movable joints with internal postures, the monitoring posture data of known large movable joints obtained by virtual-real matching of a single bracket is combined with the structural parameter Li of different models of brackets to obtain the unknown posture data of the bracket and verify the known posture data of the bracket. The inspection personnel conduct feedback verification through the verification posture data, and realize accurate matching of virtual and real models through multi-dimensional information fusion.

9. The method according to claim 8, characterized in that Aiming at the possible errors of the neighboring frame models in the hydraulic support group, the AR measurement method is used to measure the support height difference, column telescopic length difference, pitch angle difference, roll angle difference, and yaw angle difference of the head and tail supports of a specific support group to verify the actual height and telescopic length of each support and the actual attitude angle of each support determined in the detection support coordinate system in different neighboring frame models of this group; the nonlinear least squares method is used to obtain the minimum error between the measured data and the theoretical data. After removing the invalid data, the Levenberg-Marquardt optimization algorithm is used to gradually feed back these deviations to each support and perform virtual and real model correction and adjustment. After multiple iterative verifications, the error is kept within a certain range to achieve two-way matching verification between frames.

10. The method according to claim 1 or 9, characterized in that :The remote cloud data processing system stores the cloud model database, establishes the hydraulic support model database by digitally managing the hydraulic support model and related data, selects SQL Server as the hydraulic support database management system, and selects mobile devices or explosion-proof hardware as the on-site work platform for inspection personnel.

Images