Intelligent mine operation and maintenance method based on mixed reality

Abstract

The invention discloses an intelligent mine operation and maintenance method based on mixed reality, and the method specifically comprises the following steps: 1, building a model for an operation and maintenance system and equipment through mine information; 2, in combination with an MR system engine, establishing an MR operation and maintenance interaction scene, and performing targeted optimization rendering according to an underground special environment; 3, collecting mine information by using a sensing module, constructing a sensing-transmission-mapping mode, accessing actual working condition data of mine system equipment, and projecting the data to a virtual panel in a scene; and step 4, through various interaction modes of the MR equipment, detecting the operation state of the equipment, and realizing operation and maintenance of the mixed reality mode. According to the invention, the problems of huge cost and low operation and maintenance efficiency in the current operation and maintenance of the mine are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of mixed reality, and particularly relates to a method for the operation and maintenance of intelligent mines based on mixed reality. Background Technique

[0002] Over the years, underground coal mining has experienced a leap from single-machine mechanization, comprehensive mechanization to automation, and is gradually moving towards intelligence. Currently, China is vigorously promoting the construction of intelligent mines and intelligent working faces. Correspondingly, the fully-mechanized coal mining technology and equipment are also in a stage of rapid development. The intelligent level of the fully-mechanized coal mining technology and equipment has also become an important indicator for measuring intelligent working faces. An intelligent fully-mechanized working face refers to a fully-mechanized working face that uses mechatronic complete sets of equipment such as hydraulic supports, shearer loaders, and scraper conveyors with sufficient and comprehensive perception, autonomous learning and decision-making, and automatic execution functions, realizing highly automated and few-person remote monitoring of the working face, and being able to ensure safe and efficient mining of the working face; an intelligent fully-mechanized coal mining system is a comprehensive integration of multiple technologies such as electricity, hydraulics, control, network communication, and voice and video.

[0003] Due to the complexity of the working conditions in coal mines and their own particularities, as well as the complex and changeable geological conditions, the rapid development of coal mine intelligence has inevitably led to many situations that affect the continuous, safe and efficient operation of the intelligent mining system during the continuous operation process. Therefore, a large amount of human, material and other resources need to be invested in the operation and maintenance of mine system equipment. The mine system environment is complex. The current operation and maintenance of mine system equipment often rely on several modes such as manual daily maintenance, daily inspection and emergency response. All these methods require maintenance workers to go to the site to troubleshoot problems that occur during the operation of the equipment, find the fault points for subsequent maintenance. The maintenance procedures are cumbersome and the efficiency is low. At the same time, during on-site maintenance, the problems existing in the maintenance components cannot be effectively detected, the operation and maintenance efficiency is very low, and the operation and maintenance investment is relatively large.

[0004] Based on the above problems existing in the current mine operation and maintenance, such as cumbersome procedures, reliance on manual labor, large investment in human and material resources, and low operation and maintenance efficiency, a technology for the operation and maintenance of intelligent mines based on mixed reality is proposed, aiming to solve the problems of mine operation and maintenance efficiently and with low investment. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for the operation and maintenance of intelligent mines based on mixed reality, which solves the problems of huge cost and low operation and maintenance efficiency existing in the current mine operation and maintenance.

[0006] The technical solution adopted by the present invention is that the method for the operation and maintenance of intelligent mines based on mixed reality is specifically implemented according to the following steps: Step 1: Use mine information to establish models for the operation and maintenance system and equipment; Step 2: Combine with the MR system engine to establish an MR operation and maintenance interaction scenario, and perform targeted optimized rendering according to the special underground environment; Step 3: Use the sensing module to collect mine information, construct a "sensing - transmission - mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scenario; Step 4: Detect the equipment operation status through various interaction methods of the MR device to achieve operation and maintenance in the mixed reality mode.

[0007] The features of the present invention also lie in that, The beneficial effects of the present invention are as follows: The intelligent mine operation and maintenance method based on mixed reality provides more intuitive data basis for operation and maintenance personnel by accessing environmental data through the model, avoiding the problem that the equipment being maintained and its working condition information cannot be intuitively displayed to personnel during the operation and maintenance process. Through the model explosion mode, the internal component information and conditions of the equipment can be displayed, avoiding the cumbersome steps of disassembling and assembling the equipment shell during the equipment maintenance process. Through various modes such as gesture recognition, voice recognition, object tracking, gaze tracking, and tactile feedback, it assists the operation of the operation and maintenance personnel, prompts key information during the operation and maintenance process, and can effectively simplify unnecessary operations during the operation and maintenance process and optimize the operation and maintenance operation process. After the operation and maintenance are completed, the ledger information is automatically recorded in the system background, largely avoiding manual input operations, and the operation and maintenance results are also recorded more directly and accurately. Through the mixed reality remote assistance method, it intuitively and efficiently guides the technical problems encountered by on-site maintenance personnel. Utilizing the efficiency advantage of the system, the expected input for operation and maintenance can be reduced by 1 / 3, reducing the consumption of human and material resources. Brief Description of the Drawings

[0008] Figure 1 is a flowchart of the intelligent mine operation and maintenance method based on mixed reality of the present invention; Figure 2 is a non - direct gesture schematic diagram of the intelligent mine operation and maintenance method based on mixed reality of the present invention. Detailed Embodiments

[0009] The present invention will be described in detail below in conjunction with the drawings and specific embodiments.

[0010] MR is a mixed reality technology that combines the real world and the virtual world. It superimposes virtual objects onto the real environment through a display device. In MR, users can see and interact with virtual objects in the real world while maintaining the perception of the real world. MR can perceive the user's position, actions, and environmental information through sensors and computer vision, thereby enabling the interaction between virtual objects and the real environment. The background technology of mixed reality technology involves multiple aspects such as sensor technology, computer vision technology, display technology, graphics rendering technology, and data processing and algorithms.

[0011] Taking advantage of the above advantages of MR technology, studying the role it can play in mine operation and maintenance, and proposing a smart mine operation and maintenance method based on mixed reality, which is implemented according to the following steps: Step 1: Use mine information to build models for the operation and maintenance system and equipment; Step 2: Combine the MR system engine to establish an MR operation and maintenance interaction scenario, and perform targeted optimized rendering according to the special underground environment; The interaction methods in Step 2 include gesture recognition, voice control, object tracking, eye tracking, and tactile feedback.

[0012] Gesture recognition includes direct gesture recognition, non-direct gesture recognition, and touch recognition; Direct gesture means directly touching a spatial object, which belongs to a close-range reachable interaction. The user brings a finger close to the virtual object and gently touches the virtual object. For example, moving, scaling, and gently tapping on virtual objects in space. In the case of direct gestures, the user must bring the hand close to the interaction object; Non-direct gesture means interacting with the currently focused object in a long-distance situation. There are many ways to obtain focus. Commonly, it is through eye gaze, a hand ray to lock an object or component in space to obtain focus. After the user focuses on an object or control, one or more finger actions can be used to quickly operate the object. Non-direct gestures will affect the currently focused object. For example, after the user focuses on a control, one of the fingers can be used to quickly tap the thumb to make a non-direct tap gesture to activate or select the control. Because the user can quickly look in different directions, non-direct gestures allow the user to interact with any focusable object regardless of the user's position in space. The user does not have to get close to touch the object with the hand. For reference, Figure 2 several gestures.

[0013] Touch means directly selecting or activating an object, opening the associated menu in the MR system engine, touch and drag: move the object to a new position, double-touch: preview the object or file; select words in an editing context, swipe: display operations and controls; close the view; scroll; pinch and drag with both hands: zoom in or out, pinch and drag along an arc with both hands: rotate the object.

[0014] Voice control includes speech recognition, speech synthesis and control; Speech recognition collects raw audio data through a microphone, and uses MFCC (Mel Frequency Cepstral Coefficients) and PLP (Perceptual Linear Predictive) methods to extract features such as the frequency, amplitude and duration of the speech signal, and converts it into a digital signal. Specifically, a Fourier transform is performed on each frame to obtain spectral information; then the spectral information is mapped to the Mel scale to simulate the auditory perception of the human ear; then a discrete cosine transform is performed on the spectrum on the Mel scale to obtain a set of Mel frequency cepstral coefficients, which reflect the main features of the speech signal. Then, based on the neural network model RNN, the feature sequence data is processed. The RNN captures and analyzes the temporal dependencies in the speech signal, analyzes the semantics and context information of the speech signal, so as to achieve accurate speech recognition and understanding. Finally, the dynamic programming algorithm is used to select the most likely result among the candidate words for speech recognition; Speech synthesis is the process of converting text information into speech signals. It is achieved by decomposing the text into phonemes, determining the pronunciation rules of the phonemes, and then synthesizing the speech signals, including three steps: text processing, phoneme conversion and voice synthesis. Text processing is the process of preprocessing and analyzing the input text, including operations such as word segmentation, part-of-speech tagging and syntactic analysis of the text, in order to more accurately determine the phonemes and pronunciation rules. Phoneme conversion is the process of corresponding the letters and phonetic symbols in the text. By looking up the phoneme library or using phoneme conversion rules, the letters in the text are converted into corresponding phonemes for subsequent voice synthesis. Voice synthesis is the process of combining phonemes into speech signals. By concatenating the waveforms of phonemes or using synthesis rules, natural and fluent speech signals are generated. Voice synthesis uses methods based on concatenation, methods based on parameters or hybrid methods; Voice control is the process of controlling a device or performing a specific task through voice commands, which is achieved through speech recognition and speech synthesis, including three stages: voice input, voice processing and voice output. Voice input is the process by which a user inputs a speech signal into a device by speaking. The device converts the speech signal into text or commands, and performs semantic analysis and instruction matching to determine the user's intention. Voice processing is the process of parsing and processing the user's voice input, including operations such as speech recognition, semantic analysis and instruction matching, in order to convert the user's voice command into an instruction that the device can understand and execute. Voice output is the process of converting the device's response into a speech signal through speech synthesis. The device will perform corresponding operations according to the user's instructions, and convert the execution result into a speech signal through speech synthesis for feedback to the user.

[0015] Object tracking is to continuously track the target over time by leveraging the correlation between consecutive frames. The basic steps include initial target localization, target template update, and target state prediction; Object tracking locates the target by using the target bounding box in the initial frame. Then, in subsequent frames, the algorithm matches the target template with each frame to determine the position of the target in the current frame. To improve the robustness and accuracy of the algorithm, the algorithm also needs to update the target template according to the characteristics of the target motion and consider factors such as the speed and acceleration of the target when predicting the target position in the next frame. Object tracking based on correlation filters obtains the position of the target by performing correlation operations between the target template and each frame, and commonly used mean filters and kernelized correlation filters, etc.

[0016] When using the optical flow tracking algorithm for gaze tracking, when the targets and devices in the operation and maintenance scenario correspond to the motion in the two-dimensional image plane, the projection in the two-dimensional image plane forms motion. This kind of motion that is manifested as the flow of the brightness pattern in the image plane is called optical flow. The optical flow method is an important method for analyzing motion sequence images. Optical flow not only contains the motion information of the targets in the image, but also contains rich information about the three-dimensional physical structure. Therefore, it can be used to determine the operating conditions of the devices in the operation and maintenance scenario and reflect other status information.

[0017] Optical flow is the instantaneous velocity of the pixel motion of a spatially moving object on the observed imaging plane. The study of optical flow is to use the temporal variation and correlation of the pixel intensity data in the image sequence to determine the "motion" of each pixel position, that is, to study the relationship between the change of image grayscale over time and the object structure and its motion in the scene. Generally, optical flow is generated by camera motion, target motion in the scene, or the combined motion of both.

[0018] Haptic feedback allows users to receive tactile information by applying force, vibration, or touch, and interact with objects in the virtual system through haptic simulation to achieve the feeling of touching real objects.

[0019] The haptic feedback is as follows: During the operation and maintenance process, if the operation and maintenance personnel encounter problems that cannot be solved, they initiate remote assistance through the communication function of the MR device, communicate one-on-one with remote technical experts in a virtual scene, and assist the operation and maintenance personnel in troubleshooting equipment problems through visual interaction in the virtual scene. At the same time, the operation and maintenance personnel can transmit the on-site scene to the background through the remote transmission function of the MR device, and give intuitive assistance prompts, including clicking to mark equipment, tracking perspectives, and even file transmission; After the operation and maintenance is completed, the system stores the operation and maintenance date, operation and maintenance equipment, specific operation and maintenance content, list of operation and maintenance personnel, and guidance information in the corresponding data storage device to form an operation and maintenance record as an operation and maintenance ledger; The records in the ledger are beneficial as a reference basis during the later operation and maintenance process, and also serve as the data basis for summarizing and statistics of the historical operation and maintenance costs.

[0020] Step 3: Use the sensing module to collect mine information, construct a "sensing - transmission - mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scene. Step 4: Detect the operating status of the equipment through various interaction methods of the MR device, and achieve operation and maintenance in the mixed reality mode.

[0021] Embodiment 1 The intelligent mine operation and maintenance method based on mixed reality is specifically implemented according to the following steps: Step 1: Use mine information to establish models for the operation and maintenance system and equipment. Step 2: Combine with the MR system engine to establish an MR operation and maintenance interaction scene, and perform targeted optimized rendering according to the special underground environment. Step 3: Use the sensing module to collect mine information, construct a "sensing - transmission - mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scene. Step 4: Detect the operating status of the equipment through various interaction methods of the MR device, and achieve operation and maintenance in the mixed reality mode.

[0022] Embodiment 2 The intelligent mine operation and maintenance method based on mixed reality is specifically implemented according to the following steps: Step 1: Use mine information to establish models for the operation and maintenance system and equipment. Step 2: Combine with the MR system engine to establish an MR operation and maintenance interaction scene, and perform targeted optimized rendering according to the special underground environment. The interaction methods in Step 2 include gesture recognition, voice control, object tracking, eye tracking, and tactile feedback.

[0023] Gesture recognition includes direct gesture recognition, non - direct gesture recognition, and touch recognition. Direct gesture means directly touching the spatial object, which belongs to the reachable interaction at close range. The user brings the finger close to the virtual object and touches the virtual object lightly. For example, moving, scaling, and lightly clicking on the virtual object in space. In the way of direct gesture, the user must bring the hand close to the interaction object. An indirect gesture refers to interacting with the currently focused object in a long-distance scenario. There are many ways to obtain focus. Commonly, it is achieved by eye gaze, using a hand ray to lock on to an object or component in space to obtain focus. After the user focuses on an object or control, one or more finger movements can be used to quickly operate the object. Indirect gestures affect the currently focused object. For example, after the user focuses on a control, they can use one finger to quickly tap the thumb to make an indirect tap gesture to activate or select the control. Since the user can quickly look in different directions, indirect gestures allow the user to interact with any focusable object regardless of their position in space. The user does not have to get close to touch the object with their hand. Refer to Figure 2 several gestures.

[0024] Tapping refers to directly selecting or activating an object, opening the associated menu in the MR system engine. Tap and drag: Move the object to a new position. Double-tap: Preview an object or file; select words in an editing context. Swipe: Display operations and controls; close the view; scroll. Pinch and drag with both hands: Zoom in or out. Pinch and drag along an arc with both hands: Rotate the object.

[0025] Step 3: Use the sensing module to collect mine information, build a "sensing - transmission - mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scene; Step 4: Through various interaction methods of the MR device, detect the operating state of the equipment and achieve the operation and maintenance in the mixed reality mode.

[0026] Embodiment 3 A smart mine operation and maintenance method based on mixed reality is specifically implemented according to the following steps: Step 1: Use mine information to build models for the operation and maintenance system and equipment; Step 2: Combine with the MR system engine to build an MR operation and maintenance interaction scene, and perform targeted optimized rendering according to the special underground environment; The interaction methods in Step 2 include gesture recognition, voice control, object tracking, eye tracking, and tactile feedback.

[0027] Gesture recognition includes direct gesture recognition, indirect gesture recognition, and tap recognition; Direct gestures refer to directly touching spatial objects, which are interactions that can be reached at close range. The user brings their finger close to the virtual object and touches the virtual object. For example, moving, scaling, and tapping on virtual objects in space. In the way of direct gestures, the user must bring their hand close to the interaction object; Indirect gestures refer to interactions with the currently focused object at a distance. There are many ways to gain focus, the most common of which is to gain focus by gazing with the eyes or locking a hand ray on an object or component in space. After the user focuses on an object or control, they can quickly manipulate the object with one or more finger movements. Indirect gestures affect the currently focused object. For example, after the user focuses on a control, they can use one of their fingers to quickly tap their thumb to make an indirect tap gesture to activate or select the control. Because users can quickly look in different directions, indirect gestures allow users to interact with any focusable object regardless of their position in space. Users do not have to get close to touch the object with their hands. For more information, see Indirect gestures. Figure 2 Several hand gestures.

[0028] Touching means directly selecting or activating an object, opening a context menu in the MR system engine, touching and dragging: moving an object to a new location, double-tapping: previewing an object or file; selecting words in an editing context, swiping: displaying actions and controls; closing a view; scrolling; pinching and dragging with both hands: zooming in or out, pinching and dragging along an arc with both hands: rotating an object.

[0029] Voice control includes speech recognition, speech synthesis and control; Speech recognition collects raw audio data through a microphone, and uses MFCC (Mel Frequency Cepstral Coefficients) and PLP (Perceptual Linear Predictive) methods to extract features such as frequency, amplitude, and duration of speech signals, and converts them into digital signals. Specifically, Fourier transform is performed on each frame to obtain spectrum information; the spectrum information is then mapped to the Mel scale to simulate the auditory perception of the human ear; then the spectrum on the Mel scale is discrete cosine transformed to obtain a set of Mel frequency cepstral coefficients, which reflect the main characteristics of the speech signal. Then, feature sequence data is processed based on the neural network model RNN, and the time dependency in the speech signal is captured and analyzed by RNN, and the semantics and contextual information of the speech signal are analyzed to achieve accurate speech recognition and understanding. Finally, the dynamic programming algorithm is used to select the most likely result from the candidate words for speech recognition. Speech synthesis is the process of converting text information into speech signals. It is achieved by decomposing the text into phonemes, determining the pronunciation rules of the phonemes, and then synthesizing the speech signals. It includes three steps: text processing, phoneme conversion, and voice synthesis. Text processing is the process of preprocessing and analyzing the input text, including operations such as word segmentation, part-of-speech tagging, and syntactic analysis of the text, in order to more accurately determine the phonemes and pronunciation rules. Phoneme conversion is the process of corresponding the letters and phonetic symbols in the text. By looking up the phoneme library or using phoneme conversion rules, the letters in the text are converted into the corresponding phonemes for subsequent voice synthesis. Voice synthesis is the process of combining phonemes into speech signals. By concatenating the waveforms of phonemes or using synthesis rules, natural and fluent speech signals are generated. Voice synthesis adopts methods based on concatenation, methods based on parameters, or hybrid methods; Voice control is the process of controlling devices or performing specific tasks through voice commands. It is achieved through speech recognition and speech synthesis, and includes three stages: voice input, voice processing, and voice output. Voice input is the process by which a user inputs a voice signal into a device by speaking. The device converts the voice signal into text or commands, and performs semantic analysis and command matching to determine the user's intention. Voice processing is the process of parsing and processing the user's voice input, including operations such as speech recognition, semantic analysis, and command matching, in order to convert the user's voice commands into instructions that the device can understand and execute. Voice output is the process of converting the device's response into a voice signal through speech synthesis. The device performs corresponding operations according to the user's instructions, and converts the execution results into voice signals through speech synthesis for feedback to the user.

[0030] Step 3: Use the sensing module to collect mine information, construct a "sensing-transmission-mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scene; Step 4: Detect the operating state of the equipment through various interaction methods of the MR device, and realize the operation and maintenance in the mixed reality mode.

[0031] Embodiment 4 The intelligent mine operation and maintenance method based on mixed reality is specifically implemented according to the following steps: Step 1: Use mine information to establish models for the operation and maintenance system and equipment; Step 2: Combine the MR system engine to establish an MR operation and maintenance interaction scene, and perform targeted optimization rendering according to the special underground environment; The interaction methods in Step 2 include gesture recognition, voice control, object tracking, gaze tracking, and haptic feedback.

[0032] Gesture recognition includes direct gesture recognition, non-direct gesture recognition, and touch recognition; Direct gestures refer to touching objects directly in space. They are close-range interactions. Users use their fingers to approach virtual objects and touch them lightly. For example, users can move, scale, touch and click virtual objects in space. In direct gestures, users must bring their hands close to the objects to interact with. Indirect gestures refer to interactions with the currently focused object at a distance. There are many ways to gain focus, the most common of which is to gain focus by gazing with the eyes or locking a hand ray on an object or component in space. After the user focuses on an object or control, they can quickly manipulate the object with one or more finger movements. Indirect gestures affect the currently focused object. For example, after the user focuses on a control, they can use one of their fingers to quickly tap their thumb to make an indirect tap gesture to activate or select the control. Because users can quickly look in different directions, indirect gestures allow users to interact with any focusable object regardless of their position in space. Users do not have to get close to touch the object with their hands. For more information, see Indirect gestures. Figure 2 Several hand gestures.

[0033] Touching means directly selecting or activating an object, opening a context menu in the MR system engine, touching and dragging: moving an object to a new location, double-tapping: previewing an object or file; selecting words in an editing context, swiping: displaying actions and controls; closing a view; scrolling; pinching and dragging with both hands: zooming in or out, pinching and dragging along an arc with both hands: rotating an object.

[0034] Voice control includes speech recognition, speech synthesis and control; Speech recognition collects raw audio data through a microphone, and uses MFCC (Mel Frequency Cepstral Coefficients) and PLP (Perceptual Linear Predictive) methods to extract features such as frequency, amplitude, and duration of speech signals, and converts them into digital signals. Specifically, Fourier transform is performed on each frame to obtain spectrum information; the spectrum information is then mapped to the Mel scale to simulate the auditory perception of the human ear; then the spectrum on the Mel scale is discrete cosine transformed to obtain a set of Mel frequency cepstral coefficients, which reflect the main characteristics of the speech signal. Then, feature sequence data is processed based on the neural network model RNN, and the time dependency in the speech signal is captured and analyzed by RNN, and the semantics and contextual information of the speech signal are analyzed to achieve accurate speech recognition and understanding. Finally, the dynamic programming algorithm is used to select the most likely result from the candidate words for speech recognition. Speech synthesis is the process of converting text information into speech signals. It is achieved by decomposing the text into phonemes, determining the pronunciation rules of phonemes, and then synthesizing speech signals. It includes three steps: text processing, phoneme conversion, and voice synthesis. Text processing is the process of preprocessing and analyzing the input text, including operations such as word segmentation, part-of-speech tagging, and syntactic analysis of the text, in order to more accurately determine phonemes and pronunciation rules. Phoneme conversion is the process of corresponding letters and phonetic symbols in the text. By looking up the phoneme library or using phoneme conversion rules, the letters in the text are converted into corresponding phonemes for subsequent voice synthesis. Voice synthesis is the process of combining phonemes into speech signals. By concatenating the waveforms of phonemes or using synthesis rules, natural and fluent speech signals are generated. Voice synthesis uses methods based on concatenation, methods based on parameters, or hybrid methods; Voice control is the process of controlling a device or performing a specific task through voice commands, which is achieved through speech recognition and speech synthesis. It includes three stages: voice input, voice processing, and voice output. Voice input is the process by which a user inputs a speech signal into a device by speaking. The device converts the speech signal into text or commands and performs semantic analysis and instruction matching to determine the user's intention. Voice processing is the process of parsing and processing the user's voice input, including operations such as speech recognition, semantic analysis, and instruction matching, in order to convert the user's voice command into an instruction that the device can understand and execute. Voice output is the process of converting the device's response into a speech signal through speech synthesis. The device performs corresponding operations according to the user's instructions and converts the execution result into a speech signal through speech synthesis for feedback to the user.

[0035] Object tracking is the process of continuously tracking a target over time by utilizing the correlation between consecutive frames. The basic steps include target initial localization, target template update, and target state prediction; Object tracking locates the target by using the target bounding box in the initial frame. Then, in subsequent frames, the algorithm uses the target template to match each frame to determine the position of the target in the current frame. To improve the robustness and accuracy of the algorithm, the algorithm also needs to update the target template according to the characteristics of the target movement and consider factors such as the speed and acceleration of the target when predicting the target position in the next frame. Object tracking based on correlation filters obtains the position of the target by performing correlation operations between the target template and each frame, and commonly used ones include mean filters and kernelized correlation filters, etc.

[0036] When gaze tracking uses the optical flow tracking algorithm to correspond the targets and devices in the operation and maintenance scenario to the movement in the two-dimensional image plane, the projection in the two-dimensional image plane forms a motion. This kind of motion that is manifested as the flow of the brightness pattern in the image plane is called optical flow. The optical flow method is an important method for analyzing the sequence of motion images. Optical flow not only contains the motion information of the targets in the image, but also contains rich information about the three-dimensional physical structure. Therefore, it can be used to determine the operating conditions of the devices in the operation and maintenance scenario and reflect other status information.

[0037] Optical flow is the instantaneous velocity of the pixel motion of a spatially moving object on the observed imaging plane. The study of optical flow is to use the temporal variation and correlation of the pixel intensity data in the image sequence to determine the “motion” of each pixel position, that is, to study the relationship between the change of image gray level in time and the structure and motion of the objects in the scene. Generally, optical flow is generated by the camera motion, the motion of the targets in the scene, or the combined motion of both.

[0038] Step 3: Use the sensing module to collect mine information, construct a “sensing - transmission - mapping” mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scene. Step 4: Detect the operating status of the equipment through various interaction methods of the MR device to achieve operation and maintenance in the mixed reality mode.

[0039] Embodiment 5 The intelligent mine operation and maintenance method based on mixed reality is specifically implemented according to the following steps: Step 1: Use the mine information to establish models for the operation and maintenance system and equipment. Step 2: Combine the MR system engine to establish an MR operation and maintenance interaction scene, and perform targeted optimized rendering according to the special underground environment. The interaction methods in Step 2 include gesture recognition, voice control, object tracking, gaze tracking, and tactile feedback.

[0040] Voice control includes voice recognition, voice synthesis, and control. Speech recognition collects raw audio data through a microphone, and uses the MFCC (Mel Frequency Cepstral Coefficients) and PLP (Perceptual Linear Predictive) methods to extract features such as the frequency, amplitude, and duration of the speech signal, and converts it into a digital signal. Specifically, a Fourier transform is performed on each frame to obtain spectral information; then the spectral information is mapped to the Mel scale to simulate the auditory perception of the human ear; then a discrete cosine transform is performed on the spectrum on the Mel scale to obtain a set of Mel frequency cepstral coefficients, which reflect the main features of the speech signal. Then, based on the neural network model RNN, the feature sequence data is processed, and the RNN is used to capture and analyze the temporal dependencies in the speech signal, analyze the semantics and context information of the speech signal, so as to achieve accurate speech recognition and understanding. Finally, the dynamic programming algorithm is used to select the most likely result among the candidate words for speech recognition; Speech synthesis is the process of converting text information into a speech signal. It is achieved by decomposing the text into phonemes, determining the pronunciation rules of the phonemes, and then synthesizing the speech signal. It includes three steps: text processing, phoneme conversion, and sound synthesis. Text processing is the process of preprocessing and analyzing the input text, including operations such as word segmentation, part-of-speech tagging, and syntactic analysis of the text, in order to more accurately determine the phonemes and pronunciation rules. Phoneme conversion is the process of corresponding the letters and phonetic symbols in the text. By looking up the phoneme library or using phoneme conversion rules, the letters in the text are converted into corresponding phonemes for subsequent sound synthesis. Sound synthesis is the process of combining phonemes into a speech signal. By concatenating the waveforms of phonemes or using synthesis rules, a natural and fluent speech signal is generated. Sound synthesis adopts methods based on concatenation, methods based on parameters, or hybrid methods; Speech control is the process of controlling a device or performing a specific task through a voice command, which is achieved through speech recognition and speech synthesis. It includes three stages: speech input, speech processing, and speech output. Speech input is the process by which a user inputs a speech signal into a device by speaking. The device will convert the speech signal into text or a command, and perform semantic analysis and instruction matching to determine the user's intention. Speech processing is the process of parsing and processing the user's speech input, including operations such as speech recognition, semantic analysis, and instruction matching, in order to convert the user's speech command into an instruction that the device can understand and execute. Speech output is the process of converting the device's response into a speech signal through speech synthesis. The device will perform corresponding operations according to the user's command, and convert the execution result into a speech signal through speech synthesis for feedback to the user.

[0041] Object tracking is to continuously track the target in time by using the correlation between consecutive frames. The basic steps include target initial positioning, target template update, and target state prediction; Object tracking locates the target by using the target bounding box in the initial frame. Then, in subsequent frames, the algorithm matches the target template with each frame to determine the position of the target in the current frame. To improve the robustness and accuracy of the algorithm, the algorithm also needs to update the target template according to the characteristics of the target motion and consider factors such as the speed and acceleration of the target when predicting the target position in the next frame. Object tracking based on correlation filters obtains the position of the target by performing correlation operations between the target template and each frame. Commonly used ones include mean filters and kernelized correlation filters, etc.

[0042] When using the optical flow tracking algorithm for gaze tracking, when the targets and devices in the operation and maintenance scenario correspond to the motion in the two-dimensional image plane, the projection in the two-dimensional image plane forms motion. This kind of motion manifested as the flow of the brightness pattern in the image plane is called optical flow. The optical flow method is an important method for analyzing a sequence of motion images. Optical flow not only contains the motion information of the targets in the image, but also contains rich information about the three-dimensional physical structure. Therefore, it can be used to determine the operating conditions of the devices in the operation and maintenance scenario and reflect other status information.

[0043] Optical flow is the instantaneous velocity of the pixel motion of a spatially moving object on the observed imaging plane. The study of optical flow is to use the temporal variation and correlation of the pixel intensity data in the image sequence to determine the "motion" of each pixel position, that is, to study the relationship between the change of image grayscale over time and the structure and motion of the objects in the scene. Generally, optical flow is generated by camera motion, the motion of targets in the scene, or the combined motion of both.

[0044] Haptic feedback allows users to receive tactile information by applying force, vibration, or touch, and interact with objects in the virtual system through haptic simulation to achieve the feeling of touching real objects.

[0045] The haptic feedback is as follows: During the operation and maintenance process, if the operation and maintenance personnel encounter problems that cannot be solved, they initiate remote assistance through the communication function of the MR device, communicate one-on-one with remote technical experts through the virtual scene, and assist the operation and maintenance personnel in troubleshooting equipment problems through the visual interaction in the virtual scene. At the same time, the operation and maintenance personnel can transmit the on-site scene to the background through the remote transmission function of the MR device, and give intuitive assistance prompts, including selecting and marking equipment, tracking perspectives, and even file transmission; After the operation and maintenance is completed, the system stores the operation and maintenance date, operation and maintenance equipment, specific operation and maintenance content, list of operation and maintenance personnel, and guidance information in the corresponding data storage device to form an operation and maintenance record as an operation and maintenance ledger; The record of the ledger is conducive to being used as a reference basis during the later operation and maintenance process, and also provides data basis support for the summary and statistics of the historical costs of operation and maintenance.

[0046] Step 3: Use the sensing module to collect mine information, construct a "sensing - transmission - mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scene. Step 4: Detect the equipment operation status through various interaction methods of the MR device to achieve the operation and maintenance of the mixed reality mode.

[0047] Example 6 S1: Use the mine information to establish models for the operation and maintenance system and equipment.

[0048] For systems and equipment such as shearers, hydraulic supports, roadheaders, belt conveyors, pumps and valves, through on - site drawings, use the 3D modeling software SolidWorks to construct 1:1 refined models of the equipment such as shearers, hydraulic supports, roadheaders, belt conveyors, pumps and valves by points and lines. Refine the accuracy of each equipment model to the component level, and realize the virtual reality explosion diagram mode of the equipment through the translation and disassembly of the equipment models at the component level, and visually view the disassembly schematic display of various equipment. S2: Use the MR system engine to establish an MR operation and maintenance interaction scene and perform targeted optimized rendering according to the special underground environment.

[0049] Based on the mine construction information, collect the actual scene of the scene by means of camera real - shooting and CAD drawing conversion. Due to the special underground environment with dim light, targeted optimized rendering is required. Set the environmental light to match the underground light conditions, and add reflection and shadow effects on the surface of virtual objects to enhance the contrast and make it look more natural. Establish multiple operation and maintenance scenes including mine working faces, driving headings, transportation headings, substations, pump rooms, etc. through technologies such as unity. In the operation and maintenance scenes, according to various different equipment including shearers, hydraulic supports, scraper conveyors, etc. in the mine working face, develop system pages in the software system, set virtual panels for each equipment, and display the basic information of each equipment on the panel, such as the start - stop status, running direction, cutting height, speed, position and other data information of the shearer. At the same time, interact in the scene set through system development, and view the detailed information of specific equipment in the scene, including working condition information, equipment drawings, operation instructions, etc. through methods such as click interaction events set in the virtual scene.

[0050] S3: Use the sensing module to collect mine information, construct a "sensing - transmission - mapping" mode, access the working condition data of the mine system equipment, and project the data onto the virtual panel in the scene.

[0051] Among them, the sensing module matches communication protocols such as modbus, RS - 232, RS - 485, HART, MPI through various sensors on the equipment, and accesses different operation and maintenance scenes and equipment to obtain working condition data.

[0052] The transmission module classifies and filters the collected working condition data according to different policy modes. At the same time, the received time-series data is verified and de-duplicated using the wavelet transform method, cleaned and denoised (the wavelet transform method is a data denoising method based on signal processing principles. It decomposes the data into wavelet components of different frequencies, then selects appropriate wavelet components for filtering according to the characteristics of the noise, and finally synthesizes the filtered wavelet components to obtain the denoised data. The same data of the same device is screened to remove duplicate data, and the data is analyzed and processed). After data cleaning and filtering, the data is transmitted to the system background through the underground industrial ring network; The mapping module projects multiple data such as the current, temperature, pressure, position, angle, operating status, and communication status of each device obtained onto the MR virtual panel. Through the MR device, various data can be visually seen. At the same time, the system background will also perform intelligent analysis based on the working condition data, predict and judge the specific conditions of the devices, and accurately display the working condition data and operating status of each device over a period of time.

[0053] S4: Through the interaction method of the MR device, detect the operating status of the device, realize the operation and maintenance in the mixed reality mode. Directly touch and select the shearer interaction with gestures, the operating status of the shearer can be displayed, the start and stop of the device can be controlled, and voice interaction control can also be performed.

[0054] S5: During the operation and maintenance process, remote assistance can be initiated through the communication function of the MR device, communicate one-on-one with remote technical experts via video, and transmit the on-site scene to let the technical experts assist in solving equipment problems; S6: After the operation and maintenance is completed, the system stores information such as the operation and maintenance date, operation and maintenance equipment, specific operation and maintenance content, list of operation and maintenance personnel, and technical expert guidance opinions in the corresponding data storage device to form an operation and maintenance record as an operation and maintenance ledger.

Claims

1. A method for intelligent mine operation and maintenance based on mixed reality, characterized in that The implementation is specifically carried out in the following steps: Step 1: Utilize mine information to establish models for the operation and maintenance system and equipment; Step 2: Combine the MR system engine to establish an MR operation and maintenance interaction scenario, and perform targeted optimization rendering according to the special underground environment; Step 3: Use the sensing module to collect mine information, construct a "sensing - transmission - mapping" mode, access the actual working condition data of the mine system equipment, and project the data onto the virtual panel in the scenario; Step 4: Detect the equipment operation status through various interaction methods of the MR device to achieve operation and maintenance in the mixed reality mode.

2. The method for operation and maintenance of an intelligent mine based on mixed reality according to claim 1, wherein, The interaction methods in Step 2 include gesture recognition, voice control, object tracking, gaze tracking, and haptic feedback.

3. The method for intelligent mine operation and maintenance based on mixed reality according to claim 2, wherein The gesture recognition includes direct gesture recognition, non - direct gesture recognition, and tap recognition; Direct gesture means directly touching a spatial object, which belongs to a reachable interaction at close range. The user brings a finger close to the virtual object and touches the virtual object lightly; Non - direct gesture means interacting with the currently focused object in a long - distance situation. There are many ways to obtain focus. Commonly, it is by eye fixation, hand ray to lock an object or component in space to obtain focus. After the user focuses on an object or control, one or more finger movements can be used to quickly operate the object. Non - direct gestures will affect the currently focused object; Tap means directly selecting or activating an object, opening the associated menu in the MR system engine. Tap and drag: Move the object to a new position. Tap twice: Preview the object or file. Select words in an editing context. Flick: Display operations and controls; close the view; scroll. Pinch and drag with both hands: Zoom in or out. Pinch and drag along an arc with both hands: Rotate the object.

4. The method for intelligent mine operation and maintenance based on mixed reality according to claim 3, wherein The voice control includes speech recognition, speech synthesis, and control; Speech recognition collects the original audio data through a microphone and uses the MFCC and PLP methods to extract features such as the frequency, amplitude, and duration of the speech signal, and converts it into a digital signal. Specifically, perform a Fourier transform on each frame to obtain spectral information; then map the spectral information to the Mel scale to simulate the auditory perception of the human ear; then perform a discrete cosine transform on the spectrum on the Mel scale to obtain a set of Mel - frequency cepstral coefficients, and then process the feature sequence data based on the neural network model RNN. Through RNN, capture and analyze the temporal dependence relationship in the speech signal, analyze the semantics and context information of the speech signal, so as to achieve accurate speech recognition and understanding. Finally, use the dynamic programming algorithm to select the most likely result among the candidate words for speech recognition; Speech synthesis is the process of converting text information into speech signals. It is achieved by decomposing the text into phonemes, determining the pronunciation rules of the phonemes, and then synthesizing the speech signals. It includes three steps: text processing, phoneme conversion, and voice synthesis. Text processing is the process of preprocessing and analyzing the input text, including operations such as word segmentation, part-of-speech tagging, and syntactic analysis of the text, in order to more accurately determine the phonemes and pronunciation rules. Phoneme conversion is the process of corresponding the letters and phonetic symbols in the text. By looking up the phoneme library or using phoneme conversion rules, the letters in the text are converted into the corresponding phonemes for subsequent voice synthesis. Voice synthesis is the process of combining phonemes into speech signals. By concatenating the waveforms of phonemes or using synthesis rules, natural and fluent speech signals are generated. Voice synthesis uses methods based on concatenation, methods based on parameters, or hybrid methods; Voice control is the process of controlling a device or performing a specific task through voice commands. It is achieved through speech recognition and speech synthesis, and includes three stages: voice input, voice processing, and voice output. Voice input is the process by which a user inputs a speech signal into a device by speaking. The device converts the speech signal into text or commands, and performs semantic analysis and command matching to determine the user's intention. Voice processing is the process of parsing and processing the user's voice input, including operations such as speech recognition, semantic analysis, and command matching, in order to convert the user's voice command into an instruction that the device can understand and execute. Voice output is the process of converting the device's response into a speech signal through speech synthesis. The device performs corresponding operations according to the user's instructions, and converts the execution result into a speech signal through speech synthesis for feedback to the user.

5. The method for intelligent mine operation and maintenance based on mixed reality according to claim 4, wherein The object tracking is to continuously track the target in time by utilizing the correlation between consecutive frames. The basic steps include target initial positioning, target template update, and target state prediction; Object tracking locates the target by using the target bounding box in the initial frame. Then, in subsequent frames, the algorithm uses the target template to match each frame to determine the position of the target in the current frame. Object tracking based on a correlation filter obtains the position of the target by performing a correlation operation between the target template and each frame.

6. The method for intelligent mine operation and maintenance based on mixed reality according to claim 5, wherein, The gaze tracking uses an optical flow tracking algorithm. When the targets and devices in the operation and maintenance scenario correspond to the movement in the two-dimensional image plane, the projection in the two-dimensional image plane forms a movement. This movement manifested as the flow of the brightness pattern in the image plane is called optical flow, which is used to determine the operation status of the devices in the operation and maintenance scenario and reflect other status information; Optical flow is the instantaneous velocity of the pixel movement of a spatially moving object on the observed imaging plane, studying the relationship between the change of image grayscale over time and the structure and movement of objects in the scene. Optical flow is generated by camera movement, the movement of targets in the scene, or the combined movement of both.

7. The method for operation and maintenance of an intelligent mine based on mixed reality according to claim 6, wherein The haptic feedback allows users to receive haptic information by applying force, vibration, or touch, and interact with objects in the virtual system through haptic simulation to achieve the feeling of touching real objects.

8. The method for intelligent mine operation and maintenance based on mixed reality according to claim 7, characterized in that, The haptic feedback is specifically as follows: During the operation and maintenance process, if the operation and maintenance personnel encounter problems that cannot be solved, they initiate remote assistance through the communication function of the MR device, communicate one-on-one with remote technical experts in a virtual scenario, and assist the operation and maintenance personnel in troubleshooting equipment problems through visual interaction in the virtual scenario. At the same time, the operation and maintenance personnel can transmit the on-site scenario to the background through the remote transmission function of the MR device, and give intuitive assistance prompts, including clicking on and marking equipment, tracking perspectives, and even file transmission; After the operation and maintenance is completed, the system stores the operation and maintenance date, operation and maintenance equipment, specific operation and maintenance content, list of operation and maintenance personnel, and guidance information in the corresponding data storage device to form an operation and maintenance record as an operation and maintenance ledger; The record of the ledger is conducive to serving as a reference basis during the later operation and maintenance process, and also provides data basis support for the summary and statistics of the historical cost of operation and maintenance.

Images