Heading machine operation process identification method based on multi-sensor fusion and digital twinning

Abstract

The invention relates to a heading machine operation process identification method based on multi-sensor fusion and digital twinning, and belongs to the technical field of coal mine intellectualization. Comprising the following steps: acquiring and fusing an infrared image and a visible light image currently acquired by an infrared camera and a camera; inputting the fused image into a pre-trained operation process identification network, and outputting a current first operation process of the heading machine by the operation process identification network; acquiring sensing data currently acquired by a sensor mounted on the body of the heading machine; inputting the sensing data into a pre-established digital twinning model of the heading machine operation, and determining a current second operation process of the heading machine by the digital twinning model of the heading machine operation; and evaluating the current operation process of the heading machine according to the first operation process and the second operation process. According to the invention, based on the data collected by a plurality of sensors, the working procedure identification network and the heading machine working digital twin model cooperate with each other and complement each other, so that the precision, accuracy and integrity of the identification result can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent coal mine technology, and in particular to a method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twins. Background Art

[0002] In order to achieve safe and green development and clean and efficient utilization of coal resources, it is necessary to accelerate the construction of intelligent coal mines, deeply integrate advanced algorithms such as machine vision and artificial intelligence with tunneling technology, realize intelligent perception of tunneling scenes and standardized operation of tunneling machines, and improve tunneling efficiency. Due to interference factors such as noisy environment, coal dust and water mist, low illumination and large changes in lighting brightness in coal mines, the research difficulty of intelligent key technologies such as tunneling machine process identification based on coal mine monitoring video streams has increased.

[0003] Most existing technologies for identifying the working process of a roadheader on a tunneling working face use a single machine vision technology to identify the working process. However, in the dark, dusty, and misty environment of a coal mine, this machine vision-based roadheader working process identification technology has weak adaptability, low recognition effect, and is prone to errors during the recognition process, affecting the accuracy of the recognition results. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for identifying the operation process of a roadheader based on multi-sensor fusion and digital twin. The technical solution of the present invention is as follows:

[0005] A method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twin, comprising:

[0006] S1, obtaining an infrared image and a visible light image currently captured by an infrared camera and a camera, wherein the infrared camera and the camera are pre-installed on the top plate of the tunnel behind the tunnel boring machine;

[0007] S2, fusing the infrared image and the visible light image to obtain a fused image;

[0008] S3, inputting the fused image into a pre-trained operation process recognition network, and the operation process recognition network outputs the current first operation process of the tunnel boring machine;

[0009] S4, obtaining sensor data currently collected by sensors installed on the tunnel boring machine body;

[0010] S5, inputting the sensor data into a pre-established digital twin model of the tunnel boring machine operation, and using the digital twin model of the tunnel boring machine operation to determine the current second operation process of the tunnel boring machine;

[0011] S6, evaluating the current operation process of the tunnel boring machine according to the first operation process and the second operation process.

[0012] Optionally, when S2 is implemented, it includes:

[0013] S21, inputting the 224×224×1 infrared image into the CNN encoder for encoding processing to obtain an infrared image block, which is represented as a 256×14×14×1 tensor; inputting the 224×224×3 visible light image into the CNN encoder for encoding processing to obtain a visible light image block, which is represented as a 256×14×14×3 tensor;

[0014] S22, stretching the infrared image block and the visible light image block into 256×196 vectors respectively, and adding the position information of each infrared image block and each visible light image block to the stretched vectors of each infrared image block and each visible light image block respectively;

[0015] S23, inputting the 256×196 vectors of the infrared image block and the visible light image block after adding the position information into the interactive attention module, and the interactive attention module fuses them to obtain a fusion result;

[0016] S24, inputting the fusion result into the CNN decoder for decoding processing to obtain a fused image.

[0017] Optionally, the operation process identification network is an improved YOLOv5 model, a CA attention module is inserted into the backbone network of the improved YOLOv5 model, and the CA attention module is inserted between the C3-1 module and the SPPF module of the backbone network.

[0018] Optionally, the CA attention module, when processing its input feature map, includes:

[0019] S31, for the input feature map of size C×H×W, perform average pooling operations in the X direction and the Y direction respectively to generate a feature map of size C×H×1 and a feature map of size C×1×W respectively;

[0020] S32, performing a transformation operation on the C×1×W feature map to extract position information therein;

[0021] S33, concatenating the C×H×1 feature map and the transformed C×1×W feature map to fuse the position information with the channel information to obtain a fused feature map;

[0022] S34, processing the fused feature map by convolution and nonlinear activation function in sequence to obtain an intermediate feature map;

[0023] S35, splitting the intermediate feature map into two spatial vectors along the spatial dimension, and performing a dimensionality increase operation on both spatial vectors;

[0024] S36, activate the two space vectors after dimension increase through the sigmoid activation function to obtain two attention vectors;

[0025] S37, multiply the two attention vectors and multiply them by the input feature map to obtain the output feature map of the CA attention module.

[0026] Optionally, the operating procedures identified by the operating procedure identification network include one of the cutting arm lifting and rotation, the forward and backward movement of the tunnel boring machine, temporary support and permanent support; the operating procedures determined by the tunnel boring machine operation digital twin model include one of the cutting arm lifting and rotation, the tunnel boring machine forward and backward movement, the shovel board lifting and the rear support lifting.

[0027] Optionally, when S6 is implemented, it includes:

[0028] S61, when the first operation process and the second operation process are the same, determining that the current operation process of the tunnel boring machine is the first operation process, displaying the current operation process of the tunnel boring machine in text form in a first area of ​​the display, and dynamically displaying the current operation process of the tunnel boring machine in a second area of ​​the display through a tunnel boring machine operation digital twin model;

[0029] S62, when the first operation process and the second operation process are different, a prompt message for prompting manual participation in determining the current operation process of the tunnel boring machine is popped up on the display.

[0030] Optionally, the S6, when implemented, includes the following situations:

[0031] The first case: when the first operation process and the second operation process are both the cutting arm lifting and rotating, the current operation process of the tunnel boring machine is determined to be the cutting arm lifting and rotating; when the first operation process and the second operation process are both the tunnel boring machine moving forward and backward, the current operation process of the tunnel boring machine is determined to be the tunnel boring machine moving forward and backward; the current operation process of the tunnel boring machine is displayed in text form in the first area of ​​the display, and the current operation process of the tunnel boring machine is dynamically displayed in the second area of ​​the display through the tunnel boring machine operation digital twin model;

[0032] The second situation: when the first operation process is the cutting arm lifting and rotating or the tunnel boring machine moving forward and backward, and the second operation process is the tunnel boring machine moving forward and backward or the cutting arm lifting and rotating, a warning message for indicating sensor failure will pop up on the display;

[0033] The third case: when the first operation process is temporary support or permanent support and the second operation process is blank, the current operation process of the roadheader is determined to be temporary support or permanent support, and the temporary support or permanent support is displayed in text form in the first area of ​​the display, and the temporary support or permanent support is displayed in the second area of ​​the display through the digital twin model of the roadheader operation;

[0034] The fourth case: when the first operating procedure is blank and the second operating procedure is shovel lifting or rear support lifting, the current operating procedure of the tunnel boring machine is determined to be shovel lifting or rear support lifting, and the shovel lifting or rear support lifting is displayed in text form in the first area of ​​the display, and the shovel lifting or rear support lifting is dynamically displayed in the second area of ​​the display through the digital twin model of the tunnel boring machine operation.

[0035] All the above optional technical solutions can be combined arbitrarily, and the present invention does not provide detailed descriptions of the structures after the combinations.

[0036] By means of the above scheme, the beneficial effects of the present invention are as follows:

[0037] Based on the combination of data collected by cameras, infrared imagers and tunnel boring machine body sensors, and combined with the comprehensive recognition results of the operation process recognition network and the tunnel boring machine operation digital twin model, the current operation process of the tunnel boring machine is evaluated, and the operation process is identified based on data collected by multiple sensors. The operation process recognition network and the tunnel boring machine operation digital twin model cooperate and complement each other, which can reduce the impact of various harsh environments such as underground dust and increase the precision, accuracy and completeness of the recognition results.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a method for identifying the operation process of a tunnel boring machine based on multi-sensor fusion and digital twin provided in an embodiment of the present invention.

[0040] Figure 2 It is a flow chart of fusing infrared image and visible light image in an embodiment of the present invention.

[0041] Figure 3 It is a schematic diagram of the composition structure of the improved YOLOv5 model in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0043] The method for identifying the operation process of a roadheader based on multi-sensor fusion and digital twin provided in the embodiment of the present invention can be implemented by any electronic device with computing function, such as a PC, a mobile terminal or a server. The electronic device can be installed on the roadheader body or in the central control room of a coal mine. Figure 1 As shown, the method for identifying the operation process of a roadheader based on multi-sensor fusion and digital twin provided by an embodiment of the present invention includes the following steps S1 to S6:

[0044] S1, obtaining an infrared image and a visible light image currently collected by an infrared camera and a camera, wherein the infrared camera and the camera are pre-installed on a tunnel roof behind a tunnel boring machine.

[0045] It should be noted that before executing the method provided in the embodiment of the present invention, it is necessary to first install two types of sensors, an infrared camera and a camera, in the working scene of the underground tunneling machine in the coal mine, and these two types of sensors need to maintain the same angle to ensure the spatial uniformity of the two sensors and to ensure that both can collect the operating data of the tunneling machine at the same angle. Specifically, when installing the infrared camera and the camera, they can be installed on the top plate of the tunnel ten meters behind the tunneling machine. The infrared camera and the camera are both electrically connected to the electronic device. After the infrared camera and the camera collect the infrared image and the visible light image respectively, they are sent to the electronic device. The electronic device obtains the infrared image and the visible light image currently collected by the infrared camera and the camera by receiving the data sent by the infrared camera and the camera.

[0046] Furthermore, to facilitate subsequent image fusion, it is also necessary to ensure that the infrared image and visible light image currently captured by the infrared camera and the camera are at the same time, and the data captured by the camera is often the video data within the current sampling period. Therefore, after obtaining the video data captured by the camera, it is necessary to frame the video data and extract a frame of the image with the same timestamp as the infrared image as the visible light image.

[0047] S2, fuse the infrared image and the visible light image to obtain a fused image.

[0048] To facilitate image fusion, the infrared image and the visible light image can be processed into the same size first, for example, the infrared image is processed into 224×224×1 (length×width×grayscale), and the visible light image is processed into 224×224×3 (length×width×RGB).

[0049] Based on the above content, if Figure 2 As shown, the S2, when implemented specifically, includes:

[0050] S21, input the 224×224×1 infrared image into the CNN encoder for encoding processing to obtain an infrared image block, which is represented as a 256×14×14×1 tensor, which represents 256 vectors whose length, width and number of channels are 14, 14 and 1 respectively; input the 224×224×3 visible light image into the CNN encoder for encoding processing to obtain a visible light image block, which is represented as a 256×14×14×3 tensor, which represents 256 vectors whose length, width and number of channels are 14, 14 and 1 respectively.

[0051] S22, stretch the infrared image block and the visible light image block into 256×196 vectors respectively, and add the position information of each infrared image block and each visible light image block to the stretched vectors of each infrared image block and each visible light image block respectively. After adding the position information, the vector size is still 256×196.

[0052] For example, if the vector of the stretched infrared image block is represented by V ir,i , the position information of the infrared image block is represented as p i , then add the position information to the vector after the infrared image block is stretched and represent it as V ir,i +p i .

[0053] S23, input the 256×196 vectors of the infrared image block and the visible light image block after adding the position information into the interactive attention module, and the interactive attention module fuses them to obtain a fusion result.

[0054] The specific operation method of the interactive attention module for fusion is not elaborated in detail in the embodiment of the present invention. For details, please refer to the data processing method of the existing interactive attention module.

[0055] S24, inputting the fusion result into the CNN decoder for decoding processing to obtain a fused image.

[0056] S3, inputting the fused image into a pre-trained operation process recognition network, and the operation process recognition network outputs the current first operation process of the tunnel boring machine.

[0057] The embodiment of the present invention does not specifically limit the specific composition structure of the operation process identification network, and a conventional classification network can be used.

[0058] In a specific embodiment, since the YOLOv5 network structure focuses on the fusion of deep features, as the number of network layers increases and the downsampling operation is implemented, the detailed information is gradually lost, resulting in the information obtained at the output end becoming more and more abstract. This process particularly affects the recognition of small targets, because their features are easily lost in large quantities during the transmission process, thereby weakening the model's detection performance for small targets. In order to compensate for this deficiency, the job process recognition network in the embodiment of the present invention integrates an attention mechanism into the YOLOv5 model to significantly improve the job process recognition network's ability to capture key features, especially to enhance the detection accuracy of small targets. Therefore, the job process recognition network in the embodiment of the present invention adopts an improved YOLOv5 model. Figure 3 As shown, the improved YOLOv5 model includes a backbone network, a neck network and a detection head connected in sequence, and a CA attention module is inserted in the backbone network, and the CA attention module is inserted between the C3-1 module and the SPPF module of the backbone network.

[0059] In a specific embodiment, the CA attention module, when processing its input feature map, includes:

[0060] S31, for the input feature map of size C×H×W, average pooling operations are performed in the X direction and the Y direction respectively to generate a feature map of size C×H×1 and a feature map of size C×1×W respectively.

[0061] That is, in order to capture accurate position information in long-distance spatial interactions, the embodiment of the present invention decomposes the GAP (global average pooling) operation (performing average pooling operations in the X direction and the Y direction respectively). Through this decomposition, efficient capture and utilization of remote spatial features can be achieved without sacrificing spatial resolution.

[0062] S32, performing a transformation operation on the C×1×W feature map to extract the position information therein.

[0063] S33, concatenating the C×H×1 feature map and the transformed C×1×W feature map to fuse the position information with the channel information to obtain a fused feature map.

[0064] By fusing the position information with the channel information, the fused feature map can be made into a long-range spatial interaction feature with precise position information. The size of the fused feature map is C / r×1×(W+H). r represents the dimensionality reduction ratio during splicing, which is used to reduce the number of channels to reduce computational complexity and prevent overfitting. C / r represents the number of channels after dimensionality reduction. Reducing the number of channels can reduce the computational cost of subsequent operations.

[0065] S34, processing the fused feature map through convolution and nonlinear activation function in sequence to obtain an intermediate feature map.

[0066] The nonlinear activation function includes but is not limited to the ReLU function. The size of the intermediate feature map is C / r×1×(W+H).

[0067] S35, split the intermediate feature map into two spatial vectors along the spatial dimension, and perform a dimensionality increase operation on both spatial vectors.

[0068] When performing the dimension-up operation, it can be implemented through a convolutional neural network, and the convolutional neural network can use Conv2d. The sizes of the two space vectors after dimension-up are C×H×1 and C×1×W respectively.

[0069] S36, activate the two space vectors after dimensionality increase through the sigmoid activation function to obtain two attention vectors.

[0070] The sizes of the two attention vectors are C×H×1 and C×1×W respectively. These two attention vectors can be used to weightedly fuse channel information and position information, thereby enhancing the ability to detect small targets.

[0071] S37, multiply the two attention vectors and multiply them by the input feature map to obtain the output feature map of the CA attention module.

[0072] In summary, in an embodiment of the present invention, the CA attention module divides the channel attention into two independent parts, and performs feature decomposition and aggregation along two different spatial dimensions in the X direction and the Y direction respectively. This method effectively integrates the position information into the channel attention, realizes the efficient integration of channel information and position information, and can effectively enhance the key channel and spatial feature information in the feature representation. Through this improved attention mechanism, the loss of small target features in the feature extraction process can be reduced, which helps the model focus on more delicate and local features, thereby correspondingly improving the ability to accurately identify and precisely locate small targets. At the same time, the improved YOLOv5 model can also maintain the fast reasoning characteristics of the YOLOv5 model, and can achieve faster detection performance in complex environments.

[0073] It should be noted that before the fused image is input into the pre-trained operation process recognition network, the operation process recognition network needs to be trained first. Specifically, when training the operation process recognition network, it can be achieved in the following ways:

[0074] First, obtain the visible light images and infrared images collected historically in the coal mine. The infrared images and visible light images obtained should be data from the operation of the tunnel boring machine in various different operation processes to ensure that the data set has sufficient diversity and representativeness, and the infrared images and visible light images should be a one-to-one image group aligned in time and space. Then, the one-to-one corresponding infrared images and visible light images are fused to obtain a fused image set. Then, the labelling tool Labelmg is used to manually annotate the fused image set. The annotation content is the bounding box and category of the object when the cutting arm of the tunnel boring machine is lifted and rotated, the tunnel boring machine moves forward and backward, temporary support and permanent support, and the annotation information is saved in the YOLO format required by YOLOv5. Then, specify the path of the fused image set, the number of categories and other parameters in the configuration file, and adjust the parameters of the improved YOLOv5 model as needed, such as the learning rate, batch size, number of iterations, etc. Next, the improved YOLOv5 model is trained by the annotated fusion image set, and the training script of the improved YOLOv5 model is run. During the training process, the weights of the improved YOLOv5 model are saved regularly. After the training is completed, the optimal weights are loaded for subsequent evaluation or reasoning. Then, one or more sets of fusion image sets are re-collected, the trained improved YOLOv5 model is verified, and the performance of the improved YOLOv5 model is tested to ensure that the model has good generalization ability. Finally, the trained improved YOLOv5 model is deployed on the electronic equipment, and the subsequent roadheader operation process identification is carried out.

[0075] In addition, after the improved YOLOv5 model has been applied for a period of time, the improved YOLOv5 model can be updated based on the newly updated infrared images, visible light images and actual operation procedures in the coal mine, thereby improving the accuracy of the improved YOLOv5 model.

[0076] Among them, based on the installation positions of the infrared camera and the camera, the operation process recognition network can recognize the operation processes including the lifting and rotating of the cutting arm, the forward and backward movement of the tunnel boring machine, temporary support and permanent support.

[0077] S4, obtaining the sensor data currently collected by the sensor installed on the tunnel boring machine body.

[0078] The type of sensor installed on the machine body is not specifically limited in the embodiment of the present invention, and the operating process identification of the machine body can be completed based on the sensors installed on the machine body. Specifically, these sensors may include position sensors, pressure sensors, oil temperature sensors, speed sensors, displacement sensors, and ammeters. Position sensors, speed sensors, and displacement sensors can be installed on the cutting arm, shovel board, and rear support of the machine body. Ammeters and oil temperature sensors can be installed on each oil cylinder of the machine body.

[0079] Furthermore, these sensors are electrically connected to the electronic device. After collecting the data, these sensors send the sensing data to the electronic device, so that the electronic device can obtain the sensing data in real time.

[0080] S5, input the sensor data into the pre-established digital twin model of the tunnel boring machine operation, and the digital twin model of the tunnel boring machine operation determines the current second operation process of the tunnel boring machine.

[0081] It should be noted that before executing this step, a digital twin model of the tunnel boring machine operation needs to be established. Specifically, when establishing a digital twin model of the tunnel boring machine operation, it can be achieved through the following steps:

[0082] (1) Technology selection:

[0083] Modeling tool: Maya;

[0084] Programming language: C#;

[0085] Data interaction: WebSocket real-time communication protocol;

[0086] Physics engine: Unity's built-in physics engine.

[0087] (2) Establish a digital twin model of tunnel boring machine operation.

[0088] In Unity, a 3D model of a roadheader was created, including key components such as blade, rear support, cutting arm, conveyor belt, etc. Each component was simulated through Unity's physics engine to achieve realistic physical behavior.

[0089] (3) Action implementation:

[0090] 3.1. Shovel lifting:

[0091] Functional description: The shovel is used to load and unload materials. In the digital twin model of the tunnel boring machine operation, the lifting and lowering action of the shovel is achieved by controlling the rotation position of the model.

[0092] Implementation method: Create a C# script to change the position attribute of the shovel model to achieve the lifting action, and use linear interpolation (Lerp) to achieve a smooth transition.

[0093] 3.2. Rear support lifting:

[0094] Functional description: The rear support is used to support the tunnel boring machine during excavation to prevent it from tipping over. In the digital twin model of tunnel boring machine operation, the lifting and lowering of the rear support is also achieved by controlling its rotation position.

[0095] Implementation method: Similar to the lifting of the shovel board, create a C# script to achieve the lifting action by changing the position properties of the rear support model.

[0096] 3.3. Cutting arm lifting and rotation:

[0097] Function description: Control the up, down, left and right movement of the cutting arm. In the digital twin model of the tunnel boring machine operation, the lifting and rotation of the cutting arm is also achieved by controlling the position of its rotation axis.

[0098] Implementation method: Similar to the lifting of the shovel board, create a C# script to realize the lifting action by changing the coordination of each model associated with the cutting arm.

[0099] 3.4. Roadheader moves forward and backward:

[0100] Functional description: The forward and backward movements of the tunnel boring machine are achieved by controlling its overall position.

[0101] Implementation method: Create a C# script to change the position property of the tunnel boring machine model to achieve forward and backward actions. Slerp (spherical interpolation) can be used to achieve smooth transitions. At the same time, a steering action can be added to simulate the steering process of the tunnel boring machine.

[0102] The digital twin model of the tunnel boring machine operation established through the above steps has the real-time monitoring function of the digital twin of the tunnel boring machine, and transmits the actual operation data of the tunnel boring machine to the digital twin model of the tunnel boring machine operation in real time. The digital twin model of the tunnel boring machine operation can determine and dynamically display the operation process of the tunnel boring machine. Specifically, after obtaining the sensor data, the sensor data is transmitted to the digital twin model of the tunnel boring machine operation through WebSocket or other real-time communication protocols, and the digital twin model of the tunnel boring machine operation determines and displays the operation process of the tunnel boring machine.

[0103] In addition, after the digital twin model of the tunnel boring machine operation has been used for a period of time, the digital twin model of the tunnel boring machine operation can be updated based on the newly updated fuselage sensor data and actual operation procedures underground in the coal mine, thereby improving the accuracy of the digital twin model of the tunnel boring machine operation.

[0104] Based on the establishment process of the above-mentioned digital twin model of the tunnel boring machine operation, the digital twin model of the tunnel boring machine operation can at least determine the operating procedures including the lifting and rotation of the cutting arm, the forward and backward movement of the tunnel boring machine, the lifting and lowering of the shovel plate and the lifting and lowering of the rear support.

[0105] S6, describing the current operation process of the tunnel boring machine according to the first operation process and the second operation process.

[0106] In a specific embodiment, the S6, when implemented, includes:

[0107] S61, when the first operation process and the second operation process are the same, determine that the current operation process of the tunnel boring machine is the first operation process (or the second operation process), display the current operation process of the tunnel boring machine in text form in the first area of ​​the display, and dynamically display the current operation process of the tunnel boring machine in the second area of ​​the display through the tunnel boring machine operation digital twin model.

[0108] Among them, the lifting and rotating of the cutting arm and the forward and backward movement of the tunnel boring machine are the operating procedures that can be identified by both the operating procedure identification network and the digital twin model of the tunnel boring machine operation. For these two operating procedures, if the first operating procedure and the second operating procedure are the same, then the current operating procedure of the tunnel boring machine is determined to be the operating procedure identified by the two. The current operating procedure of the tunnel boring machine obtained by combining the two methods can increase the accuracy and credibility of the operating procedure identification.

[0109] By displaying the current operation process of the tunnel boring machine in different ways in different areas of the display, the operator can more intuitively obtain the current operation process of the tunnel boring machine. In the second area of ​​the display, the current operation process of the tunnel boring machine is dynamically displayed through the digital twin model of the tunnel boring machine operation, making the display effect of the current operation process of the tunnel boring machine more vivid.

[0110] S62, when the first operation process and the second operation process are different, a prompt message for prompting manual participation in determining the current operation process of the tunnel boring machine is popped up on the display.

[0111] Specifically, when the first operation process and the second operation process are different, it may be due to an abnormality in the sensor, infrared camera or camera installed on the machine body. In order to accurately determine the current operation process of the machine, a prompt message pops up on the display, and the operator evaluates the current operation process of the machine based on the prompt message. The underground equipment is checked for problems to ensure that the system can accurately identify the current operation process of the machine.

[0112] It should be noted that, generally speaking, the recognition results obtained by the fuselage sensor are more accurate and comprehensive. However, due to the complex and changeable mining environment of the mine, the fuselage sensor is easily damaged, which in turn leads to errors in the recognition results. Therefore, when there is a problem with the fuselage sensor and it cannot be repaired immediately, data can be collected through the camera and infrared imager. After the fused image is obtained through the fusion algorithm, the fused image is passed into the improved YOLOv5 model to identify the current operation process of the tunnel boring machine, ensuring that the system can accurately identify the current operation process.

[0113] More specifically, the S6, when implemented, includes the following situations:

[0114] The first case: when the first operation process and the second operation process are both the cutting arm lifting and rotation, the current operation process of the tunnel boring machine is determined to be the cutting arm lifting and rotation; when the first operation process and the second operation process are both the tunnel boring machine moving forward and backward, the current operation process of the tunnel boring machine is determined to be the tunnel boring machine moving forward and backward; the current operation process of the tunnel boring machine is displayed in text form in the first area of ​​the display, and the current operation process of the tunnel boring machine is dynamically displayed in the second area of ​​the display through the tunnel boring machine operation digital twin model.

[0115] The second situation: when the first operation process is the lifting and rotation of the cutting arm or the forward and backward movement of the tunnel boring machine, and the second operation process is the forward and backward movement of the tunnel boring machine or the lifting and rotation of the cutting arm, a warning message indicating a sensor failure will pop up on the display.

[0116] That is to say, when the first operating process is the lifting and rotating of the cutting arm, and the second operating process is the forward and backward movement of the tunnel boring machine, or when the first operating process is the forward and backward movement of the tunnel boring machine, and the second operating process is the lifting and rotating of the cutting arm, it means that the recognition results of the operating process identification network and the tunnel boring machine operation digital twin model are inconsistent. This situation is often caused by sensor failure. Therefore, the embodiment of the present invention pops up a warning message of sensor failure on the display, indicating that the sensor, infrared camera or camera installed on the tunnel boring machine body has an abnormality and requires manual intervention to maintain the underground equipment in time.

[0117] The third case: when the first operation process is temporary support or permanent support and the second operation process remains blank, the current operation process of the tunnel boring machine is determined to be temporary support or permanent support, and the temporary support or permanent support is displayed in text form in the first area of ​​the display, and the temporary support or permanent support is displayed in the second area of ​​the display through the digital twin model of the tunnel boring machine operation.

[0118] Specifically, temporary support and permanent support are the operating procedures that can be identified by the operating procedure identification network, but the digital twin model of the tunnel boring machine operation cannot identify these two operating procedures. When the current operating procedure of the tunnel boring machine is temporary support or permanent support, under normal circumstances, the second operating procedure determined by the digital twin model of the tunnel boring machine operation should be blank. In this case, the first operating procedure is used as the current operating procedure of the tunnel boring machine. In addition, the electronic device sends the operating procedure identification result of temporary support or permanent support to the digital twin model of the tunnel boring machine operation, and drives the digital twin model of the tunnel boring machine operation to display temporary support or permanent support.

[0119] The fourth case: when the first operating procedure is blank and the second operating procedure is shovel lifting or rear support lifting, the current operating procedure of the tunnel boring machine is determined to be shovel lifting or rear support lifting, and the shovel lifting or rear support lifting is displayed in text form in the first area of ​​the display, and the shovel lifting or rear support lifting is dynamically displayed in the second area of ​​the display through the digital twin model of the tunnel boring machine operation.

[0120] Specifically, blade lifting and rear support lifting are operating procedures that can be identified by the digital twin model of the tunnel boring machine operation, but the operating procedure recognition network cannot recognize these two operating procedures. When the current operating procedure of the tunnel boring machine is blade lifting or rear support lifting, under normal circumstances, the first operating procedure recognized by the operating procedure recognition network should be blank. In this case, the second operating procedure is used as the current operating procedure of the tunnel boring machine. In addition, the electronic device controls the display of blade lifting or rear support lifting in text form in the first area of ​​the display according to the blade lifting or rear support lifting recognized by the digital twin model of the tunnel boring machine operation. By dynamically displaying the blade lifting or rear support lifting in the second area of ​​the display through the digital twin model of the tunnel boring machine operation, a vivid and vivid display of the blade lifting or rear support lifting is achieved.

[0121] Furthermore, when the current operating procedure of the tunnel boring machine is an operating procedure identified by the operating procedure identification network, when the current operating procedure of the tunnel boring machine is displayed in text form in the first area of ​​the display, a fused image can also be displayed near the text to facilitate personnel to intuitively obtain the current operating procedure of the tunnel boring machine.

[0122] The method provided by the embodiment of the present invention has the following characteristics:

[0123] 1. Based on the combination of data collected by cameras, infrared imagers and sensors on the tunnel boring machine body, the current operation process of the tunnel boring machine is evaluated in combination with the comprehensive recognition results of the operation process recognition network and the tunnel boring machine operation digital twin model. The two can cooperate with each other and complement each other, which can reduce the impact of various harsh environments such as underground dust and increase the precision, accuracy and completeness of the recognition results.

[0124] 2. The use of the operation process recognition network can quickly and efficiently obtain recognition results, and the auxiliary tunnel boring machine operation digital twin model can quickly display the current operation process of the tunnel boring machine.

[0125] 3. The powerful rendering and interactive capabilities of the Unity engine enable high visualization and real-time response of the tunnel boring machine operation process, making the method provided in the embodiment of the present invention an intuitive, efficient and technologically advanced monitoring and control technology in industrial scenarios.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for identifying the operation process of a roadheader based on multi-sensor fusion and digital twin, characterized in that: include: S1, obtaining an infrared image and a visible light image currently captured by an infrared camera and a camera, wherein the infrared camera and the camera are pre-installed on the top plate of the tunnel behind the tunnel boring machine; S2, fusing the infrared image and the visible light image to obtain a fused image; S3, inputting the fused image into a pre-trained operation process recognition network, and the operation process recognition network outputs the current first operation process of the tunnel boring machine; S4, obtaining sensor data currently collected by sensors installed on the tunnel boring machine body; S5, inputting the sensor data into a pre-established digital twin model of the tunnel boring machine operation, and using the digital twin model of the tunnel boring machine operation to determine the current second operation process of the tunnel boring machine; S6, evaluating the current operation process of the tunnel boring machine according to the first operation process and the second operation process.

2. The method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twin according to claim 1 is characterized in that: When S2 is specifically implemented, it includes: S21, inputting the 224×224×1 infrared image into the CNN encoder for encoding processing to obtain an infrared image block, which is represented as a 256×14×14×1 tensor; inputting the 224×224×3 visible light image into the CNN encoder for encoding processing to obtain a visible light image block, which is represented as a 256×14×14×3 tensor; S22, stretching the infrared image block and the visible light image block into 256×196 vectors respectively, and adding the position information of each infrared image block and each visible light image block to the stretched vectors of each infrared image block and each visible light image block respectively; S23, inputting the 256×196 vectors of the infrared image block and the visible light image block after adding the position information into the interactive attention module, and the interactive attention module fuses them to obtain a fusion result; S24, inputting the fusion result into the CNN decoder for decoding processing to obtain a fused image.

3. The method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twin according to claim 1, characterized in that: The operation process identification network is an improved YOLOv5 model, and a CA attention module is inserted into the backbone network of the improved YOLOv5 model, and the CA attention module is inserted between the C3-1 module and the SPPF module of the backbone network.

4. The method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twin according to claim 3 is characterized in that: The CA attention module, when processing its input feature map, includes: S31, for the input feature map of size C×H×W, perform average pooling operations in the X direction and the Y direction respectively to generate a feature map of size C×H×1 and a feature map of size C×1×W respectively; S32, performing a transformation operation on the C×1×W feature map to extract position information therein; S33, concatenating the C×H×1 feature map and the transformed C×1×W feature map to fuse the position information with the channel information to obtain a fused feature map; S34, processing the fused feature map by convolution and nonlinear activation function in sequence to obtain an intermediate feature map; S35, splitting the intermediate feature map into two spatial vectors along the spatial dimension, and performing a dimensionality increase operation on both spatial vectors; S36, activate the two space vectors after dimension increase through the sigmoid activation function to obtain two attention vectors; S37, multiply the two attention vectors and multiply them by the input feature map to obtain the output feature map of the CA attention module.

5. The method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twin according to claim 1, characterized in that: The operating procedures identified by the operating procedure identification network include one of the cutting arm lifting and rotation, the forward and backward movement of the tunnel boring machine, temporary support and permanent support; the operating procedures determined by the tunnel boring machine operation digital twin model include one of the cutting arm lifting and rotation, the forward and backward movement of the tunnel boring machine, the shovel board lifting and the rear support lifting.

6. The method for identifying the operation process of a roadheader based on multi-sensor fusion and digital twin according to claim 1 or 5, characterized in that: When S6 is specifically implemented, it includes: S61, when the first operation process and the second operation process are the same, determining that the current operation process of the tunnel boring machine is the first operation process, displaying the current operation process of the tunnel boring machine in text form in a first area of ​​the display, and dynamically displaying the current operation process of the tunnel boring machine in a second area of ​​the display through a tunnel boring machine operation digital twin model; S62, when the first operation process and the second operation process are different, a prompt message for prompting manual participation in determining the current operation process of the tunnel boring machine is popped up on the display.

7. The method for identifying operation procedures of a roadheader based on multi-sensor fusion and digital twin according to claim 5, characterized in that: The S6, when implemented specifically, includes the following situations: The first case: when the first operation process and the second operation process are both the cutting arm lifting and rotating, the current operation process of the tunnel boring machine is determined to be the cutting arm lifting and rotating; when the first operation process and the second operation process are both the tunnel boring machine moving forward and backward, the current operation process of the tunnel boring machine is determined to be the tunnel boring machine moving forward and backward; the current operation process of the tunnel boring machine is displayed in text form in the first area of ​​the display, and the current operation process of the tunnel boring machine is dynamically displayed in the second area of ​​the display through the tunnel boring machine operation digital twin model; The second situation: when the first operation process is the cutting arm lifting and rotating or the tunnel boring machine moving forward and backward, and the second operation process is the tunnel boring machine moving forward and backward or the cutting arm lifting and rotating, a warning message for indicating sensor failure will pop up on the display; The third case: when the first operation process is temporary support or permanent support and the second operation process is blank, the current operation process of the roadheader is determined to be temporary support or permanent support, and the temporary support or permanent support is displayed in text form in the first area of ​​the display, and the temporary support or permanent support is displayed in the second area of ​​the display through the digital twin model of the roadheader operation; The fourth case: when the first operating procedure is blank and the second operating procedure is shovel lifting or rear support lifting, the current operating procedure of the tunnel boring machine is determined to be shovel lifting or rear support lifting, and the shovel lifting or rear support lifting is displayed in text form in the first area of ​​the display, and the shovel lifting or rear support lifting is dynamically displayed in the second area of ​​the display through the digital twin model of the tunnel boring machine operation.

Images