A method for positioning and optimizing personnel intrusion based on underground mobile terminals

Abstract

The present invention belongs to the field of intelligent coal mining industry and machine vision, and discloses a personnel intrusion positioning and optimization method based on an underground mobile terminal. The specific technical scheme is: collecting environmental information behind the tunnel boring machine; marking and outputting the staff in the scene; using the U‑Net module to fuse low-level and high-level semantic information, and weighting the target edge contour weight; drawing a warning line in the thermal imaging image coordinate system, calculating the pixel coordinates of the position where the person stands in the prediction frame, and comparing them with the warning line to determine whether there is a person breaking into the warning line, fusing the position relationship between the target frame and the warning line, and calculating the location area where the target person is located; obtaining an alarm result according to a multiple fusion algorithm, and when the target person is in the power-off area, sending a power-off command to the tunnel boring machine and broadcasting a voice message of timely persuasion. The present invention effectively reduces the false alarms that may be caused by environmental factors or equipment interference, and greatly improves safety and monitoring efficiency.

Description

Technical Field

[0001] The present invention belongs to the intelligent coal mining industry and the field of machine vision, and specifically discloses a method for accurately positioning personnel in a dangerous area of ​​a tunnel boring machine. Background Art

[0002] The personnel intrusion positioning and optimization algorithm based on the underground mobile terminal uses thermal imaging equipment to detect personnel at the excavation face to achieve accurate positioning and alarm of personnel intrusion in dangerous areas and protect the safety of personnel and property. First, due to the lack of visible light underground, traditional cameras cannot capture rich feature information, and the temperature radiated by the object is collected by thermal imaging equipment to construct temperature image data in a light-free environment. Secondly, since the temperature image collected by thermal imaging has a low resolution and the color information is relatively single, the U-Net segmentation network model is used to replace the traditional backbone network to achieve more accurate personnel detection. Then, the warning line of the dangerous area is manually determined in the image coordinate system as a reference line for personnel intrusion into the dangerous area, and the image area occupied by the target personnel is obtained through the segmentation algorithm, and the relative position relationship between the target personnel and the warning line is obtained based on the empirical value and the camera pitch angle. Finally, when unauthorized personnel enter the real-time monitoring environment of thermal imaging, the optimized AI model is used for accurate detection, and their location information is judged through multiple positioning algorithms. If they are in the warning area, an alarm message is sent and voice persuasion is performed. If they are in the power-off area, the excavation equipment is powered off to ensure the safety of personnel and property and reduce the risk of accidents.

[0003] The current status of algorithm research on visual intrusion detection is constantly evolving with the rapid development of artificial intelligence and computer vision technology, showing a high degree of innovation and broad application prospects. In the field of visual intrusion detection, algorithm research presents a diversified feature. Although traditional algorithms such as background difference method and frame difference method are still used in a certain range, they have gradually been replaced by more advanced and intelligent algorithms. At present, algorithms based on deep learning, especially convolutional neural network (CNN) and recurrent neural network (RNN), have become the mainstream of research. These algorithms can automatically learn feature representation from a large amount of data, achieve high-precision personnel detection and tracking, and effectively distinguish normal activities from intrusion behaviors. With the continuous optimization of the algorithm, the performance of visual intrusion detection is also significantly improved. On the one hand, researchers have improved the recognition accuracy and robustness of the algorithm by improving the network structure and optimizing the training strategy. On the other hand, the fusion processing of multimodal data (such as radar, infrared, etc.) has further enhanced the adaptability of the algorithm in complex environments. In addition, with the improvement of hardware performance, the running speed of the algorithm has also been significantly improved, making real-time detection possible. The research on visual intrusion detection algorithms is not only limited to the traditional security monitoring field, but has also gradually expanded to multiple fields such as industrial automation, intelligent transportation, and smart cities. For example, on industrial automation production lines, by deploying visual intrusion detection algorithms, the safety status of production areas can be monitored in real time to prevent unauthorized personnel from entering dangerous areas or operating key equipment; in intelligent transportation systems, combined with vehicle detection and tracking algorithms, comprehensive monitoring of pedestrians and vehicles on the road can be achieved to improve traffic safety.

[0004] Due to insufficient light sources underground and the constant movement of the excavation face, traditional visible light equipment is unable to collect rich image data, making the mainstream detection model less effective and unable to meet the functions of personnel detection and positioning. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention deeply integrates computer thermal imaging technology with a monocular positioning algorithm to meet the needs of personnel intrusion alarm in the absence of light source underground, and provides a multi-fusion positioning algorithm based on the improved yolov5, that is, improving the backbone feature extraction network to improve the accuracy of thermal imaging personnel detection, and integrating the regional positioning algorithm and the pixel ratio positioning algorithm to achieve accurate positioning of the target personnel, thereby avoiding the impact of severe shaking of the tunnel boring machine on the positioning accuracy of the target personnel.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is: a method for locating and optimizing personnel intrusion based on an underground mobile terminal, and the specific steps are as follows:

[0007] Step S1, installing the thermal imaging device on the tunnel boring machine and looking down at the rear area of ​​the tunnel boring machine so that the thermal imaging device can collect environmental information within 3-10 meters;

[0008] Step S2, obtain the excavation face scene data, collect different color bar series of thermal imaging data, and mark the workers in the scene. Before marking, divide the data into training set, verification set and test set in a ratio of 6:2:2, combine labelimg and labelme tools to jointly mark the images of the cargo on the vehicle, and output the labels in txt format;

[0009] Step S3, improve the YOLOV5 backbone network structure, use the U-Net module to fuse low-level and high-level semantic information, and weight the target edge contour weight;

[0010] Step S4, draw a 5-meter warning line in the thermal imaging image coordinate system, and calculate the pixel area occupied by the person in the thermal imaging image data when the person is at the 5-meter warning line according to the thermal imaging resolution;

[0011] Step S5: When the tunnel boring machine stops working, the thermal imaging does not shake, and the target person in the scene is detected to obtain a prediction frame of the target person, and the pixel coordinates of the position where the person stands in the prediction frame are calculated and compared with the cordon to determine whether there is a person breaking into the cordon;

[0012] Step S6: When the tunnel boring machine is working, the thermal imaging screen shakes irregularly. By calculating the image value of the target person in the thermal imaging and integrating the position relationship between the target frame and the warning line, the location area of ​​the target person is calculated;

[0013] Step S7, obtaining an alarm result according to a multiple fusion algorithm, when the target personnel are in the power-off area, sending a power-off command to the tunnel boring machine and broadcasting a voice message to persuade them to leave in time.

[0014] In step S1, the height of the thermal imaging is set to 1.8 meters, and the rear field of view of the tunnel boring machine is captured in a bird's-eye view.

[0015] In step S2, the thermal imaging device generates corresponding image data according to the ambient temperature, and converts the corresponding temperature into corresponding pixel values ​​based on the color bar information. By modifying the camera configuration parameters and setting different color bars, the morphological characteristics of the excavation face workers are collected.

[0016] In step S3, the U-Netv2 module is used to fuse low-level and high-level semantic information and weight the target edge contour weight. In the U-Netv2 module, a deep neural network encoder is used to extract multi-level features of the input image, and the semantic information from high-level features and details from low-level features are fused through the Hadamard product to enhance the feature map of each level. The improved features are transmitted to the decoder for processing and segmentation;

[0017] Step S31, the overall architecture of U-Netv2 includes an encoder, an SDI module and a decoder. Given an input image I, I∈RH×W×C, the encoder generates M levels of features, and the i-th layer feature is recorded as The collected features Transmit to SDI module for refinement;

[0018] The spatial and channel attention mechanisms are applied to the features of each layer i through the hierarchical feature maps generated by the encoder. Integrate local spatial information and global channel information, as follows:

[0019]

[0020] in, represents the feature map after level I processing, represents the level I spatial attention parameter, represents the first-level channel attention parameter, and applies 1×1 convolution to The channels are reduced to c, where c is a hyperparameter, and the obtained feature map is recorded as H i express Width, Wi represents The height of Channel;

[0021] The improved feature map is sent to the decoder, and at each decoder level i, the As the target reference, resize the feature map at each j level to match The same resolution was made:

[0022]

[0023] D represents the resolution of adaptive average pooling, I represents the resolution of identity mapping, and U represents bilinear interpolation. The resolution of , 1≤i, j≤M, 3×3 convolution is applied to smooth each resized feature map

[0024]

[0025] Among them, θ ij represents the parameters of smoothed convolution, is the jth smoothed feature map of the i-th layer;

[0026] After resizing all i-th layer feature maps to the same resolution, the element-wise Hadamard product is applied to all resized feature maps to enhance the i-th layer features as follows:

[0027]

[0028] Step S32: The network structure used is YOLOV5, and the activation function used is Mish function. Nonlinear elements are introduced into neurons, and the outputs are all linear functions of the upper layer inputs.

[0029] Step S33, select an optimizer and a loss function, and add an L2 regularization term to the loss function;

[0030] Step S34: Combine the center loss function with the focal-loss loss function to supervise the network training and find the optimal parameters. The center loss function is specifically:

[0031]

[0032] Among them, L b is the center loss, x i is the category feature, is the feature center of the i-th category, k represents the batch processing size, and the Focal Loss function is defined as:

[0033]

[0034] Among them, L pt is the Focal Loss loss function, y is the label value, the positive sample label value is 1, and the negative sample label value is 0. To balance the weights of positive and negative samples, the value range is [0,1]; P t The probability of a certain category predicted by the representative model;

[0035] The loss function of the entire network is:

[0036] L=L b +δL pt (9)

[0037] Among them, δ is the balance scalar of the two loss functions.

[0038] In step S4, the pixel ratio of the target person and the pixel ratio at the warning line are calculated to determine the location of the target person;

[0039] Step S41, line positioning method, set a reference line in the image, assume that the length of the reference line is L, the horizontal position of the reference line in the image is (xr), and calculate the relationship between the actual distance D and the length in the image;

[0040] Use the AI ​​system for target recognition to detect and mark the target object from the image. Assume that the AI ​​system generates a target box with the coordinates of the upper left corner ((x_1, y_1)) and the lower right corner ((x_2, y_2)). The position coordinates of the target box ((x_m, y_m)) are calculated by the following formula:

[0041] x_m=(x_1+x_2)\2

[0042] y_m=y_2

[0043] According to the relationship between the coordinates of the target frame position ((x_m, y_m)) and the position of the reference line (x_r), determine whether the target has entered the area specified by the reference line. If the reference line is a horizontal line, compare the (x)-coordinate of the point in the target frame with the horizontal coordinate of the reference line. The specific judgment method is as follows:

[0044] If the midpoint of the target box (x_m) is greater than the position of the reference line (x_r), it is determined that the target has entered the inner area of ​​the reference line;

[0045] If the midpoint of the target box (x_m) is less than or equal to the position of the reference line (x_r), it is determined that the target does not enter the inner area of ​​the reference line;

[0046] The line positioning method uses a fixed camera to obtain images, sets reference lines and target frames, and calculates the coordinates of the points in the target frame to determine whether the target enters the internal area of ​​the reference line;

[0047] Step S42, the camera is installed in a horizontal position, and the pixel ratio of the target at different distances is analyzed and obtained, the internal parameters of the camera, the actual size of the target, and the distance relationship between the target and the camera are determined, a mapping relationship between the distance and the pixel ratio is set, and a key pixel ratio threshold is determined according to the pixel ratio data of the target at different distances. The threshold (P_{th}) is used to calibrate the pixel ratio of the target at different distances;

[0048] The relationship between the target pixel ratio and the distance is expressed by the following formula:

[0049] P(d)=frac{A_{target}(d)}{A_{image}}

[0050] Where (P(d)) is the pixel ratio of the target at distance (d), (A_{target}(d)) is the pixel area of ​​the target region at distance (d), and (A_{image}) is the pixel area of ​​the entire image;

[0051] For the detected target area, calculate its pixel ratio (P) in the image. The calculation formula of pixel ratio (P) is:

[0052] P=\frac{A_{target}}{A_{image}}

[0053] Where A_{target} is the pixel area of ​​the target area, and A_{image} is the pixel area of ​​the entire image;

[0054] The calculated pixel ratio (P) is compared with the preset pixel ratio threshold P_{th} to determine whether the target is within the set distance range;

[0055] The distance of the target is inferred through the pixel ratio information in the image.

[0056] In step S5, when the tunnel boring machine stops working, thermal imaging monitoring is performed on the working area where the equipment is located, and the model is used to detect the target personnel in the monitoring scene, and the prediction box of each target person is obtained. The pixel coordinates of each person in the image are calculated through these prediction boxes;

[0057] Compare these pixel coordinates with the set warning line to determine whether anyone has crossed the warning line and entered the prohibited area, and detect in real time whether anyone has entered the warning area.

[0058] In step S6, assuming that the pixel coordinates of the target person in the thermal imaging image are ((x_i, y_i)), where (i) represents the i-th target person, the pixel coordinates are calculated to determine the position of the target person in the image, and the target box ((x_i^{box}, y_i^{box}, w_i, h_i)) is used to describe the boundary of the target person, where (w_i) is the width of the target box and (h_i) is the height of the target box;

[0059] Assume that the position of the warning line in normal conditions is (L_{true}), and calculate the offset of the warning line (\Delta L) as:

[0060] [\Delta L=L_{measured}-L_{true}]

[0061] In order to correct the position of the cordon, for each target person, calculate its position area in the image, using the following steps:

[0062] Determine the target area: Calculate the actual location of the target person in the image based on the coordinates and size of the target box:

[0063] [A_i={(x_i,y_i)\mid x_i\in[x_i^{box},x_i^{box}+w_i],y_i\in[y_i^{box},y_i^{box}+h_i]}]

[0064] Apply offset correction: Apply the offset of the cordon to the coordinates of the target area to correct the actual position of the target person:

[0065] [\text{Corrected Position}_i=(x_i-\Delta L,y_i)].

[0066] Compared with the prior art, the specific beneficial effects of the present invention are embodied in the following aspects: compared with the traditional visible light single positioning algorithm, the underground excavation face multiple personnel positioning method based on thermal imaging technology of the present invention has significant advantages in positioning accuracy and false alarm rate. The optimized network architecture significantly improves the processing speed and accuracy of thermal imaging data, and can identify and track target personnel more quickly. At the same time, the image pixel ratio algorithm accurately determines the position of personnel by analyzing the temperature changes in different areas of the thermal imaging image, while the line positioning algorithm overcomes the positioning deviation that may be caused by the vibration of the tunneling machine by adjusting and correcting the positioning lines in the thermal imaging image in real time. The combination of the two ensures the high accuracy of the positioning results and effectively reduces false alarms that may be caused by environmental factors or equipment interference. This multiple positioning method greatly improves the safety and monitoring efficiency of underground excavation operations.

[0067] This method can monitor the activities of personnel on the excavation face in real time and timely detect and locate the position of the operating personnel through high-precision thermal imaging technology. This greatly reduces the risk of accidents caused by equipment failure, environmental changes or other potential dangers during excavation operations, and ensures the safety of workers' lives. Timely and accurate personnel positioning can quickly respond to emergencies and reduce accident losses. Enhanced monitoring capabilities: Compared with traditional visible light positioning systems, thermal imaging technology can still work stably in low light or complex environments. This technology can penetrate obstacles such as smoke and dust, effectively improving the all-round monitoring capabilities of the excavation operation site and ensuring safety protection without blind spots. Reduced false alarm rate: By integrating the image pixel ratio algorithm and the line positioning algorithm, this method greatly reduces the false alarm rate. Accurate positioning reduces misjudgments caused by false alarms, ensures the accuracy and reliability of system alarms, thereby reducing unnecessary interference and improving overall work efficiency. Improved work efficiency: Accurate personnel positioning not only helps to improve the safety of the operation site, but also optimizes resource allocation and scheduling. Timely understanding of personnel distribution and operation status helps to arrange work tasks more efficiently and reduce unnecessary waste of time and resource consumption. Supporting intelligent management: The data collection and analysis capabilities of this method provide basic support for intelligent management. Through real-time monitoring and data analysis, better safety management strategies can be formulated and operation processes can be optimized, providing a scientific basis for decision-making and promoting the development of tunneling operations towards intelligence and automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of the method for locating multiple personnel at an underground excavation face based on thermal imaging technology.

[0069] Figure 2 On-site photo of thermal imaging equipment installation.

[0070] Figure 3 Example graph of the dataset.

[0071] Figure 4 This is a scene diagram of the underground excavation surface.

[0072] Figure 5 Schematic diagram of the overall architecture of U-Netv2.

[0073] Figure 6 This is the model training result diagram.

[0074] Figure 7 Schematic diagram of the line for human intrusion.

[0075] Figure 8 This is a schematic diagram of the percentage of pixels invaded by people.

[0076] Fig. 9 This is a schematic diagram of the alarm record for personnel intrusion at the underground excavation face. DETAILED DESCRIPTION

[0077] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0078] like Figure 1 As shown, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is described, and the specific steps are as follows:

[0079] Step S1: Due to the limitation of underground installation environment, the thermal imaging is installed on the tunnel boring machine, and the dangerous area behind the tunnel boring machine is overlooked, so that the thermal imaging can collect environmental information of 3-10 meters.

[0080] like Figure 2 As shown, the present invention is specifically applied to the underground tunnel boring machine scenario. When the tunnel boring machine is working, it will move forward and be accompanied by the body shaking. Due to the underground installation environment, the tunnel height is about 2 meters, and there is no sufficient light source. Thermal imaging is used as a sensor to replace the traditional visible light camera. It is installed on the tunnel boring machine at a height of about 1.8 meters and collects the rear field of view of the tunnel boring machine in a bird's-eye view, so that the thermal imaging can collect ground data 3-10 meters away from the tunnel boring machine.

[0081] Step S2: Obtain the excavation face scene data, collect different color bar series of thermal imaging data, and annotate the workers in the scene. Before annotation, divide the data into training set, validation set and test set in a ratio of 6:2:2, and use labelimg and labelme tools to annotate the images of cargo on the vehicle, and output the labels in txt format.

[0082] Data acquisition and annotation: Figure 3-4 As shown, the thermal imaging equipment generates corresponding image data according to the ambient temperature, and converts the corresponding temperature into corresponding pixel values ​​based on the color bar information. In order to enrich the data volume, highlight the contour features of the human body, eliminate the model overfitting phenomenon caused by a single color, and increase the data of different color bars, that is, by modifying the camera-side configuration parameters and setting different color bars, the morphological characteristics of the excavation face workers are collected, totaling 6,000 data.

[0083] To obtain the scene data of the tunneling face, different color bar series of thermal imaging data can be collected and the workers in the scene can be labeled. Before labeling, the data is divided into training set, validation set and test set in a ratio of 6:2:2, and the target image is labeled together with labelimg and labelme tools, and the labels are output in txt format.

[0084] Step S3, improving the YOLOV5 model. Aiming at the single color of thermal imaging image data and blurred internal texture of the target, the present invention improves the YOLOV5 backbone network structure, uses the U-Net module to fuse low-level and high-level semantic information, and weights the target edge contour weight, so as to enrich the model network structure and improve the model performance.

[0085] Improved YOLOV5 model: In view of the single color of thermal imaging image data and the blurred texture inside the target, the present invention improves the YOLOV5 backbone network structure, uses the U-Netv2 module to fuse low-level and high-level semantic information, weights the target edge contour weights, enriches the model network structure and optimizes the loss function, and improves the model performance. S31.U-Netv2 is a new robust and efficient U-Net variant designed for image segmentation. Its main goal is to enhance the fusion of semantic information in low-level features while using finer details to optimize high-level features. For the input image, a deep neural network encoder is first used to extract multi-level features. Next, the semantic information from high-level features and finer details from low-level features are fused through the Hadamard product to enhance the feature map of each level. The new jump connection makes all levels of features have rich semantic characteristics and complex details. The improved features are then transmitted to the decoder for further processing and segmentation. This method can be seamlessly integrated into any encoder-decoder network.

[0086] like Figure 5As shown in Figure 1, it consists of three main modules: encoder, SDI (semantic and detail injection) module and decoder. Given an input image I, where I∈RH×W×C, the encoder generates M levels of features. The i-th layer feature is recorded as These collected features It is then transferred to the SDI module for further refinement.

[0087] The hierarchical feature maps generated by the encoder are firstly applied to the features of each layer i using spatial and channel attention mechanisms. This process enables the features to integrate local spatial information and global channel information, which can be expressed as follows:

[0088]

[0089] in, represents the feature map after level I processing, and denote the first-level spatial attention and channel attention parameters respectively. In addition, the present invention applies 1×1 convolution to The channels are reduced to c, where c is a hyperparameter, and the obtained feature map is recorded as in Among them, H i , Wi, c represent Width, height and channels.

[0090] The improved feature map is sent to the decoder, and at each decoder level i, the As the target reference, resize the feature map at each j level to match The same resolution was enacted:

[0091]

[0092] D stands for adaptive average pooling, I stands for identity mapping, and U stands for bilinear interpolation Resolution H i ×W i , apply 3×3 convolution to smooth each resized feature map with 1≤i,j≤M respectively The formula is:

[0093]

[0094] Among them, θ ij represents the parameters of smoothed convolution, is the jth smoothed feature map of the i-th layer.

[0095] After resizing all i-th layer feature maps to the same resolution, element-wise Hadamard product is applied to all resized feature maps to enhance the i-th layer features with more semantic information and finer details, such as:

[0096]

[0097] Before model training, set the learning rate, batch size, number of iterations and other hyperparameters in advance. This chapter mainly introduces the activation function and loss function in the network training architecture.

[0098] S31. Select activation function:

[0099] The network structure used in this algorithm is YOLOV5, and the activation function used internally is the Mish function, which mainly introduces nonlinear elements to neurons and increases learning ability. Otherwise, each output is a linear function of the upper-level input.

[0100] S32. Select optimizer and loss function:

[0101] The optimization process is actually the process of minimizing the loss function. The smaller the loss function, the better the model performance. The model that uses only one loss function has poor performance. Therefore, the algorithm compares the effects of multiple loss functions, selects multiple appropriate loss functions, assigns weights to them, and jointly measures the quality of the model. The process of continuously adjusting parameters during model training is to minimize the loss function, so as to achieve the optimal model effect. In order to prevent the gradient from disappearing or exploding during model training, an L2 regularization term is added to the loss function.

[0102] S33. Model evaluation:

[0103] The corresponding evaluation problem uses the basic parameters of accuracy as the evaluation standard. The loss function is mainly used in the training stage of the model. After each batch of training data is sent to the model, the predicted value is obtained through forward propagation. Then the difference between the predicted value and the true value is calculated to optimize and adjust the model parameters to minimize the loss value, thereby achieving the purpose of learning. The model training results are as follows: Figure 6 shown.

[0104] The model loss function is Focal Loss, which is a loss function used to solve the problem of data imbalance. It adjusts the weights of positive and negative samples so that samples that are difficult to classify receive more attention during training. Center point loss is used to predict the center point position of the target box, helping the model to regress the position of the target box more accurately, thereby improving the accuracy of target detection. The present invention combines the center loss function with the Focal-loss loss function to jointly supervise network training and jointly supervise model training to find the optimal parameters.

[0105] The following is the definition of the center loss function:

[0106]

[0107] Among them, L b is the center loss, x i is the category feature, is the feature center of the i-th category, and k represents the batch processing size. The Focal Loss function is defined as:

[0108]

[0109] Among them, L pt is the Focal Loss loss function, y is the label value, the positive sample label value is 1, and the negative sample label value is 0. To balance the weights of positive and negative samples, the value range is [0,1]; P t It represents the probability (i.e. confidence) of a certain category predicted by the model.

[0110] The loss function of the entire network is:

[0111] L=L b +δL pt (9)

[0112] Among them, δ is the balance scalar of the two loss functions, and the scalar value is continuously optimized during the training process to make L reach the minimum value.

[0113] Step S4: Based on subjective judgment, a 5-meter line is drawn in the thermal imaging image coordinate system, which is identified as the warning line, and the pixel area occupied by a person in the 5-meter warning line in the thermal imaging image data is calculated based on the thermal imaging resolution.

[0114] S41, positioning method:

[0115] Thermal imaging is a monocular camera without depth information. It only relies on the temperature of ambient radiation to establish two-dimensional image data and cannot perceive target distance information. Therefore, the operation of drawing a cordon is used in the two-dimensional image coordinate system to increase three-dimensional perception. However, when the tunnel boring machine is working, the shaking of the fuselage will cause fluctuations in the thermal imaging equipment, causing the cordon to deviate from the actual scene distance. Therefore, the present invention determines the pixel ratio of the target person in the image when the cordon is located through empirical values. If the target person information obtained according to the line drawing operation jumps back and forth between the alarm and power-off areas, the pixel ratio of the target person at this time and the pixel ratio at the cordon are calculated to accurately determine the location of the target person.

[0116] S42, line positioning method:

[0117] Line positioning is a technique used to determine whether a target object enters a specific area. First, fix the camera in a predetermined position to ensure that its viewing angle and shooting area are stable. Set a reference line in the image, which is used as a reference for subsequent judgments. The length and position of this line in the actual scene need to be determined by actual measurement or calibration. Assume that the length of the reference line is (L) (unit: meter), and the horizontal position of the reference line in the image is (x_r) (unit: pixel). Through a known calibration process, determine the distance of the reference line in the actual scene, which converts the position in the image coordinate system to the actual physical distance. Assuming the actual length of the reference line is (L), calculate the relationship between the actual distance (D) and the length in the image.

[0118] Use artificial intelligence (AI) technology for target recognition to detect and mark target objects from images. For example, the AI ​​system generates a bounding box with the coordinates of the upper left corner ((x_1, y_1)) and the lower right corner ((x_2, y_2)). The position coordinates of the target box ((x_m, y_m)) can be calculated using the following formula:

[0119] x_m=(x_1+x_2)\2

[0120] y_m=y_2

[0121] Based on the relationship between the coordinates of the target frame position ((x_m, y_m)) and the position of the reference line (x_r), determine whether the target has entered the area specified by the reference line. If the reference line is a horizontal line, the (x)-coordinate of the point in the target frame can be compared with the horizontal coordinate of the reference line.

[0122] The specific judgment method is as follows:

[0123] If the midpoint of the target box (x_m) is greater than the position of the reference line (x_r), the target has entered the interior region of the reference line.

[0124] If the midpoint of the target box (x_m) is less than or equal to the position of the reference line (x_r), the target does not enter the interior area of ​​the reference line.

[0125] The line positioning method uses a fixed camera to acquire images, sets reference lines and target frames, and calculates the coordinates of the midpoints in the target frame to determine whether the target enters the internal area of ​​the reference line. This method combines camera calibration, AI target recognition, and geometric position judgment to provide an effective solution for dynamic scene target positioning and area judgment.

[0126] S44, pixel ratio positioning:

[0127] The pixel ratio positioning method is a technology that estimates the target distance by analyzing the target pixel ratio in the image. The key to this method is to compare the pixel ratio of the target at different distances with the empirical value, so as to achieve accurate judgment of the target distance.

[0128] The camera is installed in a horizontal position, and the pixel percentage of the target at different distances is obtained through experiments or theoretical analysis. This process requires determining the camera's internal parameters (such as focal length, sensor size, etc.), the actual size of the target, and the distance relationship between the target and the camera. Set a mapping relationship between distance and pixel percentage for subsequent distance estimation. According to the pixel percentage data of the target at different distances, determine a key pixel percentage threshold. This threshold (P_{th}) is determined after analyzing a large number of samples to calibrate the pixel percentage of the target at different distances. The process of setting the threshold can be completed by fitting functions or empirical formulas to ensure that it can accurately reflect the relationship between the target and the distance.

[0129] For example, the relationship between the target pixel ratio and the distance is expressed by the following formula:

[0130] P(d)=frac{A_{target}(d)}{A_{image}}

[0131] Among them, (P(d)) is the pixel ratio of the target at distance (d), (A_{target}(d)) is the pixel area of ​​the target region at distance (d), and (A_{image}) is the pixel area of ​​the entire image.

[0132] In practical applications, computer vision models are used to detect targets in images and obtain pixel information of the target area. For the detected target area, its pixel ratio (P) in the image is calculated. The calculation formula for pixel ratio (P) is:

[0133] P=\frac{A_{target}}{A_{image}}

[0134] Among them, A_{target} is the pixel area of ​​the target area, and A_{image} is the pixel area of ​​the entire image.

[0135] The calculated pixel ratio (P) is compared with a preset pixel ratio threshold P_{th} to determine whether the target is within the set distance range.

[0136] In practical applications, the threshold (P_{th}) can be adjusted according to the specific application scenario to adapt to different detection requirements and target types.

[0137] Through the above optimized pixel ratio positioning method, the distance of the target can be effectively inferred through the pixel ratio information in the image. This method not only combines prior knowledge and real-time detection, but also improves the accuracy and reliability of target distance estimation, and has important practical application value.

[0138] Step S5: When the tunnel boring machine stops working, the thermal imaging does not shake. The target person in the scene is detected through the model, and the prediction frame of the target person is obtained. The pixel coordinates of the position where the person stands in the prediction frame are calculated and compared with the cordon to determine whether there is a person breaking into the cordon.

[0139] S51, positioning of the tunnel boring machine when it is stopped

[0140] When the tunnel boring machine stops working, the working area of ​​the equipment is monitored by thermal imaging, and it is ensured that the thermal imaging equipment does not shake during this process. Then, the model is used to detect the target personnel in the monitoring scene to obtain the prediction box of each target person. Through these prediction boxes, the pixel coordinates of each person in the image can be accurately calculated.

[0141] Next, these pixel coordinates are compared with the set cordon to determine whether there are people crossing the cordon and entering the prohibited area. Through this comparison method, it is possible to detect in real time whether there are people breaking into the warning area, thereby ensuring the safety of the working environment. Figure 7 shown.

[0142] Step S6: When the tunnel boring machine is working, the thermal imaging screen shakes irregularly, causing the warning line to deviate from the actual warning position. Therefore, the image value of the target person in the thermal imaging is calculated, the position relationship between the target frame and the warning line is integrated, and the location area of ​​the target person is calculated.

[0143] Step S6: When the tunnel boring machine is working, the thermal imaging screen shakes irregularly, causing the warning line to deviate from the actual warning position. Therefore, the image value of the target person in the thermal imaging is calculated, the position relationship between the target frame and the warning line is integrated, and the location area of ​​the target person is calculated.

[0144] When the tunnel boring machine is working, due to mechanical vibration and other factors, the thermal imaging image may show irregular jitter. This jitter will cause the warning line in the thermal imaging image to deviate from the actual warning position, thus affecting the accurate identification and monitoring of the warning area. In order to overcome this challenge, it is necessary to take accurate calculation and correction measures in the form of image proportion, such as Figure 8 shown.

[0145] First, a detailed analysis is performed on the pixel area occupied by the target person in the thermal imaging image. Assuming that the pixel coordinates of the target person in the thermal imaging image are ((x_i, y_i)), where (i) represents the (i)th target person, the position of the target person in the image is determined by calculating these coordinates. The target box ((x_i^{box}, y_i^{box}, w_i, h_i)) describes the boundary of the target person, where (w_i) and (h_i) are the width and height of the target box, respectively.

[0146] Due to the jitter of the picture, the position of the warning line may be offset. The position of the warning line in normal conditions is set to (L_{true}), but the position of the warning line in the actual picture may be (L_{measured}). It is necessary to calculate the offset of the warning line (\Delta L), which is expressed by the following formula:

[0147] [\Delta L=L_{measured}-L_{true}]

[0148] Next, in order to correct the position of the cordon, the position of the target box and the offset are combined. Specifically, for each target person, its position area in the image is calculated using the following steps:

[0149] Determine the target area: Based on the coordinates and size of the target box, the actual location area of ​​the target person in the image can be calculated:

[0150] [A_i={(x_i,y_i)\mid x_i\in[x_i^{box},x_i^{box}+w_i],y_i\in[y_i^{box},y_i^{box}+h_i]}]

[0151] Apply offset correction: Apply the offset of the cordon to the coordinates of the target area to correct the actual position of the target person:

[0152] [\text{Corrected Position}_i=(x_i-\Delta L,y_i)]

[0153] Through these steps, the position of the target person relative to the corrected cordon can be accurately calculated in the thermal imaging image. Ultimately, this method can effectively correct the deviation of the cordon, ensure accurate identification of the warning area, and timely detect whether the target person has crossed the cordon, thereby improving the safety of the working environment.

[0154] Step S7, obtain the alarm result according to the multiple fusion algorithm. When the target personnel are in the power-off area, a power-off command will be sent to the tunnel boring machine, and a voice message of timely persuasion will be broadcast. When the target personnel are in the warning area, a voice message of warning persuasion will be broadcast. Fig. 9 The uploaded platform alarm information is shown.

[0155] S71. Algorithm deployment:

[0156] When applying multiple fusion algorithms to determine the alarm results, when the system identifies that the target person is in the power-off area, it will issue a power-off command to the tunnel boring machine and simultaneously play an emergency voice alarm for persuasion. At the same time, when the target person enters the warning area, the system will issue an alarm and broadcast a persuasion prompt message. This strategy not only ensures a rapid response to dangerous areas, but also improves the accuracy of safety management through a multi-level warning mechanism.

[0157] The present invention firstly collects underground excavation face materials through thermal imaging equipment, improves the yolov5 backbone feature network into a U-netv2 model, fully extracts the contour feature information of the target, removes the interference of single color information, and improves the accuracy of personnel detection in thermal imaging data. Secondly, the position of the actual warning line in the image coordinate system is determined by the line drawing operation to give the two-dimensional image depth information, so as to judge whether the underground excavation face staff has entered the dangerous area. When the excavator is working, it is accompanied by a slight jitter, causing the warning line position in the image coordinate system to deviate from the actual position, resulting in the positioning result jumping between power failure and warning, causing unnecessary false alarms, and the image pixel ratio positioning method obtains the pixel ratio of the person at the warning line position according to empirical information, and does not rely on the depth information of the warning line, thereby improving the stability of the personnel position information, and sending alarm data, reducing the risk of excavation face personnel invading the dangerous area, and improving the safety of underground production operations.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for locating and optimizing personnel intrusion based on an underground mobile terminal, characterized in that: The specific steps are as follows: Step S1, installing the thermal imaging device on the tunnel boring machine and looking down at the rear area of ​​the tunnel boring machine so that the thermal imaging device can collect environmental information within 3-10 meters; Step S2, obtain the excavation face scene data, collect different color bar series of thermal imaging data, and mark the workers in the scene. Before marking, divide the data into training set, verification set and test set in a ratio of 6:2:2, combine labelimg and labelme tools to mark the target image, and output the label in txt format; Step S3, improve the YOLOV5 backbone network structure, use the U-Net module to fuse low-level and high-level semantic information, and weight the target edge contour weight; Step S4, draw a 5-meter warning line in the thermal imaging image coordinate system, and calculate the pixel area occupied by the person in the thermal imaging image data when the person is at the 5-meter warning line according to the thermal imaging resolution; Step S5: When the tunnel boring machine stops working, the thermal imaging does not shake, and the target person in the scene is detected to obtain a prediction frame of the target person, and the pixel coordinates of the position where the person stands in the prediction frame are calculated and compared with the cordon to determine whether there is a person breaking into the cordon; Step S6: When the tunnel boring machine is working, the thermal imaging screen shakes irregularly. By calculating the image value of the target person in the thermal imaging and integrating the position relationship between the target frame and the warning line, the location area of ​​the target person is calculated; Step S7, obtaining an alarm result according to a multiple fusion algorithm, when the target personnel are in the power-off area, sending a power-off command to the tunnel boring machine and broadcasting a voice message to persuade them to leave in time.

2. According to claim 1, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is characterized in that: In step S1, the height of the thermal imaging is set to 1.8 meters, and the rear field of view of the tunnel boring machine is captured in a bird's-eye view.

3. According to claim 2, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is characterized in that: In step S2, the thermal imaging device generates corresponding image data according to the ambient temperature, and converts the corresponding temperature into corresponding pixel values ​​based on the color bar information. By modifying the camera configuration parameters and setting different color bars, the morphological characteristics of the excavation face workers are collected.

4. According to claim 3, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is characterized in that: In step S3, the U-Netv2 module is used to fuse low-level and high-level semantic information and weight the target edge contour weight. In the U-Netv2 module, a deep neural network encoder is used to extract multi-level features of the input image, and the semantic information from high-level features and details from low-level features are fused through the Hadamard product to enhance the feature map of each level. The improved features are transmitted to the decoder for processing and segmentation; Step S31, the overall architecture of U-Netv2 includes an encoder, an SDI module and a decoder. Given an input image I, I∈RH×W×C, the encoder generates M levels of features, and the i-th layer feature is recorded as The collected features Transmit to SDI module for refinement; The spatial and channel attention mechanisms are applied to the features of each layer i through the hierarchical feature maps generated by the encoder. Integrate local spatial information and global channel information, as follows: in, represents the feature map after level I processing, represents the level I spatial attention parameter, represents the first-level channel attention parameter, and applies 1×1 convolution to The channels are reduced to c, where c is a hyperparameter, and the obtained feature map is recorded as H i express Width, Wi represents The height of Channel; The improved feature map is sent to the decoder, and at each decoder level i, the As the target reference, resize the feature map at each j level to match The same resolution was made: D represents the resolution of adaptive average pooling, I represents the resolution of identity mapping, and U represents bilinear interpolation. The resolution of , 1≤i, j≤M, 3×3 convolution is applied to smooth each resized feature map Among them, θ ij represents the parameters of smoothed convolution, is the jth smoothed feature map of the i-th layer; After resizing all i-th layer feature maps to the same resolution, the element-wise Hadamard product is applied to all resized feature maps to enhance the i-th layer features as follows: Step S32: The network structure used is YOLOV5, and the activation function used is Mish function. Nonlinear elements are introduced into neurons, and the outputs are all linear functions of the upper layer inputs. Step S33, select an optimizer and a loss function, and add an L2 regularization term to the loss function; Step S34: Combine the center loss function with the focal-loss loss function to supervise the network training and find the optimal parameters. The center loss function is specifically: Among them, L b is the center loss, x i is the category feature, is the feature center of the i-th category, k represents the batch processing size, and the FocalLoss loss function is defined as: Among them, L pt is the Focal Loss loss function, y is the label value, the positive sample label value is 1, and the negative sample label value is 0. To balance the weights of positive and negative samples, the value range is [0,1]; P t The probability of a certain category predicted by the representative model; The loss function of the entire network is: L=L b +δL pt (9) Among them, δ is the balance scalar of the two loss functions.

5. According to claim 4, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is characterized in that: In step S4, the pixel ratio of the target person and the pixel ratio at the warning line are calculated to determine the location of the target person; Step S41, line positioning method, set a reference line in the image, assume that the length of the reference line is L, the horizontal position of the reference line in the image is (xr), and calculate the relationship between the actual distance D and the length in the image; Use the AI ​​system for target recognition to detect and mark the target object from the image. Assume that the AI ​​system generates a target box with the coordinates of the upper left corner ((x_1, y_1)) and the lower right corner ((x_2, y_2)). The position coordinates of the target box ((x_m, y_m)) are calculated by the following formula: x_m=(x_1+x_2)\2 y_m=y_2 According to the relationship between the coordinates of the target frame position ((x_m, y_m)) and the position of the reference line (x_r), determine whether the target has entered the area specified by the reference line. If the reference line is a horizontal line, compare the (x)-coordinate of the point in the target frame with the horizontal coordinate of the reference line. The specific judgment method is as follows: If the midpoint of the target box (x_m) is greater than the position of the reference line (x_r), it is determined that the target has entered the inner area of ​​the reference line; If the midpoint of the target box (x_m) is less than or equal to the position of the reference line (x_r), it is determined that the target does not enter the inner area of ​​the reference line; The line positioning method uses a fixed camera to obtain images, sets reference lines and target frames, and calculates the coordinates of the points in the target frame to determine whether the target enters the internal area of ​​the reference line; Step S42, the camera is installed in a horizontal position, and the pixel ratio of the target at different distances is analyzed and obtained, the internal parameters of the camera, the actual size of the target, and the distance relationship between the target and the camera are determined, a mapping relationship between the distance and the pixel ratio is set, and a key pixel ratio threshold is determined according to the pixel ratio data of the target at different distances. The threshold (P_{th}) is used to calibrate the pixel ratio of the target at different distances; The relationship between the target pixel ratio and the distance is expressed by the following formula: P(d)=frac{A_{target}(d)}{A_{image}} Where (P(d)) is the pixel ratio of the target at distance (d), (A_{target}(d)) is the pixel area of ​​the target region at distance (d), and (A_{image}) is the pixel area of ​​the entire image; For the detected target area, calculate its pixel ratio (P) in the image. The calculation formula of pixel ratio (P) is: P=\frac{A_{target}}{A_{image}} Where A_{target} is the pixel area of ​​the target area, and A_{image} is the pixel area of ​​the entire image; The calculated pixel ratio (P) is compared with the preset pixel ratio threshold P_{th} to determine whether the target is within the set distance range; The distance of the target is inferred through the pixel ratio information in the image.

6. According to claim 5, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is characterized in that: In step S5, when the tunnel boring machine stops working, thermal imaging monitoring is performed on the working area where the equipment is located, and the model is used to detect the target personnel in the monitoring scene, and the prediction box of each target person is obtained. The pixel coordinates of each person in the image are calculated through these prediction boxes; Compare these pixel coordinates with the set warning line to determine whether anyone has crossed the warning line and entered the prohibited area, and detect in real time whether anyone has entered the warning area.

7. According to claim 6, a method for locating and optimizing personnel intrusion based on an underground mobile terminal is characterized in that: In step S6, assuming that the pixel coordinates of the target person in the thermal imaging image are ((x_i, y_i)), where (i) represents the i-th target person, the pixel coordinates are calculated to determine the position of the target person in the image, and the target box ((x_i^{box}, y_i^{box}, w_i, h_i)) is used to describe the boundary of the target person, where (w_i) is the width of the target box and (h_i) is the height of the target box; The normal position of the warning line is set to (L_{true}), L_{measured} is the position information of the person in the image, and the offset of the warning line (\Delta L) is calculated as: [\Delta L=L_{measured}-L_{true}] In order to correct the position of the cordon, for each target person, calculate its position area in the image, using the following steps: Determine the target area: According to the coordinates and size of the target box, calculate the actual location area A_i of the target person in the image: [A_i={(x_i,y_i)\mid x_i\in[x_i^{box},x_i^{box}+w_i],y_i\in[y_i^{box},y_i^{box}+h_i]}] Apply offset correction: Apply the offset of the cordon to the coordinates of the target area to correct the actual position of the target person, whose position is \text{Corrected Position}_i. The specific calculation formula is as follows: [\text{Corrected Position}_i=(x_i-\Delta L,y_i)].

Images